Paper deep dive
From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
Yingjun Mo, Yajie Zhao, Fang Kong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature $\tau>0$ controls the approximation to greedy action selection, recovering window Q-learning as $\tau\to0$. We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor $\epsilon$ is $\Theta\left(\tau^{-5}\log(1/\varepsilon)\right)$, quantifying the increasing optimization cost as the learned update approaches window Q-learning.
Tags
Links
- Source: https://arxiv.org/abs/2610.18057v1
- Canonical: https://arxiv.org/abs/2610.18057v1
Trouble viewing inline? Open PDF directly →
Full Text
268,926 characters extracted from source content.
Expand or collapse full text
From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning Yingjun Mo Affiliation: Southern University of Science and Technology (SUSTech) Yajie Zhao Affiliation: University of Electronic Science and Technology of China (UESTC) Fang Kong †thanks: Corresponding author: kongf@sustech.edu.cn. Affiliation: Southern University of Science and Technology (SUSTech) Abstract Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature τ>0τ>0 controls the approximation to greedy action selection, recovering window Q-learning as τ→0τ→ 0. We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor ϵε is Θ(τ−5log(1/ε)) (τ^-5 (1/ ) ), quantifying the increasing optimization cost as the learned update approaches window Q-learning. 1 Introduction In-context reinforcement learning (ICRL) offers an alternative to the conventional paradigm of reinforcement learning. Rather than adapting to a new task through parameter updates, a pretrained agent can adjust its behavior directly from the interaction history provided in its context. This phenomenon has motivated an algorithmic interpretation of ICRL: a Transformer may not merely map a trajectory to a prediction, but may execute a reinforcement-learning algorithm within its forward pass. Recent theoretical work supports this view. Transformers have been shown to implement temporal-difference (TD) methods for policy evaluation, and more recent results extend this perspective to policy-improvement methods such as SARSA and actor-critic (Wang et al., 2024; Wang et al., 2025; Xie et al., 2026a; Xie et al., 2026b; Liang and Lai, 2026). A central regime remains largely unexplored: off-policy control. Existing policy-improvement constructions focus primarily on on-policy learning, where the behavior policy and the policy being improved are tightly coupled (Liang and Lai, 2026). This coupling is restrictive in real applications where an agent must learn from previously collected, heterogeneous, or suboptimal experience rather than from trajectories generated by its current policy. Off-policy control asks whether useful policy improvement can still be carried out when these two roles are decoupled. Q-learning provides a canonical setting for studying this problem. Given a transition (s,a,r,s′)(s,a,r,s ), its target is r+γmaxa′Q(s′,a′)r+γ _a Q(s ,a ), which does not depend on the successor action selected by the behavior policy. Thus, subject to sufficient state-action coverage, Q-learning can use experience generated by a suboptimal behavior policy while improving toward a greedy target policy. This makes Q-learning a natural testbed for a more demanding form of in-context adaptation: whether a Transformer can use off-policy experience to carry out control, rather than merely evaluate or improve the policy that generated its context. This transition is substantially more than a change of update rule. At a conceptual level, it marks the passage from Bellman evaluation to Bellman optimality. An in-context implementation of Q-learning must compare competing successor actions, resolve the resulting action selection, and then use that decision to construct the bootstrap target. We show that there exists a two-layer Transformer, combining masked softmax attention with linear attention, that jointly implements next-state action selection and the corresponding window semi-gradient Q-learning update in finite-temperature form, where the softmax temperature τ>0τ>0 controls the degree of approximation to greedy action selection, and standard window (hard-max) Q-learning is recovered in the limit τ→0τ→ 0. In this sense, the architecture of transformer does not merely reproduce the numerical output of Q-learning; its internal computation mirrors the decomposition of the algorithm itself. What kind of pretraining can give rise to such a Transformer? We develop a gradient-based training algorithm in which a soft window-Q update serves as the teacher for each sampled context. Within the prescribed tokenization and attention mask, training supervises only the output update and imposes no numerical targets on the internal parameter blocks to implement either window Q-learning or action selection. Nevertheless, the constructed parameter structure emerges under training, recovering the underlying computation, including the internal action-selection mechanism. On unseen tasks, the trained Transformer achieves performance comparable to window Q-learning with the same step size. To understand why a Q-learning implementation can emerge under this training procedure, we provide a theoretical analysis of its convergence. The nonlinear dependence of softmax attention on the action-score parameters makes the analysis substantially more intricate than in linear-attention settings. Meanwhile, the Q-learning realization is inherently non-identifiable, with different parameter configurations implementing the same update. This makes the relevant optimization target a manifold of functionally equivalent Q-learning realizations rather than a single parameter vector. We characterize this realization manifold and show that, under verifiable conditions on the training distribution, the discrete-time training dynamics converge locally to this manifold at an exponential rate. The analysis further reveals a quantitative difficulty in approaching greedy Q-learning: for the random-task model considered here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor ϵε is Θ(τ−5log(1/ε)) (τ^-5 (1/ ) ), quantifying the increasing optimization cost as the learned update approaches standard Q-learning. Our results make three concrete advances. First, we give an explicit attention-only construction that jointly realizes successor-action comparison and the window Q-learning update. Second, we design a gradient-based training procedure and characterize a smooth manifold of functionally equivalent Q-learning realizations, showing local exponential convergence of the training dynamics to this manifold. Third, we quantify how the softmax temperature governs the tradeoff between approximating greedy action selection and optimization difficulty. Together, these results extend the mechanistic analysis of ICRL from policy evaluation and on-policy control to off-policy Q-learning. 2 Related work Transformer-based agents have been studied extensively for sequential decision making. Decision Transformer and Trajectory Transformer formulate reinforcement learning as sequence modeling (Chen et al., 2021; Janner et al., 2021), while Algorithm Distillation and related ICRL methods show that pretrained sequence models can adapt to unseen tasks through interaction histories without test-time parameter updates (Laskin et al., 2023; Lee et al., 2023; Grigsby et al., 2024). On the theoretical side, Lin et al. (2024) establish provable ICRL under supervised pretraining and show that Transformers can approximate classical online decision-making algorithms. Recent work has taken a mechanistic view of ICRL, asking which RL algorithms can be implemented by Transformer forward computation and whether the corresponding algorithmic parameters can emerge through pretraining. Wang et al. (2024); Wang et al. (2025) show that linear-attention Transformers can implement TD updates for policy evaluation, while Xie et al. (2026b) extend this result to standard softmax attention via a kernel-weighted TD construction. Beyond policy evaluation, Liang and Lai (2026) show that Transformers can realize semi-gradient SARSA and actor-critic updates for policy improvement. However, analogous results for off-policy control, including classical Q-learning, where trajectory data are coupled with subsequent action selection, remain largely unexplored. We study the emergence of Q-learning updates within Transformers, including the Transformer construction, pretraining algorithm, and its convergence analysis. Unlike Liang and Lai (2026), we focus on the effect of the action-selection mechanism on convergence, and our analysis accounts for discrete-time training iterates and provides verifiable convergence conditions. Off-policy learning is particularly relevant to ICRL when context trajectories are offline, heterogeneous, or generated by suboptimal behavior policies. In such settings, the quality of the learned policy need not be tied to that of the behavior generating the trajectories, making off-policy value-based learning particularly relevant. Dong et al. (2025) study ICRL from suboptimal historical trajectories and propose an advantage-weighted policy-learning framework to improve upon the behavior policies in the data. Scalable In-Context Q-Learning (Liu et al., 2026) combines context-conditioned value estimation, dynamic programming, and policy extraction to enable iterative improvement from suboptimal offline experience. In-Context Compositional Q-Learning (Xu et al., 2026) instead formulates Q-function estimation as contextual inference and uses retrieved transitions to infer local Q-functions for offline RL. Our work differs fundamentally from these studies by taking a mechanistic view of how Transformers realize Q-learning in ICRL, characterizing both the forward-pass implementation of the Q-learning update and the emergence of the corresponding parameter structure through training. 3 Background 3.1 Transformers and Self-Attention All vectors are treated as column vectors. Let InI_n denote the n×n× n identity matrix, 0m×n0_m× n the m×nm× n zero matrix, and 0m0_m the m-dimensional zero vector. Let =[h1,…,hn]∈ℝD×nH=[h_1,…,h_n] ^D× n denote a prompt representation, where each column hi∈ℝDh_i ^D corresponds to one token. The standard single-head softmax self-attention (Vaswani et al., 2017) processes the prompt via SoftAttn,()≜()softmax(⊤)SoftAttn_P,V(H) (VH)softmax (H PH ), where ∈ℝD×DV ^D× D is the value projection matrix, :=K⊤Q∈ℝD×DP:=W_K W_Q ^D× D collects the key and query projection parameters, the conventional scaling factor in scaled dot-product attention is absorbed into P, and the softmax is applied independently to each query column, normalizing the attention scores over all key positions. As a widely studied alternative to softmax self-attention, linear self-attention replaces the softmax-normalized attention weights with unnormalized bilinear attention scores (Ahn et al. (2023), Gatmiry et al. (2024), Zhang et al. (2024)). Its formulation is given by LinAttn,()≜1n()(⊤)LinAttn_P,V(H) 1n(VH)(H PH), where the factor 1n 1n normalizes the aggregation with respect to the prompt length. To model a multi-layer Transformer structure, we stack L attention layers with residual connections. Let (0)=∈ℝD×nH^(0)=H ^D× n denote the initial input prompt representation. For each layer ℓ∈1,2,…,L ∈\1,2,…,L\, the layer-wise update rule reads (ℓ):=(ℓ−1)+Attn(ℓ),(ℓ)((ℓ−1))∈ℝD×n,H^( ):=H^( -1)+Attn_P^( ),V^( ) (H^( -1) ) ^D× n, (1) where AttnAttn denotes either SoftAttnSoftAttn or LinAttnLinAttn, and (ℓ),(ℓ)\P^( ),V^( )\ are the layer-specific projection parameters at layer ℓ . After stacking L attention layers, the final Transformer output is defined as TFL(,(ℓ),(ℓ)ℓ=1L):=fread((L)), _L (H;\P^( ),V^( )\_ =1^L ):=f_read (H^(L) ), (2) where fread:ℝD×n→ℝDoutf_read ^D× n ^D_out denotes a fixed readout map. Equation (2) establishes the notation convention we adopt for defining the output of an L-layer Transformer. Specifically, the attention mechanism produces a matrix, while we require a vector output for policy improvement. Following prior works (Xie et al. (2026b), Liang and Lai (2026),Wang et al. (2024)), we use the bottom-right d-dimensional vector of the output matrix as this vector. 3.2 Reinforcement Learning and Q-learning We consider an infinite-horizon discounted Markov decision process (MDP) ℰ=(,,P,R,γ)E=(S,A,P,R,γ), where S is the state space, A is the action space, P(⋅∣s,a)∈()P(· s,a) (S) is the transition kernel, R:×→ℝR:S×A×S is the reward function, and γ∈[0,1)γ∈[0,1) is the discount factor. Here, ()P(X) denotes the set of probability distributions over X. At time t, an agent in state StS_t selects an action At∼π(⋅∣St)A_t π(· S_t) according to a policy π:→()π:S (A), receives a reward Rt+1=R(St,At,St+1)R_t+1=R(S_t,A_t,S_t+1), and transitions to a successor state St+1∼P(⋅∣St,At)S_t+1 P(· S_t,A_t). The discounted return from time t is defined as Gt:=∑k=0∞γkRt+k+1G_t:= _k=0^∞γ^kR_t+k+1. The state-value and action-value functions associated with a policy π are respectively defined as vπ(s):=π[Gt∣St=s]v_π(s):=E_π\! [G_t S_t=s ], qπ(s,a):=π[Gt∣St=s,At=a]q_π(s,a):=E_π\! [G_t S_t=s,A_t=a ]. The objective of RL is to find a policy that maximizes the expected discounted return. Q-learning (Watkins, 1989; Watkins and Dayan, 1992) is a classical off-policy TD control algorithm for estimating the optimal action-value function. In the tabular setting, an estimate Qt(s,a)Q_t(s,a) is maintained explicitly for each state-action pair. Given a transition (St,At,Rt+1,St+1)(S_t,A_t,R_t+1,S_t+1), Q-learning updates the value of the visited state-action pair according to Qt+1(St,At)=(1−α)Qt(St,At)+α[Rt+1+γmaxa′∈Qt(St+1,a′)],Q_t+1(S_t,A_t)=(1-α)Q_t(S_t,A_t)+α [R_t+1+γ _a Q_t(S_t+1,a ) ], (3) where α∈(0,1]α∈(0,1] is the learning rate. All other state-action values remain unchanged. To handle large state-action spaces, we consider linear function approximation (Tsitsiklis and Van Roy, 1997; Melo et al., 2008). Given a feature map ϕ:×→ℝdφ:S×A ^d and a parameter vector t∈ℝdw_t ^d at time t, the action-value function is approximated by qt(s,a):=t⊤ϕ(s,a)q_w_t(s,a):=w_t φ(s,a). Given a transition (St,At,Rt+1,St+1)(S_t,A_t,R_t+1,S_t+1), the semi-gradient Q-learning update is t+1=t+α(Rt+1+γmaxa′∈t⊤ϕ(St+1,a′)−t⊤ϕ(St,At))ϕ(St,At).w_t+1=w_t+α (R_t+1+γ _a w_t φ(S_t+1,a )-w_t φ(S_t,A_t) )φ(S_t,A_t). (4) The tabular setting is recovered as a special case when S and A are finite, d=||||d=|S||A|, and each state-action pair (s,a)(s,a) is assigned a distinct one-hot feature vector ϕ(s,a)=(s,a)φ(s,a)=e_(s,a). In this case, each coordinate of tw_t corresponds to a single state-action value Qt(s,a)Q_t(s,a), and the above semi-gradient update reduces exactly to the standard tabular Q-learning update. 4 Transformer can implement In-context soft Q-learning We consider a Boltzmann-smoothed Q-learning update that replaces the hard maximum over successor actions with a Boltzmann-weighted average. Given the current parameter w, trajectories (s0,a0,r1,…,sn−1,an−1,rn,sn)(s_0,a_0,r_1,…,s_n-1,a_n-1,r_n,s_n) can be obtained by interacting with environment under its behavioral policy, where behavioral policy is a behavior policy πb _b, which may differ from greedy target policy. Let z denote a context consisting of an n-transition window, parameter w, and the features of all candidate successor actions. For i=0,…,n−1i=0,…,n-1, define ϕi=ϕ(si,ai) _i=φ(s_i,a_i), ϕi,a+=ϕ(si+1,a)φ^+_i,a=φ(s_i+1,a), a∈a and πi,aτ=exp(⊤ϕi,a+/τ)∑b∈exp(⊤ϕi,b+/τ),ϕ¯i+,τ=∑a∈πi,aτϕi,a+.π^τ_i,a= (w φ^+_i,a/τ) _b (w φ^+_i,b/τ), φ_i^+,τ= _a π^τ_i,aφ^+_i,a. (5) The corresponding batched semi-gradient update is soft(τ)=+αn∑i=0n−1ϕi(ri+1+γ⊤ϕ¯i+,τ−⊤ϕi)w^(τ)_soft=w+ αn _i=0^n-1 _i(r_i+1+ φ_i^+,τ-w _i). Here, ⊤ϕ¯i+,τw φ_i^+,τ is the Boltzmann-weighted average of the successor action values, which replaces the hard maximum maxa∈⊤ϕi,a+ _a w φ_i,a^+ in Q-learning. This soft backup recovers the Q-learning backup in the zero-temperature limit τ↓0τ 0; see Lemma A.1. We now give an explicit Transformer construction and show that its forward pass implements the soft Q-learning update. Each token is partitioned into six blocks, [Φ;G;R;Ψ;U;W][ ;G;R; ;U;W] of dimensions [d,d,1,d,d,d+1][d,d,1,d,d,d+1], respectively. The prompt consists of three types of tokens: transition tokens, which encode the observed transitions; candidate-action tokens, which encode the features of candidate actions at the successor states; and a query token, which carries the current parameter vector. Letting ~=[1;] w=[1;w], for each transition i=0,…,n−1i=0,…,n-1 and candidate action a∈a , we define ti=[ϕi;0;ri+1;0;;0],ci,a=[0;0;0;ϕi,a+;0;0],p=[0;0;0;0;0;~] splitt_i=[ _i;0;r_i+1;0;w;0], c_i,a=[0;0;0;φ^+_i,a;0;0], p=[0;0;0;0;0; w] split (6) Thus, the token dimension is D=5d+2D=5d+2, and the prompt contains N=n(m+1)+1N=n(m+1)+1 tokens. For each transition i, we group the candidate-action tokens as i=[ci,a1,ci,a2,…,ci,am]C_i=[c_i,a^1,c_i,a^2,…,c_i,a^m], where m:=||<∞m:=|A|<∞, so that iC_i contains exactly the candidate-action tokens associated with all actions available at the successor state si+1s_i+1. The resulting input prompt is (0)=[t0,0,t1,1,…,tn−1,n−1,p]. ^(0)= [t_0,C_0,\,t_1,C_1,\,…,\,t_n-1,C_n-1,\,p ]. (7) Our Transformer consists of two successive attention layers. The first layer uses single-head softmax attention to compute a Boltzmann-weighted aggregation over candidate successor actions, thereby implementing a smooth approximation to the maximization operator in Q-learning. The second layer then uses linear attention to realize the corresponding update. Specifically, the first layer is defined as (1)=(0)+(1(0))softmax(((0))⊤1(0)+). ^(1)=H^(0)+ (V_1H^(0) )softmax ((H^(0)) P_1H^(0)+M ). (8) We use ⋆ to denote an arbitrary matrix block of the corresponding dimension. These blocks need not vanish and may take arbitrary values, since they do not affect the entries relevant to the construction under the prescribed attention mask and token structure. We choose 1=[⋆0d×d⋆0d×1⋆θ⋆],1=[⋆Φ⋆0⋆0⋆R⋆0⋆0⋆0⋆U⋆0⋆0⋆ρ] _1= bmatrix & & & & & \\ & & & & & \\ & & & & & \\ 0_d× d& &0_d× 1& &A_θ& \\ & & & & & \\ & & & & & bmatrix, _1= bmatrix & & &D_ & &0\\ & & &L& &0\\ & & &D_R& &0\\ & & &0& &0\\ & & &D_U& &0\\ & & &0& &E_ρ bmatrix (9) where θ=τ−1IdA_θ=τ^-1I_d, ΦD_ , U∈ℝd×dD_U ^d× d, ∈GL(d)L (d), R∈ℝ1×dD_R ^1× d, and ρ=−Id+1E_ρ=Q-I_d+1. Here =[ρb⊤0d×1Id]Q= bmatrixρ&b \\ 0_d× 1&I_d bmatrix, b∈ℝdb ^d, ρ≠0ρ≠ 0, and GL(d)GL(d) denotes the set of invertible d×d× d matrices. We define the attention mask ∈0,−∞N×NM∈\0,-∞\^N× N, where rows index keys and columns index queries. The mask is chosen such that each transition token tit_i attends only to its corresponding candidate-action tokens ci,aa∈\c_i,a\_a , while each ci,ac_i,a and the query token p attend only to p. All other pairs are masked out, ensuring that each tit_i aggregates only its corresponding successor action candidates. The second layer employs single-head linear attention: (2)=(1)+1n(2(1))(((1))⊤2(1)). ^(2)=H^(1)+ 1n(V_2H^(1))\; ((H^(1)) P_2H^(1) ). (10) We choose 2=[⋆−⋆+⋆r⋆U⋆],2=[⋆(4d+2)×d⋆(4d+2)×(d)⋆(4d+2)×(3d+2)G0d×(3d+2)]. _2= bmatrix & & & & &B_-\\ & & & & &B_+\\ & & & & &B_r\\ & & & & & \\ & & & & &B_U\\ & & & & & bmatrix,V_2= bmatrix _(4d+2)× d& _(4d+2)×(d)& _(4d+2)×(3d+2)\\ C&C_G&0_d×(3d+2) bmatrix. (11) where −=−c−1B_-=-cJQ^-1, +=c−⊤[−R⊤,γId+Φ⊤−U⊤Ω]−1B_+=cL^- [-D_R ,γ I_d+D_ -D_U ]Q^-1, r=ce0⊤−1B_r=ce_0 Q^-1, U=cΩ−1B_U=c ^-1, =(α/c)IdC=(α/c)I_d, and G=(−α/c)Φ−1C_G=(-α/c)D_ L^-1. Here, Ω⊤=−Ω =- is arbitrary and may, in particular, be nonzero, =[0,Id]J=[0,I_d], e0⊤=[1,01×d]e_0 =[1,0_1× d], and c≠0c≠ 0. Theorem 1 (Forward pass as soft Q-learning). Consider two-layer Transformer defined above, with masked softmax attention (8) parameterized by 1,1P_1,V_1 in the first layer, and linear attention (10) parameterized by 2,2P_2,V_2 in the second layer. For the input prompt (0)H^(0) in (7), the following hold. (i) The Transformer exactly implements the soft Q-learning update: TF(τ):=TF(τ)((0),1,1,2,2)=+αn∑i=0n−1ϕi[ri+1+γ⊤ϕ¯i+,τ−⊤ϕi],w_TF^(τ):=TF^(τ)(H^(0);P_1,V_1,P_2,V_2)=w+ αn _i=0^n-1 _i [r_i+1+ φ_i^+,τ-w _i ], where ϕ¯i+,τ=∑a∈πi,aτϕi,a+ φ_i^+,τ= _a _i,a^τ _i,a^+, πi,aτ=exp(⊤ϕi,a+/τ)/∑b∈exp(⊤ϕi,b+/τ) _i,a^τ= (w _i,a^+/τ)/ _b (w _i,b^+/τ). (i) For each i, define qi,a:=⊤ϕi,a+q_i,a:=w _i,a^+, qi⋆:=maxa∈qi,aq_i := _a q_i,a, and i⋆:=argmaxa∈qi,aA_i := _a q_i,a. Let m:=||<∞m:=|A|<∞ and Ki:=|i⋆|K_i:=|A_i |. Suppose that ‖ϕ(s,a)‖2⩽Bϕ\|φ(s,a)\|_2 B_φ and |q(s,a)|⩽Bq|q_w(s,a)| B_q for all state-action pairs. For every i such that i⋆≠A_i , define the action gap Δi:=qi⋆−maxa∉i⋆qi,a>0 _i:=q_i - _a _i q_i,a>0. Then limτ↓0TF(τ)=Q _τ 0w_TF^(τ)=w_Q and ‖Q−TF(τ)‖2⩽2αγBqBϕ∑i:i⋆≠m−KinKie−Δi/τ, \|w_Q-w_TF^(τ) \|_2 2αγ B_qB_φ _i:\,A_i m-K_inK_ie^- _i/τ, where Q=+αn∑i=0n−1ϕi[ri+1+γmaxa∈⊤ϕi,a+−⊤ϕi]w_Q=w+ αn _i=0^n-1 _i [r_i+1+γ _a w _i,a^+-w _i ] is window Q-learning update. The proof of Theorem 1 is provided in Appendix A, together with the corresponding result for the tabular setting. Moreover, if each maximizer is unique and Δmin:=miniΔi>0 _ := _i _i>0, then ∥Q−TF(τ)∥2⩽2αγBqBϕ(m−1)e−Δmin/τ\|w_Q-w_TF^(τ)\|_2 2αγ B_qB_φ(m-1)e^- _ /τ, m>1m>1, showing that the Transformer update converges exponentially fast to the standard window Q-learning update as τ↓0τ 0. The theorem thus establishes that a two-layer Transformer can exactly implement soft Q-learning in its forward pass and recover standard Q-learning in the zero-temperature limit. We call this in-context Q-learning. 5 IC-Q-Learning algorithm and training dynamics analysis 5.1 IC-Q-Learning Algorithm and Training Dynamics We now design an ICRL training procedure for the proposed soft Q-learning Transformer, inspired by prior ICRL training frameworks (Laskin et al., 2023; Liang and Lai, 2026), as summarized in Algorithm 1. The training objective follows the teacher-mimicking principle: for each sampled context, the Transformer is trained to predict the parameter obtained after one soft Q-learning update. In contrast to directly supervising the internal attention matrices, the loss is imposed only on the final parameter prediction. Hence, the Transformer is not explicitly provided with the analytical parameter construction derived in the preceding section. Algorithm 1 IC-Q-Learning Transformer Training with Independent-Context Minibatches 0: Transformer parameters (0)=(1(0),1(0),2(0),2(0)) θ^(0)=(P_1^(0),V_1^(0),P_2^(0),V_2^(0)), trajectory window length n, number of optimizer updates U, minibatch size B, ϵε-greedy parameter ϵε, Transformer learning rate η, Q-learning step size α, and temperature τ. 1: for k=0,…,U−1k=0,…,U-1 do 2: for b=1,…,Bb=1,…,B do 3: Independently sample a new MDP ℳb(k)M_b^(k), feature map ϕb(k) _b^(k), b(k)w_b^(k), and initial state sbs_b. 4: Estimate qb(k,b)(s,a)=(b)⊤ϕb(k)(s,a)q_w_b^(k,b)(s,a)=(w_b) _b^(k)(s,a), ∀(s,a)∈×∀(s,a) ×A. 5: Sample an n-step trajectory (sb,0,ab,0,rb,1,sb,1,ab,1,…,rb,n,sb,n)(s_b,0,a_b,0,r_b,1,s_b,1,a_b,1,…,r_b,n,s_b,n) from ℳb(k)M_b^(k) according to the ϵε-greedy behavior policy derived from q(k,b)q_w^(k,b). 6: Formulate bH_b as in (7) using the sampled trajectory, the candidate features ϕb(k)(sb,i+1,a):a∈\ _b^(k)(s_b,i+1,a):a \, and the current parameter bw_b. 7: Compute ~b=b+(1b)softmax((b)⊤1b+) H_b=H_b+(V_1H_b)softmax ((H_b) P_1H_b+M ). 8: Compute TF,b=[~b+1n(2~b)((~b)⊤2~b)]−d:,−1w_TF,b= [ H_b+ 1n(V_2 H_b) (( H_b) P_2 H_b ) ]_-d:,-1. 9: Compute the teacher update soft,b(τ)w^(τ)_soft,b according to soft Q-learning. 10: end for 11: Compute ℒ^B(k)=12B∑b=1B‖TF,b−soft,b(τ)‖22 L_B^(k)= 12B _b=1^B\|w_TF,b-w^(τ)_soft,b\|_2^2. 12: Update (k+1)←(k)−η∇θℒ^B(k) θ^(k+1)← θ^(k)-η _θ L_B^(k) 13: end for 14: return trained Transformer parameters θ. Fix integers d⩾1d 1, m⩾2m 2, n⩾1n 1 and constants α>0α>0, 0<γ<10<γ<1, τ>0τ>0. Let :=(,ϕi,ri+1,ϕi,j+:0⩽i<n,j∈[m])∼nz:= (w,\ _i,r_i+1,φ^+_i,j:0 i<n,\ j∈[m]\ ) _n. Let soft(τ)()w^(τ)_soft(z) denote the corresponding soft Q-learning target. We define the population teacher-mimicking objective as ℒ():=[12‖TFθ(())−soft(τ)()‖22].L( θ):=E_z [ 12\|TF_θ(H(z))-w^(τ)_soft(z)\|_2^2 ]. (12) where the expectation is taken jointly over the training-MDP distribution, random feature maps, behavior trajectories, and independently sampled input Q-function parameters. We optimize the population objective in (12) using independently sampled training contexts. At each optimization step, we draw a minibatch 1,…,B∼i.i.d.nz_1,…,z_B i.i.d. D_n, where each context is generated from an independently sampled MDP and feature map. The empirical loss is ℒ^B():=12B∑b=1B‖TFθ((b))−soft(τ)(b)‖22 L_B( θ):= 12B _b=1^B\|TF_θ(H(z_b))-w^(τ)_soft(z_b)\|_2^2. Here, B controls the amount of averaging across independently sampled training contexts within each parameter update, and hence the stochasticity of the resulting gradient estimate. Further details are provided in Proposition B.1. Write =(Φ,R,U)K=( ,R,U) and ℐ=(Φ,G,R,U)I=( ,G,R,U). Let WoutW_out denote the last d coordinates of W. Let out θ_out collect all entries of =(1,1,2,2) θ=(P_1,V_1,P_2,V_2) outside the following 1616 blocks: 1,ΨP_1, , 1,ℐΨV_1,I , 1,WWV_1,W, 2,ℐWP_2,IW, 2,WoutℐV_2,W_outI. Under the full-matrix population gradient flow ˙=−∇ℒ() θ=- ( θ), if out(0)=0 θ_out(0)=0, then out(t)=0 θ_out(t)=0 for all t⩾0t 0 (Proposition B.2). Consequently, the training dynamics are entirely determined by the retained blocks. We refer to these parameter blocks as valid parameter blocks. However, the resulting parameterization remains highly redundant, with additional degrees of freedom associated with ΦD_ , UD_U, ρ, b, c, L, and Ω . To isolate the essential parameter blocks, we define sup:=(,,−,+,r,)∈Θsup θ_sup:=(A_ θ,L,B_-,B_+,B_r,C)∈ _sup. For any ∈GL(d)L (d) and c≠0c≠ 0, let ¯τ(,c)=(τ−1Id,,[0,−cId],[0,cγ−⊤],[c,0],αcId). θ_τ(L,c)= (τ^-1I_d,\,L,\,[0,-cI_d],\,[0,c ^- ],\,[c,0],\, αcI_d ). (13) These six blocks form an inclusion-minimal sufficient support within the specified architecture: they realize the target update in Theorem 1 on all admissible prompts, whereas no proper subset does so when every other block is fixed to zero (Proposition B.3). We call these six parameter blocks the support parameter blocks. 5.2 Empirical Analysis Each MDP has nS=9n_S=9 states and nA=m=4n_A=m=4 actions, with discount γ=0.5γ=0.5. For each MDP, the initial-state distribution P0P_0 and transition kernel P(⋅∣s,a)P(· s,a) are drawn independently from Dirichlet(1,…,1)Dirichlet(1,…,1) (uniform over the probability simplex on nSn_S states), and the rewards r(a,s′)∼Unif(−1,1)r(a,s ) (-1,1) are sampled i.i.d. per (a,s′)(a,s ). For each MDP, the state-action feature map ϕ:×→ℝdφ ×A ^d (d=nSnA=36d=n_Sn_A=36) and the initial weight ∈ℝdw ^d have entries drawn i.i.d. from Unif(−1,1)Unif(-1,1). The behavior policy is ϵε-greedy (ϵ=0.1ε=0.1) with Q(s,a)=ϕ(s,a)⊤Q(s,a)=φ(s,a) w, and the soft window Q-learning target soft(τ)w^(τ)_soft is computed from the trajectory window with step size α=0.2α=0.2. The Transformer uses embedding dimension D=5d+2=182D=5d+2=182. The prompt supplies w, current features ϕi _i, rewards ri+1r_i+1, and all next-state candidate features ϕi,a+φ^+_i,a. Tabular tests replace features by ϕ(s,a)=e(s−1)m+aφ(s,a)=e_(s-1)m+a and are explicitly separated from dense-feature tests. Training is performed over 46.40 million tasks per model and five paired seeds. Adam uses cosine learning-rate decay from the first update; the default temperature, batch size and context length are τ=0.2τ=0.2, B=32B=32 and n=20n=20. By Proposition B.2, the sixteen valid parameter blocks are initialized using Xavier normal initialization with gain 0.10.1. All remaining entries of the two-layer attention construction are initialized to zero, as they remain fixed during training. We conduct the following experiments, with full details provided in Appendix C.1. Experiments show that parameter blocks other than the sixteen valid parameter blocks stay at zero as training progresse, confirming Proposition B.2. We evaluate the learned update using three complementary metrics: Soft error measures how accurately the Transformer realizes the soft Q update, Hard error measures its discrepancy from standard Q-learning, and Greedy agreement measures how often it selects the same greedy action as Q-learning. The learned Transformer closely approximates the soft Q-learning update and achieves nearly 100% greedy agreement. Figure 1 presents the final learned matrices 1P_1, 1V_1, 2P_2, and 2V_2 obtained after training. Remarkably, the learned matrices recover the parameter structure predicted in Theorem 1. In particular, the diagonal entries of θA_θ in 1P_1 approach 1/τ1/τ, indicating that the Transformer has learned the internal action-selection computation required by the soft Q-learning update. This provides direct evidence of parameter emergence. To quantitatively assess parameter recovery, we examine several construction relations. The learned parameters closely approximate analytical relations, and the task-exposure curves show improved recovery as training progresses. See Appendix C.1 for details of the heatmaps. Figure 1: Final learned 1P_1, 1V_1, 2P_2 and 2V_2 matrices after training at τ=0.2τ=0.2, B=32B=32, and n=20n=20. the non-zero structure matches the theoretical constructions (1,1,2,2)(P_1,V_1,P_2,V_2) in (9) and (11). The left panel of Figure 2 deploys frozen dense-feature-trained Transformers on unseen tabular MDPs and measures exact policy returns under a common interaction budget. Comparisons include hard and soft window Q-learning, a model-informed Oracle, and uniform Random. The Transformer achieves returns close to those of the soft window teacher and a higher sample-mean return than the hard window-Q baseline using the same fixed step size. Figure 2: Left: Performance of Transformers on unseen tabular MDPs (sixteen valid parameter blocks). Right: The learned scalar projection βatt _att and its target 1/τ1/τ (six support parameter blocks). The effect of the action-selection operator. To separately study how temperature affects the emergence of parameters, we analyze the six-block parameterization, which retains the complete soft Q-learning update mechanism and is inclusion-minimal (Proposition B.3). All six blocks remain jointly trainable under output-only supervision. Training is performed over 20.48 million tasks per model. At a given temperature τ, the trained Transformer is shown to approximate the soft Q-learning update and produce emergent parameters, verifying Proposition B.3. Decreasing τ brings the soft Q update closer to standard Q-learning, but increases the approximation error between the Transformer and the soft update. With a fixed training budget, the recovery error of the predicted parameter structure increases gradually as τ decreases. The most pronounced discrepancy appears in the scalar attention scale βatt:=tr(θ)/d _att:=tr(A_θ)/d. As shown in the right panel of Figure 2, the learned scale falls below its theoretical target 1/τ1/τ at low temperatures. This implies that smaller τ requires greater task exposure for accurate parameter recovery. We further examine the effects of batch size and training-context length. See Appendix C.2 for details. 5.3 Theoretical Analysis In this section, we provide a theoretical account of the above empirical findings by explaining how parameter emergence arises from the pretraining algorithm. Our primary focus is to investigate the effect of the action-selection mechanism on convergence. To this end, we restrict optimization to the support parameter blocks, reducing parameter redundancy and isolating the Transformer’s Q-learning mechanism. Define the optimal parameter manifold as ℳτ:=¯τ(,c):∈GL(d),c≠0M_τ:=\ θ_τ(L,c):L (d),\ c≠ 0\, we study whether deterministic gradient descent k+1=k−η∇ℒ(k) θ_k+1= θ_k-η ( θ_k) converges to ℳτM_τ and characterize how its convergence rate and iteration complexity depend on τ. Let V be a vector space with dim=4d2+3d =4d^2+3d and ζ:=(EC,eq,e−,E−,e+,E+,EA)∈ζ:=(E_C,e_q,e_-,E_-,e_+,E_+,E_A) , where the four E-symbols are d×d× d matrices and the three lowercase e-symbols are vectors in ℝdR^d. Let the teacher-weighted covariance be Σiτ=∑a=1mπi,aτ(ϕi,a+−ϕ¯i+,τ)(ϕi,a+−ϕ¯i+,τ)⊤∈ℝd×d _i^τ= _a=1^m _i,a^τ(φ^+_i,a- φ_i^+,τ)(φ^+_i,a- φ_i^+,τ) ^d× d and ℬτ,n()[ζ]= _τ,n(z)[ζ]= 1n∑i=0n−1ECϕiδiτ+ϕi[ϕi⊤(e−+E−)+(ϕ¯i+,τ)⊤(e++E+) 1n _i=0^n-1 \E_C _i _i^τ+ _i [ _i (e_-+E_-w)+( φ_i^+,τ) (e_++E_+w) +ri+1eq⊤+⊤ΣiτEA]. +r_i+1e_q w+w _i^τE_Aw ] \. (14) where δiτ()=ri+1−⊤ϕi+γ⊤ϕ¯i+,τ() _i^τ(z)=r_i+1-w _i+ φ_i^+,τ(z). In these coordinates, the adjoint ℬτ,n()∗:ℝd→B_τ,n(z)^*:R^d is represented by the transpose of ℬτ,n()B_τ,n(z). We define the dim×dim × data Gram matrix as n(τ)=[ℬτ,n()∗ℬτ,n()] H_n(τ)=E[B_τ,n(z)^*B_τ,n(z)], and denote its smallest eigenvalue by χ=λmin(n(τ))χ= _ ( H_n(τ)). We now state the richness assumptions on the training-MDP distribution Assumption 1. Let Π denote the class of behavior policies used to generate training trajectories. There exist positive constants Bw,Bϕ,BrB_w,B_φ,B_r such that, for every π∈Ππ∈ , nD_n is independent of θ and (i) Boundedness. ‖2⩽Bw\|w\|_2 B_w, ‖ϕi‖2,‖ϕi,a+‖2⩽Bϕ\| _i\|_2,\|φ^+_i,a\|_2 B_φ, |ri+1|⩽Br|r_i+1| B_r, n-a.s.D_n-a.s. (i) Joint observable richness. For the fixed (n,τ)(n,τ), n(τ)≻0 H_n(τ) 0, meaning ⟨ζ,n(τ)ζ⟩>0 ζ, H_n(τ)ζ >0 for every ζ≠0ζ≠ 0. Equivalently, ℬτ,n()[ζ]=0B_τ,n(z)[ζ]=0, n-a.s.⟹ζ=0D_n-a.s. ζ=0. Assumption 1 holds for a broad class of task populations with a nonzero absolutely continuous component and sufficient trajectory support, including the distribution used in our experiments. Explicit sufficient conditions and their verification are given in Appendix E. Fix a nonempty relatively compact open set ⋐GL(d)×(ℝ∖0) (d)×(R \0\). Its closure ¯ is compact and remains inside GL(d)×(ℝ∖0)GL(d)×(R \0\). Define its images τ=¯τ(,c):(,c)∈U_τ=\ θ_τ(L,c):(L,c)∈ \, ¯τ=¯τ(,c):(,c)∈¯ U_τ=\ θ_τ(L,c):(L,c)∈ \. The set ¯τ U_τ is compact at each fixed τ. At a point ¯∈ℳτ θ _τ, the tangent space T¯ℳτT_ θM_τ consists of derivatives at zero of smooth curves lying in ℳτM_τ and passing through ¯ θ. The normal space is its orthogonal complement, N¯ℳτ=(T¯ℳτ)⟂N_ θM_τ=(T_ θM_τ) , define the tubular neighborhood of τU_τ as :=r(τ)=¯+Δ:¯∈τ,Δ∈N¯ℳτ,∥Δ∥<r.T:=T_r(U_τ)=\ θ+ : θ _τ,\ ∈ N_ θM_τ,\ \| \|<r\. (15) A perturbation ∈ΘsupU∈ _sup has blocks denoted by ΔA,…,ΔC A,…, C in the same order as (13). Let κ denote the minimum local curvature normal to ℳτM_τ, defined by κ:=inf¯∈τ,⟂T¯ℳτ,‖=1⟨,∇2ℒ(¯)⟩>0.κ:= _ subarrayc θ∈U_τ,U T_ θM_τ,\|U\|=1 subarray ,∇^2L( θ)U >0. (16) Let Tη()=−η∇ℒ()T_η( θ)= θ-η ( θ) and d():=dist(,ℳτ)d( θ):=dist( θ,M_τ), define KGD(ε,0):=minK∈ℕ0:d(TηK(0))⩽εd(0).K_GD( ; θ_0):= \K _0:d (T_η^K( θ_0) ) d( θ_0) \. Theorem 2. Under Assumption 1, there exist a neighborhood T, a constant MJ>0M_J>0 such that, for any ε∈(0,1/2] ∈(0,1/2], 0<η⩽12MJ20<η 12M_J^2, and 0∈τ:=∈:d()>0,4MJκd()<dist(,c) θ_0 _τ:=\ θ :d( θ)>0, 4M_J κd( θ)<dist( θ,T^c)\, (i) The gradient descent k+1=k−η∇ℒ(k) θ_k+1= θ_k-η ( θ_k), converges to some ∞∈τ θ_∞ _τ, and ‖k−∞‖⩽4MJκd(0)(1−14ηκ)k/2\| θ_k- θ_∞\| 4M_J κd( θ_0)(1- 14ηκ)^k/2, where 1−14ηκ∈(0,1)1- 14ηκ∈(0,1). (i) KGD(ε,0)=O(1ηκlog1ε)K_GD( ; θ_0)=O( 1ηκ 1 ) for every 0∈τ θ_0 _τ, and sup0∈τKGD(ε,0)=Θ(1ηκlog1ε) _ θ_0 _τK_GD( ; θ_0)= _ ( 1ηκ 1 ). In particular, under our experimental setting, we obtain the following corollary. Corollary 3. Let feature table Φ:×→ℝd :S×A ^d, reward table R:×→ℝR:A×S , and an input weight w, ⊂ℕN be a nonempty finite set of context lengths. Suppose nS,d,m⩾2n_S,d,m 2, P0P_0 and all transition rows are mutually independent Dirichlet(1,…,1)Dirichlet(1,…,1), independent of (,Φ,R)(w, ,R), whose entries are mutually independent Unif(−1,1)Unif(-1,1). The behavior policy is ϵb _b-greedy with uniform exploration, 0<εb⩽10< _b 1. There exists τ∗>0 _*>0 such that, for 0<τ⩽τ∗0<τ _*, n∈n and η satisfying Theorem 2, KGD(ε,0)=O(1ητ5log1ε)K_GD( ; θ_0)=O( 1ητ^5 1 ) for every 0∈τ θ_0 _τ, and sup0∈τKGD(ε,0,τ,n)=Θ,(1ητ5log1ε) _ θ_0 _τK_GD( ; θ_0,τ,n)= _ ,N( 1ητ^5 1 ). The proofs of Theorem 2 and Corollary 3 are provided in Appendices D and E. Corollary 3 shows that as τ decreases, the local curvature diminishes, slowing convergence and requiring more iterations to achieve the same optimization accuracy, consistent with our empirical results. 6 Discussion and limitations This work takes a step toward white-boxing in-context reinforcement learning for off-policy control. We show that an attention-only Transformer can implement a finite-temperature Q-learning update in its forward pass and recover standard window Q-learning as the temperature approaches zero. We further show that the corresponding parameter structure can emerge through teacher-mimicking training, and establish local exponential convergence to a manifold of functionally equivalent soft Q-learning realizations, with particular attention to how action selection in soft Q-updates affects parameter emergence. These results provide a mechanistic account of how Q-learning can be represented and learned within Transformer computation. Our analysis has several limitations. First, our theoretical analysis is restricted to the support parameter blocks because the valid parameter blocks exhibit substantial redundancy; extending the convergence analysis to all valid blocks remains future work. Second, we do not analyze minibatch SGD, which would be needed to characterize the dependence of convergence on minibatch size. AI use statement Generative AI tools were used to assist with code review, experimental implementation and analysis, verification of mathematical derivations, and linguistic refinement of the manuscript. In Appendix E, AI tools contributed key proof steps that were adopted by the authors. The core ideas of the other proofs were developed by the authors, with AI assistance in checking mathematical arguments, carrying out partial algebraic calculations, and identifying and correcting errors. The authors checked the proofs step by step and executed and tested the AI-assisted programs. All numerical results reported in the paper were obtained from executable simulation or model-evaluation code and its saved outputs. All AI-assisted content was reviewed and verified by the authors, who retain full responsibility for the accuracy, validity, and final presentation of this work. Ethics Statement This work investigates the representation and learning of Q-learning updates in structured Transformers through mathematical analysis and experiments on synthetically generated Markov decision processes. The study does not involve human participants, personal or sensitive data, or real-world deployment. We do not identify specific ethical concerns arising from the reported experiments. The results apply under the stated architectural and mathematical assumptions and do not constitute a safety guarantee for real-world reinforcement-learning deployment. Reproducibility Statement The Transformer architecture, prompt construction, attention mask, and teacher-mimicking objective are specified in Sections 4 and 5 and Algorithm 1. The assumptions and proofs supporting the theoretical results are provided in Appendices A, B, D, and E. Section 5.2 and Appendix C describe the synthetic task generation, training configurations, evaluation protocols, and definitions of the reported metrics. Appendix C also reports results across five paired random seeds and describes the uncertainty estimates used in the experiments. An anonymized implementation is provided in the supplementary material. References Ahn et al. (2023) K. Ahn, X. Cheng, H. Daneshmand, and S. Sra Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems 36, p. 45614–45650. Cited by: §3.1. Chen et al. (2021) L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch Decision transformer: reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, Vol. 34. Cited by: §2. Dong et al. (2025) J. Dong, M. Guo, E. X. Fang, Z. Yang, and V. Tarokh In-context reinforcement learning from suboptimal historical data. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 14021–14039. Cited by: §2. Gatmiry et al. (2024) K. Gatmiry, N. Saunshi, S. J. Reddi, S. Jegelka, and S. Kumar Can looped transformers learn to implement multi-step gradient descent for in-context learning?. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §3.1. Grigsby et al. (2024) J. Grigsby, L. Fan, and Y. Zhu AMAGO: scalable in-context reinforcement learning for adaptive agents. In International Conference on Learning Representations, Cited by: §2. Janner et al. (2021) M. Janner, Q. Li, and S. Levine Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, Vol. 34, p. 1273–1286. Cited by: §2. Laskin et al. (2023) M. Laskin, L. Wang, J. Oh, E. Parisotto, S. Spencer, R. Steigerwald, D. Strouse, S. Hansen, A. Filos, E. Brooks, M. Gazeau, H. Sahni, S. Singh, and V. Mnih In-context reinforcement learning with algorithm distillation. In International Conference on Learning Representations, Cited by: §2, §5.1. Lee (2013) J. M. Lee Introduction to smooth manifolds. 2 edition, Graduate Texts in Mathematics, Vol. 218, Springer, New York. External Links: Document Cited by: §D.1. Lee et al. (2023) J. Lee, A. Xie, A. Pacchiano, Y. Chandak, C. Finn, O. Nachum, and E. Brunskill Supervised pretraining can learn in-context reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §2. Liang and Lai (2026) H. Liang and L. Lai Transformers provably implement in-context reinforcement learning with policy improvement. arXiv preprint arXiv:2605.05755. Cited by: §1, §1, §2, §3.1, §5.1. Lin et al. (2024) L. Lin, Y. Bai, and S. Mei Transformers as decision makers: provable in-context reinforcement learning via supervised pretraining. In International Conference on Learning Representations, Cited by: §2. Liu et al. (2026) J. Liu, F. Liu, Z. Sun, J. Hao, H. Li, B. Wang, D. Dong, C. Chen, and Z. Wang Scalable in-context q-learning. In International Conference on Learning Representations, Cited by: §2. Melo et al. (2008) F. S. Melo, S. P. Meyn, and M. I. Ribeiro An analysis of reinforcement learning with function approximation. In Proceedings of the 25th International Conference on Machine Learning, p. 664–671. External Links: Document Cited by: §3.2. Tsitsiklis and Van Roy (1997) J. N. Tsitsiklis and B. Van Roy An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control 42 (5), p. 674–690. Cited by: §3.2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §3.1. Wang et al. (2024) J. Wang, E. H. Blaser, H. Daneshmand, and S. Zhang Transformers learn temporal difference methods for in-context reinforcement learning. In ICML 2024 Workshop on In-Context Learning, Cited by: §1, §2, §3.1. Wang et al. (2025) J. Wang, R. Chandra, and S. Zhang Towards provable emergence of in-context reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §1, §2. Watkins and Dayan (1992) C. J. C. H. Watkins and P. Dayan Q-learning. Machine Learning 8 (3–4), p. 279–292. External Links: Document Cited by: §3.2. Watkins (1989) C. J. C. H. Watkins Learning from delayed rewards. Ph.D. Thesis, King’s College, University of Cambridge. Cited by: §3.2. Xie et al. (2026a) Z. Xie, X. Liu, R. Chandra, and S. Zhang Convergence and emergence of in-context reinforcement learning with chain of thought. External Links: 2605.07123, Link Cited by: §1. Xie et al. (2026b) Z. Xie, X. Liu, C. Chen, S. D. Liu, R. Chandra, and S. Zhang Beyond linear attention: softmax transformers implement in-context reinforcement learning. arXiv preprint arXiv:2605.07333. Cited by: §1, §2, §3.1. Xu et al. (2026) Q. Xu, Y. Huang, Y. Jiang, W. Zheng, L. Song, J. Wang, and J. Bian In-context compositional q-learning for offline reinforcement learning. In International Conference on Learning Representations, Cited by: §2. Zhang et al. (2024) R. Zhang, S. Frei, and P. L. Bartlett Trained transformers learn linear models in-context. Journal of Machine Learning Research 25 (49), p. 1–55. Cited by: §3.1. Appendix A Two-layer Transformer for soft Q-learning Softmax weights at finite temperature τ are not exactly equivalent to max . Recall the definition of πi,aτπ^τ_i,a in (5). Lemma A.1 states that softmax operator can approximate max arbitrarily well, with the true hard-max limit recovered only as the softmax temperature τ→0τ→ 0. Lemma A.1. For each i, define qi,a:=⊤ϕi,a+q_i,a:=w _i,a^+, qi⋆:=maxa∈qi,aq_i := _a q_i,a, and i⋆:=argmaxa∈qi,aA_i := _a q_i,a. Let m:=||<∞m:=|A|<∞ and Ki:=|i⋆|K_i:=|A_i |. Then, as τ↓0τ 0, πi,aτ→1Kia∈i⋆ _i,a^τ→ 1K_i1\a _i \. If i⋆≠A_i , define the action gap Δi:=qi⋆−maxa∉i⋆qi,a>0 _i:=q_i - _a _i q_i,a>0, then 1−∑a∈i⋆πi,aτ⩽[(m−Ki)/Ki]e−Δi/τ1- _a _i _i,a^τ [(m-K_i)/K_i]e^- _i/τ. Proof of Lemma A.1. Fix i. Dividing the numerator and denominator of πi,aτ _i,a^τ by exp(qi⋆/τ) (q_i /τ) gives πi,aτ=exp(−(qi⋆−qi,a)/τ)∑b∈exp(−(qi⋆−qi,b)/τ). _i,a^τ= (-(q_i -q_i,a)/τ) _b (-(q_i -q_i,b)/τ). For every a∈i⋆a _i , we have qi⋆−qi,a=0q_i -q_i,a=0, whereas for a∉i⋆a _i , qi⋆−qi,a>0q_i -q_i,a>0. Hence, as τ→0τ→ 0, ∑b∈exp(−(qi⋆−qi,b)/τ)→Ki _b (-(q_i -q_i,b)/τ)→ K_i, which yields πi,aτ→1/Ki _i,a^τ→ 1/K_i, for a∈i⋆a _i , πi,aτ→0 _i,a^τ→ 0, for a∉i⋆a _i . Therefore, ∑a∈i⋆πi,aτ→1 _a _i _i,a^τ→ 1. Now suppose i⋆≠A_i and define Δi=qi⋆−maxa∉i⋆qi,a>0 _i=q_i - _a _i q_i,a>0. Then 1−∑a∈i⋆πi,aτ=∑a∉i⋆e−(qi⋆−qi,a)/τKi+∑a∉i⋆e−(qi⋆−qi,a)/τ.1- _a _i _i,a^τ= _a _i e^-(q_i -q_i,a)/τK_i+ _a _i e^-(q_i -q_i,a)/τ. Since qi⋆−qi,a⩾Δiq_i -q_i,a _i for all a∉i⋆a _i , 1−∑a∈i⋆πi,aτ⩽m−KiKie−Δi/τ.1- _a _i _i,a^τ m-K_iK_ie^- _i/τ. When Ki=1K_i=1, this reduces to 1−πi,ai⋆τ⩽(m−1)e−Δi/τ1- _i,a_i ^τ (m-1)e^- _i/τ. ∎ We proceed to prove Theorem 1. Our Transformer architecture is two-layered, where the first layer adopts single-head softmax attention to produce Boltzmann-weighted aggregation across all candidate successor actions. To facilitate the proof, we first formally characterize the output behavior of this layer in Lemma A.2, before establishing the full result of Theorem 1. We first recall the first-layer attention mask. Let TiT_i, Ci,aC_i,a, and PoP_o denote the positions of tit_i, ci,ac_i,a, and p, respectively. For each transition query TiT_i, only its associated candidate-action tokens are unmasked: Cj,a,Ti=0,j=i,−∞,j≠i,Tj,Ti=Po,Ti=−∞.M_C_j,a,T_i= cases0,&j=i,\\ -∞,&j≠ i, cases _T_j,T_i=M_P_o,T_i=-∞. For each candidate-action query and for the parameter query, only the parameter token p is unmasked: Po,Ci,a=0M_P_o,C_i,a=0, Po,Po=0M_P_o,P_o=0, and all remaining entries are set to −∞-∞. Hence, each transition token tit_i attends exactly to its own candidate-action set ci,aa∈\c_i,a\_a . Lemma A.2 (First layer computes the Boltzmann-weighted successor feature). Consider the single-head masked softmax attention layer in (8) with mask M and parameters 1,1P_1,V_1 specified in (9). Define ζQ=(−Id+1)~=[ρ−1+b⊤0d],~Q=~=[ρ+b⊤]. _Q=(Q-I_d+1) w= bmatrixρ-1+b w\\ 0_d bmatrix, w_Q=Q w= bmatrixρ+b w\\ w bmatrix. For the input prompt (0)H^(0) in (7), the output is (1)=[t0(1),0(1),t1(1),1(1),…,tn−1(1),n−1(1),p(1)],H^(1)= [t_0^(1),C_0^(1),t_1^(1),C_1^(1),…,t_n-1^(1),C_n-1^(1),p^(1) ], where i(1)=[ci,a(1)]a∈C_i^(1)=[c_i,a^(1)]_a and ti(1)=[ϕi+Φϕ¯i+,τϕ¯i+,τri+1+Rϕ¯i+,τ0d+Uϕ¯i+,τ0d+1],ci,a(1)=[0d0d0ϕi,a+0dζQ],p(1)=[0d0d00d0d~Q].t_i^(1)= bmatrix _i+D_ φ_i^+,τ\\ L φ_i^+,τ\\ r_i+1+D_R φ_i^+,τ\\ 0_d\\ w+D_U φ_i^+,τ\\ 0_d+1 bmatrix, c_i,a^(1)= bmatrix0_d\\ 0_d\\ 0\\ φ^+_i,a\\ 0_d\\ _Q bmatrix, p^(1)= bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ w_Q bmatrix. (A.1) Proof. Write (0)=[h1,…,hN]H^(0)=[h_1,…,h_N]. Let ¯:=softmax(((0))⊤1(0)+) A:=softmax ((H^(0)) P_1H^(0)+M ), where the softmax is applied columnwise over the key positions. Thus, if hkh_k and hqh_q denote the tokens at key position k and query position q, respectively, then ¯k,q=exp(hk⊤1hq+k,q)∑ℓ=1Nexp(hℓ⊤1hq+ℓ,q). A_k,q= (h_k P_1h_q+M_k,q) _ =1^N (h_ P_1h_q+M_ ,q). (A.2) Since the kkth column of 1(0)V_1H^(0) is 1hkV_1h_k, the qqth column of the attention output satisfies [(1(0))¯]:,q=∑k=1N¯k,q1hk [(V_1H^(0)) A ]_:,q= _k=1^N A_k,qV_1h_k. Therefore, using the residual connection, the output corresponding to query token hqh_q is hq(1)=hq+∑k=1N¯k,q1hk.h_q^(1)=h_q+ _k=1^N A_k,qV_1h_k. (A.3) We first consider a transition token tit_i. By construction of the mask, the query tit_i can attend only to its associated candidate-action tokens ci,aa∈\c_i,a\_a . Hence, the only finite attention scores in the column corresponding to tit_i are ci,a⊤1tic_i,a P_1t_i, a∈a . Recall that ti=[ϕi;0d;ri+1;0d;;0d+1]t_i=[ _i;0_d;r_i+1;0_d;w;0_d+1], cj,a=[0d;0d;0;ϕj,a+;0d;0d+1]c_j,a=[0_d;0_d;0;φ^+_j,a;0_d;0_d+1]. Since cj,ac_j,a is nonzero only in its Ψ block, the inner product cj,a⊤1tic_j,a P_1t_i depends only on the Ψ block of 1tiP_1t_i. From the fourth block row of 1P_1, (1ti)Ψ=0d×dϕi+⋆0d+0d×1ri+1+⋆0d+θ+⋆0d+1(P_1t_i)_ =0_d× d _i+ 0_d+0_d× 1r_i+1+ 0_d+A_θw+ 0_d+1. All terms except the fifth vanish, so (1ti)Ψ=θ(P_1t_i)_ =A_θw. Since θ=τ−1IdA_θ=τ^-1I_d, we obtain (1ti)Ψ=τ−1(P_1t_i)_ =τ^-1w. Therefore, for any candidate token cj,ac_j,a, cj,a⊤1ti=(ϕj,a+)⊤(1ti)Ψ=1τ⊤ϕj,a+c_j,a P_1t_i=(φ^+_j,a) (P_1t_i)_ = 1τw φ^+_j,a. The arbitrary ⋆ blocks in 1P_1 do not affect the relevant attention scores. Although 1tiP_1t_i may have nonzero components in other blocks, the candidate token cj,ac_j,a has zeros in all of those coordinates and therefore removes them in the inner product. The mask now restricts the query tit_i to the candidates with the same transition index i. Hence, MCi,a,Ti=0M_C_i,a,T_i=0, and Mk,Ti=−∞M_k,T_i=-∞ for every other key position k. It follows from (A.2) that ¯Ci,a,Ti=exp(ci,a⊤1ti)∑b∈exp(ci,b⊤1ti)=exp(⊤ϕi,a+/τ)∑b∈exp(⊤ϕi,b+/τ)=πi,aτ. A_C_i,a,T_i= (c_i,a P_1t_i) _b (c_i,b P_1t_i)= (w φ^+_i,a/τ) _b (w φ^+_i,b/τ)= _i,a^τ. (A.4) All other attention weights in the TiT_ith column are zero. We next compute the corresponding value vectors. Since ci,ac_i,a is nonzero only in its Ψ block, only the fourth block column of 1V_1 contributes to 1ci,aV_1c_i,a. From the prescribed form of 1V_1, this block column is (1):,Ψ=[ΦR0d×dU0(d+1)×d],1ci,a=[Φϕi,a+ϕi,a+Rϕi,a+0dUϕi,a+0d+1](V_1)_:, = bmatrixD_ \\ L\\ D_R\\ 0_d× d\\ D_U\\ 0_(d+1)× d bmatrix, V_1c_i,a= bmatrixD_ φ^+_i,a\\ Lφ^+_i,a\\ D_Rφ^+_i,a\\ 0_d\\ D_Uφ^+_i,a\\ 0_d+1 bmatrix Hence, the attention contribution to the transition query tit_i is [(1(0))¯]:,Ti [(V_1H^(0)) A ]_:,T_i =∑a∈¯Ci,a,Ti1ci,a=∑a∈πi,aτ1ci,a = _a A_C_i,a,T_iV_1c_i,a= _a _i,a^τV_1c_i,a =[Φ∑a∈πi,aτϕi,a+∑a∈πi,aτϕi,a+R∑a∈πi,aτϕi,a+0dU∑a∈πi,aτϕi,a+0d+1]=[Φϕ¯i+,τϕ¯i+,τRϕ¯i+,τ0dUϕ¯i+,τ0d+1], = bmatrixD_ _a _i,a^τφ^+_i,a\\ L _a _i,a^τφ^+_i,a\\ D_R _a _i,a^τφ^+_i,a\\ 0_d\\ D_U _a _i,a^τφ^+_i,a\\ 0_d+1 bmatrix= bmatrixD_ φ_i^+,τ\\ L φ_i^+,τ\\ D_R φ_i^+,τ\\ 0_d\\ D_U φ_i^+,τ\\ 0_d+1 bmatrix, (A.5) where we used ϕ¯i+,τ=∑a∈πi,aτϕi,a+ φ_i^+,τ= _a _i,a^τφ^+_i,a in the last equality. Finally, applying the residual connection gives ti(1)=ti+[(1(0))¯]:,Ti=[ϕi0dri+10d0d+1]+[Φϕ¯i+,τϕ¯i+,τRϕ¯i+,τ0dUϕ¯i+,τ0d+1]=[ϕi+Φϕ¯i+,τϕ¯i+,τri+1+Rϕ¯i+,τ0d+Uϕ¯i+,τ0d+1]. t_i^(1)=t_i+ [(V_1H^(0)) A ]_:,T_i= bmatrix _i\\ 0_d\\ r_i+1\\ 0_d\\ w\\ 0_d+1 bmatrix+ bmatrixD_ φ_i^+,τ\\ L φ_i^+,τ\\ D_R φ_i^+,τ\\ 0_d\\ D_U φ_i^+,τ\\ 0_d+1 bmatrix= bmatrix _i+D_ φ_i^+,τ\\ L φ_i^+,τ\\ r_i+1+D_R φ_i^+,τ\\ 0_d\\ w+D_U φ_i^+,τ\\ 0_d+1 bmatrix. (A.6) It remains to compute the candidate-action tokens and the parameter token. By the mask construction, each candidate-action query ci,ac_i,a and the parameter query p can attend only to p. There is exactly one unmasked key in each of these columns, so ¯Po,Ci,a=exp(p⊤1ci,a)exp(p⊤1ci,a)=1,¯Po,Po=exp(p⊤1p)exp(p⊤1p)=1. A_P_o,C_i,a= (p P_1c_i,a) (p P_1c_i,a)=1, A_P_o,P_o= (p P_1p) (p P_1p)=1. All other attention weights in these columns are zero. Recall that p=[0d;0d;0;0d;0d;~]p=[0_d;0_d;0;0_d;0_d; w]. The last block column of 1V_1, corresponding to the W block, is zero except for its last block −Id+1Q-I_d+1. Hence 1p=[0d0d00d0d(−Id+1)~]=[0d0d00d0dζQ],V_1p= bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ (Q-I_d+1) w bmatrix= bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ _Q bmatrix, where, explicitly, (−Id+1)~=[ρ−1b⊤0d×10d×d][1]=[ρ−1+b⊤0d]=ζQ(Q-I_d+1) w= bmatrixρ-1&b \\ 0_d× 1&0_d× d bmatrix bmatrix1\\ w bmatrix= bmatrixρ-1+b w\\ 0_d bmatrix= _Q. Therefore, the attention outputs for all candidate-action queries and for the parameter query are [(1(0))¯]:,Ci,a=1p [(V_1H^(0)) A ]_:,C_i,a=V_1p, [(1(0))¯]:,Po=1p [(V_1H^(0)) A ]_:,P_o=V_1p. Using the residual connection once again, ci,a(1)=ci,a+1p=[0d0d0ϕi,a+0d0d+1]+[0d0d00d0dζQ]=[0d0d0ϕi,a+0dζQ],p(1)=p+1p=[0d0d00d0d~]+[0d0d00d0dζQ]=[0d0d00d0d~Q], c_i,a^(1)=c_i,a+V_1p= bmatrix0_d\\ 0_d\\ 0\\ φ^+_i,a\\ 0_d\\ 0_d+1 bmatrix+ bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ _Q bmatrix= bmatrix0_d\\ 0_d\\ 0\\ φ^+_i,a\\ 0_d\\ _Q bmatrix, p^(1)=p+V_1p= bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ w bmatrix+ bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ _Q bmatrix= bmatrix0_d\\ 0_d\\ 0\\ 0_d\\ 0_d\\ w_Q bmatrix, where ~+ζQ=(Id+1+−Id+1)~=~=~Q w+ _Q= (I_d+1+Q-I_d+1 ) w=Q w= w_Q. Thus, (1)=[t0(1),0(1),t1(1),1(1),…,tn−1(1),n−1(1),p(1)]H^(1)= [t_0^(1),C_0^(1),t_1^(1),C_1^(1),…,t_n-1^(1),C_n-1^(1),p^(1) ], with all three types of tokens as stated. In particular, the candidate-action tokens and the parameter token remain unchanged for every w precisely when ρ=1ρ=1 and b=0b=0, that is, when =Id+1Q=I_d+1. ∎ With the above lemma established, we proceed to prove Theorem 1. Proof of Theorem 1. (i) By Lemma A.2, after the first attention layer, each transition token takes the form ti(1)=[ϕi+Φϕ¯i+,τϕ¯i+,τri+1+Rϕ¯i+,τ0d+Uϕ¯i+,τ0d+1],ϕ¯i+,τ=∑a∈πi,aτϕi,a+.t_i^(1)= bmatrix _i+D_ φ_i^+,τ\\ L φ_i^+,τ\\ r_i+1+D_R φ_i^+,τ\\ 0_d\\ w+D_U φ_i^+,τ\\ 0_d+1 bmatrix, φ_i^+,τ= _a _i,a^τφ^+_i,a. The candidate-action tokens and the parameter token have the forms ci,a(1)=[0d;0d;0;ϕi,a+;0d;ζQ]c_i,a^(1)=[0_d;0_d;0;φ^+_i,a;0_d; _Q], p(1)=[0d;0d;0;0d;0d;~Q]p^(1)=[0_d;0_d;0;0_d;0_d; w_Q]. In particular, the Φ and G blocks of these latter tokens are still zero, and the only possibly nonzero block of p(1)p^(1) is its W block. We now consider the second linear-attention layer. Write (1)=[h1(1),…,hN(1)]H^(1)=[h_1^(1),…,h_N^(1)]. From (2)=(1)+1n(2(1))(((1))⊤2(1))H^(2)=H^(1)+ 1n(V_2H^(1)) ((H^(1)) P_2H^(1) ), the update to the parameter token is Δp:=p(2)−p(1)=1n∑j=1N2hj(1)((hj(1))⊤2p(1)). p:=p^(2)-p^(1)= 1n _j=1^NV_2h_j^(1) ((h_j^(1)) P_2p^(1) ). (A.7) Indeed, this is simply the column of the matrix product corresponding to the query token p(1)p^(1): the jjth entry of the corresponding score column is (hj(1))⊤2p(1)(h_j^(1)) P_2p^(1), and the jjth value column is 2hj(1)V_2h_j^(1). We only need the last d coordinates of Δp p, since the readout extracts these coordinates of the parameter token. Recall Πout=[0d×(4d+2),Id] _out=[0_d×(4d+2),I_d], =αcIdC= αcI_d, G=−Φ−1C_G=-CD_ L^-1. By the block structure of 2V_2, for any token h, Πout2h=hΦ+GhG, _outV_2h=Ch_ +C_Gh_G, (A.8) where hΦh_ and hGh_G denote its Φ and G blocks. The arbitrary entries in the first 4d+24d+2 rows of 2V_2 do not appear in this expression. Therefore, Πout2ti(1) _outV_2t_i^(1) =(ϕi+Φϕ¯i+,τ)+Gϕ¯i+,τ =C ( _i+D_ φ_i^+,τ )+C_GL φ_i^+,τ =ϕi+Φϕ¯i+,τ−Φ−1ϕ¯i+,τ =C _i+CD_ φ_i^+,τ-CD_ L^-1L φ_i^+,τ =ϕi+Φϕ¯i+,τ−Φϕ¯i+,τ=ϕi=αcϕi. =C _i+CD_ φ_i^+,τ-CD_ φ_i^+,τ=C _i= αc _i. (A.9) Here the second equality uses G=−Φ−1C_G=-CD_ L^-1, and the third uses −1=IdL^-1L=I_d. On the other hand, both ci,a(1)c_i,a^(1) and p(1)p^(1) have zero Φ and G blocks. Applying (A.8) gives Πout2ci,a(1)=0d+G0d=0d,Πout2p(1)=0d+G0d=0d. _outV_2c_i,a^(1)=C0_d+C_G0_d=0_d, _outV_2p^(1)=C0_d+C_G0_d=0_d. (A.10) Thus, only the transition tokens contribute to the last d coordinates of the parameter update. The additional W block ζQ _Q in each candidate token does not affect this conclusion, since the last d rows of the W block column of 2V_2 are zero. No assumption that the candidate or parameter scores vanish is needed: their projected value vectors already vanish. It remains to compute the scalar score (ti(1))⊤2p(1)(t_i^(1)) P_2p^(1). Since p(1)=[0d;0d;0;0d;0d;~Q]p^(1)=[0_d;0_d;0;0_d;0_d; w_Q], ~Q=~=[1] w_Q=Q w=Q bmatrix1\\ w bmatrix, only the last block column of 2P_2 contributes to 2p(1)P_2p^(1). Because Q is invertible, −1~Q=−1~=~=[1]Q^-1 w_Q=Q^-1Q w= w= bmatrix1\\ w bmatrix. From the definition of the specified blocks, we obtain −~Q B_- w_Q =−c−1~=−c[0d×1,Id][1]=−c, =-cJQ^-1Q w=-c[0_d× 1,I_d] bmatrix1\\ w bmatrix=-cw, (A.11) +~Q B_+ w_Q =c−⊤[−R⊤,γId+Φ⊤−U⊤Ω]−1~ =cL^- [-D_R ,γ I_d+D_ -D_U ]Q^-1Q w =c−⊤[−R⊤,γId+Φ⊤−U⊤Ω][1] =cL^- [-D_R ,γ I_d+D_ -D_U ] bmatrix1\\ w bmatrix =c−⊤[−R⊤+(γId+Φ⊤−U⊤Ω)], =cL^- [-D_R +(γ I_d+D_ -D_U )w ], (A.12) r~Q B_r w_Q =ce0⊤−1~=c[1,01×d][1]=c, =ce_0 Q^-1Q w=c[1,0_1× d] bmatrix1\\ w bmatrix=c, (A.13) U~Q B_U w_Q =cΩJ−1~=cΩ[0d×1,Id][1]=cΩ. =c JQ^-1Q w=c [0_d× 1,I_d] bmatrix1\\ w bmatrix=c w. (A.14) The fourth and sixth block rows of 2p(1)P_2p^(1) may contain arbitrary terms arising from the corresponding ⋆ blocks. However, the Ψ and W blocks of ti(1)t_i^(1) are both zero, so these terms do not contribute to the bilinear score. Consequently, (ti(1))⊤2p(1)= (t_i^(1)) P_2p^(1)= (ϕi+Φϕ¯i+,τ)⊤−~Q+(ϕ¯i+,τ)⊤+~Q+(ri+1+Rϕ¯i+,τ)r~Q ( _i+D_ φ_i^+,τ ) B_- w_Q+ (L φ_i^+,τ ) B_+ w_Q+ (r_i+1+D_R φ_i^+,τ )B_r w_Q +(+Uϕ¯i+,τ)⊤U~Q + (w+D_U φ_i^+,τ ) B_U w_Q = = −c(ϕi+Φϕ¯i+,τ)⊤+c(ϕ¯i+,τ)⊤−⊤[−R⊤+(γId+Φ⊤−U⊤Ω)] -c ( _i+D_ φ_i^+,τ ) w+c (L φ_i^+,τ ) L^- [-D_R +(γ I_d+D_ -D_U )w ] +c(ri+1+Rϕ¯i+,τ)+c(+Uϕ¯i+,τ)⊤Ω. +c (r_i+1+D_R φ_i^+,τ )+c (w+D_U φ_i^+,τ ) w. (A.15) We simplify these terms one by one. First, −c(ϕi+Φϕ¯i+,τ)⊤=−cϕi⊤−c(ϕ¯i+,τ)⊤Φ⊤-c ( _i+D_ φ_i^+,τ ) w=-c _i w-c( φ_i^+,τ) D_ w. Next, since ⊤−⊤=IdL L^- =I_d, c(ϕ¯i+,τ)⊤−⊤[−R⊤+(γId+Φ⊤−U⊤Ω)] c (L φ_i^+,τ ) L^- [-D_R +(γ I_d+D_ -D_U )w ] =c(ϕ¯i+,τ)⊤⊤−⊤[−R⊤+(γId+Φ⊤−U⊤Ω)] =c( φ_i^+,τ) L L^- [-D_R +(γ I_d+D_ -D_U )w ] =−c(ϕ¯i+,τ)⊤R⊤+cγ(ϕ¯i+,τ)⊤+c(ϕ¯i+,τ)⊤Φ⊤−c(ϕ¯i+,τ)⊤U⊤Ω. =-c( φ_i^+,τ) D_R +cγ( φ_i^+,τ) w+c( φ_i^+,τ) D_ w-c( φ_i^+,τ) D_U w. The remaining two terms satisfy c(ri+1+Rϕ¯i+,τ) c (r_i+1+D_R φ_i^+,τ ) =cri+1+cRϕ¯i+,τ, =cr_i+1+cD_R φ_i^+,τ, c(+Uϕ¯i+,τ)⊤Ω c (w+D_U φ_i^+,τ ) w =c⊤Ω+c(ϕ¯i+,τ)⊤U⊤Ω. =cw w+c( φ_i^+,τ) D_U w. The terms involving ΦD_ , RD_R, and UD_U therefore cancel in pairs: −c(ϕ¯i+,τ)⊤Φ⊤+c(ϕ¯i+,τ)⊤Φ⊤=0, -c( φ_i^+,τ) D_ w+c( φ_i^+,τ) D_ w=0, −c(ϕ¯i+,τ)⊤R⊤+cRϕ¯i+,τ=0, -c( φ_i^+,τ) D_R +cD_R φ_i^+,τ=0, −c(ϕ¯i+,τ)⊤U⊤Ω+c(ϕ¯i+,τ)⊤U⊤Ω=0. -c( φ_i^+,τ) D_U w+c( φ_i^+,τ) D_U w=0. For the second cancellation, we used the fact that Rϕ¯i+,τD_R φ_i^+,τ is a scalar, so it equals its transpose (ϕ¯i+,τ)⊤R⊤( φ_i^+,τ) D_R . Moreover, since Ω⊤=−Ω =- , ⊤Ω=(⊤Ω)⊤=⊤Ω⊤=−⊤Ω,w w=(w w) =w w=-w w, and therefore ⊤Ω=0w w=0. Substituting these identities into (A.15) yields (ti(1))⊤2p(1) (t_i^(1)) P_2p^(1) =−cϕi⊤+cγ(ϕ¯i+,τ)⊤+cri+1 =-c _i w+cγ( φ_i^+,τ) w+cr_i+1 =c(ri+1+γ⊤ϕ¯i+,τ−⊤ϕi). =c (r_i+1+γw φ_i^+,τ-w _i ). (A.16) Define δi(τ):=ri+1+γ⊤ϕ¯i+,τ−⊤ϕi _i^(τ):=r_i+1+γw φ_i^+,τ-w _i. Then (ti(1))⊤2p(1)=cδi(τ)(t_i^(1)) P_2p^(1)=c _i^(τ). Therefore, the last d coordinates of the second-layer update are ΠoutΔp _out p =1n∑j=1NΠout2hj(1)((hj(1))⊤2p(1))=1n∑i=0n−1Πout2ti(1)((ti(1))⊤2p(1)) = 1n _j=1^N _outV_2h_j^(1) ((h_j^(1)) P_2p^(1) )= 1n _i=0^n-1 _outV_2t_i^(1) ((t_i^(1)) P_2p^(1) ) =1n∑i=0n−1(αcϕi)cδi(τ)=αn∑i=0n−1ϕiδi(τ). = 1n _i=0^n-1 ( αc _i )c _i^(τ)= αn _i=0^n-1 _i _i^(τ). (A.17) The second equality follows from (A.10), and the third uses (A.9) and (A.16). Since the W block of the first-layer parameter token is ~Q=[ρ+b⊤;] w_Q=[ρ+b w;w], its last d coordinates are Πoutp(1)=~Q=~=~=[0d×1,Id][1]=. _outp^(1)=J w_Q=JQ w=J w=[0_d× 1,I_d] bmatrix1\\ w bmatrix=w. Here =JQ=J follows directly from the prescribed form of Q. The residual connection thus gives TFτ((0)) _τ(H^(0)) =Πoutp(2)=Πoutp(1)+ΠoutΔp=+αn∑i=0n−1ϕiδi(τ) = _outp^(2)= _outp^(1)+ _out p=w+ αn _i=0^n-1 _i _i^(τ) =+αn∑i=0n−1ϕi[ri+1+γ⊤ϕ¯i+,τ−⊤ϕi]. =w+ αn _i=0^n-1 _i [r_i+1+γw φ_i^+,τ-w _i ]. The readout extracts precisely these last d coordinates of p(2)p^(2). Hence we have proved part (i). Notice that this argument does not prescribe the first coordinate of the W block of p(2)p^(2); that coordinate is outside the readout. (i) Recall that qi,a:=⊤ϕi,a+q_i,a:=w φ^+_i,a, qi⋆:=maxa∈qi,aq_i := _a q_i,a, i⋆:=argmaxa∈qi,aA_i := _a q_i,a, Ki:=|i⋆|K_i:=|A_i |. The soft backup value is q¯i(τ):=⊤ϕ¯i+,τ=∑a∈πi,aτqi,a q_i^(τ):=w φ_i^+,τ= _a _i,a^τq_i,a. Since ∑a∈πi,aτ=1 _a _i,a^τ=1 and qi,a=qi⋆q_i,a=q_i for every a∈i⋆a _i , qi⋆−q¯i(τ) q_i - q_i^(τ) =∑a∈πi,aτqi⋆−∑a∈πi,aτqi,a=∑a∈πi,aτ(qi⋆−qi,a)=∑a∉i⋆πi,aτ(qi⋆−qi,a). = _a _i,a^τq_i - _a _i,a^τq_i,a= _a _i,a^τ(q_i -q_i,a)= _a _i _i,a^τ(q_i -q_i,a). (A.18) In particular, qi⋆−q¯i(τ)⩾0q_i - q_i^(τ) 0. Since |qi,a|⩽Bq|q_i,a| B_q and |qi⋆|⩽Bq|q_i | B_q, we have 0⩽qi⋆−qi,a⩽|qi⋆|+|qi,a|⩽2Bq0 q_i -q_i,a |q_i |+|q_i,a| 2B_q. Therefore, 0⩽qi⋆−q¯i(τ)⩽2Bq∑a∉i⋆πi,aτ0 q_i - q_i^(τ) 2B_q _a _i _i,a^τ. If i⋆≠A_i , Lemma A.1 gives ∑a∉i⋆πi,aτ=1−∑a∈i⋆πi,aτ⩽m−KiKie−Δi/τ. _a _i _i,a^τ=1- _a _i _i,a^τ m-K_iK_ie^- _i/τ. (A.19) Hence 0⩽qi⋆−q¯i(τ)⩽2Bqm−KiKie−Δi/τ.0 q_i - q_i^(τ) 2B_q m-K_iK_ie^- _i/τ. (A.20) If i⋆=A_i =A, then all actions have the same value and qi⋆−q¯i(τ)=0q_i - q_i^(τ)=0 for every τ>0τ>0. The difference between the hard Q-learning update and the Transformer update is therefore Q−TF(τ) w_Q-w_TF^(τ) =αn∑i=0n−1ϕi[(ri+1+γqi⋆−⊤ϕi)−(ri+1+γq¯i(τ)−⊤ϕi)]=αγn∑i=0n−1ϕi(qi⋆−q¯i(τ)). = αn _i=0^n-1 _i [(r_i+1+γ q_i -w _i)-(r_i+1+γ q_i^(τ)-w _i) ]= αγn _i=0^n-1 _i (q_i - q_i^(τ) ). By the triangle inequality and ‖ϕi‖2⩽Bϕ\| _i\|_2 B_φ, ‖Q−TF(τ)‖2 \|w_Q-w_TF^(τ) \|_2 ⩽αγn∑i=0n−1∥ϕi∥2|qi⋆−q¯i(τ)|⩽αγBϕn∑i:i⋆≠(qi⋆−q¯i(τ)) αγn _i=0^n-1\| _i\|_2 |q_i - q_i^(τ) | αγ B_φn _i:\,A_i (q_i - q_i^(τ) ) ⩽2αγBqBϕn∑i:i⋆≠m−KiKie−Δi/τ. 2αγ B_qB_φn _i:\,A_i m-K_iK_ie^- _i/τ. (A.21) The second inequality uses the nonnegativity of the backup gap and the fact that this gap is zero whenever i⋆=A_i =A; the last inequality uses (A.20). Since the number of transitions is finite and each Δi>0 _i>0 in the displayed sum, the right-hand side converges to zero as τ↓0τ 0. Thus limτ↓0TF(τ)=Q _τ 0w_TF^(τ)=w_Q. In particular, if m⩾2m 2 and every maximizer is unique, then Ki=1K_i=1 and 1−πi,ai⋆τ⩽(m−1)e−Δi/τ1- _i,a_i ^τ (m-1)e^- _i/τ. Since there are finitely many transitions, Δmin:=min0⩽i⩽n−1Δi>0 _ := _0 i n-1 _i>0. Consequently, e−Δi/τ⩽e−Δmin/τe^- _i/τ e^- _ /τ, and hence ‖Q−TF(τ)‖2 \|w_Q-w_TF^(τ) \|_2 ⩽2αγBqBϕ(m−1)n∑i=0n−1e−Δi/τ⩽2αγBqBϕ(m−1)nne−Δmin/τ 2αγ B_qB_φ(m-1)n _i=0^n-1e^- _i/τ 2αγ B_qB_φ(m-1)nne^- _ /τ =2αγBqBϕ(m−1)e−Δmin/τ. =2αγ B_qB_φ(m-1)e^- _ /τ. If m=1m=1, all backup gaps are zero and TF(τ)=Qw_TF^(τ)=w_Q for every τ>0τ>0. This proves the claimed exponential approximation bound. ∎ We next consider whether emergence of tabular Q-learning algorithms can be achieved within Transformers. Let e(s,a)e_(s,a) denote the one-hot basis vector for state-action pair (s,a)(s,a). Corollary A.3 (Transformer implements tabular Q-learning). Under the conditions of the preceding theorem, let S and A be finite, and consider ϕ(s,a)=e(s,a)∈ℝ||||φ(s,a)=e_(s,a) ^|S||A|, (s,a)=Q(s,a)w_(s,a)=Q(s,a). The two-layer Transformer constructed in the preceding theorem implements the batch tabular Q-learning update up to softmax approximation error. Proof. According to Theorem 1, the batch-update formula using n samples is TFτ((0),1,1,2,2)=+αn∑i=0n−1e(si,ai)[ri+1+γ∑b∈πi,bτ(si+1,b)−(si,ai)],TF_τ(H^(0);P_1,V_1,P_2,V_2)=w+ αn _i=0^n-1e_(s_i,a_i) [r_i+1+γ _b _i,b^τw_(s_i+1,b)-w_(s_i,a_i) ], where πi,aτ≔exp(Q(si+1,a)/τ)/∑b∈exp(Q(si+1,b)/τ) _i,a^τ (Q(s_i+1,a)/τ )/ _b (Q(s_i+1,b)/τ ). Only the weight components corresponding to visited (si,ai)(s_i,a_i) samples are updated; all remaining coordinates are multiplied by zero and thus remain unchanged. The equivalent update rule for the Q-values reads [TFτ((0);1,1,2,2)](s,a)=Q(s,a)+αn∑i:(si,ai)=(s,a)[ri+1+γ∑b∈πi,bτQ(si+1,b)−Q(si,ai)].[TF_τ(H^(0);P_1,V_1,P_2,V_2)]_(s,a)=Q(s,a)+ αn _i:(s_i,a_i)=(s,a) [r_i+1+γ _b _i,b^τQ(s_i+1,b)-Q(s_i,a_i) ]. Notably, only state-action pairs (s,a)(s,a) appearing within the mini-batch receive updates; all other entries stay fixed. ∎ Appendix B Training Algorithm B.1 Independent-context gradient variance Let ℓ(,):=12‖TFθ(())−soft(τ)()‖22 ( θ;z):= 12\|TF_θ(H(z))-w^(τ)_soft(z)\|_2^2, and ℒ():=∼[ℓ(,)]L( θ):=E_z [ ( θ;z)]. Fix θ, and let 1,…,B∼i.i.d.z_1,…,z_B i.i.d. D be independent of θ. Define g(,)=∇ℓ(,)g( θ;z)= _ θ ( θ;z), g^B()=1B∑b=1Bg(,b) g_B( θ)= 1B _b=1^Bg( θ;z_b). The following proposition shows that independent-context minibatching preserves the population-gradient direction in expectation while reducing context-specific gradient fluctuations. Proposition B.1 (Independent-context minibatch gradient). Assume ‖g(,)‖22<∞E\|g( θ;z)\|_2^2<∞ and that differentiation and expectation can be interchanged. Then [g^B()∣] [ g_B( θ) θ] =∇ℒ(), = ( θ), (B.1) Cov(g^B()∣) ( g_B( θ) θ) =1BCov(g(,)∣), = 1BCov(g( θ;z) θ), (B.2) [‖g^B()−∇ℒ()‖22|] \! [ \| g_B( θ)- ( θ) \|_2^2 | θ ] =1B[‖g(,)−∇ℒ()‖22|]. = 1BE\! [ \|g( θ;z)- ( θ) \|_2^2 | θ ]. (B.3) Equivalently, for every deterministic direction v, Var(v⊤g^B()∣)=1BVar(v⊤g(,)∣)Var(v g_B( θ) θ)= 1BVar(v g( θ;z) θ). Proof. (i) Let μ():=∼[g(,)∣]μ( θ):=E_z [g( θ;z) θ]. By the interchange of differentiation and expectation, μ()=∼[∇ℓ(,)]=∇∼[ℓ(,)]=∇ℒ().μ( θ)=E_z [ _ θ ( θ;z)]= _ θE_z [ ( θ;z)]= ( θ). Therefore, [g^B()∣]=1B∑b=1B[g(,b)∣]=μ()=∇ℒ()E[ g_B( θ) θ]= 1B _b=1^BE[g( θ;z_b) θ]=μ( θ)= ( θ), which proves (B.1). (i) Now define the centered gradients ξb:=g(,b)−μ() _b:=g( θ;z_b)-μ( θ). Conditioned on θ, the random vectors ξ1,…,ξB _1,…, _B are independent and identically distributed, with [ξb∣]=0E[ _b θ]=0. Moreover, g^B()−∇ℒ()=1B∑b=1Bξb g_B( θ)- ( θ)= 1B _b=1^B _b. Hence Cov(g^B∣) ( g_B θ) =[(1B∑b=1Bξb)(1B∑b′=1Bξb′)⊤|] =E\! [ ( 1B _b=1^B _b ) ( 1B _b =1^B _b ) | θ ] =1B2∑b=1B[ξbξb⊤∣]+1B2∑b≠b′[ξbξb′⊤∣]. = 1B^2 _b=1^BE[ _b _b θ]+ 1B^2 _b≠ b E[ _b _b θ]. For b≠b′b≠ b , conditional independence and centering give [ξbξb′⊤∣]=[ξb∣][ξb′⊤∣]=0E[ _b _b θ]=E[ _b θ]E[ _b θ]=0. Thus Cov(g^B∣)=B2Cov(g(,)∣)=1BCov(g(,)∣)Cov( g_B θ)= BB^2Cov(g( θ;z) θ)= 1BCov(g( θ;z) θ), which proves (B.2). (i) Taking the trace of the covariance identity yields [‖g^B−∇ℒ()‖22|]=1B[‖g(,)−∇ℒ()‖22|],E [\| g_B- ( θ)\|_2^2 | θ ]= 1BE [ \|g( θ;z)- ( θ) \|_2^2 | θ ], which is (B.3). Finally, for any deterministic v, Var(v⊤g^B∣)=v⊤Cov(g^B∣)v=1Bv⊤Cov(g(,)∣)v,Var(v g_B θ)=v Cov( g_B θ)v= 1Bv Cov(g( θ;z) θ)v, giving the directional variance identity. ∎ B.2 Invariant parameter groups Proposition B.2 (Invariant parameter groups). Assume that the context distribution nD_n is independent of θ and satisfies the boundedness condition in Assumption 1 (i). Let out θ_out collect all entries of =(1,1,2,2) θ=(P_1,V_1,P_2,V_2) outside the following 1616 blocks: 1,Ψ,1,ℐΨ,1,WW,2,ℐW,2,Woutℐ.P_1, ,V_1,I ,V_1,W,P_2,IW,V_2,W_outI. Under the update k+1=k−ηk∇ℒ^B(k) θ^k+1= θ^k- _k∇ L_B( θ^k), if out0=0 θ_out^0=0, then outk=0 θ_out^k=0 for all k>0k>0. Proof. Let 16=:out=0S_16=\ θ: θ_out=0\. We will prove the pointwise identity ∇outℓ(,)=0,for every context and every ∈16, _ θ_out ( θ;z)=0, every context z and every θ _16, (B.4) where ℓ(,)=12‖f()−yτ()‖22 ( θ;z)= 12\|f_ θ(z)-y_τ(z)\|_2^2, f()=TFθ(())f_ θ(z)=TF_θ(H(z)), and yτ()=soft(τ)()y_τ(z)=w_soft^(τ)(z). Here the context and teacher target are fixed when differentiating with respect to θ. Throughout the proof, derivatives are taken in the full space of the four matrices. The condition ∈16 θ _16 is imposed only when evaluating these derivatives. Fix z and ∈16 θ _16. The input tokens tit_i, ci,ac_i,a, and p are supported in K, Ψ , and W, respectively. The first-layer mask allows tit_i to attend only to ci,a:a∈\c_i,a:a \. Write ui,a=ci,a⊤1ti,πi,a=eui,a∑b∈eui,b,c¯i=∑a∈πi,aci,a.u_i,a=c_i,a P_1t_i, _i,a= e^u_i,a _b e^u_i,b, c_i= _a _i,ac_i,a. (B.5) The probabilities πi,a _i,a are the student’s current attention weights; no teacher parameter value is imposed on 1P_1. Since each ci,ac_i,a is supported in Ψ , so is c¯i c_i. Every candidate query and the parameter query have exactly one allowed key, namely p. Their attention weights are identically one, for arbitrary full matrices 1P_1 and 1V_1. Consequently, the first-layer tokens satisfy hi=ti+1c¯i,c^i,a=ci,a+1p,q=p+1p.h_i=t_i+V_1 c_i, c_i,a=c_i,a+V_1p, q=p+V_1p. (B.6) On 16S_16, the only possibly nonzero blocks of 1V_1 are 1,ℐΨV_1,I and 1,WWV_1,W. Therefore supp(1c¯i)⊆ℐsupp(V_1 c_i) , supp(1p)⊆Wsupp(V_1p) W. Because ⊆ℐK and the coordinate sets ℐ,Ψ,WI, ,W are pairwise disjoint, (B.6) gives supp(hi)⊆ℐ,supp(c^i,a)⊆Ψ∪W,supp(q)⊆W.supp(h_i) , ( c_i,a) ∪ W, (q) W. (B.7) Neither q=pq=p nor c^i,a=ci,a c_i,a=c_i,a is required. In particular, 1,WWV_1,W can be any (d+1)×(d+1)(d+1)×(d+1) matrix. Let Π=[0d×(4d+2),Id] =[0_d×(4d+2),I_d], ~2=Π2 V_2= V_2. Write h1,…,hNh_1,…,h_N for all first-layer tokens, in their original order, with hN=qh_N=q. For arbitrary full matrices, the second-layer readout is f()=Πq+1n∑j=1N(~2hj)(hj⊤2q).f_ θ(z)= q+ 1n _j=1^N( V_2h_j)(h_j P_2q). (B.8) To compute its full differential, define vj=~2hjv_j= V_2h_j and σj=hj⊤2q _j=h_j P_2q. The product rule gives d(vjσj)=σj(Πd2hj+~2dhj)+vj((dhj)⊤2q+hj⊤(d2)q+hj⊤2dq). \,d(v_j _j)= _j ( \,\,dV_2\,h_j+ V_2\,\,dh_j )+v_j ((\,dh_j) P_2q+h_j (\,dP_2)q+h_j P_2\,\,dq ). (B.9) We now evaluate this differential at ∈16 θ _16. Since 2V_2 has nonzero entries only in (Wout,ℐ)(W_out,I), ~2 V_2 has nonzero columns only in ℐI. Similarly, 2P_2 has nonzero entries only in (ℐ,W)(I,W), and q is supported in W, so supp(2q)⊆ℐsupp(P_2q) . For every candidate token and for the parameter token considered as a key/value token, (B.7) therefore implies vj=~2hj=0d,σj=hj⊤2q=0.v_j= V_2h_j=0_d, _j=h_j P_2q=0. (B.10) Substituting (B.10) into (B.9), every term vanishes: d(vjσj)=0for every full-matrix perturbation d.\,d(v_j _j)=0 every full-matrix perturbation \,d θ. For a transition token, write ai=~2hia_i= V_2h_i, si=hi⊤2qs_i=h_i P_2q, and e=f()−yτ()e=f_ θ(z)-y_τ(z). The full differential evaluated on 16S_16 is df() \,df_ θ(z) =Πdq+1n∑i=0n−1si(Πd2hi+~2dhi) = \,\,dq+ 1n _i=0^n-1s_i ( \,\,dV_2\,h_i+ V_2\,\,dh_i ) +1n∑i=0n−1ai((dhi)⊤2q+hi⊤(d2)q+hi⊤2dq). + 1n _i=0^n-1a_i ((\,dh_i) P_2q+h_i (\,dP_2)q+h_i P_2\,\,dq ). (B.11) The sample-loss differential is dℓ=e⊤df()\,d =e \,df_ θ(z). Gradient of 2P_2 and 2V_2. Holding the first-layer tokens fixed, the terms involving d2\,dP_2 in (B.11) give d2ℓ=1n∑i=0n−1(e⊤ai)hi⊤(d2)q.\,d_P_2 = 1n _i=0^n-1(e a_i)h_i (\,dP_2)q. Since ⟨A,B⟩F=tr(A⊤B) A,B _F=tr(A B) and hi⊤(d2)q=⟨hiq⊤,d2⟩Fh_i (\,dP_2)q= h_iq ,\,dP_2 _F, we obtain ∇2ℓ(,)=1n∑i=0n−1(e⊤ai)hiq⊤ _P_2 ( θ;z)= 1n _i=0^n-1(e a_i)h_iq . Each hih_i is supported in ℐI and q is supported in W. Thus hiq⊤h_iq has nonzero entries only in rows ℐI and columns W, and supp(∇2ℓ)⊆(ℐ,W)supp( _P_2 ) (I,W). Entrywise, for a row coordinate r and column coordinate s, (∇2ℓ)rs=1n∑i(e⊤ai)(hi)rqs=0( _P_2 )_rs= 1n _i(e a_i)(h_i)_rq_s=0 if r∉ℐ or s∉Wr or s∉ W. Likewise, the terms involving d2\,dV_2 in (B.11) give d2ℓ=1n∑i=0n−1sie⊤Π(d2)hi=⟨1n∑i=0n−1siΠ⊤ehi⊤,d2⟩F.\,d_V_2 = 1n _i=0^n-1s_ie (\,dV_2)h_i= 1n _i=0^n-1s_i e\,h_i ,\,dV_2 _F. Hence ∇2ℓ(,)=1n∑i=0n−1siΠ⊤ehi⊤ _V_2 ( θ;z)= 1n _i=0^n-1s_i e\,h_i . The vector Π⊤e e is supported in WoutW_out, while hih_i is supported in ℐI. Therefore supp(∇2ℓ)⊆(Wout,ℐ)supp( _V_2 ) (W_out,I). Every row outside WoutW_out, including the first row of the W block, has zero gradient; within WoutW_out, the columns in Ψ and W also have zero gradient. To propagate the loss gradient through the first layer, collect the dhi\,dh_i and dq\,dq terms in (B.11). For each transition token, dhiℓ=1n[sie⊤~2dhi+(e⊤ai)(dhi)⊤2q]=:λi⊤dhi,\,d_h_i = 1n[s_ie V_2\,\,dh_i+(e a_i)(\,dh_i) P_2q]=: _i \,dh_i, Both ~2⊤e V_2 e and 2qP_2q are supported in ℐI, so supp(λi)⊆ℐsupp( _i) . For the parameter token, its residual contribution is e⊤Πdqe \,\,dq, and its query contribution is n−1∑i(e⊤ai)hi⊤2dqn^-1 _i(e a_i)h_i P_2\,\,dq. Thus dqℓ=[Π⊤e+1n∑i=0n−1(e⊤ai)2⊤hi]⊤dq=:λq⊤dq.\,d_q =[ e+ 1n _i=0^n-1(e a_i)P_2 h_i] \,dq=: _q \,dq. There is no additional key/value contribution from hN=qh_N=q, because its summand has zero differential by (B.10). Since 2P_2 is supported in (ℐ,W)(I,W), its transpose maps hih_i into W. Also Π⊤e e is supported in Wout⊆W_out W. Consequently, supp(λq)⊆Wsupp( _q) W. Each candidate token appears in the second layer only as a key/value token in its own summand. That summand has zero differential, so ∇c^i,aℓ=0. _ c_i,a =0. (B.12) Here q and the candidate tokens are treated as separate intermediate variables; their shared dependence on 1pV_1p is included in the next step. Gradient of 1V_1. When varying 1V_1 and keeping 1P_1 fixed, the attention weights and c¯i c_i are fixed. Differentiating (B.6) yields dhi=(d1)c¯i\,dh_i=(\,dV_1) c_i, dc^i,a=(d1)p\,d c_i,a=(\,dV_1)p, dq=(d1)p\,dq=(\,dV_1)p. By the chain rule, including all three token types, d1ℓ \,d_V_1 =∑i=0n−1λi⊤(d1)c¯i+∑i,a(∇c^i,aℓ)⊤(d1)p+λq⊤(d1)p = _i=0^n-1 _i (\,dV_1) c_i+ _i,a( _ c_i,a ) (\,dV_1)p+ _q (\,dV_1)p =∑i=0n−1λi⊤(d1)c¯i+λq⊤(d1)p, = _i=0^n-1 _i (\,dV_1) c_i+ _q (\,dV_1)p, where the candidate terms vanish by (B.12). It follows that ∇1ℓ(,)=∑i=0n−1λic¯i⊤+λqp⊤ _V_1 ( θ;z)= _i=0^n-1 _i c_i + _qp . For the first sum, λi _i is supported in ℐI and c¯i c_i in Ψ , so its support is contained in (ℐ,Ψ)(I, ). For the second term, both λq _q and p are supported in W, so its support is contained in (W,W)(W,W). Hence supp(∇1ℓ)⊆(ℐ,Ψ)∪(W,W).supp( _V_1 ) (I, )∪(W,W). Equivalently, the only possibly nonzero gradient blocks of 1V_1 are (Φ,Ψ)( , ), (G,Ψ)(G, ), (R,Ψ)(R, ), (U,Ψ)(U, ), (W,W)(W,W). Every entry outside these five blocks therefore has zero sample gradient. Gradient of 1P_1 (including the softmax derivative). Only transition queries have nonconstant first-layer attention weights. For a candidate query or the parameter query, the unique allowed-key weight is eu/eu=1e^u/e^u=1, whose derivative with respect to every entry of 1P_1 is zero. For a transition query, differentiating (B.5) gives dπi,a=πi,a(dui,a−∑b∈πi,bdui,b), \,d _i,a= _i,a(\,du_i,a- _b _i,b\,\,du_i,b), dc¯i=∑a∈ci,adπi,a=∑a∈πi,aci,adui,a−c¯i∑b∈πi,bdui,b=∑a∈πi,a(ci,a−c¯i)dui,a. \,d c_i= _a c_i,a\,\,d _i,a= _a _i,ac_i,a\,\,du_i,a- c_i _b _i,b\,\,du_i,b= _a _i,a(c_i,a- c_i)\,\,du_i,a. When only 1P_1 varies, dhi=1dc¯i\,dh_i=V_1\,\,d c_i, while dq=0\,dq=0 and dc^i,a=0\,d c_i,a=0. Therefore d1ℓ=∑i=0n−1λi⊤1dc¯i=∑i=0n−1∑a∈πi,aλi⊤1(ci,a−c¯i)dui,a. \,d_P_1 = _i=0^n-1 _i V_1\,\,d c_i= _i=0^n-1 _a _i,a _i V_1(c_i,a- c_i)\,\,du_i,a. Define the scalar κi,a:=πi,aλi⊤1(ci,a−c¯i) _i,a:= _i,a _i V_1(c_i,a- c_i). Since the input tokens are fixed, dui,a=ci,a⊤(d1)ti=⟨ci,ati⊤,d1⟩F\,du_i,a=c_i,a (\,dP_1)t_i= c_i,at_i ,\,dP_1 _F. Consequently, ∇1ℓ(,)=∑i=0n−1∑a∈κi,aci,ati⊤ _P_1 ( θ;z)= _i=0^n-1 _a _i,ac_i,at_i . Each ci,ac_i,a is supported in Ψ and each input transition token tit_i is supported in =(Φ,R,U)K=( ,R,U). Hence supp(∇1ℓ)⊆(Ψ,).supp( _P_1 ) ( ,K). Explicitly, only 1,ΨΦP_1, , 1,ΨRP_1, R, and 1,ΨUP_1, U can have nonzero gradients. Population gradients and zero coordinates. The calculations above hold for every fixed context z. Taking expectations gives the full-matrix population gradient formulas, evaluated at any ∈16 θ _16: ∇1ℒ() _P_1L( θ) =[∑i,aκi,aci,ati⊤],∇1ℒ()=[∑iλic¯i⊤+λqp⊤], =E_z [ _i,a _i,ac_i,at_i ], _V_1L( θ)=E_z [ _i _i c_i + _qp ], ∇2ℒ() _P_2L( θ) =1n[∑i(e⊤ai)hiq⊤],∇2ℒ()=1n[∑isiΠ⊤ehi⊤]. = 1nE_z [ _i(e a_i)h_iq ], _V_2L( θ)= 1nE_z\! [ _is_i e\,h_i ]. Their supports are therefore contained in supp(∇1ℒ) ( _P_1L) ⊆(Ψ,),supp(∇1ℒ)⊆(ℐ,Ψ)∪(W,W), ( ,K), ( _V_1L) (I, )∪(W,W), (B.13) supp(∇2ℒ) ( _P_2L) ⊆(ℐ,W),supp(∇2ℒ)⊆(Wout,ℐ). (I,W), ( _V_2L) (W_out,I). These are exactly the 1616 retained blocks. Thus ∇outℒ()=0whenever out=0. _ θ_outL( θ)=0 θ_out=0. (B.14) The boundedness condition in Assumption 1 (i) justifies differentiation under the expectation: on each compact parameter set, the fixed-mask softmax and the two attention layers have uniformly bounded derivatives of every fixed order on these bounded inputs. The teacher target is bounded as well. Thus the sample loss and its first two parameter derivatives admit an integrable bound there, so ℒL is C2C^2 and the displayed expectation formulas are valid. It remains to show that (B.14) holds along the entire training trajectory, rather than only at initialization. Write =(x,y) θ=(x,y), where x collects the retained coordinates and y=outy= θ_out. Then the full gradient-flow system is x˙=−∇xℒ(x,y),y˙=−∇yℒ(x,y). x=- _xL(x,y), y=- _yL(x,y). (B.15) By (B.14), ∇yℒ(x,0)=0 _yL(x,0)=0 for every x, not merely for the initial value x(0)x(0). Consider the restricted system x¯˙=−∇xℒ(x¯,0) x=- _xL( x,0), x¯(0)=x(0) x(0)=x(0), y¯(t)≡0 y(t)≡ 0. It satisfies y¯˙(t)=0=−∇yℒ(x¯(t),0) y(t)=0=- _yL( x(t),0), so (x¯(t),y¯(t))( x(t), y(t)) solves the full system (B.15) with the same initial condition as (x(t),y(t))(x(t),y(t)). Since ℒL is C2C^2, its gradient is locally Lipschitz, and the initial-value problem has a unique local solution. The full solution therefore coincides with the restricted solution. Repeating this uniqueness argument along the trajectory gives y(t)=0y(t)=0 throughout its maximal interval of existence. For completeness, this interval contains every t⩾0t 0. Along a gradient-flow solution, using the product Frobenius norm, dtℒ((t))=⟨∇ℒ((t)),˙(t)⟩=−‖∇ℒ((t))‖2=−‖˙(t)‖2 ddtL( θ(t))= ( θ(t)), θ(t) =-\| ( θ(t))\|^2=-\| θ(t)\|^2. Integrating and using ℒ⩾0L 0 yields ∫0t‖˙(s)‖2s=ℒ((0))−ℒ((t))⩽ℒ((0)) _0^t\| θ(s)\|^2\,ds=L( θ(0))-L( θ(t)) ( θ(0)). By the triangle inequality and Cauchy–Schwarz, ‖(t)−(0)‖⩽∫0t‖˙(s)‖s⩽t(∫0t‖˙(s)‖2s)1/2⩽tℒ((0)). \| θ(t)- θ(0)\| _0^t\| θ(s)\|\,ds t ( _0^t\| θ(s)\|^2\,ds )^1/2 t\,L( θ(0)). If the maximal existence time were finite, this bound would keep the trajectory inside a compact subset of the full parameter space up to that time. The locally Lipschitz vector field then permits continuation, contradicting maximality. Thus the solution is global, and out(t)=0 θ_out(t)=0 for all t⩾0t 0. For any batch 1,…,Bz_1,…,z_B, at every ∈16 θ _16, ∇outℒ^B()=1B∑b=1B∇outℓ(,b)=0 _ θ_out L_B( θ)= 1B _b=1^B _ θ_out ( θ;z_b)=0. Hence an update k+1=k−ηk∇ℒ^B(k) θ^k+1= θ^k- _k∇ L_B( θ^k) gives outk+1=outk−ηk 0=0 θ_out^k+1= θ_out^k- _k\,0=0 whenever outk=0 θ_out^k=0. Induction proves invariance at every discrete iteration, without projecting the gradients or explicitly freezing these coordinates. ∎ B.3 Support parameter blocks Denote the six blocks by =(1)Ψ,U, =(P_1)_ ,U, =(1)G,Ψ,−=(2)Φ,W, =(V_1)_G, , _-=(P_2)_ ,W, +=(2)G,W, _+=(P_2)_G,W, r=(2)R,W,=(2)Wout,Φ. _r=(P_2)_R,W, =(V_2)_W_ out, . Let 6=,,−,+,r,S_6=\A,L,B_-,B_+,B_r,C\. For ⊆6S _6, let Θ() (S) be the parameter space in which the blocks in S are unrestricted and every other entry of the four matrices is zero. Set Θsup=Θ(6) _sup= (S_6). Define Fτ()=+αn∑i=1nϕi(ri+γ(μiτ)⊤−ϕi⊤). F_τ(H)=w+ αn _i=1^n _i (r_i+γ( _i^τ) w- _i w ). (B.16) where μi()=∑a=1mexp((ϕi,a+)⊤)∑b=1mexp((ϕi,b+)⊤)ϕi,a+ _i(A)= _a=1^m (( _i,a^+) Aw) _b=1^m (( _i,b^+) Aw)φ^+_i,a, and μiτ=μi(τ−1Id) _i^τ= _i(τ^-1I_d). The following Proposition shows that the six-block support realizes the target update Fτ()F_τ(H) exactly, whereas none of its proper subsets does so on all original prompts. Proposition B.3 (Sufficiency and inclusion-minimality of the six-block support). Let ℋH denote the class of prompts of the form (7), d,n⩾1d,n 1, m⩾2m 2, α>0α>0, 0<γ<10<γ<1, and 0<τ<∞0<τ<∞. Then (i) There exists ∈Θsup θ∈ _sup, such that for all ∈ℋH , TFθ()=Fτ()TF_θ(H)=F_τ(H). (i) For all ⊊6S _6, and ∈Θ() θ∈ (S), there exists ∈ℋH , such that TFθ()≠Fτ()TF_θ(H)≠ F_τ(H). Proof. (i) For arbitrary parameters in Θsup _sup, the first-layer transition token is ti(1)=[ϕi;μi();ri;0d;;0d+1].t_i^(1)= bmatrix _i;L _i(A);r_i;0_d;w;0_d+1 bmatrix. Indeed, the mask restricts transition i to its own successor candidates, whose scores are ψia⊤ _ia Aw. The candidates and final query are unchanged, since each attends only to the final query p and 1p=0V_1p=0. In the second layer, 2pP_2p is supported on Φ,G,R ,G,R. Consequently, the candidate and final-query scores against p are zero; only the transition tokens contribute. Their scores are σi=ϕi⊤−~+μi()⊤⊤+~+rir~ _i= _i B_- w+ _i(A) L B_+ w+r_iB_r w, and their value vectors in the output coordinates are ϕiC _i. Including the residual readout gives TF()−=1n∑i=1nϕi(ϕi⊤−~+μi()⊤⊤+~+rir~).TF_ θ(H)-w= 1n _i=1^nC _i ( _i B_- w+ _i(A) L B_+ w+r_iB_r w ). (B.17) For any c≠0c≠ 0 and any ∈GL(d)L (d), choose =τ−1IdA=τ^-1I_d, −=−cB_-=-cJ, r=ce0⊤B_r=ce_0 , +=γc−⊤B_+=γ cL^- J, =αcIdC= αcI_d. Then ~=J w=w, e0⊤~=1e_0 w=1, and ⊤+=γcL B_+=γ cJ. Thus σi=c(ri+γ(μiτ)⊤−ϕi⊤) _i=c (r_i+γ( _i^τ) w- _i w ), ϕi=αcϕiC _i= αc _i. Equation (B.17) becomes (B.16). (i) Fix a unit vector e∈ℝde ^d. In every counterexample below, take ϕ1=e _1=e and ϕi=0 _i=0 for i⩾2i 2. Accordingly only transition 1 contributes to either update, regardless of the fixed context length n. These inputs can be realized by a trajectory that moves from an initial state to a second state and subsequently selects a zero-feature action there. The successor feature lists used below are either all zero or (e,0,…,0)(e,0,…,0), so such an action exists because m⩾2m 2. Set the later rewards to zero. If =0C=0, (B.17) gives TF()=TF_ θ(H)=w for every input. Choose =0w=0, r1=1r_1=1, and all first-transition successor features zero. The target increment is αe/n≠0α e/n≠ 0, which cannot be reproduced. If r=0B_r=0, the output in (B.17) does not depend on any reward. Fix =0w=0 and the first-transition successor features at zero, and compare otherwise identical inputs with r1=0r_1=0 and r1=1r_1=1. Their target outputs differ by αe/nα e/n, while the model outputs coincide. No fixed choice of the other blocks can fit both inputs; Set =ew=e, r1=0r_1=0, and all first-transition successor features to zero. Then μ1()=0 _1(A)=0 for every A. If −=0B_-=0, the model increment is zero, but the target increment is −αe/n≠0-α e/n≠ 0. If =0L=0 or +=0B_+=0, then ⊤+=0L B_+=0, so the output is independent of the successor features. Fix =ew=e and r1=0r_1=0, and compare successor lists (0,…,0)(0,…,0) and (e,0,…,0)(e,0,…,0). The target outputs differ by αγnexp(1/τ)exp(1/τ)+m−1e≠0 αγn (1/τ) (1/τ)+m-1\,e≠ 0. Again the model outputs coincide, so they cannot both equal the targets. If =0A=0, attention weights are uniform. Holding the features and rewards fixed, μi(0) _i(0) is independent of w, and the increment in (B.17) is an affine function of w. Choose the first-transition successor list (e,0,…,0)(e,0,…,0), r1=0r_1=0, and =tew=te, with t varying in an interval about zero. The target increment is g(t)=αne[γtp(t)−t]g(t)= αne [γ tp(t)-t ], p(t)=exp(t/τ)exp(t/τ)+m−1p(t)= (t/τ) (t/τ)+m-1. Since p′(0)=(m−1)/(τm2)p (0)=(m-1)/(τ m^2), g′(0)=2αγ(m−1)nτm2e≠0g (0)= 2αγ(m-1)nτ m^2\,e≠ 0. Thus the target increment is not affine in t, whereas the model increment is affine. They cannot agree for all these inputs. Every proper subset of 6S_6 omits at least one of these blocks. The corresponding case applies irrespective of the values of the remaining blocks. ∎ Appendix C Experimental reproducibility and implementation details We evaluate parameter recovery and cross-task generalization in a structured two-layer Q-learning Transformer trained by output-only supervision. All models use cosine learning-rate decay from the first Adam update. Batch-size and context-length comparisons further reveal distinct effects on parameter recovery and functional accuracy under fixed task budgets. Frozen models transfer to 512 unseen tabular tasks, achieving returns close to their soft teachers. Table 1: Notation. A task is one independently sampled MDP/feature realization; a context is a valid trajectory window generated in that task. In the training, one task supplies one context. Symbol Meaning and default S,m,dS,m,d State count S=9S=9, action count m=4m=4, Q-feature dimension d=36d=36. ,w, θ Q parameter ∈ℝdw ^d; distinct Transformer parameter vector θ. τ Teacher temperature. α,γ,ϵ,ηα,γ,ε,η Q-update step 0.20.2, discount 0.50.5, behavior exploration 0.10.1, Adam learning rate. B,UB,U Independent contexts per optimizer update; number of optimizer updates. n Transitions within one context, default 2020; not minibatch size B. Ntask,NobsN_task,N_obs Tasks/contexts seen: Ntask=BUN_task=BU; supplied transitions: Nobs=nBUN_obs=nBU. D,NtokD,N_ tok Token width D=5d+2=182D=5d+2=182; tokens per prompt Ntok=n(m+1)+1N_ tok=n(m+1)+1. Task budget and optimization. Each independently sampled task contributes one context. Let B denote the number of contexts per optimizer update, n the number of transitions within a context, and UBU_B the number of optimizer updates. At update k=0,…,UB−1k=0,…,U_B-1, let mk=kBm_k=kB be the number of tasks already consumed. The learning rate is η(mk)=10−6+10−3−10−62[1+cos(πmkNtask)]η(m_k)=10^-6+ 10^-3-10^-62 [1+ (π m_kN_ task) ]. All blocks follow this schedule from the first update at 10−310^-3, without warm-up, restarts, or an additional phase. Its minimum is 10−610^-6 at the budget boundary; the last actual update occurs at mk=Ntask−Bm_k=N_ task-B and uses a slightly larger value. Adam uses (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), ϵAdam=10−8 _ Adam=10^-8, no weight decay, and float32 parameters and moments. The study uses five paired seeds for each configuration. For experiments on valid parameter blocks, the cosine schedule terminates at 20.48×10620.48× 10^6 tasks. Training continues at a learning rate of 10−610^-6 to 25.60×10625.60× 10^6 tasks, then 10−410^-4, with the rate lowered to 10−510^-5 beyond 37.76×10637.76× 10^6 tasks. The rate is finally switched to 10−610^-6, ending at 46.40×10646.40× 10^6 tasks per seed. For experiments on structured parameter blocks, the cosine schedule terminates at 20.48×10620.48× 10^6 tasks. C.1 Experiments on Valid Parameter Blocks Evaluating the accuracy of approximate Q updates. We adopt three metrics, namely Soft error errsofterr_soft, Hard error errharderr_hard, and Greedy agreement, to quantify how closely the Transformer proposed in this work approximates the Q-learning update rule. Specifically, errsoft=∑∈‖u^()−uτ()‖22∑∈‖uτ()‖22,errhard=∑‖u^()−umax()‖22∑‖umax()‖22.err_soft= _z \| u(z)-u_τ(z)\|_2^2 _z \|u_τ(z)\|_2^2, _hard= _z\| u(z)-u_ (z)\|_2^2 _z\|u_ (z)\|_2^2. (C.1) where uτ()=αn∑i=0n−1ϕi[ri+1+γVτ,i−⊤ϕi]u_τ(z)= αn _i=0^n-1 _i[r_i+1+γ V_τ,i-w _i], umax()=αn∑i=0n−1ϕi[ri+1+γmaxaqi,a+−⊤ϕi]u_max(z)= αn _i=0^n-1 _i[r_i+1+γ _aq_i,a^+-w _i], and qi,a+=⊤ϕi,aq_i,a^+=w _i,a, Vτ,i=∑aπiqi,a+V_τ,i= _a _iq_i,a^+. The two greedy metrics quantify action agreement at distinct granularities: GreedytrGreedy_ tr reports the fraction of individual transitions for which the model’s attention-selected action matches the teacher’s greedy action, i.e., the action with the maximum Q-value; GreedyctxGreedy_ ctx is a stricter metric, defined as the fraction of contexts where such a match holds across all 20 transitions within the context. The Teacher bias is obtained by replacing the numerator of errharderr_hard with ∑‖uτ−umax‖2 _z\|u_τ-u_ \|^2. Teacher bias is deterministic on the common corpus and is not assigned an artificial seed confidence interval. Update errors and greedy agreement are reported as percentages. The identity/zero-update predictor yields a 100% update error: the residual w cannot mask an unlearned update. ± denotes a 95% Student-t interval across trained seeds. Table 2 (dense features) demonstrates close reproduction of the soft-Q update by the model, yielding 0.01256% NRMSE and 99.990% transition-level greedy agreement; by contrast, the one-hot feature setting (Table 3) exhibits noticeably weaker performance, attaining 0.07950% NRMSE and 99.989% transition-level greedy agreement. Table 2: Dense held-out contexts: update errors and greedy agreement Tasks (10610^6) Soft error (%) Hard error (%) Greedytr (%) Greedyctx (%) 0.32 9.0157±0.65089.0157± 0.6508 9.0465±0.67169.0465± 0.6716 95.314±0.10295.314± 0.102 56.587±0.77656.587± 0.776 5.12 7.1287±0.22257.1287± 0.2225 7.1792±0.21187.1792± 0.2118 96.708±0.15796.708± 0.157 67.205±0.89567.205± 0.895 10.24 4.4694±0.69524.4694± 0.6952 4.5328±0.67504.5328± 0.6750 98.246±0.08798.246± 0.087 80.769±1.04580.769± 1.045 20.48 0.0657±0.00020.0657± 0.0002 0.6655±0.00190.6655± 0.0019 99.920±0.00599.920± 0.005 98.840±0.08298.840± 0.082 25.60 0.0625±0.00020.0625± 0.0002 0.6633±0.00200.6633± 0.0020 99.927±0.00399.927± 0.003 98.936±0.03598.936± 0.035 37.76 0.7393±0.05570.7393± 0.0557 0.9834±0.04440.9834± 0.0444 99.654±0.01299.654± 0.012 96.003±0.19796.003± 0.197 45.76 0.0846±0.01070.0846± 0.0107 0.6468±0.00440.6468± 0.0044 99.971±0.00299.971± 0.002 99.636±0.04699.636± 0.046 46.40 0.0126±0.00050.0126± 0.0005 0.6394±0.00080.6394± 0.0008 99.990±0.00499.990± 0.004 99.861±0.04599.861± 0.045 Table 3: One-hot held-out contexts: update errors and greedy agreement Tasks (10610^6) Soft error (%) Hard error (%) Greedytr (%) Greedyctx (%) 0.32 55.7678±4.549355.7678± 4.5493 57.4641±4.824457.4641± 4.8244 95.752±0.15195.752± 0.151 66.636±0.89966.636± 0.899 5.12 44.0258±1.960744.0258± 1.9607 45.6216±1.997145.6216± 1.9971 97.076±0.08597.076± 0.085 75.928±0.83375.928± 0.833 10.24 27.8956±4.680427.8956± 4.6804 29.2407±4.782029.2407± 4.7820 98.406±0.06798.406± 0.067 86.035±0.48386.035± 0.483 20.48 0.4427±0.00660.4427± 0.0066 5.1372±0.00715.1372± 0.0071 99.985±0.00299.985± 0.002 99.834±0.02399.834± 0.023 25.60 0.4181±0.00730.4181± 0.0073 5.1137±0.00975.1137± 0.0097 99.987±0.00399.987± 0.003 99.875±0.02299.875± 0.022 37.76 4.5953±0.33534.5953± 0.3353 6.7774±0.22006.7774± 0.2200 99.681±0.01299.681± 0.012 97.070±0.07797.070± 0.077 45.76 0.5257±0.06390.5257± 0.0639 4.8381±0.01714.8381± 0.0171 99.978±0.00499.978± 0.004 99.783±0.02099.783± 0.020 46.40 0.0795±0.00370.0795± 0.0037 4.8091±0.00294.8091± 0.0029 99.989±0.00499.989± 0.004 99.890±0.03999.890± 0.039 Emergence of Parameters. To distinguish scalar calibration from nonscalar distortion, let βatt=tr(θ)/d _att=tr(A_θ)/d. Frobenius orthogonality of θ−βattIA_θ- _attI and I gives EA2=(τβatt−1)2⏟EA,scale2+τ2d‖θ−βattI‖F2⏟EA,shape2.E_A^2= (τ _att-1)^2_E_A,scale^2+ τ^2d\|A_θ- _attI\|_F^2_E_A,shape^2. When βatt>0 _att>0, 1/βatt1/ _att summarizes the effective temperature of the scalar projection; it is not an exact temperature when nonscalar components remain. A visible diagonal in a heatmap is therefore insufficient evidence of inverse-temperature recovery. As can be seen in Table 4, both residual components decrease substantially toward the end of training, demonstrating the emergence of an action-selection mechanism. Figures 3 presents heatmaps of the key matrices under τ=0.2τ=0.2, revealing that the training algorithm successfully induces the constructed parameters to emerge. Table 4: Attention residual and scale across training. Tasks (10610^6) EAE_A βatt _ att EA,shapeE_A, shape 0.32 0.729464±0.0009240.729464± 0.000924 1.353730±0.0045961.353730± 0.004596 0.017515±0.0004740.017515± 0.000474 5.12 0.435038±0.0012730.435038± 0.001273 2.828350±0.0063502.828350± 0.006350 0.024805±0.0006130.024805± 0.000613 10.24 0.303830±0.0004360.303830± 0.000436 3.483203±0.0022333.483203± 0.002233 0.016895±0.0003790.016895± 0.000379 20.48 0.081884±0.0001920.081884± 0.000192 4.590583±0.0009604.590583± 0.000960 0.000274±0.0000090.000274± 0.000009 25.60 0.077982±0.0001890.077982± 0.000189 4.610089±0.0009444.610089± 0.000944 0.000148±0.0000070.000148± 0.000007 37.76 0.023377±0.0001880.023377± 0.000188 4.885535±0.0010094.885535± 0.001009 0.004731±0.0000920.004731± 0.000092 45.76 0.009809±0.0000930.009809± 0.000093 4.951000±0.0004624.951000± 0.000462 0.000428±0.0000160.000428± 0.000016 46.40 0.009781±0.0000910.009781± 0.000091 4.951105±0.0004574.951105± 0.000457 0.000185±0.0000080.000185± 0.000008 Figure 3: Terminal detail of all 16 initialized blocks for seed 0. Each block has its own color range, shared across seeds for that block. This view complements the full matrices; it does not replace structural zeros by cropped or normalized values in the full-matrix figures. Generalization to unseen tabular tasks We evaluate frozen Transformers on a common bank of 512 independent unseen MDPs. Deployment features are one-hot, ϕ(s,a)=e4s+aφ(s,a)=e_4s+a, whereas training uses dense features. These experiments assess dense-to-tabular transfer, not direct tabular training. Each learner recursively updates its own Q parameter w and generates its own ϵε-greedy actions and successor states. The Transformer parameters θ remain fixed, with no teacher parameters, actions, or states injected into its rollout. The primary comparison uses 0=0w_0=0, ntest=20n_ test=20, and 20,000 environment interactions per learner. Every temperature comparison includes the Transformer, hard and soft window-Q teachers, Oracle, and Random. The window teachers, or their hard-max counterpart, maintain a fixed w for each trajectory segment. Standard online Q-learning instead updates after every transition; we report both a constant step size of 0.20.2 and a visit-count step size of 1/Nsa1/N_sa. Oracle uses the true transition and reward model to compute an optimal policy and is therefore a privileged upper reference. Random uses a uniform action policy rather than a greedy policy induced by a random Q table. We evaluate the exact discounted return J(π)=P0⊤(I−γPπ)−1rπJ(π)=P_0 (I-γ P_π)^-1r_π, r(s,a)=∑s′P(s′∣s,a)R(a,s′)r(s,a)= _s P(s s,a)R(a,s ), where PπP_π and rπr_π are the policy-induced transition matrix and expected reward vector. For learned methods and Q-learning baselines, π is the current greedy policy; Oracle and Random use their specified policies. Thus, J is a policy-evaluation metric. Table 5: Exact greedy-policy return after 20,000 interactions on 512 unseen MDPs. Method Mean J(π)J(π) 95% interval Transformer 0.479225 [0.461365,0.497690][0.461365,0.497690] Soft window-Q 0.480136 [0.462090,0.498943][0.462090,0.498943] Hard window-Q 0.465381 [0.447874,0.483684][0.447874,0.483684] Online Q, constant 0.449844 [0.426708,0.473394][0.426708,0.473394] Online Q, visit count 0.522897 [0.502016,0.544255][0.502016,0.544255] Oracle 0.529490 [0.508685,0.550915][0.508685,0.550915] Random -0.002155 [−0.019109,0.015196][-0.019109,0.015196] Figure 4 deploys frozen dense-feature-trained Transformers on unseen tabular MDPs and measures exact policy returns under a common interaction budget. Comparisons include hard and soft window Q-learning, a model-informed Oracle, and uniform Random. The Transformer achieves returns close to those of the soft window teacher and a higher sample-mean return than the hard window-Q baseline using the same fixed step size. Figure 4: Frozen dense-feature-trained Transformers on unseen tabular MDPs and measures exact policy returns under a common interaction budget. C.2 Experiments on Structured Parameter Blocks Evaluating the accuracy of approximate Q updates. Learning error across different temperatures τ. Table 6 gives an analysis of the impacts of τ. The τ=0.1τ=0.1 model has a soft-update error of 0.07575%0.07575\% despite its appreciable parameter residual; the τ=1τ=1 model closely matches its soft teacher but has a large hard-update error, reflecting the finite-temperature teacher discrepancy. Here, Greedy agreement quantifies the fraction of contexts where the argmax action from the attention module matches the optimal maximizing next action of the Q-learning target. Table 6: Update accuracy on 8,192 common dense-feature contexts. τ Soft error (%) Hard error (%) Teacher bias (%) Greedy (%) 1 0.00283±0.001070.00283± 0.00107 10.03718±0.0002010.03718± 0.00020 10.0370 99.9985 0.8 0.00261±0.000710.00261± 0.00071 7.114048±0.0000497.114048± 0.000049 7.1140 99.9974 0.5 0.00276±0.000780.00276± 0.00078 3.21972±0.000133.21972± 0.00013 3.2198 99.9962 0.2 0.00393±0.000190.00393± 0.00019 0.63950±0.000140.63950± 0.00014 0.6385 99.9990 0.1 0.07575±0.000250.07575± 0.00025 0.25509±0.000400.25509± 0.00040 0.1960 99.9707 Table 7 illustrates how different batch sizes B affect the approximation error. The soft error decreases first and then rises with increasing batch size, minimized at B=64B=64. Greedy agreement stays near 100%. The fitted effective temperature τ^eff=1/βatt τ_eff=1/ _att, with βatt=tr()/d>0 _att=tr(A_ θ)/d>0 acting as a scalar projection diagnostic, remains around 0.2. Table 7: Final checkpoints on 8,192 new shared contexts. B Soft error (%) Hard error (%) Greedy (%\%) τ^eff τ_ eff 1 0.0202±0.00080.0202± 0.0008 0.6348±0.00070.6348± 0.0007 99.994 0.200113 2 0.0135±0.00130.0135± 0.0013 0.6347±0.00040.6347± 0.0004 99.995 0.200128 4 0.0110±0.00140.0110± 0.0014 0.6349±0.00040.6349± 0.0004 99.995 0.200161 8 0.0077±0.00090.0077± 0.0009 0.6351±0.00020.6351± 0.0002 99.999 0.200223 16 0.0052±0.00050.0052± 0.0005 0.6352±0.00030.6352± 0.0003 99.999 0.200299 32 0.0041±0.00030.0041± 0.0003 0.6355±0.00010.6355± 0.0001 100.000 0.200373 64 0.0036±0.00010.0036± 0.0001 0.6357±0.00010.6357± 0.0001 100.000 0.200459 128 0.0045±0.00010.0045± 0.0001 0.6367±0.00010.6367± 0.0001 100.000 0.200784 256 0.0123±0.00010.0123± 0.0001 0.6424±0.00030.6424± 0.0003 99.998 0.202525 To compare the effects of different optimizations on experimental results, we also evaluated the following methods. 1. Constant Adam: All six blocks retain learning rate 10−310^-3, with moments (0.9,0.999)(0.9,0.999) and numerical constant 10−810^-8; 2. Uniform cosine decay: separately labeled optimizer change. After the common 100,000-step prefix, all block learning rates decrease from 10−310^-3 to 10−510^-5. For an update indexed by j∈0,…,219999j∈\0,…,219999\ after the fork, the rate for block k is ηk(j)=10−5+ηk,0−10−52[1+cos(πj219999)] _k(j)=10^-5+ _k,0-10^-52[1+ ( π j219999)]; 3. Low-rate extension: a larger, adaptively chosen budget. After observing the completed uniform-cosine runs and the aggressive schedule’s training/validation overscaling, we continue all five uniform-cosine endpoints to 20.48 million tasks. Every block uses η(j)=10−6+10−5−10−62[1+cos(πj319999)]η(j)=10^-6+ 10^-5-10^-62[1+ ( π j319999)], j=0,…,319999j=0,…,319999. Table 8 reports three metrics under τ=0.2τ=0.2 with above learning rate schedules. Each dense-feature experiment uses B=32B=32 and n=20n=20. Table 8: Results of Soft error, Hard error, and Greedy agreement with three learning rate schedules. Condition NtaskN_task Soft error (%) Hard error (%) Greedy (%) Constant Adam 10,240,000 3.242±0.7323.242± 0.732 3.332±0.7173.332± 0.717 98.100±0.14298.100± 0.142 Uniform cosine 10,240,000 0.04028±0.004190.04028± 0.00419 0.64309±0.000760.64309± 0.00076 99.980±0.00699.980± 0.006 Low-rate extension 20,480,000 0.00415±0.000230.00415± 0.00023 0.63428±0.000120.63428± 0.00012 100.000±0.000100.000± 0.000 Emergence of Parameters. Figure 5 presents the final learned matrices 1P_1, 1V_1, 2P_2, and 2V_2 obtained after training with τ=0.2τ=0.2. Remarkably, the learned matrices recover the parameter structure predicted in Theorem 1. In particular, the diagonal entries of θA_θ in 1P_1 approach 1/τ1/τ, indicating that the Transformer has learned the internal action-selection computation required by the soft Q-learning update. This provides direct evidence of parameter emergence. (a) Learned 1P_1 (b) Learned 1V_1 (c) Learned 2P_2 (d) Learned 2V_2 Figure 5: Final learned 1P_1, 1V_1, 2P_2 and 2V_2 matrices after training at τ=0.2τ=0.2, B=32B=32, and n=20n=20. the non-zero structure matches the theoretical constructions (1,1,2,2)(P_1,V_1,P_2,V_2) in (9) and (11). We define some evaluation metrics to demonstrate whether the final parameters emerge. Write −=[b−,K−]B_-=[b_-,K_-] and +=[b+,K+]B_+=[b_+,K_+]. The analytical construction defines a sufficient family satisfying θ=τ−1IA_θ=τ^-1I, K−=−cIK_-=-cI, r=c[1,0]B_r=c[1,0], ⊤K+=γcIL K_+=γ cI, c=αIcC=α I, b−=0b_-=0, ⊤b+=0L b_+=0, where c≠0c≠ 0 may be positive or negative. For each trained model, we estimate c^=((r)0−trK−)/(d+1) c=((B_r)_0-trK_-)/(d+1) separately for each model, allowing either sign. The product ⊤K+L K_+ removes the arbitrary invertible representation of the intermediate greedy feature. Likewise ⊤b+L b_+, rather than b+b_+ alone, is the appropriate product entering the output relation. Low residuals indicate agreement with these sufficient relations. The complete diagnostics used in every table are Eθ E_A_θ =‖τθ−I‖Fd, = \| _θ-I\|_F d, E− E_- =‖K−+c^I‖F|c^|d, = \|K_-+ cI\|_F| c| d, (C.2) Er E_r =‖r−c^[1,0]‖F|c^|, = \|B_r- c[1,0]\|_F| c|, E+ E_+ =‖⊤K+−γc^I‖Fγ|c^|d, = \|L K_+-γ cI\|_Fγ| c| d, (C.3) EC E_C =‖c^−αI‖Fαd, = \| cC-α I\|_Fα d, E−,0 E_-,0 =‖b−‖2|c^|d, = \|b_-\|_2| c| d, (C.4) E+,0 E_+,0 =‖⊤b+‖2γ|c^|d, = \|L b_+\|_2γ| c| d, ETD2 E_ TD^2 =E−2+Er2+E+2+EC2+E−,02+E+,02. =E_-^2+E_r^2+E_+^2+E_C^2+E_-,0^2+E_+,0^2. (C.5) and Eall=Eθ2+ETD2E_ all= E_A_θ^2+E_ TD^2, τ^eff=1/βatt τ_ eff=1/ _att with βatt=tr()/d>0 _att=tr(A_ θ)/d>0 is a scalar projection diagnostic, not an exact temperature when A_ θ has nonscalar components. The scalar estimate is allowed to be negative. The full completed models have nondegenerate c c; a normalization near zero is not interpreted as a zero residual. For arbitrary active blocks, EallE_ all is not an established Euclidean distance to all minimizers. Figures 6 and 7 present heatmaps of the key matrices and their relations under τ=0.2τ=0.2, respectively, revealing that the training algorithm successfully induces the constructed parameters to emerge. Figure 6: All six actual trained blocks, seed 0. ,,A_ θ,L,C are 36×3636× 36; −,+B_-,B_+ are 36×3736× 37; rB_r is 1×371× 37. The fitted scalar is c^=0.967893 c=0.967893. Dense L and +B_+ are compatible with a nontrivial invertible gauge and are not replaced by identity-shaped pictures. Figure 7: Normalized relations for the same model. The first four targets are I, the reward-row target is [1,0,…,0][1,0,…,0], and the two bias targets are zero. These are display-only diagnostics, not raw weights. Common limits are used for each named relation across seeds and methods, including the failed control. For the relationship between the emergence of parameters and cumulative task exposure. Figure 8 tracks recovery along one training trajectory under growing cumulative task exposure. Reduced final TD residual shows attention-scale calibration dominates aggregate residual. Since task exposure and optimizer updates rise together, the curves cannot isolate task diversity’s fixed-compute effect. Figure 8: Parameter recovery versus cumulative fresh tasks at B=32B=32, n=20n=20, and τ=0.2τ=0.2. Left: attention residual EAE_A. Right: the aggregate of the six remaining TD and bias residuals, ETDE_ TD. Table 9 shows close agreement with the analytical relations at temperature τ=1,0.8,0.5τ=1,0.8,0.5, and 0.20.2, while recovery remains incomplete at τ=0.1τ=0.1. At τ=0.2τ=0.2, the learned scalar projection is βatt=4.992236 _att=4.992236 against a target of 55. At τ=0.1τ=0.1, it is 8.1743968.174396 against a target of 1010, and Eall=0.1826E_ all=0.1826. Figure 9 makes this scale discrepancy explicit rather than inferring recovery from matrix appearance alone. No temperature-specific tuning is performed; the tested values do not establish a critical temperature or an impossibility result for lower-temperature recovery. Table 9: Final parameter recovery at 20.48 million tasks, B=32B=32, and n=20n=20. Entries are five-seed means; EallE_ all includes a pointwise 95% interval half-width. Residuals are dimensionless. The scalar projection βatt _att is compared with 1/τ1/τ without removing nonscalar distortion. τ 1/τ1/τ βatt _att EAE_A ETDE_ TD EallE_ all 1 1 1.000000 3.808×10−53.808× 10^-5 5.410×10−55.410× 10^-5 (6.624±2.102)×10−5(6.624± 2.102)× 10^-5 0.8 1.25 1.250000 3.150×10−53.150× 10^-5 4.741×10−54.741× 10^-5 (5.715±1.049)×10−5(5.715± 1.049)× 10^-5 0.5 2 2.000002 2.424×10−52.424× 10^-5 4.882×10−54.882× 10^-5 (5.466±1.323)×10−5(5.466± 1.323)× 10^-5 0.2 5 4.992236 1.553×10−31.553× 10^-3 6.384×10−56.384× 10^-5 (1.554±0.017)×10−3(1.554± 0.017)× 10^-3 0.1 10 8.174396 0.18260.1826 6.590×10−46.590× 10^-4 0.18256±0.000200.18256± 0.00020 Figure 9: Temperature-dependent recovery at B=32B=32 and n=20n=20. Left: attention residual versus cumulative tasks, with five paired seeds per temperature. Right: the learned scalar projection βatt _att and its target 1/τ1/τ. The target is shown only for evaluation; no matrix target is supplied during training. For the relationship between the emergence of parameters and batch size. We report scale calibration and matrix shape in Figure 10. The lowest endpoint mean of EallE_ all occurs at B=128B=128, with value 7.204×10−47.204× 10^-4. By contrast, mean soft-update error continues to decrease up to B=64B=64 in Table 7. Thus, the best observed parameter recovery and the lowest observed functional error do not coincide in this sweep. At fixed NtaskN_ task, reducing B increases the number of Adam updates, whereas increasing B averages more tasks in each update. Adam’s moment memories are defined in update units, so their span in task units also changes. The observed ranking is consequently specific to the shared schedule, numerical precision, architecture, and task budget; it does not establish a universally optimal batch size. Figure 10: Batch-size effects under a fixed task budget. Left: attention scale and traceless-shape residuals, whose squares sum to EA2E_A^2 for each model. Right: aggregate residuals at four checkpoints of the same training runs. Intermediate curves are not independently retrained short-budget experiments. For the relationship between the emergence of parameters and Training context length. For each training length, only the first ntrainn_ train transitions of the common 80-step trajectory enter the loss. The teacher increment is recomputed from that prefix with normalization α/ntrainα/n_ train. The ntrain=20n_ train=20 default uses the same paired generator and is shared across the three sweeps. Table 10: Training-length comparison at τ=0.2τ=0.2 and B=32B=32. Every model receives 20.48 million tasks and 640,000 updates; supplied transitions differ. Residuals are five-seed means, with the 95% interval half-width for EallE_ all in the final column. The default is shared across sweeps. ntrainn_ train Transitions (M) EAE_A ETDE_ TD EallE_ all 95% interval half-width 1 20.48 0.01280.0128 2.033×10−42.033× 10^-4 0.01280.0128 1.969×10−41.969× 10^-4 5 102.4 2.155×10−32.155× 10^-3 8.059×10−58.059× 10^-5 2.156×10−32.156× 10^-3 8.661×10−58.661× 10^-5 10 204.8 1.699×10−31.699× 10^-3 6.440×10−56.440× 10^-5 1.700×10−31.700× 10^-3 5.381×10−55.381× 10^-5 20 409.6 1.553×10−31.553× 10^-3 6.384×10−56.384× 10^-5 1.554×10−31.554× 10^-3 1.728×10−51.728× 10^-5 40 819.2 1.739×10−31.739× 10^-3 6.393×10−56.393× 10^-5 1.740×10−31.740× 10^-3 5.290×10−55.290× 10^-5 80 1638.4 2.019×10−32.019× 10^-3 7.050×10−57.050× 10^-5 2.020×10−32.020× 10^-3 3.846×10−53.846× 10^-5 Table 11: Length-specific models on matched-length and common-ntest=20n_ test=20 dense tests, and matched-length one-hot tests. Entries are update NRMSE percentages with pointwise 95% seed-interval half-widths. The tabular column measures feature transfer without one-hot training. ntrainn_ train Dense matched (%) Dense common 20 (%) Tabular matched (%) 1 0.01893±0.000320.01893± 0.00032 0.01573±0.000250.01573± 0.00025 0.08467±0.000950.08467± 0.00095 5 0.00561±0.000390.00561± 0.00039 0.00525±0.000380.00525± 0.00038 0.01558±0.000770.01558± 0.00077 10 0.00431±0.000430.00431± 0.00043 0.00417±0.000420.00417± 0.00042 0.01284±0.000490.01284± 0.00049 20 0.00393±0.000190.00393± 0.00019 0.00393±0.000190.00393± 0.00019 0.01222±0.000360.01222± 0.00036 40 0.00367±0.000200.00367± 0.00020 0.00384±0.000210.00384± 0.00021 0.01413±0.000600.01413± 0.00060 80 0.00380±0.000120.00380± 0.00012 0.00411±0.000120.00411± 0.00012 0.01679±0.000440.01679± 0.00044 Among the tested lengths, ntrain=20n_ train=20 yields the lowest sample mean of EallE_ all, 1.554×10−31.554× 10^-3 (Table 10). This is a fixed-task, fixed-update comparison, not a fixed-transition or fixed-compute comparison: n=80n=80 supplies 80 times as many transitions per task as n=1n=1. Context length also changes the distribution and variance of the window-averaged target. Table 11 therefore reports both matched-length tests and tests at the common length ntest=20n_ test=20. The common-length evaluation holds the test window fixed when comparing models trained with different lengths. The one-hot results measure dense-to-tabular feature transfer, not performance after one-hot training. Figure 11: Training context length at τ=0.2τ=0.2 and B=32B=32, with 20.48 million tasks per model. Left: final parameter residuals. Right: update errors on matched-length and common-length dense-feature tests. Generalization to unseen tabular tasks. At τ=0.2τ=0.2, the Transformer achieves mean return 0.48800.4880, compared with 0.48610.4861 for its soft teacher and 0.53740.5374 for Oracle. The paired Transformer-minus-soft difference is 0.001880.00188. This comparison does not resolve a return advantage and does not establish equivalence or noninferiority. The visit-count online Q-learning baseline reaches 0.53180.5318, exceeding the Transformer means in the primary comparison. The evidence therefore supports useful self-fed transfer (Figure 12). Deployment intervals use 5,000 percentile-bootstrap replicates with independent resampling of the training-seed and test-task axes, preserving paired methods and complete trajectories. Figure 12: Online Q-learning is included in the comparisons; the variant with decaying visit-count learning rate yields the best performance. Table 12 and Figure 13 complete the five-temperature comparison; τ=0.2τ=0.2 appears in the main text. All curves use the same task bank, interaction axis, and evaluation protocol, and include Transformer, hard window-Q, soft window-Q, Oracle, and Random. Table 12: Final returns on 512 unseen tabular tasks: ntest=20n_ test=20, zero Q initialization. τ Transformer [95% CI] Hard Q (window) Soft Q (window) Oracle Random 1 0.5036 [0.4835, 0.5237] 0.4740 0.5036 0.5374 0.0033 0.8 0.5033 [0.4829, 0.5227] 0.4740 0.5025 0.5374 0.0033 0.5 0.5003 [0.4804, 0.5198] 0.4740 0.4996 0.5374 0.0033 0.2 0.4880 [0.4686, 0.5074] 0.4740 0.4861 0.5374 0.0033 0.1 0.4833 [0.4644, 0.5022] 0.4740 0.4795 0.5374 0.0033 Figure 13: Tabular transfer at τ=1τ=1, τ=0.8τ=0.8, τ=0.5τ=0.5, and τ=0.1τ=0.1, with ntest=20n_ test=20, zero Q initialization, and 20,000 interactions. Error bars denote pointwise 95% intervals. Transformer weights are frozen throughout. All experiments use the same 512 tasks and 20,000-interaction budget. For training and deployment context lengths. We distinguish three evaluations. The frozen-length test varies ntestn_ test for models trained with ntrain=20n_ train=20. The matched-length test deploys each length-specific model at its training length. The common-length test deploys all length-specific models at ntest=20n_ test=20. Every learner receives 20,000 interactions, so a window-based learner performs 20,000/ntest20,000/n_ test updates. Varying the deployment length therefore changes both the window-averaging coefficient and the update frequency. In the frozen-length test, ntest=5n_ test=5 yields the highest observed Transformer mean return, 0.51940.5194, whereas ntest=20n_ test=20 yields 0.48800.4880. The window teachers exhibit a similar dependence on deployment length. With common ntest=20n_ test=20, the length-specific models instead have mean returns between 0.48680.4868 and 0.48840.4884. Thus, training-context effects should be distinguished from deployment update frequency; longer windows do not uniformly improve finite-budget control. Figure 14: Separating training-length and deployment-length effects at τ=0.2τ=0.2. Left: models trained at ntrain=20n_ train=20 and deployed with different windows. Right: models trained at different lengths and all deployed at ntest=20n_ test=20. Both evaluations use the same task bank and 20,000-interaction budget. Appendix D Theoretical analysis Fix integers d⩾1d 1, m⩾2m 2, n⩾1n 1 and real constants α>0α>0, 0<γ<10<γ<1, τ>0τ>0. The meanings of these fixed quantities are given below. The temperature τ is fixed during optimization, let =(,ϕi,ri+1,ψi,a:0⩽i<n,a∈[m])∼n.z= (w,\ _i,r_i+1, _i,a:0 i<n,\ a∈[m]\ ) _n. Here ∈ℝdw ^d is an input vector, held fixed within a context. The vector ϕi∈ℝd _i ^d is the current state–action feature, ri+1∈ℝr_i+1 is the observed reward, and ψi,a∈ℝd _i,a ^d is the feature of candidate action a at the successor state. These are allowed to be statistically dependent. The symbol nD_n denotes their joint probability law, let πi,aτ() _i,a^τ(z) =exp(⊤ψi,a/τ)∑b=1mexp(⊤ψi,b/τ), = (w _i,a/τ) _b=1^m (w _i,b/τ), ψ¯iτ() ψ_i^τ(z) =∑a=1mπi,aτ()ψi,a, = _a=1^m _i,a^τ(z) _i,a, (D.1) δiτ() _i^τ(z) =ri+1−⊤ϕi+γ⊤ψ¯iτ(), =r_i+1-w _i+ ψ_i^τ(z), Fτ,n() F_τ,n(z) =αn∑i=0n−1ϕiδiτ(). = αn _i=0^n-1 _i _i^τ(z). (D.2) The probabilities πi,aτ _i,a^τ average successor features; they are not the behavior policy that generated the trajectory. The scalar δiτ _i^τ is the specified temporal-difference target minus the current score. The vector Fτ,n∈ℝdF_τ,n ^d is the update increment to imitate. Thus the teacher’s full prediction is +Fτ,nw+F_τ,n. In this appendix, we explicitly restrict optimization to the six-block support of Proposition B.3, with every other block fixed at zero. The six-block support sup=(,,−,+,r,)∈Θsup θ_sup=(A_ θ,L,B_-,B_+,B_r,C)∈ _sup can be broken down into following blocks =(A,L,b−,K−,b+,K+,β,q,C)∈Θ,dimΘ=5d2+3d+1. θ=(A,L,b_-,K_-,b_+,K_+,β,q,C)∈ , =5d^2+3d+1. (D.3) where A=θA=A_θ, L=L=L, −=[b−,K−]B_-=[b_-,K_-], +=[b+,K+]B_+=[b_+,K_+], r=[β,q⊤]B_r=[β,q ]. The trainable matrix L is not assumed invertible; invertibility is imposed only on the exact-realization manifold. pi,a(A,)=exp(ψi,a⊤A)∑b=1mexp(ψi,b⊤A),ψ¯i(A,)=∑api,a(A,)ψi,a, p_i,a(A;z)= ( _i,a Aw) _b=1^m ( _i,b Aw), ψ_i(A;z)= _ap_i,a(A;z) _i,a, (D.4) si(,)=ϕi⊤(b−+K−)+(Lψ¯i(A,))⊤(b++K+)+ri+1(β+q⊤), s_i( θ;z)= _i (b_-+K_-w)+(L ψ_i(A;z)) (b_++K_+w)+r_i+1(β+q w), (D.5) F,n()=1n∑i=0n−1Cϕisi(,). F_ θ,n(z)= 1n _i=0^n-1C _is_i( θ;z). (D.6) Both full outputs add the same w, so their difference is the increment error e():=F,n()−Fτ,n()e_ θ(z):=F_ θ,n(z)-F_τ,n(z). For the fixed context length n, we write the loss as Ln(θ)=ℒ(θ)L_n(θ)=L(θ), i.e. Ln()=12∼n‖F,n()−Fτ,n()‖22. L_n( θ)= 12E_z _n\|F_ θ,n(z)-F_τ,n(z)\|_2^2. (D.7) We prove Theorem 2 in three parts. First, we characterize the local geometry and normal curvature of the optimal parameter manifold. Second, we establish a one-step contraction of gradient descent toward this manifold. Finally, we use these results to prove convergence to a single optimal parameter and derive the matching worst-case iteration complexity over the admissible initialization class. D.1 Geometry and Curvature of the Optimal Parameter Manifold A perturbation ∈ΘU∈ has blocks denoted by ΔA,…,ΔC A,…, C in the same order as (D.3). ‖Θ2= \|U\|_ ^2= ‖ΔA‖F2+‖ΔL‖F2+‖Δb−‖22+‖ΔK−‖F2+‖Δb+‖22 \| A\|_F^2+\| L\|_F^2+\| b_-\|_2^2+\| K_-\|_F^2+\| b_+\|_2^2 +‖ΔK+‖F2+|Δβ|2+‖Δq‖22+‖ΔC‖F2. +\| K_+\|_F^2+| β|^2+\| q\|_2^2+\| C\|_F^2. For a vector, ‖x‖22=∑jxj2\|x\|_2^2= _jx_j^2; for a matrix, ‖M‖F2=tr(M⊤M)\|M\|_F^2=tr(M M). The product inner product is the corresponding sum of vector dot products and matrix Frobenius inner products. We abbreviate ‖Θ\|U\|_ to ‖\|U\| when the space is unambiguous. Norms of operators are induced by their domain and codomain norms, not entrywise norms unless specified. For a fixed context z, let J,n():=DF,n():Θ→ℝdJ_ θ,n(z):=D_ θF_ θ,n(z): ^d denote the Jacobian of the Transformer output with respect to its parameters. Thus, for a parameter perturbation ∈ΘU∈ , J,n()[]=dtF+t,n()|t=0J_ θ,n(z)[U]= . ddtF_ θ+tU,n(z) |_t=0. Equivalently, F+t,n()=F,n()+tJ,n()[]+o(t)F_ θ+tU,n(z)=F_ θ,n(z)+tJ_ θ,n(z)[U]+o(t). Hence J,n()[]J_ θ,n(z)[U] measures the first-order change of the Transformer output, on the particular context z, when the parameters are perturbed in direction U. Since our training objective is the population loss (D.7), we now introduce the population Jacobian. We therefore consider the Hilbert space ℋn=L2(n,ℝd),⟨f,h⟩ℋn=⟨f(),h()⟩2,H_n=L^2(D_n;R^d), f,h _H_n=E f(z),h(z) _2, with norm |f|ℋn2=|f()|22|f|_H_n^2=E|f(z)|_2^2, and define the population Jacobian ,n:Θ→ℋnJ_ θ,n: _n, (,n)()=J,n()[](J_ θ,nU)(z)=J_ θ,n(z)[U]. Thus |,n|ℋn2=|J,n()[]|22|J_ θ,nU|_H_n^2=E|J_ θ,n(z)[U]|_2^2 measures the average squared first-order output change caused by the parameter perturbation U over the entire context distribution. Let ,n∗J_ θ,n^* denote the adjoint of the population Jacobian, defined by ⟨,n,f⟩ℋn=⟨,,n∗f⟩Θ _ θ,nU,f _H_n= ,J_ θ,n^*f _ . Equivalently, ,n∗f=[J,n()∗f()]J_ θ,n^*f=E [J_ θ,n(z)^*f(z) ]. At an exact realization ¯∈ℳτ θ _τ, define the population Gauss-Newton operator G(¯):=¯,n∗¯,nG( θ):=J_ θ,n^*J_ θ,n. For every parameter direction U, ⟨,G(¯)⟩=|¯,n|ℋn2=|J¯,n()[]|22. ,G( θ)U =|J_ θ,nU|_H_n^2=E|J_ θ,n(z)[U]|_2^2. Therefore G(¯)G( θ) quantifies how strongly the Transformer output responds, to first order, to perturbations of its parameters around an exact realization. The minimum normal curvature over the chosen target patch τ⊂ℳτU_τ _τ is defined by κ:=inf¯∈τ,τ∈N¯ℳτ,||=1|¯,n|ℋn2κ:= _ subarrayc θ _τ,U_τ∈ N_ θM_τ,|U|=1 subarray|J_ θ,nU|_H_n^2. The quantity κ has a direct optimization interpretation. A large κ means that every unit perturbation away from the exact-realization manifold produces a substantial change in the Transformer output, so the population loss has appreciable curvature in every identifiable parameter direction. A small κ means that at least one identifiable direction is nearly flat and is therefore harder to learn by training. Throughout the remainder of Appendix D, n and τ are fixed. We use the abbreviations Jθ=Jθ,nJ_θ=J_θ,n, G=GnG=G_n, ℋ=ℋnH=H_n, and =n(τ) H= H_n(τ). Below we mainly analyze the geometric and curvature information of the optimal parameter manifold, as detailed in Lemmas D.1, D.2 and D.3 below. Structure of the optimal parameter manifold. The structure of the optimal parameter manifold establishes that every parameter on the manifold exactly realizes the target update and achieves zero loss, that the collection of such optimal parameters forms a smooth embedded manifold, and that the Hessian ∇2Ln(¯)∇^2L_n( θ) is degenerate exactly along the tangent directions of this manifold, which means motion within ℳτM_τ preserves zero loss, whereas every nonzero normal direction has strictly positive Hessian curvature. Lemma D.1 (Structure of the optimal parameter manifold). For every ¯=¯τ(L,c)∈ℳτ θ= θ_τ(L,c) _τ, F¯,n()=Fτ,n()F_ θ,n(z)=F_τ,n(z) for every admissible input z, and (i) Ln(¯)=0=min∈ΘLn()L_n( θ)=0= _ θ∈ L_n( θ), and ℳτ⊆argmin∈ΘLn()M_τ _ θ∈ L_n( θ). (i) ℳτM_τ is a closed smooth embedded submanifold of Θ , dim(ℳτ)=d2+1 (M_τ)=d^2+1. (i) Under Assumption 1, kerGn(¯)=T¯ℳτ G_n( θ)=T_ θM_τ and ∇2Ln(¯)=Gn(¯)∇^2L_n( θ)=G_n( θ), where kerT T denotes the set of directions that the operator T maps to zero, i.e., the directions along which T has no effect. Proof. (i) Since the population loss is Ln()=12[‖F,n()−Fτ,n()‖22]L_n( θ)= 12E [\|F_ θ,n(z)-F_τ,n(z)\|_2^2 ], Theorem 1 immediately gives Ln(¯)=0L_n( θ)=0 and Ln()⩾0L_n( θ) 0 for every ∈Θ θ∈ . Therefore Ln(¯)=0=min∈ΘLn()L_n( θ)=0= _ θ∈ L_n( θ). Since ¯∈ℳτ θ _τ was arbitrary, ℳτ⊆argmin∈ΘLn()M_τ _ θ∈ L_n( θ). (i) Define :=GL(d)×(ℝ∖0)G:=GL(d)×(R \0\). Since GL(d)=L∈ℝd×d:det(L)≠0GL(d)=\L ^d× d: (L)≠ 0\, and the determinant map is continuous, GL(d)GL(d) is an open subset of ℝd×d≃ℝd2R^d× d ^d^2. Likewise, ℝ∖0R \0\ is open in ℝR. Therefore ⊂ℝd2+1G ^d^2+1 is an open smooth manifold of dimension dim()=d2+1 (G)=d^2+1. Define the parameterization Φτ:→Θ _τ:G→ , Φτ(L,c):=¯τ(L,c) _τ(L,c):= θ_τ(L,c). Then, by definition, ℳτ=Φτ()M_τ= _τ(G). Matrix inversion L↦L−1L L^-1 is smooth on GL(d)GL(d), while the scalar map c↦c−1c c^-1 is smooth on ℝ∖0R \0\. Consequently, (L,c)↦cγL−1(L,c) cγ L^-1, (L,c)↦αcId(L,c) αcI_d are smooth, and hence Φτ _τ is a smooth map. In order to prove ℳτ=Φτ()M_τ= _τ(G) is an embedded submanifold of Θ , it suffices to verify that Φτ _τ is an injective immersion which is a homeomorphism onto its image. Since Φτ(L1,c1)=Φτ(L2,c2)→(L1,c1)=(L2,c2) _τ(L_1,c_1)= _τ(L_2,c_2)→(L_1,c_1)=(L_2,c_2), Φτ _τ is injective. For a perturbation (Z,ξ)∈ℝd×d×ℝ≃T(L,c)(Z,ξ) ^d× d×R T_(L,c)G, where T(L,c)T_(L,c)G is the tangent space of G at the point (L,c)(L,c), the differential of Φτ _τ is DΦτ(L,c)[Z,ξ]=(0,Z, 0,−ξId, 0,γξL−⊤−cγL−⊤Z⊤L−⊤,ξ, 0,−αξc−2Id). D _τ(L,c)[Z,ξ]= (0,\,Z,\,0,\,-ξ I_d,\,0,γξ L^- -cγ L^- Z L^- ,\,ξ,\,0,\,-αξ c^-2I_d ). This follows from differentiating L−⊤L⊤=IdL^- L =I_d in the direction Z, which gives D(L−⊤)[Z]=−L−⊤Z⊤L−⊤D(L^- )[Z]=-L^- Z L^- . Suppose DΦτ(L,c)[Z,ξ]=0D _τ(L,c)[Z,ξ]=0, we have (Z,ξ)=(0,0)(Z,ξ)=(0,0), which means DΦτ(L,c)D _τ(L,c) is injective at every (L,c)∈(L,c) , i.e. Φτ _τ is an immersion. For any =(A,L,b−,K−,b+,K+,β,q,C)∈ℳτ θ=(A,L,b_-,K_-,b_+,K_+,β,q,C) _τ, The inverse parameterization on ℳτM_τ is simply Φτ−1()=(L,β) _τ^-1( θ)=(L,\,β). Since coordinate projections are continuous, Φτ−1 _τ^-1 is continuous on its image. Therefore, ℳτ=Φτ()M_τ= _τ(G) is an embedded submanifold of Θ , and dim(ℳτ)=d2+1 (M_τ)=d^2+1 by the fact that an embedding preserves dimension. It remains to show that ℳτM_τ is closed as a subset of Θ . Consider an arbitrary convergent sequence j=¯τ(Lj,cj)∈ℳτ θ_j= θ_τ(L_j,c_j) _τ, j→⋆ θ_j→ θ_ in Θ . We show that ⋆∈ℳτ θ_ _τ. Since convergence in Θ implies convergence of every parameter block, Lj→L⋆L_j→ L_ . Since the seventh block satisfies βj=cj _j=c_j, cj→c⋆:=β⋆c_j→ c_ := _ . We need to verify that c⋆≠0c_ ≠ 0 and L⋆∈GL(d)L_ ∈ GL(d). Suppose, for contradiction, that c=0c=0, then cj→0c_j→ 0. However, the ninth parameter block is Cj=αcjIdC_j= αc_jI_d. Since α>0α>0, ‖Cj‖F=α|cj|‖Id‖F=αd|cj|→∞\|C_j\|_F= α|c_j|\|I_d\|_F= α d|c_j|→∞. This contradicts the assumed convergence of j θ_j in the finite-dimensional parameter space Θ , thus c⋆≠0c_ ≠ 0. Similarly, suppose that L⋆L_ is singular. Since Lj∈GL(d)L_j∈ GL(d), Lj→L⋆L_j→ L_ , with L⋆L_ singular, the smallest singular value satisfies σmin(Lj)→0 _ (L_j)→ 0. Consequently, ‖Lj−1‖op=1σmin(Lj)→∞\|L_j^-1\|_op= 1 _ (L_j)→∞. On the other hand, cj→c⋆≠0c_j→ c_ ≠ 0, so for sufficiently large j, |cj|⩾|c⋆|2>0|c_j| |c_ |2>0. The sixth parameter block is K+,j=cjγLj−⊤K_+,j=c_jγ L_j^- . Therefore ‖K+,j‖op=γ|cj|‖Lj−⊤‖op=γ|cj|σmin(Lj)⟶∞\|K_+,j\|_op=γ|c_j|\,\|L_j^- \|_op= γ|c_j| _ (L_j) ∞, contradicting convergence of the sixth block of j θ_j, thus L⋆∈GL(d)L_ ∈ GL(d). Since L⋆∈GL(d)L_ ∈ GL(d) and c⋆≠0c_ ≠ 0, the pair (L⋆,c⋆)(L_ ,c_ ) belongs to G. Moreover, continuity of matrix inversion on GL(d)GL(d) gives Lj−1→L⋆−1L_j^-1→ L_ ^-1, and continuity of c⋆↦1/c⋆c_ 1/c_ gives 1cj→1c⋆ 1c_j→ 1c_ . Therefore every block of j θ_j converges to the corresponding block of ¯τ(L,c) θ_τ(L,c), so that ⋆=¯τ(L⋆,c⋆)∈ℳτ θ_ = θ_τ(L_ ,c_ ) _τ. Hence every convergent sequence in ℳτM_τ whose limit exists in Θ has its limit again in ℳτM_τ. Since Θ is finite dimensional, this proves that ℳτM_τ is closed. (i) We prove kerG(¯)=T¯ℳτ G( θ)=T_ θM_τ and ∇2Ln(¯)=G(¯):=J¯∗J¯∇^2L_n( θ)=G( θ):=J_ θ^*J_ θ in the following four steps. Step 1: Tangent space of the optimal parameter manifold ℳτM_τ. Consider a smooth curve t↦(L(t),c(t))t (L(t),c(t)) in GL(d)×(ℝ∖0)GL(d)×(R \0\) such that L(0)=L(0)=L, c(0)=c(0)=c. Write Z:=L˙(0)Z:= L(0), ξ:=c˙(0)ξ:= c(0). Differentiating L(t)−1L(t)=IdL(t)^-1L(t)=I_d at t=0t=0 gives dtL(t)−⊤|t=0=−L−⊤Z⊤L−⊤. ddtL(t)^- |_t=0=-L^- Z L^- . Hence differentiating ¯τ(L(t),c(t)) θ_τ(L(t),c(t)) gives ΔA=0,ΔL=Z,Δb−=0,ΔK−=−ξId, A=0, L=Z, b_-=0, K_-=-ξ I_d, Δb+=0,Δq=0,Δβ=ξ,ΔC=−αξc−2Id, b_+=0, q=0, β=ξ, C=-αξ c^-2I_d, ΔK+=dt[γc(t)L(t)−⊤]t=0=γξL−⊤−cγL−⊤Z⊤L−⊤. K_+= ddt [γ c(t)L(t)^- ]_t=0=γξ L^- -cγ L^- Z L^- . Therefore T¯ℳτ=(0,Z,0,−ξId,0,γξL−⊤−cγL−⊤Z⊤L−⊤,ξ,0,−αξc−2Id):Z∈ℝd×d,ξ∈ℝ. T_ θM_τ= \ (0,Z,0,-ξ I_d,0,γξ L^- -cγ L^- Z L^- ,ξ,0,-αξ c^-2I_d ):Z ^d× d,\ ξ \. (D.8) These are precisely the infinitesimal gauge directions. Step 2: Sample Jacobian and collect the observable parameter combinations. Recall pi,a(A)p_i,a(A) and ψ¯i(A) ψ_i(A) defined in (D.4). Let Σiτ=∑aπi,aτ(ψi,a−ψ¯iτ)(ψi,a−ψ¯iτ)⊤ _i^τ= _a _i,a^τ( _i,a- ψ_i^τ)( _i,a- ψ_i^τ) . The softmax derivative gives Dpi,a(τ−1Id)[ΔA]=πi,aτ(ψi,a−ψ¯iτ)⊤ΔADp_i,a(τ^-1I_d)[ A]= _i,a^τ( _i,a- ψ_i^τ) A\,w. Consequently, Dψ¯i(τ−1Id)[ΔA]=∑aψi,aDpi,a(τ−1Id)[ΔA]=ΣiτΔA. D ψ_i(τ^-1I_d)[ A]= _a _i,aDp_i,a(τ^-1I_d)[ A]= _i^τ A\,w. (D.9) Recall si(,)=ϕi⊤(b−+K−)+b+⊤Lψ¯i(A)+⊤K+⊤Lψ¯i(A)+ri+1(β+q⊤)s_i( θ;z)= _i (b_-+K_-w)+b_+ L ψ_i(A)+w K _+L ψ_i(A)+r_i+1(β+q w) defined in (D.5). At θ¯=θ¯τ(L,c) θ= θ_τ(L,c), we have b+=0b_+=0 and K+=cγL−⊤K_+=cγ L^- . For an arbitrary perturbation U, write ξ:=Δβξ:= β. Using the product rule and (D.9), we obtain s˙i[]= s_i[U]= ϕi⊤(Δb−+ΔK−)+(ψ¯iτ)⊤[L⊤Δb++(L⊤ΔK++cγΔL⊤L−⊤)] _i ( b_-+ K_-w)+( ψ_i^τ) [L b_++ (L K_++cγ L L^- )w ] +ri+1(ξ+Δq⊤)+γc⊤ΣiτΔA. +r_i+1 (ξ+ q w )+γ c\,w _i^τ A\,w. (D.10) Since F,n()=1n∑iCϕisi(,)F_ θ,n(z)= 1n _iC _is_i( θ;z) and C=αcIdC= αcI_d, si(¯,)=cδiτs_i( θ;z)=c _i^τ, the sample Jacobian satisfies J¯()[]=1n∑i[cΔCϕiδiτ+αcϕis˙i[]].J_ θ(z)[U]= 1n _i [c C\, _i _i^τ+ αc _i s_i[U] ]. (D.11) Then we collect the observable parameter combinations. Define the linear map QL,c:Θ→Q_L,c: , QL,c=(EC,eq,e−,E−,e+,E+,EA)Q_L,cU=(E_C,e_q,e_-,E_-,e_+,E_+,E_A), by EC=c2αΔC+ξId,eq=Δq,e−=Δb−,E−=ΔK−+ξId, E_C= c^2α C+ξ I_d, e_q= q, e_-= b_-, E_-= K_-+ξ I_d, (D.12) e+=L⊤Δb+,EA=γcΔA,E+=cγΔL⊤L−⊤+L⊤ΔK+−γξId. e_+=L b_+, E_A=γ c\, A, E_+=cγ L L^- +L K_+-γξ I_d. (D.13) Subtracting ξδiτξ _i^τ from (D.10), where δiτ=ri+1−⊤ϕi+γ⊤ψ¯iτ _i^τ=r_i+1-w _i+ ψ_i^τ, gives s˙i[]−ξδiτ=ϕi⊤(e−+E−)+(ψ¯iτ)⊤(e++E+)+ri+1eq⊤+⊤ΣiτEA. s_i[U]-ξ _i^τ= _i (e_-+E_-w)+( ψ_i^τ) (e_++E_+w)+r_i+1e_q w+w _i^τE_Aw. Likewise, cΔC=αc(EC−ξId)c C= αc(E_C-ξ I_d). Substitution into (D.11) shows that the terms proportional to ξδiτξ _i^τ cancel exactly. Thus J¯()[]=αcℬτ,n()[QL,c].J_ θ(z)[U]= αcB_τ,n(z)[Q_L,cU]. (D.14) Step 3: The kernels of QL,cQ_L,c and Gn(¯)G_n( θ) are exactly the tangent space. Suppose QL,c=0Q_L,cU=0. Then EA=eq=e−=e+=0E_A=e_q=e_-=e_+=0. Since γ>0γ>0, c≠0c≠ 0, and L is invertible, this implies ΔA=0 A=0, Δq=0 q=0, Δb−=0 b_-=0, Δb+=0 b_+=0. Moreover, EC=0E_C=0 implies ΔC=−αξc−2Id C=-αξ c^-2I_d, while E−=0E_-=0 gives ΔK−=−ξId K_-=-ξ I_d. Finally, E+=0E_+=0 gives L⊤ΔK+=γξId−cγΔL⊤L−⊤L K_+=γξ I_d-cγ L L^- . Then ΔK+=γξL−⊤−cγL−⊤ΔL⊤L−⊤ K_+=γξ L^- -cγL^- L L^- . Set Z:=ΔLZ:= L. Together with the preceding identities, this is precisely the tangent form in (D.8). Hence kerQL,c⊂T¯ℳτ Q_L,c⊂ T_ θM_τ. Conversely, take any ∈T¯ℳτU∈ T_ θM_τ. Then for some Z and ξ, direct substitution of (D.8) into (D.12)-(D.13) gives QL,c=0Q_L,cU=0. Therefore T¯ℳτ⊂kerQL,cT_ θM_τ⊂ Q_L,c, which means kerQL,c=T¯ℳτ. Q_L,c=T_ θM_τ. (D.15) Since ℋ=L2(n,ℝd)H=L^2(D_n;R^d) is the population output space, and G(¯)=J¯∗J¯G( θ)=J_ θ^*J_ θ, then, for every perturbation U, by (D.14), we have ⟨,G(¯)⟩=‖J¯‖ℋ2=α2c2[‖ℬτ,n()[QL,c]‖22]=α2c2⟨QL,c,HQL,c⟩, ,G( θ)U =\|J_ θU\|_H^2= α^2c^2\,E [\|B_τ,n(z)[Q_L,cU]\|_2^2 ]= α^2c^2 Q_L,cU,\,H\,Q_L,cU _V, (D.16) where H=[ℬτ,n()∗ℬτ,n()]H=E[B_τ,n(z)^*B_τ,n(z)]. By Assumption 1 (i), H≻0H 0. Therefore ⟨QL,c,HQL,c⟩=0 Q_L,cU,HQ_L,cU =0 if and only if QL,c=0Q_L,cU=0. Since G(¯)=J¯∗J¯G( θ)=J_ θ^*J_ θ is positive semi-definite, ⟨,G(¯)⟩=0⟺G(¯)=0. ,G( θ)U =0 G( θ)U=0. Combining this with (D.16) and (D.15), we obtain kerG(¯)=kerQL,c=T¯ℳτ. G( θ)= Q_L,c=T_ θM_τ. (D.17) Step 4: Prove ∇2Ln(¯)=G(¯)∇^2L_n( θ)=G( θ). Recall that Ln()=12‖e()‖22L_n( θ)= 12E\|e_ θ(z)\|_2^2, where e():=F,n()−Fτ,n()e_ θ(z):=F_ θ,n(z)-F_τ,n(z). Under Assumption 1 (i), the data are bounded. On a sufficiently small bounded parameter neighborhood, softmax smoothness gives common integrable bounds on the predictor and its first two derivatives. Hence differentiation under the expectation is valid. For arbitrary directions ,U,V, D2Ln()[,]=[⟨J()[],J()[]⟩]+[⟨e(),D2F,n()[,]⟩]. D^2L_n( θ)[U,V]=E [ J_ θ(z)[U],J_ θ(z)[V] ]+E [ e_ θ(z),D^2F_ θ,n(z)[U,V] ]. (D.18) By part (i), at every correct parameter ¯∈ℳτ θ _τ we have the samplewise identity e¯()=0e_ θ(z)=0. Therefore the second term in (D.18) vanishes identically, and D2Ln(¯)[,]=[⟨J¯()[],J¯()[]⟩]=⟨,J¯∗J¯⟩. D^2L_n( θ)[U,V]=E [ J_ θ(z)[U],J_ θ(z)[V] ]= ,J_ θ^*J_ θV . Hence, as operators, ∇2Ln(¯)=J¯∗J¯=G(¯).∇^2L_n( θ)=J_ θ^*J_ θ=G( θ). (D.19) Combining (D.19) with (D.17) yields ker∇2Ln(¯)=T¯ℳτ. ∇^2L_n( θ)=T_ θM_τ. Equivalently, if ∉T¯ℳτU∉ T_ θM_τ, then ⟨,∇2Ln(¯)⟩>0 ,∇^2L_n( θ)U >0. In particular, for every nonzero normal direction ∈N¯ℳτ∖0U∈ N_ θM_τ \0\, we have ⟨,∇2Ln(¯)⟩>0 ,∇^2L_n( θ)U >0. ∎ Uniform normal curvature of the optimal parameter manifold By Lemma D.1, we have ∇2Ln(¯)=G(¯)∇^2L_n( θ)=G( θ). It follows that the curvature κ defined in (16) characterizes the weakest second-order growth of the loss when moving away from the optimal manifold. Lemma D.2 shows that the loss has uniformly positive curvature in all normal, non-gauge directions around the optimal parameter manifold, and that this curvature is quantitatively comparable to the data-richness constant: κ≍χ:=λmin(n(τ))κ χ:= _ ( H_n(τ)), where :=[ℬτ,n()∗ℬτ,n()] H:=E [B_τ,n(z)^*B_τ,n(z) ]. Thus, stronger data identifiability leads to stronger local curvature and faster contraction toward the optimal manifold. Recall that ¯⋐GL(d)×(ℝ∖0) GL(d)×(R \0\), τ:=¯τ(L,c):(L,c)∈⊂ℳτU_τ:= \ θ_τ(L,c):(L,c)∈ \ _τ. Lemma D.2 (Uniform normal curvature). Under Assumptions 1, there exists constants 0<cΩ⩽CΩ<∞0<c_ C_ <∞ such that, uniformly over ¯=¯τ(L,c)∈τ θ= θ_τ(L,c) _τ, cΩχ⩽λmin(∇2Ln(¯)|N¯ℳτ)⩽CΩχc_ χ _ (∇^2L_n( θ)|_N_ θM_τ) C_ χ. Equivalently, 0<cΩχ⩽κ⩽CΩχ0<c_ χ κ C_ χ. Proof. Fix ¯=¯τ(L,c)∈τ θ= θ_τ(L,c) _τ. From (D.14), J¯()[]=αcℬτ,n()[QL,c]J_ θ(z)[U]= αcB_τ,n(z)[Q_L,cU], where QL,c:T¯Θ→Q_L,c:T_ θ is the linear map collecting the effective observable coefficients. Moreover, kerQL,c=T¯ℳτ Q_L,c=T_ θM_τ. We first show that the restriction QL,c|N¯ℳτ:N¯ℳτ→Q_L,c|_N_ θM_τ:N_ θM_τ is an isomorphism (a bijective linear map that preserves linear operations). Since N¯ℳτ∩kerQL,c=0N_ θM_τ∩ Q_L,c=\0\, the restriction is injective. Furthermore, dimN¯ℳτ=(5d2+3d+1)−(d2+1)=4d2+3d N_ θM_τ=(5d^2+3d+1)-(d^2+1)=4d^2+3d. The dimension of observable coefficient space is dim=4d2+3d =4d^2+3d, thus the restriction is bijective. For completeness, one may construct an explicit right inverse RL,c:→T¯ΘR_L,c:V→ T_ θ . Specifically, for ζ=(EC,eq,e−,E−,e+,E+,EA)∈ζ=(E_C,e_q,e_-,E_-,e_+,E_+,E_A) , set ΔA A =(γc)−1EA, =(γ c)^-1E_A, ΔL L =0, =0, Δβ β =0, =0, ΔC C =αc−2EC, =α c^-2E_C, Δb− b_- =e−, =e_-, ΔK− K_- =E−, =E_-, Δq q =eq, =e_q, Δb+ b_+ =L−⊤e+, =L^- e_+, ΔK+ K_+ =L−⊤E+. =L^- E_+. Then QL,cRL,c=IQ_L,cR_L,c=I_V. Let PNP_N denote the orthogonal projection onto N¯ℳτN_ θM_τ. Since QL,cQ_L,c vanishes on T¯ℳτT_ θM_τ, QL,cPN=QL,cQ_L,cP_N=Q_L,c. Therefore PNRL,cP_NR_L,c is the inverse of QL,c|N¯ℳτQ_L,c|_N_ θM_τ. Since ¯ is compact and L,L−1,c,c−1L,\ L^-1,\ c,\ c^-1 are uniformly bounded on ¯ , the operator norms of QL,cQ_L,c and RL,cR_L,c are uniformly bounded. Hence there exist constants 0<t−⩽t+<∞0<t_- t_+<∞ such that, for every ¯∈τ θ _τ and every ∈N¯ℳτU∈ N_ θM_τ, t−‖⩽‖QL,c‖⩽t+‖.t_-\|U\| \|Q_L,cU\|_V t_+\|U\|. (D.20) where one may take t−=(sup(L,c)∈¯‖RL,c‖op)−1t_-=( _(L,c)∈ \|R_L,c\|_op)^-1, t+=sup(L,c)∈¯‖QL,c‖opt_+= _(L,c)∈ \|Q_L,c\|_op. We now transfer Assumption 1 from coefficient space to parameter space. Since ⪰χI H χ I_V, we can obtain ⟨QL,c,QL,c⟩⩾χ‖QL,c‖2 Q_L,cU,\, HQ_L,cU χ\|Q_L,cU\|_V^2 by (D.16). Thus, for every unit normal vector ∈N¯ℳτU∈ N_ θM_τ, ‖=1\|U\|=1, yields ⟨,G(¯)⟩⩾α2c2χt−2 ,G( θ)U α^2c^2χ t_-^2 from (D.20). Let c+:=max(L,c)∈¯|c|c_+:= _(L,c)∈ |c|, then 1c2⩾1c+2 1c^2 1c_+^2, and hence ⟨,G(¯)⟩⩾α2t−2c+2χ ,G( θ)U α^2t_-^2c_+^2χ. Therefore, by taking cΩ=α2t−2c+2c_ = α^2t_-^2c_+^2, we have λmin(G(¯)|N¯ℳτ)⩾cΩχ. _ (G( θ)|_N_ θM_τ) c_ χ. (D.21) Let ζ∗∈ _* , ‖ζ∗‖=1\| _*\|_V=1, be a unit eigenvector of H corresponding to its smallest eigenvalue ζ∗=χζ∗ H _*=χ _*. Since QL,c|N¯ℳτQ_L,c|_N_ θM_τ is invertible, define ∗=(QL,c|N¯ℳτ)−1ζ∗U_*=(Q_L,c|_N_ θM_τ)^-1 _*. Then QL,c∗=ζ∗Q_L,cU_*= _*. By the upper estimate in (D.20), 1=‖QL,c∗‖⩽t+‖∗‖1=\|Q_L,cU_*\|_V t_+\|U_*\|, so ‖∗‖⩾1t+\|U_*\| 1t_+. Using (D.16), ⟨^∗,G(¯)^∗⟩=α2c2⟨ζ∗,ζ∗⟩‖∗‖2=α2c2χ‖∗‖2⩽α2t+2c2χ,^∗=∗‖∗‖ U_*,G( θ) U_* = α^2c^2 _*, H _* \|U_*\|^2= α^2c^2 χ\|U_*\|^2 α^2t_+^2c^2χ, U_*= U_*\|U_*\| Let c−:=min(L,c)∈¯|c|>0c_-:= _(L,c)∈ |c|>0. Then 1c2⩽1c−2 1c^2 1c_-^2, and therefore, by taking CΩ=α2t+2c−2C_ = α^2t_+^2c_-^2, λmin(G(¯)|N¯ℳτ)⩽CΩχ. _ (G( θ)|_N_ θM_τ) C_ χ. (D.22) Finally, by Lemma D.1, ∇2Ln(¯)=G(¯)∇^2L_n( θ)=G( θ) for every ¯∈ℳτ θ _τ. Combining (D.21) and (D.22) gives cΩχ⩽λmin(∇2Ln(¯)|N¯ℳτ)⩽CΩχ.c_ χ _ (∇^2L_n( θ)|_N_ θM_τ) C_ χ. Taking the infimum over ¯∈τ θ _τ and over unit normal directions yields cΩχ⩽κ⩽CΩχc_ χ κ C_ χ. Since χ>0χ>0 by Assumption 1 and cΩ>0c_ >0, we conclude that κ>0κ>0. ∎ Local tubular geometry and unique projection. Lemma D.3 provides a precise local notion of distance to the optimal parameter manifold. For any parameter θ sufficiently close to ℳτM_τ, define, d():=dist(,ℳτ)=infϑ∈ℳτ‖−ϑ‖.d( θ):=dist( θ,M_τ)= _ _τ\| θ- \|. Lemma D.3 guarantees that this distance is attained at a unique nearest point ¯=Π() θ= ( θ). Moreover, the displacement Δ=−Π() = θ- ( θ) is normal to the manifold, i.e. d()=‖−Π()‖d( θ)=\| θ- ( θ)\|. Lemma D.3 (Local tubular geometry and unique projection). There exists a constant rgeo>0r_ geo>0 such that the following statements hold. (i) For every ¯∈¯τ θ∈ U_τ and Δ∈N¯ℳτ ∈ N_ θM_τ with ‖Δ‖<rgeo\| \|<r_ geo, the point =¯+Δ θ= θ+ has ¯ θ as its unique nearest point on the full manifold ℳτM_τ. (i) Define the normal tube rgeo(τ)=¯+Δ:¯∈τ,Δ∈N¯ℳτ,∥Δ∥<rgeoT_r_ geo(U_τ)= \ θ+ : θ _τ,\ ∈ N_ θM_τ,\ \| \|<r_ geo \. For every ∈rgeo(τ) θ _r_ geo(U_τ), the nearest-point projection Π() ( θ) onto ℳτM_τ is uniquely defined and satisfies −Π()∈NΠ()ℳτ θ- ( θ)∈ N_ ( θ)M_τ and d()=‖−Π()‖d( θ)=\| θ- ( θ)\|. Moreover, rgeo(τ)T_r_ geo(U_τ) is an open neighborhood of τU_τ in the ambient parameter space. Proof. By Lemma D.1, ℳτM_τ is a smooth closed embedded submanifold of the finite-dimensional Euclidean parameter space, and ¯τ U_τ is compact. By the tubular neighborhood theorem (Lee, 2013, Theorem 6.24), every point sufficiently close to ℳτM_τ can be uniquely represented as =¯+Δ θ= θ+ , where ¯∈ℳτ θ _τ and Δ∈N¯ℳτ ∈ N_ θM_τ is a sufficiently small normal displacement. For each ¯∈¯τ θ∈ U_τ, there exist a neighborhood ¯⊂ℳτV_ θ _τ and a radius r¯>0r_ θ>0 such that the above normal representation remains unique for all normal perturbations of size smaller than r¯r_ θ from points in ¯V_ θ. Since ¯τ U_τ is compact, finitely many such neighborhoods, say 1,…,LV_1,…,V_L, cover ¯τ U_τ. Define =⋃j=1LjV= _j=1^LV_j and r0=min1⩽j⩽Lrjr_0= _1 j Lr_j. Since the minimum is taken over finitely many positive radii, r0>0r_0>0. Thus every normal perturbation of norm smaller than r0r_0 from a point in V lies in the region where the normal representation is unique. (i) We choose the radius small enough so that the nearest point cannot leave V. Since ¯τ⊂ U_τ , ¯τ U_τ is compact, and ℳτ∖M_τ is closed, we have δ:=dist(¯τ,ℳτ∖)>0δ:=dist( U_τ,M_τ )>0. Choose rgeo>0r_ geo>0 such that rgeo<minr0,δ/2r_ geo< \r_0,δ/2\. Now take ¯∈¯τ θ∈ U_τ and Δ∈N¯ℳτ ∈ N_ θM_τ with ‖Δ‖<rgeo\| \|<r_ geo, and set =¯+Δ θ= θ+ . Since ℳτM_τ is closed, there exists at least one nearest point ⋆∈ℳτ θ _τ to θ. Because ¯∈ℳτ θ _τ, ‖−⋆‖⩽‖−¯‖=‖Δ‖<rgeo.\| θ- θ \| \| θ- θ\|=\| \|<r_ geo. The triangle inequality then gives ‖⋆−¯‖<2rgeo<δ\| θ - θ\|<2r_ geo<δ. Hence ⋆∈ θ . Since ⋆ θ minimizes the Euclidean distance from θ to ℳτM_τ, the first-order optimality condition implies −⋆∈N⋆ℳτ θ- θ ∈ N_ θ M_τ. Therefore θ has two normal representations: =¯+Δ θ= θ+ and =⋆+(−⋆) θ= θ +( θ- θ ). Both base points lie in V, and both normal displacements have norm smaller than r0r_0. By uniqueness of the tubular representation, we must have ⋆=¯ θ = θ. Thus ¯ θ is the unique nearest point to θ on the full manifold ℳτM_τ. Consequently, Π()=¯,−Π()∈NΠ()ℳτ,d()=‖−Π()‖. ( θ)= θ, θ- ( θ)∈ N_ ( θ)M_τ, d( θ)=\| θ- ( θ)\|. (i) Since τU_τ is relatively open in ℳτM_τ, the normal perturbations of norm smaller than rgeor_ geo based at points of τU_τ form an open set in the tubular coordinates. The tubular-neighborhood map sends this set to an open subset of the ambient parameter space containing τU_τ. Hence the corresponding normal tube is an open neighborhood of τU_τ. ∎ D.2 A Direct Parameter-Distance Estimate for One GD Step The identity d()=‖Δ‖d( θ)=\| \| is essential for the convergence analysis, where Δ=−Π() = θ- ( θ) is the normal displacement from the current parameter to its nearest point on the optimal manifold. The operator Π(),n[Δ]J_ ( θ),n[ ] describes the first-order change of the model output when the parameter moves from the nearest optimal point Π() ( θ) in the direction Δ . Since Δ is normal to the manifold, the normal-curvature bound in Lemma D.2 can be applied directly, giving ‖Π(),n[Δ]‖L2(Dn)2⩾κ‖Δ‖2=κd()2. \|J_ ( θ),n[ ] \|_L^2(D_n)^2 κ\| \|^2=κ d( θ)^2. Lemma D.4 shows that one gradient-descent step moves the current parameter closer to the optimal manifold. We first prove that the gradient has a sufficiently large component in the direction that reduces the normal displacement from the manifold. We then control the overall gradient norm to ensure that the update does not overshoot. Combining these two estimates yields a one-step contraction of the distance to ℳτM_τ. Lemma D.4 (One-step contraction toward the optimal manifold). Under Assumption 1, (i) There exist constants MJ>0M_J>0, LJ>0L_J>0, and 0<ρ0<rgeo0< _0<r_ geo such that, for all ,′∈ρ0(τ) θ, θ _ _0(U_τ), ‖,n‖op⩽MJ\|J_ θ,n\|_ op M_J, and ‖,n−′,n‖op⩽LJ‖−′‖\|J_ θ,n-J_ θ ,n\|_ op L_J\| θ- θ \|. (i) Let r:=minρ0,κ4LJr:= \ _0, κ4L_J \, :=r(τ)T:=T_r(U_τ). For any ∈ θ and every step size 0<η⩽1/(2MJ2)0<η 1/(2M_J^2), d(−η∇Ln())2⩽(1−ηκ4)d()2d ( θ-η∇ L_n( θ) )^2 (1- ηκ4 )d( θ)^2, and η‖∇Ln()‖⩽4MJκ[d()−d(−η∇Ln())]η \|∇ L_n( θ) \| 4M_J κ [d( θ)-d ( θ-η∇ L_n( θ) ) ], where 1−ηκ/4∈[7/8,1)1-ηκ/4∈[7/8,1). Proof. (i) Write the parameter as =(A,x) θ=(A,x), where x collects all parameter blocks other than A. Since ¯τ U_τ is compact, there exists M0<∞M_0<∞ such that ‖x‖⩽M0\|x\| M_0 on ¯τ U_τ. Choose 0<ρ0<minrgeo,10< _0< \r_geo,1\ sufficiently small so that ρ0(τ)T_ _0(U_τ) and the line segment joining each point in this tube to its projection onto ℳτM_τ are contained in =(A,x):A∈ℝd×d,∥x∥<M0+1.C= \(A,x):A ^d× d,\ \|x\|<M_0+1 \. Although A is unrestricted in C, it enters the model through the softmax logits. The derivatives of the softmax are uniformly bounded, the features and weights are bounded by Assumption 1, and all remaining parameter blocks are bounded on C. Hence there exist constants MJ>0M_J>0 and LJ>0L_J>0 such that, for all ,′∈ρ0(τ) θ, θ _ _0(U_τ), ‖,n‖op⩽MJ,‖,n−′,n‖op⩽LJ‖−′‖.\|J_ θ,n\|_ op M_J, \|J_ θ,n-J_ θ ,n\|_ op L_J\| θ- θ \|. (D.23) Since κ is the minimum curvature over unit normal directions on the selected optimal patch, the first bound in (D.23) also gives κ⩽MJ2κ M_J^2. For a fixed gauge domain Ω and temperature-independent data bounds, the constants MJM_J and LJL_J can be chosen independently of τ. Indeed, the non-A coordinates of the realization patch do not depend on τ, while A enters the model only through the softmax logits, whose derivatives with respect to those logits are uniformly bounded. (i) Fix ∈r(τ) θ _r(U_τ) and let ¯=Π() θ= ( θ) and Δ=−¯ = θ- θ. By Lemma D.3, Δ∈N¯ℳτ ∈ N_ θM_τ and ‖Δ‖=d()\| \|=d( θ). If d()=0d( θ)=0, then ∈ℳτ θ _τ and ∇Ln()=0∇ L_n( θ)=0, so the conclusions are immediate. Hence assume d()>0d( θ)>0. Define +=−η∇Ln() θ^+= θ-η∇ L_n( θ) and d+:=d(+)d_+:=d( θ^+). Since ¯∈ℳτ θ _τ is a valid comparison point in the definition of d+d_+, d+2⩽‖−η∇Ln()−¯‖2=d()2−2η⟨∇Ln(),Δ⟩+η2∥∇Ln()∥2. d_+^2 \| θ-η∇ L_n( θ)- θ \|^2=d( θ)^2-2η ∇ L_n( θ), +η^2\|∇ L_n( θ)\|^2. (D.24) Thus, it is enough to show that the directional component ⟨∇Ln(),Δ⟩ ∇ L_n( θ), is uniformly positive and that the total gradient norm is controlled by this same quantity. We establish these two bounds below. For ⟨∇Ln(),Δ⟩ ∇ L_n( θ), . Since ¯∈ℳτ θ _τ, F¯,n=Fτ,nF_ θ,n=F_τ,n, and the fundamental theorem of calculus gives F,n−Fτ,n=∫01¯+tΔ,n[Δ]t.F_ θ,n-F_τ,n= _0^1J_ θ+t ,n[ ]\,dt. (D.25) Moreover, for every parameter direction U, ⟨∇Ln(),⟩=⟨F,n−Fτ,n,,n[]⟩L2(n). ∇ L_n( θ),U = F_ θ,n-F_τ,n,J_ θ,n[U] _L^2(D_n). (D.26) Because Δ is normal to ℳτM_τ, Lemma D.2 gives ‖¯,n[Δ]‖L2(n)⩾κd() \|J_ θ,n[ ] \|_L^2(D_n) κ\,d( θ). Using the Jacobian Lipschitz bound in (D.23), ‖,n[Δ]‖L2(n)⩾‖¯,n[Δ]‖L2(n)−LJ‖−¯‖‖Δ‖⩾κd()−LJd()2. \|J_ θ,n[ ] \|_L^2(D_n) \|J_ θ,n[ ] \|_L^2(D_n)-L_J\| θ- θ\|\| \| κ\,d( θ)-L_Jd( θ)^2. Since d()<rd( θ)<r and r⩽κ/(4LJ)r κ/(4L_J), ‖,n[Δ]‖L2(n)⩾34κd(). \|J_ θ,n[ ] \|_L^2(D_n) 34 κ\,d( θ). (D.27) Using (D.25), write F,n−Fτ,n=,n[Δ]+R,F_ θ,n-F_τ,n=J_ θ,n[ ]+R, (D.28) where R:=∫01(¯+tΔ,n−,n)[Δ]tR:= _0^1 (J_ θ+t ,n-J_ θ,n )[ ]\,dt. By the Jacobian Lipschitz bound, ‖R‖L2(n)⩽∫01LJ(1−t)‖Δ‖2t=LJ2d()2⩽18κd(). \|R\|_L^2(D_n) _0^1L_J(1-t)\| \|^2\,dt= L_J2d( θ)^2 18 κ\,d( θ). (D.29) Combining this with (D.27) gives ‖R‖L2(n)⩽16‖,n[Δ]‖L2(n)\|R\|_L^2(D_n) 16\|J_ θ,n[ ]\|_L^2(D_n). Taking =ΔU= in (D.26) and using (D.28), we have ⟨∇Ln(),Δ⟩=⟨,n[Δ]+R,,n[Δ]⟩L2(n)⩾56‖,n[Δ]‖L2(n)2. ∇ L_n( θ), = _ θ,n[ ]+R,J_ θ,n[ ] _L^2(D_n) 56\|J_ θ,n[ ]\|_L^2(D_n)^2. (D.30) Together with (D.27), this yields ⟨∇Ln(),Δ⟩⩾1532κd()2. ∇ L_n( θ), 1532κ d( θ)^2. (D.31) For ‖∇Ln()‖\|∇ L_n( θ)\|. By (D.28), ‖F,n−Fτ,n‖L2(n)⩽76‖,n[Δ]‖L2(n)\|F_ θ,n-F_τ,n\|_L^2(D_n) 76\|J_ θ,n[ ]\|_L^2(D_n). Using (D.26), (D.23), and (D.30), we obtain ‖∇Ln()‖2⩽4930MJ2⟨∇Ln(),Δ⟩.\|∇ L_n( θ)\|^2 4930M_J^2 ∇ L_n( θ), . (D.32) We now return to the distance expansion (D.24). Using (D.32), d+2 d_+^2 ⩽d()2−η(2−4930ηMJ2)⟨∇Ln(),Δ⟩. d( θ)^2-η (2- 4930η M_J^2 ) ∇ L_n( θ), . (D.33) If η⩽1/(2MJ2)η 1/(2M_J^2), then 2−(49/30)ηMJ2⩾71/60>12-(49/30)η M_J^2 71/60>1. Therefore, by (D.31), d+2⩽d()2−η⟨∇Ln(),Δ⟩⩽(1−1532ηκ)d()2⩽(1−ηκ4)d()2. d_+^2 d( θ)^2-η ∇ L_n( θ), (1- 1532ηκ )d( θ)^2 (1- ηκ4 )d( θ)^2. (D.34) Since κ⩽MJ2κ M_J^2 and η⩽1/(2MJ2)η 1/(2M_J^2), the contraction factor satisfies 1−ηκ/4∈[7/8,1)1-ηκ/4∈[7/8,1). It remains to control the step length. From (D.33) and the same step-size condition, η⟨∇Ln(),Δ⟩⩽d()2−d+2⩽2d()(d()−d+),η ∇ L_n( θ), d( θ)^2-d_+^2 2d( θ) (d( θ)-d_+ ), (D.35) where d+⩽d()d_+ d( θ) follows from (D.34). Furthermore, by (D.30), (D.27), and the bound ‖F,n−Fτ,n‖L2(n)⩽(7/6)‖,n[Δ]‖L2(n)\|F_ θ,n-F_τ,n\|_L^2(D_n) (7/6)\|J_ θ,n[ ]\|_L^2(D_n), ‖∇Ln()‖⟨∇Ln(),Δ⟩⩽28MJ15κd(). \|∇ L_n( θ)\| ∇ L_n( θ), 28M_J15 κ\,d( θ). (D.36) Combining (D.35) and (D.36) yields η∥∇Ln()∥⩽56MJ15κ(d()−d+)⩽4MJκ[d()−d(−η∇Ln())]. η\|∇ L_n( θ)\| 56M_J15 κ (d( θ)-d_+ ) 4M_J κ [d( θ)-d ( θ-η∇ L_n( θ) ) ]. This proves both the one-step contraction and the step-length estimate. ∎ D.3 The proof of Theorem 2 Let qη:=1−ηκ/4q_η:=1-ηκ/4. By Lemma D.4, if 0<η⩽1/(2MJ2)0<η 1/(2M_J^2), then qη∈(0,1)q_η∈(0,1). Convergence of the full parameter sequence and its limit. Let dk:=d(k)d_k:=d( θ_k). Assume d0>0d_0>0. Define Λ0:=4MJκd0 _0:= 4M_J κ\,d_0. By the initialization condition, Λ0<b(0) _0<b_T( θ_0), where b():=dist(,c)b_T( θ):=dist( θ,T^c). Therefore the closed ball B¯(0,Λ0) B( θ_0, _0) is contained in T. We prove inductively that every iterate and every update segment remain in this ball and that dk⩽qηk/2d0d_k q_η^k/2d_0. Suppose k∈ θ_k . By Lemma D.4, dk+1⩽qηdk,‖k+1−k‖⩽4MJκ(dk−dk+1).d_k+1 q_η\,d_k, \| θ_k+1- θ_k\| 4M_J κ(d_k-d_k+1). (D.37) The first inequality gives dk+1⩽qη(k+1)/2d0d_k+1 q_η^(k+1)/2d_0. Now consider any point =k+s(k+1−k) y= θ_k+s( θ_k+1- θ_k), with 0⩽s⩽10 s 1, on the new update segment. Using (D.37) and the induction hypothesis, ‖−0‖ \| y- θ_0\| ⩽∑j=0k−1‖j+1−j‖+s‖k+1−k‖ _j=0^k-1\| θ_j+1- θ_j\|+s\| θ_k+1- θ_k\| ⩽4MJκ∑j=0k(dj−dj+1)=4MJκ(d0−dk+1)⩽Λ0. 4M_J κ _j=0^k(d_j-d_j+1)= 4M_J κ(d_0-d_k+1) _0. (D.38) Hence the entire update segment remains in B¯(0,Λ0)⊂ B( θ_0, _0) . Every iterate remains in T, and dk⩽qηk/2d0d_k q_η^k/2d_0 holds for all k⩾0k 0. The decay dk→0d_k→ 0 alone does not guarantee convergence of the full parameter sequence, since the iterates could in principle drift along the optimal manifold. The step-length estimate in (D.37) rules out this possibility. For integers ℓ>k >k, summing the step-length bound gives ∑j=kℓ−1‖j+1−j‖ _j=k -1\| θ_j+1- θ_j\| ⩽4MJκ(dk−dℓ)⩽4MJκdk⩽Λ0qηk/2. 4M_J κ(d_k-d_ ) 4M_J κd_k _0q_η^k/2. (D.39) The right-hand side tends to zero as k→∞k→∞. Hence (k)( θ_k) is a Cauchy sequence. Since the parameter space is finite-dimensional, there exists ∞ θ_∞ such that k→∞ θ_k→ θ_∞. Letting ℓ→∞ →∞ in (D.39) yields ‖k−∞‖⩽Λ0qηk/2.\| θ_k- θ_∞\| _0q_η^k/2. (D.40) Together with dk⩽qηk/2d0d_k q_η^k/2d_0, this proves geometric convergence both to the optimal manifold and to a single parameter limit. The distance function d()=dist(,ℳτ)d( θ)=dist( θ,M_τ) is 11-Lipschitz. Since k→∞ θ_k→ θ_∞ and dk→0d_k→ 0, we obtain d(∞)=0d( θ_∞)=0. Because ℳτM_τ is closed, ∞∈ℳτ θ_∞ _τ. Moreover, all iterates lie in the closed ball B¯(0,Λ0) B( θ_0, _0), so the limit also lies in this ball and therefore in T. Since =r(τ)T=T_r(U_τ), there exist ¯∈τ θ _τ and Δ∈N¯ℳτ ∈ N_ θM_τ, with ‖Δ‖<r\| \|<r, such that ∞=¯+Δ θ_∞= θ+ . By Lemma D.3, ¯ θ is the unique nearest point of ∞ θ_∞ on ℳτM_τ. But ∞∈ℳτ θ_∞ _τ, so ∞ θ_∞ itself is a nearest point. Uniqueness therefore implies Δ=0 =0 and ∞=¯∈τ θ_∞= θ _τ. Hence there exist (L∞,c∞)∈(L_∞,c_∞)∈ such that ∞=¯τ(L∞,c∞) θ_∞= θ_τ(L_∞,c_∞). By exact realization on ℳτM_τ, F∞,n=Fτ,nF_ θ_∞,n=F_τ,n, and therefore Ln(∞)=0=min∈ΘLn()L_n( θ_∞)=0= _ θ∈ L_n( θ). Iteration-complexity bound. By dk⩽qηk/2d0d_k q_η^k/2d_0, d(K)⩽qηK/2d0d( θ_K) q_η^K/2d_0. Since qη=1−ηκ/4q_η=1-ηκ/4 and 1−x⩽e−x1-x e^-x, d(K)⩽exp(−ηκK8)d0d( θ_K) (- ηκ K8)d_0. Thus K⩾8ηκlog(1/ε)K 8ηκ (1/ ) is sufficient to guarantee d(K)⩽εd0d( θ_K) d_0. Consequently, KGD(ε,0)⩽⌈8ηκlog1ε⌉K_ GD( ; θ_0) 8ηκ 1 . We prove the lower bound by considering a small perturbation along the normal direction with the smallest local curvature. This direction is the locally slowest contraction direction of gradient descent. Choose an interior point ¯∘∈τ θ _τ and define ℓ:=λmin(∇2Ln(¯∘)|N¯∘ℳτ). := _ (∇^2L_n( θ ) |_N_ θ M_τ ). (D.41) Thus, ℓ is the smallest Hessian curvature over the normal directions at ¯∘ θ . By Lemma D.2, κ⩽ℓ⩽(CΩ/cΩ)κ (C_ /c_ )κ. At an optimal parameter, ∇2Ln(¯∘)=¯∘,n∗¯∘,n∇^2L_n( θ )=J_ θ ,n^*J_ θ ,n. Hence the local Jacobian bound gives ℓ⩽MJ2 M_J^2. Since the Hessian is symmetric and ker(∇2Ln(¯∘))=T¯∘ℳτker(∇^2L_n( θ ))=T_ θ M_τ, the normal space is invariant under the Hessian. Therefore, there exists a unit vector ∘∈N¯∘ℳτU ∈ N_ θ M_τ such that ∇2Ln(¯∘)∘=ℓ∘.∇^2L_n( θ )U = . (D.42) Thus, ∘U is a normal direction with the smallest local curvature. For t>0t>0, define t=¯∘+t∘ θ_t= θ +tU . For sufficiently small t, Lemma D.3 gives d(t)=td( θ_t)=t. We first verify that these perturbations are admissible initializations. Let b∘:=b(¯∘)>0b :=b_T( θ )>0. Since the distance to cT^c is 11-Lipschitz, b(t)⩾b∘−tb_T( θ_t) b -t. Thus, if t is sufficiently small so that t<rt<r, t<b∘/2t<b /2, and t<b∘κ/(8MJ)t<b κ/(8M_J), then 4MJκd(t)<b(t). 4M_J κd( θ_t)<b_T( θ_t). (D.43) Hence all sufficiently small nonzero t θ_t are admissible initializations. We next determine how fast gradient descent contracts this perturbation. Define Tη()=−η∇Ln()T_η( θ)= θ-η∇ L_n( θ). Since ∇Ln(¯∘)=0∇ L_n( θ )=0, the point ¯∘ θ is a fixed point of TηT_η. Moreover, DTη(¯∘)=I−η∇2Ln(¯∘)DT_η( θ )=I-η∇^2L_n( θ ). Therefore, using (D.42), for every fixed finite integer K, D(TηK)(¯∘)∘=(1−ηℓ)K∘.D(T_η^K)( θ )U =(1-η )^KU . (D.44) Hence, to first order, the perturbation along ∘U is reduced by the factor (1−ηℓ)K(1-η )^K after K gradient-descent steps. We now make this local linearization precise. Fix K and then let t tend to zero. Since LnL_n is twice continuously differentiable in the local regularity region, TηT_η is continuously differentiable there. For sufficiently small t, the first K iterates remain in this region, and TηK(t)=¯∘+t(1−ηℓ)K∘+RK(t),‖RK(t)‖t⟶0as t↓0.T_η^K( θ_t)= θ +t(1-η )^KU +R_K(t), \|R_K(t)\|t 0 t 0. (D.45) Here K is fixed before taking the limit t↓0t 0. Since ℓ⩽MJ2 M_J^2 and η⩽1/(2MJ2)η 1/(2M_J^2), we have 0<ηℓ⩽1/20<η 1/2, and therefore (1−ηℓ)K>0(1-η )^K>0. For sufficiently small t, Lemma D.3 gives d(¯∘+t(1−ηℓ)K∘)=t(1−ηℓ)Kd( θ +t(1-η )^KU )=t(1-η )^K. Because the distance to ℳτM_τ is 11-Lipschitz, (D.45) implies limt↓0d(TηK(t))d(t)=(1−ηℓ)K. _t 0 d(T_η^K( θ_t))d( θ_t)=(1-η )^K. (D.46) Suppose that K iterations guarantee d(TηK(0))⩽εd(0)d(T_η^K( θ_0)) d( θ_0) for every admissible initialization. Since all sufficiently small t θ_t constructed above are admissible, (D.46) implies (1−ηℓ)K⩽ε(1-η )^K . Therefore, K⩾log(1/ε)/[−log(1−ηℓ)]K (1/ )/[- (1-η )]. Since 0<ηℓ⩽1/20<η 1/2 and −log(1−x)⩽2x- (1-x) 2x for 0<x⩽1/20<x 1/2, we obtain sup0∈τKGD(ε,θ0)⩾12ηℓlog1ε⩾cΩ2CΩ1ηκlog1ε. _ θ^0 _τK_GD( ;θ^0) 12η 1 c_ 2C_ 1ηκ 1 . (D.47) Appendix E Verification for a Concrete Population Model In this section, we provide simple sufficient conditions under which the richness condition in Assumption 1 holds, showing that it is satisfied by a broad class of task distributions, including our experimental setting. We then quantify its dependence on the temperature parameter for the experimental distribution. Let a task consist of an initial distribution P0P_0, transition kernel P, feature table Φ:×→ℝd :S×A ^d, reward table R(a,s′)R(a,s ), and an input weight w. Given a task, a trajectory is generated by a fixed behavior policy πb _b, and ϕi=Φ(si,ai) _i= (s_i,a_i), ψi,a=Φ(si+1,a) _i,a= (s_i+1,a), and ri+1=R(ai,si+1)r_i+1=R(a_i,s_i+1). E.1 Qualitative verification of observable richness Theorem E.1 (Continuous-population verification). Under Assumption 1(i), assume that: (i) the finite-dimensional random vector formed by w, the entries of the feature Φ , and the entries of the reward R has a nonzero absolutely continuous component with respect to Lebesgue measure; (i) there exist distinct states s0,s1s_0,s_1 and, when n>1n>1, distinct actions ax,aza_x,a_z such that, almost surely over tasks, P0(s0)>0P_0(s_0)>0, P(s1∣s0,ax)>0P(s_1 s_0,a_x)>0, P(s1∣s1,az)>0P(s_1 s_1,a_z)>0, and πb(ax∣s0)>0 _b(a_x s_0)>0, πb(az∣s1)>0 _b(a_z s_1)>0. For n=1n=1, only P0(s0)>0P_0(s_0)>0, P(s1∣s0,ax)>0P(s_1 s_0,a_x)>0, and πb(ax∣s0)>0 _b(a_x s_0)>0 are required. Then, for every fixed n⩾1n 1 and τ>0τ>0, Assumption 1 (i) holds. Proof. Fix n⩾1n 1 and τ>0τ>0. Recall that τ,n=[ℬτ,n()∗ℬτ,n()] H_τ,n=E_z[B_τ,n(z)^*B_τ,n(z)]. Hence, for every coefficient vector ζ, ⟨ζ,τ,nζ⟩=[‖ℬτ,n()[ζ]‖2] ζ, H_τ,nζ =E[\|B_τ,n(z)[ζ]\|^2]. Therefore, it is enough to prove that ℬτ,n()[ζ]=0B_τ,n(z)[ζ]=0 almost surely implies ζ=0ζ=0. Let TnT_n denote the random discrete trajectory and ℰn=Tn=wnE_n=\T_n=w_n\ denote the calibration family. Since |ℬτ,n()[ζ]|2⩾[|ℬτ,n()[ζ]|2ℰn]E|B_τ,n(z)[ζ]|^2 [|B_τ,n(z)[ζ]|^21_E_n ], it suffices to show that every ζ≠0ζ≠ 0 is detected on ℰnE_n, i.e., [|ℬτ,n()[ζ]|2ℰn]>0E [|B_τ,n(z)[ζ]|^21_E_n ]>0. Consider the deterministic calibration path ωn _n consisting of s0→axs1s_0 a_xs_1, followed, when n>1n>1, by n−1n-1 repetitions of s1→azs1s_1 a_zs_1. For n=1n=1, only the first transition is retained. Let XnX_n collect the continuous coordinates W=W=w, Y=Φ(s0,ax)Y= (s_0,a_x), Ψa=Φ(s1,a) _a= (s_1,a), a∈[m]a∈[m], together with R0=R(ax,s1)R_0=R(a_x,s_1) and, when n>1n>1, R1=R(az,s1)R_1=R(a_z,s_1). Define the subprobability measure νn(E):=ℙ(Xn∈E,Tn=ωn) _n(E):=P (X_n∈ E,\,T_n= _n ). The calibration path ωn _n is not a null event. Let μ denote the task distribution, and define pωn(Ξ):=Pr(Tn=ωn∣Ξ)p_ _n( ):= (T_n= _n ), where Ξ denotes the sampled task variables. By assumption (i), every factor in pωn(Ξ)p_ _n( ) is strictly positive almost surely. Hence pωn(Ξ)>0p_ _n( )>0 for μ-almost every task, and Pr(Tn=ωn)=μ[pωn(Ξ)]>0 (T_n= _n)=E_μ[p_ _n( )]>0. Let μX _X denote the marginal distribution of XnX_n, and define ℓn(x):=μ[pωn(Ξ)∣Xn=x] _n(x):=E_μ [p_ _n( ) X_n=x ]. Since pωn(Ξ)>0p_ _n( )>0 almost surely, we have ℓn(x)>0 _n(x)>0 for μX _X-almost every x. By the tower property, for every Borel set E, νn(E)=Pr(Xn∈E,Tn=ωn)=∫Eℓn(x)μX(x) _n(E)= (X_n∈ E,\ T_n= _n)= _E _n(x)\, _X(\,dx). By assumption (i), the joint distribution of (,Φ,R)(w, ,R) has a nonzero absolutely continuous component. Since XnX_n is a coordinate subvector of (,Φ,R)(w, ,R), its marginal distribution μX _X also has a nonzero absolutely continuous component, denoted by μXac _X^ac. The measure ν~n(dx):=ℓn(x)μXac(dx) ν_n(\,dx):= _n(x)\, _X^ac(\,dx) is absolutely continuous with respect to Lebesgue measure. Moreover, it is nonzero because μXac≠0 _X^ac≠ 0 and ℓn>0 _n>0 μXac _X^ac-almost everywhere. Since μXac⩽μX _X^ac _X, we also have ν~n⩽νn ν_n _n. Therefore, νn _n has a nonzero absolutely continuous component. Suppose that ℬτ,n()[ζ]=0B_τ,n(z)[ζ]=0 almost surely under the joint task-trajectory distribution. Restricting to the event Tn=ωnT_n= _n gives ℬτ,n(ωn(x))[ζ]=0B_τ,n (z_ _n(x) )[ζ]=0 for νn _n-almost every x. Each component of the left-hand side is real analytic in the continuous coordinates x, since it is formed from finite sums, products, exponentials, and divisions by strictly positive softmax denominators. Because νn _n has a nonzero absolutely continuous component, each component vanishes on a set of positive Lebesgue measure. A nonzero real-analytic scalar function on ℝqR^q has a Lebesgue-null zero set. Hence every component must vanish identically. Therefore, ℬτ,n()[ζ]=0B_τ,n(z)[ζ]=0 implies ℬτ,n(ωn(x))[ζ]=0, for all x.B_τ,n(z_ _n(x))[ζ]=0, for all x. We now use this identity to identify the coefficient blocks in ζ. Let ax=1a_x=1, set Y=yY=y, Ψ1=x _1=x, and Ψa=0 _a=0 for every a≠1a≠ 1. Since az≠axa_z≠ a_x when n>1n>1, all later current-state features on the calibration path vanish. Writing r=R0r=R_0, nℬτ,n[ζ]=0nB_τ,n[ζ]=0 reduces to 0=ECy(r−⊤y+γ⊤ψ¯)+y[y⊤(e−+E−)+ψ¯⊤(e++E+)+req⊤+⊤ΣEA], 0=E_Cy (r-w y+ ψ )+y [y (e_-+E_-w)+ ψ (e_++E_+w)+r\,e_q w+w E_Aw ], for every x,y,∈ℝdx,y,w ^d and r∈ℝr . The coefficient of r is [EC+(eq⊤)Id]y[E_C+(e_q w)I_d]y. Setting =0w=0 and varying y gives EC=0E_C=0. The remaining identity (eq⊤)y=0(e_q w)y=0 for every ,yw,y then gives eq=0e_q=0. After substituting EC=eq=0E_C=e_q=0, the scalar multiplying y must vanish for every y≠0y≠ 0. Its dependence on y is affine, so its linear coefficient satisfies e−+E−=0e_-+E_-w=0 for every w. Setting =0w=0 gives e−=0e_-=0, and varying w then yields E−=0E_-=0. The remaining identity is ψ¯⊤(e++E+)+⊤ΣEA=0. ψ (e_++E_+w)+w E_Aw=0. (E.1) For the candidate feature set (x,0,…,0)(x,0,…,0), define p=exp(⊤x/τ)exp(⊤x/τ)+m−1p= (w x/τ) (w x/τ)+m-1. Then ψ¯=px ψ=px and Σ=p(1−p)xx⊤ =p(1-p)x . Since m⩾2m 2, we have 0<p<10<p<1. Dividing (E.1) by p gives x⊤(e++E+)+(1−p)(⊤x)(x⊤EA)=0.x (e_++E_+w)+(1-p)(w x)(x E_Aw)=0. (E.2) Set x=tvx=tv with t≠0t≠ 0 and divide (E.2) by t. The second term is O(t)O(t) as t→0t→ 0, while the first term becomes v⊤(e++E+)v (e_++E_+w). Letting t→0t→ 0 gives v⊤(e++E+)=0v (e_++E_+w)=0 for every v,v,w. Hence e++E+=0e_++E_+w=0 for every w, and therefore e+=0e_+=0 and E+=0E_+=0. Equation (E.2) now reduces to (⊤x)(x⊤EA)=0for every x,.(w x)(x E_Aw)=0 every x,w. (E.3) Suppose, for contradiction, that EA≠0E_Aw≠ 0 for some w. Then ≠0w≠ 0, and the two proper hyperplanes x:⊤x=0\x:w x=0\ and x:x⊤EA=0\x:x E_Aw=0\ cannot cover ℝdR^d. Hence there exists x for which both factors in (E.3) are nonzero, a contradiction. Therefore EA=0E_Aw=0 for every w, and thus EA=0E_A=0. We have shown that all seven coefficient blocks vanish: EC=eq=e−=E−=e+=E+=EA=0E_C=e_q=e_-=E_-=e_+=E_+=E_A=0. Hence ζ=0ζ=0. ∎ Corollary E.2 (Verification under the experimental generator). Let Ns⩾2N_s 2 and m⩾2m 2. Suppose that P0P_0 and all transition rows are mutually independent Dirichlet(1,…,1)Dirichlet(1,…,1) random vectors, independent of (,Φ,R)(w, ,R), whose scalar entries are mutually independent Unif(−1,1)Unif(-1,1) random variables. Let the behavior policy be ϵb _b-greedy with uniform exploration for some 0<ϵb⩽10< _b 1. Then the hypotheses of Theorem E.1 are satisfied. Consequently, for every fixed n⩾1n 1 and τ>0τ>0, Assumption 1 holds. Proof. We verify the two conditions of Theorem E.1. First, collect all scalar coordinates of w, Φ , and R into a finite-dimensional random vector X=(,vec(Φ),vec(R))∈ℝqX=(w,vec( ),vec(R)) ^q. By assumption, the coordinates of X are mutually independent Unif(−1,1)Unif(-1,1) random variables. Hence X has density fX(x)=2−q(−1,1)q(x)f_X(x)=2^-q1_(-1,1)^q(x) with respect to q-dimensional Lebesgue measure. In particular, the law of X is absolutely continuous, with a nonzero density on the open set (−1,1)q(-1,1)^q. Thus condition (i) of Theorem E.1 holds. We next verify condition (i). A Dirichlet(1,…,1)Dirichlet(1,…,1) random vector is uniformly distributed on the probability simplex and lies in its relative interior almost surely. Therefore every coordinate is strictly positive almost surely. In particular, for any two distinct states s0,s1s_0,s_1, P0(s0)>0P_0(s_0)>0 a.s. and, for every action a, P(s1∣s0,a)>0P(s_1 s_0,a)>0, P(s1∣s1,a)>0P(s_1 s_1,a)>0 a.s. It remains to check the behavior-policy probabilities. Because the behavior policy is ϵb _b-greedy with uniform exploration, every action receives at least its exploration probability: πb(a∣s)⩾ϵbm>0 _b(a s) _bm>0 for every (s,a)(s,a), independently of which action is greedy and of how ties are broken. Hence, when n>1n>1, choose any distinct states s0,s1s_0,s_1 and any distinct actions ax,aza_x,a_z. Then, almost surely, P0(s0)>0P_0(s_0)>0, P(s1∣s0,ax)>0P(s_1 s_0,a_x)>0, P(s1∣s1,az)>0P(s_1 s_1,a_z)>0, and πb(ax∣s0)>0 _b(a_x s_0)>0, πb(az∣s1)>0 _b(a_z s_1)>0. Thus condition (i) holds. For n=1n=1, only the first transition is needed, and the same argument gives P0(s0)>0P_0(s_0)>0, P(s1∣s0,ax)>0P(s_1 s_0,a_x)>0, πb(ax∣s0)>0 _b(a_x s_0)>0 a.s. Therefore all hypotheses of Theorem E.1 are satisfied. Applying that theorem yields Assumption 1 for every fixed n⩾1n 1 and τ>0τ>0. ∎ E.2 Effect of Temperature on Convergence Rate In this section, we establish the lower bound on κ(τ)κ(τ) and then construct a matching normal direction for the upper bound. By the normal-curvature comparison established in Lemma D.2, it is sufficient for the lower bound to control the smallest eigenvalue χτ,n:=λmin(τ,n) _τ,n:= _ ( H_τ,n) of the coefficient-space Gram matrix. Equivalently, we need to prove a uniform lower bound of n|ℬτ,n(z)[ζ]|22E_D_n |B_τ,n(z)[ζ] |_2^2 for every coefficient direction ζ. Lemma E.3 further reduces this task to the reference law UnU_n on the calibration path. Hence it suffices to establish the corresponding coercivity bound for Un|ℬτ,n(zωn)[ζ]|22E_U_n|B_τ,n(z_ _n)[ζ]|_2^2. The only coefficient block whose response degenerates is the attention-score block in the first layer of Transformer. We therefore first isolate the successor-value and attention contribution Kτ=Ψ¯τ⊤(e++E+W)+W⊤ΣτEAWK_τ= _τ (e_++E_+W)+W _τE_AW and derive a quantitative lower bound for UnKτ2E_U_nK_τ^2, which shows in Lemma E.4. We then combine this estimate with orthogonal projections controlling the remaining coefficient blocks (EC,eq,e−,E−)(E_C,e_q,e_-,E_-), which yields the lower bound of κ(τ)κ(τ). Lemma E.3 (Calibration-path reduction). Under the experimental generator, let ωn _n be the calibration path used in Theorem E.1, and let UnU_n denote the product-uniform law of the continuous coordinates appearing along this path. Then, for every coefficient direction ζ, n‖ℬτ,n()[ζ]‖22⩾pnUn‖ℬτ,n(ωn)[ζ]‖22,E_D_n \|B_τ,n(z)[ζ] \|_2^2 p_n\,E_U_n \|B_τ,n(z_ _n)[ζ] \|_2^2, where pn=pminnNs2Γ(Ns)Γ(n)Γ(Ns+n−1)>0,pmin:=ϵbm.p_n= p_ ^nN_s^2 (N_s) (n) (N_s+n-1)>0, p_ := _bm. Proof. We use the same calibration path as in Theorem E.1, s0→axs1s_0 a_xs_1, s1→azs1s_1 a_zs_1 for the remaining n−1n-1 transitions, with the second part omitted when n=1n=1. Let UnU_n denote the product-uniform law of the continuous coordinates appearing along this path, and let ωnz_ _n be the corresponding context. Because the behavior policy selects every action with probability at least pminp_ , the conditional probability of following the prescribed action sequence is at least pminnp_ ^n. The remaining state-transition contribution is Qn=P0(s0)P(s1∣s0,ax)P(s1∣s1,az)n−1,Q_n=P_0(s_0)P(s_1 s_0,a_x)P(s_1 s_1,a_z)^n-1, where the final factor is omitted for n=1n=1. Hence the probability of the calibration path, conditional on the task, is bounded below by pminnQnp_ ^nQ_n. Since the initial-state probability P0(s0)P_0(s_0) and the transition probability P(s1∣s0,ax)P(s_1 s_0,a_x) each have mean 1/Ns1/N_s. The repeated self-transition probability P(s1∣s1,az)P(s_1 s_1,a_z) has the Beta(1,Ns−1)Beta(1,N_s-1) distribution, so Qn=P0(s0)P(s1∣s0,ax)P(s1∣s1,az)n−1=1Ns2Γ(Ns)Γ(n)Γ(Ns+n−1).EQ_n=EP_0(s_0)EP(s_1 s_0,a_x)EP(s_1 s_1,a_z)^n-1= 1N_s^2 (N_s) (n) (N_s+n-1). Under the experimental generator, the environment variables in QnQ_n are independent of the selected continuous coordinates (,Φ,R)(w, ,R). Therefore, for every nonnegative measurable function f, nf() _D_nf(Z) ⩾n[f()Tn=ωn]=[f(ωn(Xn))ℙ(Tn=ωn∣Ξ)] _D_n [f(Z)1_\T_n= _n\ ]=E [f(z_ _n(X_n))P(T_n= _n ) ] ⩾pminn[f(ωn(Xn))Qn]=pminn[f(ωn(Xn))][Qn]⩾pnUnf(ωn). p_ ^nE [f(z_ _n(X_n))Q_n ]=p_ ^nE [f(z_ _n(X_n)) ]E[Q_n] p_nE_U_nf(z_ _n). where pn=pminnNs2Γ(Ns)Γ(n)Γ(Ns+n−1)>0p_n= p_ ^nN_s^2 (N_s) (n) (N_s+n-1)>0. Let f()=‖ℬτ,n()[ζ]‖2f(z)=\|B_τ,n(z)[ζ]\|^2, we have the desired result. ∎ Let ζ=(EC,eq,e−,E−,e+,E+,EA)ζ=(E_C,e_q,e-,E_-,e_+,E_+,E_A) be the observable coefficient direction induced by U. We decompose the attention coefficient EAE_A as D:=EA=D0+aIdD:=E_A=D_0+aI_d, a:=tr(D)da:= tr(D)d, tr(D0)=0tr(D_0)=0. We first isolate the part of the response associated with the successor-feature and attention coefficients. Under the reference law UnU_n, let W denote the weight vector and Ψ1,…,Ψm _1,…, _m the successor features. Their contribution is summarized by Kτ:=Ψ¯τ⊤(e++E+W)+W⊤ΣτDWK_τ:= τ (e^++E_+W)+W _τDW. Lemma E.4 shows that the temperature dependence of KτK_τ determines the weakest curvature contributed by the attention block and, in particular, reveals the τ5τ^5 scaling of the scalar attention direction. Lemma E.4. Assume d⩾2d 2. Under the reference law UnU_n, there exist constants batt>0b_ att>0 and τ∗>0 _*>0, independent of τ and of the coefficient direction, such that for every 0<τ⩽τ∗0<τ _*, UnKτ2⩾batt(‖e+‖2+‖E+‖F2+τ3‖D0‖F2+τ5da2). _U_nK_τ^2 b_ att (\|e_+\|^2+\|E_+\|_F^2+τ^3\|D_0\|_F^2+τ^5da^2 ). (E.4) Proof. (i) We first lower-bound [(W⊤ΣτDW)2]E[(W _τDW)^2]. The difficulty is that D0D_0 may point in an arbitrary trace-free matrix direction. We first construct a finite collection of measurements u⊤D0vu D_0v that controls the entire matrix D0D_0. Define ℱ=(ei,ej):i≠j∪(ei−ej2,ei+ej2),(ei+ej2,ei−ej2):i<j. =\(e_i,e_j):i≠ j\∪ \ ( e_i-e_j 2, e_i+e_j 2 ), ( e_i+e_j 2, e_i-e_j 2 ):i<j \. The pairs (ei,ej)(e_i,e_j) directly measure the off-diagonal entries of D0D_0, while the rotated pairs also measure differences between its diagonal entries. Since tr(D0)=0tr(D_0)=0, we have ∑(u,v)∈ℱ(u⊤D0v)2 _(u,v) (u D_0v )^2 =∑i≠j(D0)ij2+∑i<j12((D0)ii−(D0)jj)2+12((D0)ij−(D0)ji)2 = _i≠ j(D_0)_ij^2+ _i<j 12 ((D_0)_i-(D_0)_j )^2+ 12 ((D_0)_ij-(D_0)_ji )^2 =∑i≠j(D0)ij2+d2∑i(D0)ii2+12∑i<j((D0)ij−(D0)ji)2 = _i≠ j(D_0)_ij^2+ d2 _i(D_0)_i^2+ 12 _i<j ((D_0)_ij-(D_0)_ji )^2 ⩾∑i≠j(D0)ij2+∑i(D0)ii2=‖D0‖F2. _i≠ j(D_0)_ij^2+ _i(D_0)_i^2=\|D_0\|_F^2. (E.5) Hence the finite family ℱF detects every nonzero trace-free direction. For each (u,v)∈ℱ(u,v) , we construct a local configuration in which the attention response contains the measurement u⊤D0vu D_0v. Choose fixed sufficiently small constants s,t,b>0s,t,b>0 and set W=svW=sv, V:=Ψ1−Ψ2=tuV:= _1- _2=tu, and M:=(Ψ1+Ψ2)/2=bvM:=( _1+ _2)/2=bv, with Ψa=−bv _a=-bv for a⩾3a 3. Equivalently, Ψ1=bv+(t/2)u _1=bv+(t/2)u and Ψ2=bv−(t/2)u _2=bv-(t/2)u. Since u⊤v=0u v=0, actions 11 and 22 have the same score, W⊤Ψ1=W⊤Ψ2=sbW _1=W _2=sb, whereas every action a⩾3a 3 has score −sb-sb. Thus actions 11 and 22 are tied, while all remaining actions are separated from them by a fixed positive margin. Let g:=W⊤V=W⊤(Ψ1−Ψ2)g:=W V=W ( _1- _2) denote the score gap between actions 11 and 22. At the center, g=0g=0. In a sufficiently small neighborhood of each center, we use g, together with the remaining local coordinates υ , as coordinates. The corresponding reference density can be written as jℓ(υ,g)dυdgj_ ( ,g)\,\,d \,\,dg, with 0<j−⩽jℓ(υ,g)⩽j+<∞0<j_- j_ ( ,g) j_+<∞. Since only finitely many charts are used, these constants can be chosen uniformly. We restrict each chart to the near-tie region τ⩽|g|⩽2τ |g| 2τ. In this region, actions 11 and 22 both receive nonnegligible softmax mass, whereas all other actions remain separated by a fixed score gap and contribute only exponentially small terms. Using the pairwise representation of Στ _τ, we obtain W⊤ΣτDW=π1τπ2τgV⊤DW+Rτ,|Rτ|⩽Ce−g0/τ∥D∥F. W _τDW= _1^τ _2^τ\,g\,V DW+R_τ,|R_τ| Ce^-g_0/τ\|D\|_F. (E.6) Since D=D0+aIdD=D_0+aI_d and V⊤W=gV W=g, gV⊤DW=gV⊤D0W+ag2. gV DW=gV D_0W+ag^2. (E.7) The trace-free component therefore contains one factor of the small gap g, whereas the scalar component contains two. To make the dependence on τ explicit, set g=τxg=τ x and ae:=τa_e:=τ a. Then gV⊤D0W+ag2=τ(xV⊤D0W+aex2). gV D_0W+ag^2=τ (xV D_0W+a_ex^2 ). (E.8) Squaring contributes a factor τ2τ^2, while dg=τdx\,dg=τ\,\,dx contributes one additional factor τ. Hence the near-tie contribution has an overall factor τ3τ^3. After removing this common factor, the corresponding quadratic forms converge to G0(D0,ae) G_0(D_0,a_e) =∑ℓ∫Zℓjℓ(υ,0)∫1⩽|x|⩽2k(x)(xfℓ(υ,D0)+aex2)2xυ, = _ _Z_ j_ ( ,0) _1 |x| 2k(x) (xf_ ( ;D_0)+a_ex^2 )^2\,\,dx\,\,d , (E.9) where k(x):=e2x/(1+ex)4>0k(x):=e^2x/(1+e^x)^4>0 and fℓ(υ,D0):=Vℓ(υ,0)⊤D0Wℓ(υ,0)f_ ( ;D_0):=V_ ( ,0) D_0W_ ( ,0). The role of G0G_0 is to rule out cancellation between the trace-free part D0D_0 and the scalar part aIdaI_d. Suppose G0(D0,ae)=0G_0(D_0,a_e)=0. Since the integrand is nonnegative and the weights are strictly positive, we must have xfℓ(υ,D0)+aex2=0xf_ ( ;D_0)+a_ex^2=0 throughout each chart. Evaluating this identity at x and −x-x, with 1<x<21<x<2, gives xfℓ+aex2=0,−xfℓ+aex2=0xf_ +a_ex^2=0,-xf_ +a_ex^2=0. Hence ae=0a_e=0 and fℓ=0f_ =0. At the center of the ℓ -th chart, Vℓ=tuℓV_ =tu_ and Wℓ=svℓW_ =sv_ , and therefore fℓ(0,D0)=stuℓ⊤D0vℓ=0f_ (0;D_0)=st\,u_ D_0v_ =0. Since this holds for every (uℓ,vℓ)∈ℱ(u_ ,v_ ) , (E.5) implies D0=0D_0=0. Thus G0G_0 is positive definite. Because the coefficient space is finite dimensional, compactness of its unit sphere gives a constant λ0>0 _0>0 such that G0(D0,ae) G_0(D_0,a_e) ⩾λ0(‖D0‖F2+dae2). _0 (\|D_0\|_F^2+da_e^2 ). (E.10) Moreover, after the rescaling g=τxg=τ x and a=ae/τa=a_e/τ, the contribution of the remaining actions to the rescaled response is uniformly O(τ−2e−g0/τ)=o(1)O(τ^-2e^-g_0/τ)=o(1) on the coefficient unit sphere tr(D0)=0tr(D_0)=0, ‖D0‖F2+dae2=1\|D_0\|_F^2+da_e^2=1. The coordinate maps and densities also converge uniformly on the fixed chart domains, chosen with compact closures. Thus GτG_τ converges uniformly to G0G_0 on this unit sphere, equivalently in operator norm for the associated Gram operators. Hence, for all sufficiently small τ>0τ>0, Gτ(D0,ae):=τ−3∑ℓ∫Cℓ,τ[W⊤Στ(D0+aeτId)W]2P⩾λ02(‖D0‖F2+dae2). G_τ(D_0,a_e):=τ^-3 _ _C_ ,τ [W ^τ (D_0+ a_eτI_d )W ]^2\,dP _02 (\|D_0\|_F^2+da_e^2 ). (E.11) Let ℓ,τC_ ,τ denote the near-tie region in the ℓ -th chart and define ℰtie(τ):=⋃ℓ,τE_ tie(τ):= _ C_ ,τ. Since there are only L:=|ℱ|L:=|F| charts, their overlap multiplicity is at most L. Therefore, for every nonnegative function h, ∑ℓ∫ℓ,τhP _ _C_ ,τh\,dP ⩽L[h 1ℰtie(τ)]. L\,E [h\,1_E_ tie(τ) ]. (E.12) Combining this finite-overlap bound with (E.11), and recalling that ae=τa_e=τ a, gives [(W⊤ΣτDW)2ℰtie(τ)] [ (W _τDW )^21_E_ tie(τ) ] ⩾b3τ3∥D0∥F2+b5τ5da2−Ce−g0/τ∥D∥F2. b_3τ^3\|D_0\|_F^2+b_5τ^5da^2-Ce^-g_0/τ\|D\|_F^2. (E.13) Thus the near-tie regions provide order-τ3τ^3 control of the trace-free component D0D_0 and the weaker order-τ5τ^5 control of the scalar component aIdaI_d. (i) We next control cancellation with E+E_+. All expectations below are under the reference law UnU_n. Write Xτ:=Ψ¯τE+WX_τ:= _τ TE_+W, Yτ:=WΣτDWY_τ:=W T _τDW, Zτ:=Ψ¯τe+Z_τ:= _τ Te_+, where D=D0+aIdD=D_0+aI_d and tr(D0)=0tr(D_0)=0. In particular, Kτ=Zτ+Xτ+YτK_τ=Z_τ+X_τ+Y_τ, ∥D∥F 2=∥D0∥F 2+da2 D _F^\,2= D_0 _F^\,2+da^2. We choose all constants independently of τ and the coefficient direction. The parameters d and m are fixed. Choose fixed, sufficiently small positive numbers r∗,a∗,b∗r_*,a_*,b_*, and consider the following d2d^2 configurations in the interior of the product-uniform support: W=r∗ej,Ψ1=b∗ej,Ψa=−b∗ej(a⩾2),j=1,…,d; W=r_*e_j, _1=b_*e_j, _a=-b_*e_j (a 2), j=1,…,d; W=r∗ej,Ψ1=a∗ei+b∗ej,Ψa=−b∗ej(a⩾2),i≠j. W=r_*e_j, _1=a_*e_i+b_*e_j, _a=-b_*e_j (a 2), i≠ j. At each configuration, action 1 has a fixed positive winning gap. Take sufficiently small, fixed open neighborhoods of these configurations whose closures lie inside the support, and denote their union by ℰsepE_sep. On this union, every gap W(Ψ1−Ψa)W T( _1- _a), a⩾2a 2, is bounded below by a common constant gsep>0g_sep>0. For X0(E):=Ψ1EWX_0(E):= _1 TEW, the center measurements are b∗r∗Ejjb_*r_*E_j and a∗r∗Eij+b∗r∗Ejja_*r_*E_ij+b_*r_*E_j. They jointly determine every entry of E. Hence the quadratic form [X0(E)2ℰsep]E[X_0(E)^21_E_sep] is positive definite: if it vanished, positivity of the reference density and continuity would force X0(E)X_0(E) to vanish on each neighborhood, including at every center, and therefore E=0E=0. The diagonal centers similarly show that [(Ψ1e)2ℰsep]E[( _1 Te)^21_E_sep] is positive definite in e. Compactness of the respective unit spheres therefore gives constants λE,λe>0 _E, _e>0 such that [X0(E)2ℰsep]⩾λE∥E∥F 2,[(Ψ1e)2ℰsep]⩾λe‖e‖2. [X_0(E)^21_E_sep] _E E _F^\,2, [( _1 Te)^21_E_sep] _e e ^2. (E.14) All features and W are bounded. The uniform winning gap gives ‖Ψ¯τ−Ψ1‖+‖Στ‖op⩽Ce−gsep/τon ℰsep. _τ- _1 + _τ _op Ce^-g_sep/τ E_sep. (E.15) For example, the total softmax mass outside action 1 is at most (m−1)e−gsep/τ(m-1)e^-g_sep/τ, which bounds both quantities in (E.15) up to fixed constants. The reverse triangle inequality in the restricted L2L^2 norm, together with (E.14), implies that, for all sufficiently small τ, [Xτ2ℰsep]⩾λE4∥E+∥F 2,[Zτ2ℰsep]⩾λe4‖e+‖2. [X_τ^21_E_sep] _E4 E_+ _F^\,2, [Z_τ^21_E_sep] _e4 e_+ ^2. (E.16) These estimates hold simultaneously for all coefficient directions, because the L2L^2 approximation errors are bounded by Ce−gsep/τ∥E+∥FCe^-g_sep/τ E_+ _F and Ce−gsep/τ‖e+‖Ce^-g_sep/τ e_+ , respectively. Also, (E.15) yields [Yτ2ℰsep]⩽Csepe−2gsep/τ∥D∥F 2E[Y_τ^21_E_sep] C_sepe^-2g_sep/τ D _F^\,2. The elementary inequality (u+v)2⩾12u2−v2(u+v)^2 12u^2-v^2 for u,v∈ℝu,v follows by completing the square. Applying it with u=Xτu=X_τ and v=Yτv=Y_τ, and weakening the exponential remainder if necessary, gives [(Xτ+Yτ)2ℰsep]⩾bE∥E+∥F 2−Csepe−gsep/τ∥D∥F 2,bE:=λE8>0.E[(X_τ+Y_τ)^21_E_sep] b_E E_+ _F^\,2-C_sepe^-g_sep/τ D _F^\,2, b_E:= _E8>0. (E.17) Let ℰtie(τ)E_tie(τ) be the union of the near-tie charts from part (i). Their bounded densities, fixed bounded coordinate domains, and gap restriction τ⩽|g|⩽2τ |g| 2τ imply ℙ(ℰtie(τ))⩽CtτP(E_tie(τ)) C_tτ. Indeed, each chart contributes at most its upper density bound times its coordinate-domain volume times the gap-interval length 2τ2τ; summing over the finitely many charts gives this bound regardless of overlap. Since ‖Ψ¯τ‖,‖W‖⩽d _τ , W d, [Xτ2ℰtie(τ)]⩽CXτ∥E+∥F 2,CX:=d2Ct.E[X_τ^21_E_tie(τ)] C_Xτ E_+ _F^\,2, C_X:=d^2C_t. (E.18) Rewrite (E.13), retaining its coefficient-independent constants, as [Yτ2ℰtie(τ)]⩾b3τ3∥D0∥F 2+b5τ5da2−Ctiee−g0/τ∥D∥F 2.E[Y_τ^21_E_tie(τ)] b_3τ^3 D_0 _F^\,2+b_5τ^5da^2-C_tiee^-g_0/τ D _F^\,2. (E.19) Apply (u+v)2⩾12u2−v2(u+v)^2 12u^2-v^2 with u=Yτu=Y_τ and v=Xτv=X_τ. Equations (E.18) and (E.19) give [(Xτ+Yτ)2ℰtie(τ)]⩾b32τ3∥D0∥F 2+b52τ5da2−CXτ∥E+∥F 2−Ctie2e−g0/τ∥D∥F 2. [(X_τ+Y_τ)^21_E_tie(τ)] b_32τ^3 D_0 _F^\,2+ b_52τ^5da^2-C_Xτ E_+ _F^\,2- C_tie2e^-g_0/τ D _F^\,2. (E.20) No disjointness of the two regions is needed: their indicators sum to at most two. Consequently, [(Xτ+Yτ)2]⩾12([(Xτ+Yτ)2ℰsep]+[(Xτ+Yτ)2ℰtie(τ)]). [(X_τ+Y_τ)^2] 12 (E[(X_τ+Y_τ)^21_E_sep]+E[(X_τ+Y_τ)^21_E_tie(τ)] ). Set c=ming0,gsep>0c= \g_0,g_sep\>0 and C0=Csep/2+Ctie/4C_0=C_sep/2+C_tie/4. Combining the two regional lower bounds yields [(Xτ+Yτ)2]⩾(bE2−CXτ2)∥E+∥F 2+b34τ3∥D0∥F 2+b54τ5da2−C0e−c/τ(∥D0∥F 2+da2). [(X_τ+Y_τ)^2] ( b_E2- C_Xτ2 ) E_+ _F^\,2+ b_34τ^3 D_0 _F^\,2+ b_54τ^5da^2-C_0e^-c/τ ( D_0 _F^\,2+da^2 ). (E.21) Choose τ∗>0 _*>0 smaller than the thresholds used above and in part (i), so that for every 0<τ⩽τ∗0<τ _*, τ⩽1,CXτ⩽bE2,C0e−c/τ⩽minb3,b58τ5.τ 1, C_Xτ b_E2, C_0e^-c/τ \b_3,b_5\8τ^5. (E.22) Such a choice exists because e−c/τ/τ5→0e^-c/τ/τ^5→ 0 as τ↓0τ 0. Every quantity used to choose this threshold is independent of e+e_+, E+E_+, and D. Using τ5⩽τ3τ^5 τ^3 and (E.22) in (E.21) gives [(Xτ+Yτ)2]⩾bE4∥E+∥F 2+b38τ3∥D0∥F 2+b58τ5da2. [(X_τ+Y_τ)^2] b_E4 E_+ _F^\,2+ b_38τ^3 D_0 _F^\,2+ b_58τ^5da^2. (E.23) The product-uniform law of (W,Ψ1,…,Ψm)(W, _1,…, _m) is invariant under simultaneous sign reversal. This transformation preserves all softmax logits and Στ _τ, reverses Ψ¯τ _τ, and makes ZτZ_τ odd while Xτ+YτX_τ+Y_τ is even. Therefore [Zτ(Xτ+Yτ)]=0E[Z_τ(X_τ+Y_τ)]=0, [Kτ2]=[Zτ2]+[(Xτ+Yτ)2]E[K_τ^2]=E[Z_τ^2]+E[(X_τ+Y_τ)^2]. The second estimate in (E.16) implies [Zτ2]⩾(λe/4)‖e+‖2E[Z_τ^2] ( _e/4) e_+ ^2. Combining this with (E.23) proves [Kτ2]⩾batt(‖e+‖2+∥E+∥F 2+τ3∥D0∥F 2+τ5da2),E[K_τ^2] b_att ( e_+ ^2+ E_+ _F^\,2+τ^3 D_0 _F^\,2+τ^5da^2 ), (E.24) where one may take batt=minλe4,bE4,b38,b58>0b_att= \ _e4, b_E4, b_38, b_58 \>0. Both battb_att and τ∗ _* are independent of the coefficient direction. They can also be chosen independently of n, since all quantities in this argument, including the near-tie construction in part (i), depend only on the same product-uniform marginal of (W,Ψ1,…,Ψm)(W, _1,…, _m) under every UnU_n. ∎ The proof of Corollary 3. (i) Since ℬτ,n()[ζ]=0B_τ,n(z)[ζ]=0, n-a.s.⟹ζ=0D_n-a.s. ζ=0 has already been established in Corollary E.2; This step is quantitative only. Lemma E.4 analyzed e+,E+,EAe_+,E_+,E_A, we now incorporate EC,eq,e−,E−E_C,e_q,e_-,E_-. On the calibration path, let Y:=Φ(s0,ax)Y:= (s_0,a_x), R0:=R(ax,s1)R_0:=R(a_x,s_1), ZY:=YY⊤−13IdZ_Y:=Y - 13I_d, and vτ:=W⊤Ψ¯τv_τ:=W _τ. Conditional on all variables other than Y and R0R_0, the observable response can be written as nℬτ,n(zωn)[ζ]=R0Ar(Y)+Ao(Y)+A2(Y)+Aconst,nB_τ,n(z_ _n)[ζ]=R_0A_r(Y)+A_o(Y)+A_2(Y)+A_ const, (E.25) where Ar(Y)=[EC+(eq⊤W)Id]YA_r(Y)=[E_C+(e_q W)I_d]Y, Ao(Y)=γvτECY+KτYA_o(Y)=γ v_τE_CY+K_τY, A2(Y)=−ECZYW+ZY(e−+E−W)A_2(Y)=-E_CZ_YW+Z_Y(e_-+E_-W), and Aconst=13(−ECW+e−+E−W)+AlaterA_ const= 13(-E_CW+e_-+E_-W)+A_ later. Here AlaterA_ later is the sum of the contributions from transitions 1,…,n−11,…,n-1, and is zero when n=1n=1. The calibration path has s0≠s1s_0≠ s_1, and all later transitions use features associated with s1s_1. Hence, under the product reference law, AlaterA_ later is independent of Y=Φ(s0,ax)Y= (s_0,a_x) and of R0R_0. Thus AconstA_ const is constant with respect to the conditional variables (Y,R0)(Y,R_0). The four terms in (E.25) are mutually orthogonal in L2(Y,R0)L^2(Y,R_0). Indeed, R0=0ER_0=0 and R0R_0 independent on Y; AoA_o is odd in Y, i.e., Ao(−Y)=−Ao(Y)A_o(-Y)=-A_o(Y); Since Z−Y=ZYZ_-Y=Z_Y, A2A_2 is even in Y and centered because ZY=0EZ_Y=0; and AconstA_ const is constant in (Y,R0)(Y,R_0). Odd and even functions are orthogonal under symmetric distributions. In particular, later transitions cannot cancel any of the first three components. The uniform moments satisfy [YY⊤]=[WW⊤]=13IdE[Y ]=E[W ]= 13I_d, R02=13ER_0^2= 13, and ZY2=νIdEZ_Y^2=ν I_d, ν:=5d−145>0ν:= 5d-145>0. For the reward projection, ‖R0Ar(Y)‖2=19‖EC+(eq⊤W)Id‖F2=19‖EC‖F2+d27‖eq‖2, \|R_0A_r(Y)\|^2= 19E \|E_C+(e_q W)I_d \|_F^2= 19\|E_C\|_F^2+ d27\|e_q\|^2, (E.26) where the cross term vanishes because W=0EW=0. For 0<δ<10<δ<1, we use ‖x+y‖2⩾δ‖y‖2−δ1−δ‖x‖2\|x+y\|^2 δ\|y\|^2- δ1-δ\|x\|^2. Since |vτ|⩽d|v_τ| d, ‖Ao(Y)‖2⩾δod3Kτ2−δo1−δoγ2d23‖EC‖F2.E\|A_o(Y)\|^2 _od3\,EK_τ^2- _o1- _o γ^2d^23\|E_C\|_F^2. (E.27) Moreover, ZYe−Z_Ye_- is orthogonal to the terms linear in the centered variable W. Using ‖ZYE−W‖2=ν3‖E−‖F2E\|Z_YE_-W\|^2= ν3\|E_-\|_F^2, ‖ECZYW‖2=ν3‖EC‖F2E\|E_CZ_YW\|^2= ν3\|E_C\|_F^2, we obtain ‖A2(Y)‖2⩾ν‖e−‖2+δ2ν3‖E−‖F2−δ21−δ2ν3‖EC‖F2.E\|A_2(Y)\|^2 ν\|e_-\|^2+ _2ν3\|E_-\|_F^2- _21- _2 ν3\|E_C\|_F^2. (E.28) Choose δo=min12,148γ2d2 _o= \ 12, 148γ^2d^2 \, δ2=min12,148ν _2= \ 12, 148ν \. Since δ/(1−δ)⩽2δ/(1-δ) 2δ for δ⩽1/2δ 1/2, each negative coefficient multiplying ‖EC‖F2\|E_C\|_F^2 in (E.27) and (E.28) is at most 1/721/72. Hence the coefficient 1/91/9 in (E.26) retains at least 112 112. Combining the orthogonal decomposition, (E.4), (E.26), (E.27), and (E.28), there exists bref>0b_ ref>0, independent of n,τn,τ, and ζ, such that for all sufficiently small 0<τ⩽10<τ 1, Un‖ℬτ,n(ωn)[ζ]‖22⩾brefn2(CLOSE _U_n \|B_τ,n(z_ _n)[ζ] \|_2^2 b_ refn^2 ( ‖EC‖F2+‖eq‖2+‖e−‖2+‖E−‖F2 \|E_C\|_F^2+\|e_q\|^2+\|e_-\|^2+\|E_-\|_F^2 OPEN+‖e+‖2+‖E+‖F2+τ3‖D0‖F2+τ5da2). +\|e_+\|^2+\|E_+\|_F^2+τ^3\|D_0\|_F^2+τ^5d\,a^2 ). (E.29) Since ‖D‖F2=‖D0‖F2+da2\|D\|_F^2=\|D_0\|_F^2+da^2 and 0<τ⩽10<τ 1, the right-hand side is bounded below by brefn2τ5‖ζ‖2 b_ refn^2τ^5\|ζ\|_V^2. Using Lemma E.3, n‖ℬτ,n()[ζ]‖22⩾pnbrefn2τ5‖ζ‖2.E_D_n \|B_τ,n(z)[ζ] \|_2^2 p_nb_ refn^2τ^5\|ζ\|_V^2. (E.30) Therefore χτ,n⩾pnbrefn2τ5 _τ,n p_nb_ refn^2τ^5. By Lemma D.2, κ(τ,n)⩾cΩpnbrefn2τ5κ(τ,n) c_ p_nb_ refn^2τ^5. Since N is finite, cexp:=cΩbrefminn∈pnn2>0c_ exp:=c_ b_ ref _n p_nn^2>0, and hence κ(τ)⩾cexpτ5κ(τ) c_ expτ^5, n∈n . (i) For the upper bound, it suffices to exhibit a single unit normal direction whose response is O(τ5)O(τ^5). Consider the pure scalar attention perturbation sc=(Idd,0,…,0)U_ sc=( I_d d,0,…,0). Every tangent vector to ℳτM_τ has zero A-block, so sc∈Nθ¯ℳτU_ sc∈ N_ θM_τ at every θ¯∈ℳτ θ _τ, and ‖sc‖=1\|U_ sc\|=1. For a fixed successor state, define gab=W⊤(Ψa−Ψb)g_ab=W ( _a- _b). This random variable has a uniformly bounded density. Conditional on W≠0W≠ 0, let MW:=maxj|Wj|M_W:= _j|W_j| and choose a coordinate attaining this maximum. Since the difference of two independent Unif(−1,1)Unif(-1,1) variables has density bounded by 1/21/2, convolution with the remaining independent coordinates yields ‖fgab|W‖∞⩽12MW\|f_g_ab W\|_∞ 12M_W. Furthermore, ℙ(MW⩽t)=tdP(M_W t)=t^d, MW−1=d−1EM_W^-1= dd-1, so, because d⩾2d 2, ‖fgab‖∞⩽Ld:=d2(d−1)<∞\|f_g_ab\|_∞ L_d:= d2(d-1)<∞. Set Vτ:=W⊤ΣτWV_τ:=W _τW. The pairwise covariance identity gives Vτ=∑a<bπaτπbτgab2V_τ= _a<b _a^τ _b^τg_ab^2. Moreover, πaτπbτ⩽e−|gab|/τ _a^τ _b^τ e^-|g_ab|/τ. Indeed, if the larger of the two logits is qaq_a, the softmax denominator is at least e2qa/τe^2q_a/τ after multiplying the two probabilities. Let Pm:=m(m−1)/2P_m:=m(m-1)/2. By Cauchy–Schwarz, Vτ2⩽Pm∑a<b[gab4e−2|gab|/τ]⩽2LdPm2∫0∞x4e−2x/τdx=CVτ5, _τ^2 P_m _a<bE [g_ab^4e^-2|g_ab|/τ ] 2L_dP_m^2 _0^∞x^4e^-2x/τ\,\,dx=C_Vτ^5, (E.31) where CV<∞C_V<∞ depends only on d and m. Direct differentiation in the direction scU_ sc gives Jθ¯()[sc]=αγnd∑i=0n−1ϕiVτ,iJ_ θ(z)[U_ sc]= αγn d _i=0^n-1 _iV_τ,i. Using convexity of the squared norm and ‖ϕi‖2⩽d\| _i\|^2 d, ‖Jθ¯()[sc]‖22⩽α2γ2n∑i=0n−1Vτ,i2E\|J_ θ(z)[U_ sc]\|_2^2 α^2γ^2n _i=0^n-1EV_τ,i^2. The visited successor state may depend on the task, so we do not condition on it. If Vτ(s)V_τ(s) denotes the same scalar covariance computed using the feature table at a fixed state s, then Vτ,i2=∑s∈[si+1=sVτ(s)2]⩽∑s∈Vτ(s)2⩽NsCVτ5.EV_τ,i^2= _s E [1_\s_i+1=s\V_τ(s)^2 ] _s EV_τ(s)^2 N_sC_Vτ^5. Therefore ‖Jθ¯()[sc]‖22⩽α2γ2NsCVτ5E\|J_ θ(z)[U_ sc]\|_2^2 α^2γ^2N_sC_Vτ^5. Since κ(τ)κ(τ) is the minimum squared response over unit normal directions, κ(τ,n)⩽Cexpτ5κ(τ,n) C_ expτ^5, Cexp:=α2γ2NsCVC_ exp:=α^2γ^2N_sC_V, uniformly over all n⩾1n 1. Finally, by the uniform local derivative bounds established in Lemma D.4, the same admissible step size η may be used throughout the small-temperature regime. Applying the parameter-distance complexity result of Theorem 2 and substituting κ(τ)=Θ(τ5)κ(τ)= (τ^5) yields sup0∈τKGD(ε,0,τ,n)=Θ,(1ητ5log1ε) _ θ^0 _τK_GD( ; θ^0,τ,n)= _ ,N( 1ητ^5 1 ). ∎