Paper deep dive
Attention is Just Another Name for Coupling?: A Fast-Slow ODE Perspective on Hierarchical Pretraining
Zhengyuan Gao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 6/20/2026, 8:48:06 AM
Summary
The paper proposes a hierarchical pretraining architecture for language models based on a fast-slow ordinary differential equation (ODE) perspective. It introduces a 'fast path' of standard causal attention and a 'slow path' operating on a temporally downsampled (block-mean pooled) view of the sequence. The two paths are coupled via a zero-initialized additive gate, which acts as a forward-Euler forcing term. The authors prove that under a linear-generator assumption, the equilibrium manifold of the fast-slow system corresponds to the master-equation (ME) stationary distribution. The architecture is designed to be computationally efficient (using a $P^2$ reduction in slow-path cost) and structurally guarantees causality, timescale separation, and zero-initialization identity.
Entities (8)
Relation Signals (5)
Fast Path → iscoupledwith → Slow Path
confidence 100% · a fast path of standard causal attention over T tokens, a slow path of full attention over T/P pooled tokens... and a zero-initialised additive gate.
Equilibrium Manifold → isidenticalto → Master Equation Stationary Distribution
confidence 100% · the equilibrium manifold x = ϕ(y) is exactly the master-equation (ME) stationary distribution p_st(y)
Fast-Slow ODE → describes → Fast Path
confidence 90% · The paper instantiates the fast-slow ODE formalism as a concrete neural network.
Causal Block-Mean Pooling → generates → Slow Path
confidence 90% · The slow variable y is derived from x by causal block-mean pooling with factor P.
Zero-Initialized Gate → connects → Slow Path
confidence 50% · the slow path... fed back into the fast path through a zero-initialised gate.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Causal self-attention is a coupling mechanism: each token's hidden state is updated by a learned mixture of preceding tokens at the same timescale. This paper asks whether a second, temporally slower coupling-a slow sub-system operating on a temporally-downsampled view of the sequence and fed back into the fast path through a zero-initialised gate-complements it. The question is framed in the language of singularly perturbed ordinary differential equations (ODEs), where the fast variable $x$ evolves at the token rate, the slow variable $y$ evolves at one update per $P$ tokens, and the timescale ratio $\varepsilon = 1/P$ is enforced structurally by causal block-mean pooling. The paper instantiates the fast-slow ODE formalism as a concrete neural network: a fast path of standard causal attention over $T$ tokens, a slow path of full attention over $T/P$ pooled tokens ($P^2 \times$ cheaper per layer), and a zero-initialised additive gate. In addition, under a linear-generator assumption on the fast dynamics, we prove that the equilibrium manifold $x = \phi(y)$ is exactly the master-equation (ME) stationary distribution $p_{\mathrm{st}}(y)$; in that regime a learned MLP $\phi_\theta(y)$ is a variational approximation of it (the trained block is not a generator, so this identity is the structured limit, not a claim about the network as trained). Empirically, at $500$k tokens the coupling is neutral -- the gate stays closed and the coupled and frozen ablations are within run-to-run noise -- at a wall-clock cost comparable to a dense baseline. The contribution is the precise, gap-marked mapping itself, not a performance gain.
Tags
Links
- Source: https://arxiv.org/abs/2606.16730v1
- Canonical: https://arxiv.org/abs/2606.16730v1
Trouble viewing inline? Open PDF directly →
Full Text
44,823 characters extracted from source content.
Expand or collapse full text
Attention is just another name for coupling? A Fast–Slow ODE Perspective on Hierarchical Pretraining Zhengyuan Gao Independent scholar (June 2026) Abstract Causal self-attention is a coupling mechanism: each token’s hidden state is updated by a learned mixture of preceding tokens at the same timescale. This paper asks whether a second, temporally slower coupling—a slow sub-system operating on a temporally-downsampled view of the sequence and fed back into the fast path through a zero-initialised gate—complements it. The question is framed in the language of singularly perturbed ordinary differential equations (ODEs), where the fast variable x evolves at the token rate, the slow variable y evolves at one update per P tokens, and the timescale ratio ε=1/P =1/P is enforced structurally by causal block-mean pooling. The paper instantiates the fast–slow ODE formalism as a concrete neural network: a fast path of standard causal attention over T tokens, a slow path of full attention over T/PT/P pooled tokens (P2×P^2× cheaper per layer), and a zero-initialised additive gate. In addition, under a linear-generator assumption on the fast dynamics, we prove that the equilibrium manifold x=ϕ(y)x=φ(y) is exactly the master-equation (ME) stationary distribution pst(y)p_st(y); in that regime a learned MLP ϕθ(y) _θ(y) is a variational approximation of it (the trained block is not a generator, so this identity is the structured limit, not a claim about the network as trained). Empirically, at 500500k tokens the coupling is neutral—the gate stays closed and the coupled and frozen ablations are within run-to-run noise—at a wall-clock cost comparable to a dense baseline. The contribution is the precise, gap-marked mapping itself, not a performance gain. ∈0,…,V−1Tx∈\0,…,V-1\^T token idsx←Pre(x)x← Pre(x) npren_pre fast layers, T tokens×nrounds×\,n_roundsy←1P∑j=0P−1x[kP+j]y← 1P _j=0^P-1x[kP+j] ε=1/P =1/P (HMM average)y~←Slow(y) y← Slow(y) nslown_slow layers, T/PT/P tokensyup←upsample(y~)y_up← upsample( y) causal, t/P−1t/P-1x←x+tanh(γ)⋅Norm(ϕ(yup))x← x+ (γ)· Norm(φ(y_up)) Δt=tanh(γ) t= (γ), zero-initx←Post(x)x← Post(x) npostn_post fast layers, T tokens=Head(Norm(x))p= Head( Norm(x)) output logitsFast xxSlow yyx˙=f(x,y) x=f(x,y) (Euler forcing)εy˙=g(x,y) y=g(x,y) (HMM average) Figure 1: The hierarchical fast–slow architecture. The fast variable x (blue) is updated by standard causal attention. The slow variable y (brown) is the causal block-mean (an HMM-style average of the fast trajectory), processed by attention over T/PT/P tokens and upsampled back. The injection x←x+Δt⋅ϕ(yup)x← x+ t·φ(y_up) is the slow→ forcing term of a forward-Euler step, with Δt=tanh(γ) t= (γ) initialised at zero. The function ϕφ may be a linear projection WsfW_sf or a learned nonlinear projector ϕθ _θ (Section˜11). 1 Introduction Causal self-attention [1] is a coupling mechanism: each token representation is updated by a learned mixture of all preceding representations, xt←xt+∑s<tαtsvsx_t← x_t+ _s<t _tsv_s, with αts=softmax(qt⊤ks/d) _ts=softmax(q_t k_s/ d). This coupling is dense, applied identically at every layer, and operates at the token rate. This paper asks whether attention is the only coupling we need, or whether a second, temporally slower coupling—operating on a block-averaged view of the sequence and fed back through a zero-initialised gate—complements it. The language we use is that of singularly perturbed ODEs (fast–slow systems) [2, 3]: x˙=f(x,y,t),εy˙=g(x,y,t),0<ε≪1. x=f(x,y,t), \, y=g(x,y,t), 0< 1. (1) We identify xt∈ℝDx_t ^D with the per-token hidden state and yk∈ℝDy_k ^D with a block average of P tokens, giving ε=1/P =1/P. The paper provides a precise mapping between LLM components and ODE variables, listing the ODE properties the architecture formally instantiates (Table˜1). It then gives the equilibrium manifold a concrete identity: under a linear generator-structure assumption on the fast dynamics, the quasi-steady-state (QSS) equilibrium x=ϕ(y)x=φ(y) is exactly the master equation’s (ME) stationary distribution pst(y)p_st(y) (Theorem˜28), and a forward-Euler iteration recovers it to numerical precision. In that regime a learned MLP ϕθ(y) _θ(y) is a variational approximation of pst(y)p_st(y); we are explicit that the trained block is not a generator, so this is the structured limit rather than a property of the network as trained (Remark˜30). The paper is organised as follows. §2 establishes the fast–slow ODE formalism. §3 instantiates it as a concrete neural network. §4 proves causality and timescale separation by construction. §5 clarifies the relationship between three uses of “manifold” in prior work. §6 and §7 state and prove that, under a linear-generator assumption, the QSS equilibrium manifold equals the ME stationary distribution. §8 situates attention within this identity. §9 reports the empirical results, §11 describes the four QSS approximation methods, and §12 lists the open questions. 2 Fast–Slow ODEs and the Multirate Coupling 2.1 The singularly perturbed system and QSS reduction In (1) the fast variable x evolves on timescale τf _f and the slow variable y on τs=ετf≫τf _s= _f _f. When ε is small, y appears frozen on the timescale of x. The standard analytical technique is the quasi-steady-state (QSS) reduction [2]: 1. Hold y fixed. Solve f(x,y)=0f(x,y)=0 for the equilibrium x=ϕ(y)x=φ(y). 2. Substitute into the slow equation: εy˙=g(ϕ(y),y,t) \, y=g(φ(y),y,t). This decomposes the full dynamics into fast equilibration toward the manifold ℳ=(x,y):f(x,y)=0M=\(x,y):f(x,y)=0\, followed by slow evolution on that manifold. We stress at the outset that the present architecture does not solve f(x,y)=0f(x,y)=0; it performs one forward-Euler step per round, and the question of how close that step brings x to equilibrium is treated empirically in §9 and theoretically in §7. 2.2 The injection is the forcing term of a forward-Euler step Over a single block of P tokens, the full fast update is a forward-Euler step applied to x˙=f(x,y) x=f(x,y). Writing the right-hand side explicitly: x←x+[Attn(Normx)+MLP(Normx)]⏟x-dependent part of f+tanh(γ)Norm(ϕ(yup))⏟slow forcing term.x\;←\;x\;+\; [Attn(Norm\,x)+MLP(Norm\,x) ]_$x$-dependent part of $f$\;+\; (γ)\,Norm(φ(y_up))_slow forcing term. (2) Two additive terms make up f(x,y)f(x,y): the attention and MLP layers are the x-dependent dynamics; the gated injection is the slow forcing term (the only place y enters f). The effective step size Δt=tanh(γ) t= (γ) is a learned scalar with γ=0γ=0 at initialisation, so the slow forcing vanishes at step 0 and the model must learn to open the gate. The function ϕφ maps the slow context to an update direction; by default it is a linear projection ϕ(y)=Wsfyφ(y)=W_sf\,y, but we also test a learned nonlinear projector ϕθ _θ (Section˜11.1). 2.3 Pooling implements the data-aggregation step of the HMM (Heterogeneous Multiscale Method) framework How is y obtained? Not by running k fast steps with y frozen and then advancing y (that update-count Lie splitting, ε=1/k =1/k, is the regime of HRM [9]). Instead we average the fast trajectory over a block of P tokens and run the slow dynamics on that average. This is a particular realisation of the data-aggregation step in the heterogeneous multiscale method (HMM) framework of E and Engquist [4]: estimate the slow evolution from aggregated fast data, where the macro variable is the block mean of the micro trajectory. Causal block-mean pooling (Definition˜3) implements the data-aggregation step of the HMM framework in its simplest form (macro = block mean of the micro trajectory), and P is the sequence-length timescale ratio ε=1/P =1/P. 2.4 An explicit ODE↔ correspondence To keep the mapping systematic rather than decorative, Table˜1 lists each correspondence that the present architecture instantiates (formally true of the computation). ODE concept LLM realisation Fast variable x per-token hidden state Slow variable y causal block-mean of P tokens Timescale ratio ε=1/P =1/P pooling factor (sequence-length ratio) Multirate averaging (HMM) pool → slow attention on T/PT/P tokens Forward-Euler forcing term gated injection tanh(γ)ϕ(yup) (γ)\,φ(y_up) Causality of the flow causal masks + past-block upsample Table 1: ODE↔ correspondence. 3 Concrete Architecture 3.1 Fast path The fast variable x∈ℝB×T×Dx ^B× T× D is initialised from token embeddings and processed by standard transformer layers (GQA causal attention with QK-RMSNorm, RoPE θ=104θ=10^4, GELU MLP, RMSNorm). We use a standard GELU MLP (matching the reference implementation). Let Block(x) Block(x) denote one such layer. The fast path has: • npren_pre layers applied once, before any slow interaction. • npostn_post layers applied per round, after slow-to-fast injection. 3.2 Slow path: causal temporal pooling The slow variable y is derived from x by causal block-mean pooling with factor P: Definition 3 (Causal pool). Given x∈ℝB×T×Dx ^B× T× D, the pooled representation is y[b,k,:]=1P∑j=0P−1x[b,kP+j,:],k=0,…,⌈T/P⌉−1.y[b,k,:]= 1P _j=0^P-1x[b,kP+j,:], k=0,…, T/P -1. (4) This is a causal low-pass filter with cutoff frequency approximately 1/(2P)1/(2P). Token pooling for hierarchical representations has been explored earlier for efficient long-sequence modelling [11, 12]; our block-mean pooling instantiates this idea with a fixed window of P tokens, used causally to preserve autoregressive validity. The slow variable y∈ℝB×(T/P)×Dy ^B×(T/P)× D is then processed by nslown_slow standard transformer layers: y~←Slow(y)(nslow layers, T/P tokens). y← Slow(y) ($n_ slow$ layers, $T/P$ tokens). (5) Because the pooled sequence has T/PT/P tokens, each slow attention layer costs O(T2/P2)O(T^2/P^2)—a factor P2P^2 less than the fast path’s O(T2)O(T^2). Full attention over T/PT/P positions has perfect long-range recall (the softmax spans all pooled positions), so no decay gate or retention mechanism is needed. 3.3 Upsample and Euler coupling The processed slow variable y~ y is mapped back to the fast timescale by causal upsample: Definition 6 (Causal upsample). yup[b,t,:]=y~[b,⌊t/P⌋−1,:],t≥P,,t<P.y_up[b,t,:]= cases y[b,\, t/P -1,\,:],&t≥ P,\\ 0,&t<P. cases (7) Tokens in the first block (t<Pt<P) receive zero slow context—the causal cold start. Using block ⌊t/P⌋ t/P would leak future tokens; using ⌊t/P⌋−1 t/P -1 always references a fully-past block. (The tail block, when TmodP≠0T P≠ 0, is averaged over its real token count in the reference implementation; the dilation TmodPT P leftover is absorbed by a divisor correction, so the slow attention sees a ⌈T/P⌉ T/P -length sequence regardless of T.) The fast variable is then updated: x←x+tanh(γ)⋅RMSNorm(Wsfyup),x← x+ (γ)· RMSNorm(W_sf\,y_up), (8) where γ∈ℝγ is a learnable scalar initialised at zero (γ=0γ=0 at step 0), and Wsf∈ℝD×DW_sf ^D× D projects the slow representation into the fast space. By default the injection uses the linear projection WsfW_sf; in §11.1 we replace it with a learned nonlinear projector ϕθ _θ. 3.4 Forward pass and cost Proposition 9 (Forward pass). Given token indices ∈0,…,V−1Tx∈\0,…,V-1\^T, the architecture computes: x x ←Embed() ← Embed(x) (10) x x ←Pre(x)(npre layers) ← Pre(x) ($n_ pre$ layers) (11) For r r =1,…,nrounds: =1,…,n_rounds: y y ←causal_pool(x,P) ← causal\_pool(x,P) (12) y~ y ←Slow(y)(nslow layers) ← Slow(y) ($n_ slow$ layers) (13) yup y_up ←causal_upsample(y~,P,T) ← causal\_upsample( y,P,T) (14) x x ←x+tanh(γ)⋅RMSNorm(ϕ(yup)) ← x+ (γ)· RMSNorm(φ(y_up)) (15) x x ←Post(x)(npost layers) ← Post(x) ($n_ post$ layers) (16) x x ←RMSNorm(x) ← RMSNorm(x) (17) =Head(RMSNorm(x)). = Head( RMSNorm(x)). (18) Proposition 9 is the executable specification of the architecture. It defines the complete computation graph in a single, self-contained sequence of operations. (The Lie operator splitting [3] of HRM-style update-count architectures is not used here; the present architecture is an HMM-style aggregation as described in §2.3, and the pool-then-attend-then-inject-then-post sequence in the Proposition is its discrete-time realisation, not a Lie splitting.) Thus, every line of the training code can be traced back to a specific step in the proposition. Causality, timescale separation, and zero-init identity are consequences of this specific sequence of operations. It isolates the slow→ interface at a single line (x←x+tanh(γ)Norm(ϕ(yup))x← x+ (γ)\,Norm(φ(y_up))), making the coupling mechanism cleanly separable from the rest of the architecture for ablation studies. 3.5 Cost Proposition 19 (Layer-equivalent cost). The total fast-layer equivalents (cost measured in full-T attention layers) is: Leq=npre+nrounds⋅(npost+nslowP2).L_eq=n_pre+n_rounds· (n_post+ n_slowP^2 ). (20) With default values (1,1,2,2,4)(1,1,2,2,4): Leq=1+2(1+2/16)=3.25L_eq=1+2(1+2/16)=3.25, versus 44 for a standard 44-layer dense baseline. The wall-clock ratio measured on Apple-Silicon MPS (T=256T=256, D=128D=128, H=4H=4, batch 44) is ≈0.97×≈ 0.97× the same-width (D=128D=128) dense baseline (Table˜2; the narrower D=96D=96 Gemma-style baseline is faster). The LeqL_eq advantage is partially offset by per-step constant overhead from RMSNorm, the upsample projection, and the slow-to-fast linear projection. The P2P^2 factor in the denominator is the structural reason the slow path is cheap: each slow attention layer costs 1/161/16 of a fast layer at P=4P=4. Without this factor, a system with nroundsn_rounds rounds and nslown_slow slow layers per round would be prohibitively expensive (as was the case in prior recurrent implementations where every layer operated on the full T-token sequence). The Leq=3.25L_eq=3.25 figure tells us that the entire architecture costs less than a 44-layer dense baseline—this is what makes real-scale experiments (≥10≥ 10M tokens, T≥1024T≥ 1024) affordable. The proposition also provides a compute budget: given a target LeqL_eq, the designer can trade off nroundsn_rounds (more iterative refinement) against nslown_slow (deeper slow processing) against P (stronger timescale separation) while staying within a fixed compute envelope. Together, Propositions 9 and 19 establish that the architecture is (a) precisely defined and (b) cheap enough to be a practical research instrument. 4 Structural Properties 4.1 Causality Proposition 21 (End-to-end causality). In the forward pass of Proposition 9, perturbing input token xtx_t does not change any output logit pt′p_t for t′<t <t. Proof. All fast layers (lines (11), (16)) use causal attention (is_causal=True). The pool (12) groups tokens into disjoint blocks; yky_k depends only on xkP,…,xkP+P−1x_kP,…,x_kP+P-1. The slow attention (13) over T/PT/P positions uses causal masking. The upsample (14) references block ⌊t/P⌋−1 t/P -1, whose last token is at position t−(tmodP)−1≤t−1<t-(t P)-1≤ t-1<t. The injection (15) is per-position and additive. By composition, the full forward pass is causal. ∎ Causality is a hard requirement for autoregressive language modelling. If the architecture leaked future tokens into past predictions, two failures would occur: (i) during training, the model would achieve artificially low loss by “cheating” from the future, making the loss an unreliable measure of learning; (i) at inference, when future tokens are unavailable, the model’s behaviour would diverge from its training behaviour, causing distribution shift. The causal upsample (Definition˜6) is the only non-trivial part of the proof: using block ⌊t/P⌋−1 t/P -1 instead of ⌊t/P⌋ t/P prevents the slow path from leaking block-level future information into the fast path. This is the kind of subtle bug that could go undetected in loss curves but would invalidate the entire architecture as a language model. This is verified empirically in the reference implementation: modifying xT−1x_T-1 produces zero change in p0:T−1p_0:T-1 within numerical precision (<10−4<10^-4), even when the gate is forced wide open (coupling scale = 10.010.0). The latter check is non-trivial: a real implementation bug that depended on the magnitude of tanh(γ) (γ) would be caught by this test but not by the closed-gate test alone. 4.2 Timescale Separation Proposition 22 (Genuine ε=1/P =1/P). The architecture enforces a timescale ratio ε=1/P =1/P. The slow variable y changes at most once every P fast tokens. The slow attention operates over at most T/PT/P positions. Proof. The pool (4) compresses P tokens into one pooled vector. The slow layers (13) operate on these T/PT/P vectors. The upsample (14) maps each pooled position to P fast positions, making yupy_up piecewise-constant over blocks of P tokens. A change in y can occur only at block boundaries, i.e., at most once per P tokens. ∎ In a learned-decay-gate variant of this architecture (gated linear attention [10] in the slow path), we observed the decay parameter collapse during training, producing an effective memory of ∼2 2 tokens—the “slow” subsystem became a fast-decaying buffer indistinguishable from the fast attention. Proposition 22 eliminates this failure mode structurally. The block-mean pool enforces that y cannot change faster than once per P tokens, regardless of what parameters the model learns. This means that any experiment using this architecture can rule out “the slow state wasn’t slow” as an explanation for null results. The timescale separation is guaranteed by the architecture, not by training dynamics. 4.3 Zero-Init Gate Proposition 23 (Zero-init identity). At initialisation (γ=0γ=0), the coupled forward pass (Proposition 9) is numerically identical to the variant with line (15) removed (no coupling). The model begins with no slow-to-fast modulation. If the coupling gate were initialised at a non-zero value (e.g. tanh(0.05)≈0.05 (0.05)≈ 0.05), the model would inject random slow representations from the first training step. Any observed coupling effect could then be an artefact of initialisation rather than learned behaviour. Proposition 23 guarantees that at step 0, the coupled model is numerically identical to the frozen (no-coupling) ablation. Any difference that emerges later is a learned effect. This makes --freeze_coupling a clean ablation: comparing coupled vs. frozen after training tells us whether the model chose to use coupling, not whether it inherited an initialisation bias. Together, these three propositions define the trust boundary of the architecture. They tell the experimenter what failure modes have been ruled out by construction (causality violation, timescale collapse, initialisation bias) and which remain as empirical questions. 5 The Manifold Question A natural idea—motivated by DeepSeek’s manifold-constrained connections [13]—is to constrain the slow state to a manifold (e.g. ‖y‖=1\|y\|=1). Our own prior variants applied such constraints and found no benefit. The ODE lens explains why: the word “manifold” names three distinct objects, and only one carries the fast–slow content. ℳ1M_1 (operator). The Birkhoff polytope of doubly stochastic mixing matrices (mHC). A Sinkhorn projection gives ‖ℋx‖≤‖x‖\|Hx\|≤\|x\|. This constrains signal propagation—a stability property. ℳ2M_2 (state). The unit sphere SD−1S^D-1, i.e. ‖y‖=1\|y\|=1. This constrains the magnitude of the slow state, making y a pure direction whose injected size is set entirely by Δt=tanh(γ) t= (γ). Also a stability property. ℳ3M_3 (dynamical). The Fenichel slow manifold ℳ=(x,y):f(x,y)=0M=\(x,y):f(x,y)=0\—the equilibrium set onto which the fast variable relaxes. This is the object the fast–slow reduction is about. Remark 24 (Prior manifold constraints did not help). mHC constrains ℳ1M_1; an ‖y‖=1\|y\|=1 constraint constrains ℳ2M_2. Both are norm/stability constraints and are orthogonal to ℳ3M_3: bounding an operator norm, or pinning a state to a sphere, cannot create a useful equilibrium manifold or make the slow variable informative. The RMSNorm already in the injection path bounds ‖ϕ(yup)‖\|φ(y_up)\| to D D, so ‖y‖=1\|y\|=1 adds no new constraint. This explains the consistent finding that coupling is neutral regardless of such constraints. Remark 25 (The principled role of norm constraints). Fenichel’s theorem [2] requires the slow manifold ℳ3M_3 to be normally hyperbolic—the fast dynamics must be contractive transverse to the manifold—for it to persist under perturbation. Norm constraints (ℳ1M_1, ℳ2M_2, RMSNorm, HRM’s MagicNorm) keep the fast subsystem bounded and contractive, i.e. they enforce the precondition for a slow manifold to exist and persist. They are necessary for a well-posed fast–slow reduction, but not sufficient to make the coupling load-bearing. Norm constraints are about persistence, not usefulness. 6 The Meaning of the Equilibrium Manifold Section˜5 identified the dynamical manifold ℳ3=(x,y):f(x,y)=0M_3=\(x,y):f(x,y)=0\ as the object the fast–slow reduction is about, but left its content unspecified: what is the equilibrium x=ϕ(y)x=φ(y), concretely? This section gives it a precise meaning by specialising the fast dynamics to a regime where ϕ(y)φ(y) is an identifiable probabilistic object—a master-equation stationary distribution—and the next section (Section˜7) proves the identity. This both sharpens the ODE framework and ties the equilibrium manifold directly to a familiar object in language-model computation (Section˜8). 6.1 The master equation The master equation (ME) describes the evolution of a probability distribution P∈Δn−1P∈ ^n-1 over n discrete states [7, 5, 6]: P˙=AP,Aij≥0(i≠j),⊤A=⊤, P=A\,P, A_ij≥ 0\ (i≠ j), 1 A=0 , (26) where the generator A has non-negative off-diagonal rates and zero column sums. The latter is conservation: dt⊤P=⊤AP=0 ddt1 P=1 AP=0. A stationary distribution solves Apst=0A\,p_st=0, pst≥0p_st≥ 0, ⊤pst=11 p_st=1, and the relative entropy S(t)=−∑nPnlog(Pn/pnst)S(t)=- _nP_n (P_n/p^st_n) is a Lyapunov function (S˙≤0 S≤ 0; the ME H-theorem). 6.2 The slow variable parametrises a generator We read the fast subsystem, in its linear and conservative limit, as such an ME whose generator is modulated by the slow context: f(x,y)≈A(y)x,A(y)a generator as in (26).f(x,y)\;≈\;A(y)\,x, A(y)\ a generator as in eq:me. (27) The slow variable y is exactly the ME’s external control parameter: it varies on the slow timescale (ε=1/P =1/P) and sets the transition rates of the fast chain. Under (27) the equilibrium manifold ϕ(y)φ(y) acquires a concrete identity, stated and proved next. We are explicit (Remark˜30) that (27) is a structured limit, not the trained block itself; the value of the reading is that it gives the abstract set ℳ3M_3 a name. 7 The Equilibrium Manifold is a Stationary Distribution Theorem 28 (QSS manifold == ME stationary distribution). Let the fast dynamics for fixed y be linear with generator structure, f(x,y)=A(y)xf(x,y)=A(y)\,x, where A(y)A(y) satisfies Aij(y)≥0A_ij(y)≥ 0 (i≠j)(i≠ j) and ⊤A(y)=⊤1 A(y)=0 . Then: (i) the QSS manifold is the kernel of the generator, ϕ(y)=kerA(y)φ(y)= A(y); (i) if A(y)A(y) is irreducible, kerA(y)=spanpst(y) A(y)=span\p_st(y)\ with pst(y)>0p_st(y)>0 the unique stationary distribution; hence ϕ(y)=pst(y)φ(y)=p_st(y); (i) under the same irreducibility assumption, the fast flow relaxes onto it: eA(y)tx(0)→pst(y)e^A(y)tx(0)→ p_st(y) for every distribution x(0)x(0), and the forward-Euler iterate (I+ΔtA(y))kx(0)→pst(y)(I+ t\,A(y))^kx(0)→ p_st(y) for 0<Δt<1/maxj|Ajj(y)|0< t<1/ _j|A_j(y)|. Proof. Fix y and write A=A(y)A=A(y). (i) By definition the QSS manifold is x:f(x,y)=0=x:Ax=0=kerA\x:f(x,y)=0\=\x:Ax=0\= A. (i) The column-sum condition ⊤A=⊤1 A=0 gives ∈kerA⊤1∈ A , so A is singular and kerA≠0 A≠\0\. Pick Δt t with 0<Δt<1/maxj|Ajj|0< t<1/ _j|A_j| and set M=I+ΔtAM=I+ t\,A. Then Mij=ΔtAij≥0M_ij= t\,A_ij≥ 0 for i≠ji≠ j, Mjj=1+ΔtAjj=1−Δt∑i≠jAij>0M_j=1+ t\,A_j=1- t _i≠ jA_ij>0 by the step bound, and ⊤M=⊤+Δt 1⊤A=⊤1 M=1 + t\,1 A=1 . So M is non-negative, column-stochastic, and has strictly positive diagonal. If A is irreducible, so is M, and the positive diagonal makes M aperiodic—hence primitive. By the Perron–Frobenius theorem for primitive stochastic matrices [7], M has spectral radius 11, the eigenvalue 11 is simple, and its right eigenvector p may be taken strictly positive. Normalising ⊤p=11 p=1 yields the unique stationary distribution pstp_st. Since Mp=p⇔Ap=0Mp=p Ap=0, we have kerA=spanpst A=span\p_st\ and ϕ(y)=pst(y)φ(y)=p_st(y). (i) By Gershgorin’s theorem applied to the columns of A (valid because σ(A)=σ(A⊤)σ(A)=σ(A )), every eigenvalue of A lies in a disc centred at Ajj=−∑i≠jAijA_j=- _i≠ jA_ij with radius ∑i≠jAij _i≠ jA_ij, hence in Rez≤0\Rez≤ 0\, touching the imaginary axis only at 0. For irreducible A the zero eigenvalue is simple and all others have Re<0Re<0; since the left/right 0-eigenvectors are 1 and pstp_st (with ⊤pst=11 p_st=1), the spectral projection onto the 0-eigenspace is pst⊤p_st1 , so eAt→pst⊤e^At→ p_st1 and eAtx(0)→pst(⊤x(0))=pste^Atx(0)→ p_st(1 x(0))=p_st for any distribution x(0)x(0) (no renormalisation is needed: eAte^At conserves ⊤x1 x). For the discrete map, M primitive gives Mk→pst⊤M^k→ p_st1 , so Mkx(0)→pst(⊤x(0))=pstM^kx(0)→ p_st(1 x(0))=p_st. ∎ Corollary 29 (ϕθ _θ as a variational stationary-distribution approximator). In the generator regime of Theorem˜28, ϕ(y)=pst(y)φ(y)=p_st(y) is a structured target, so a network ϕθ(y) _θ(y) trained to predict it is a variational approximation of the stationary distribution—it returns the equilibrium without iterating the chain (verified in Remark˜31). In the trained network, which is not a generator (Remark˜30), there is no probabilistic pst(y)p_st(y) to approximate; there ϕθ _θ is the corresponding learned nonlinear coupling. Its significance is structural rather than numerical: among the approximations of §11, the fidelity-only methods (truncated iteration, Anderson, auxiliary loss) change how closely x reaches a fixed point, whereas ϕθ _θ changes what is injected (a learned, nonlinear function of y). To the extent that the obstacle is the usefulness of y rather than QSS fidelity, this makes ϕθ _θ the distinguished rung—a hypothesis to be tested, not a result established here. Remark 30 (The boundary of the identity). Theorem˜28 requires the generator structure (27). A trained attention++MLP block does not satisfy it: its linearised one-step map I+ΔtJI+ t\,J (with J the Jacobian of the block) has column sums that would be expected to differ from 11 in general, since a trained block is not a generator; a numerical check on a random non-generator linear map gives [0.892,1.096]≠1[0.892,1.096]≠ 1. The identity therefore holds on the generator-structured submanifold of fast dynamics; the trained network is the unconstrained, nonlinear generalisation, and its fixed point is a least-contracting eigendirection rather than a stationary distribution. Consequently, methods that only raise QSS fidelity (converging more accurately to the block’s fixed point) cannot add probabilistic content when that fixed point is not a stationary distribution—a structural reason to expect such methods not to help, distinct from changing what is injected (Corollary˜29). Remark 31 (Numerical verification). A numerical direct check confirms each part on random generators (n=12n=12): forward-Euler power iteration matches the null vector pstp_st to max|⋅|=9.7×10−8 |·|=9.7× 10^-8 with generator residual |Ap|=1.2×10−7|Ap|=1.2× 10^-7 (geometric convergence); a small MLP ϕθ:y↦pst(y) _θ:y p_st(y) reaches held-out mean error 0.0310.031 without iterating; and a random non-generator map yields I+ΔtWI+ tW column sums in [0.892,1.096]≠1[0.892,1.096]≠ 1, confirming Remark˜30. 8 The Master Equation and Attention Theorem˜28 makes the equilibrium manifold a stationary distribution. This section connects the dynamics that move toward it—attention— to the master equation, stating each correspondence at its correct strength. The connection extends the meaning of ℳ3M_3 from a fast–slow abstraction to an object in everyday LLM computation. Attention is a discrete-time Markov transition. Self-attention computes xt′=∑s≤tαtsvsx _t= _s≤ t _tsv_s with α row-stochastic (∑sαts=1 _s _ts=1). This is one application of a stochastic matrix—a single Chapman–Kolmogorov step of a discrete-time chain, i.e. the integrated form eAΔte^A t—not the generator A of (26) (which has negative diagonal and zero column sums). “Attention is a master-equation step” should therefore read “attention is one step of a discrete-time Markov transition,” the exponentiated form of an ME rather than the ME itself. (The two conventions used in the paper—attention’s row-stochasticity and Theorem˜28’s column-stochasticity—are related by transposition: 1 is a right eigenvector of α in the former and a left eigenvector of M=I+ΔtAM=I+ tA in the latter; the stationary distribution pstp_st is in both cases the eigenvector of the appropriate side for eigenvalue 11.) Softmax is the Gibbs form. A single softmax yields a Boltzmann-shaped vector ∝eq⊤k/d e^q k/ d with inverse temperature 1/d1/ d. It is the shape of a Gibbs distribution. Attention entropy is a diagnostic. The ME H-theorem decreases the relative entropy of the state distribution P(t)P(t) to pstp_st. The entropy of the attention rows H(αt)=−∑sαtslogαtsH( _t)=- _s _ts _ts is a different object (the dispersion of the transition weights), so a regulariser on H˙attn H_attn is not the ME Lyapunov functional. Attention entropy remains a cheap, useful diagnostic—does an opened gate (tanhγ>0 γ>0) correlate with sharper attention and lower loss?—but it must be reported as such, not as the system descending its Lyapunov function. Pooling is approximate coarse-graining. The pooling factor P corresponds to lumping P micro-states into one macro-state. Exact lumpability of a Markov chain requires the Kemeny–Snell conditions [7]; causal block-mean pooling is an approximate lumping, consistent with the restriction/prolongation (HMM-/multigrid-style) reading of §2.3. Remark 32 (Summary of the ME correspondence). The two correspondences established above are (1) the QSS manifold under the generator-structure assumption is the ME stationary distribution (Theorem˜28 and Corollary˜29), and (2) attention is one discrete-time Markov transition, not the generator itself. The remaining analogies in this section (softmax as Gibbs form, attention entropy as diagnostic, pooling as approximate coarse-graining) are weaker, by-analogy statements; the table that v3 of this paper used to enumerate them is omitted here because the formal content is carried by (1) and (2). The master-equation reading gives the equilibrium manifold ℳ3M_3 a concrete identity (a stationary distribution), reframes the learned projector ϕθ _θ in the generator regime as a stationary-distribution approximator (Corollary˜29), and gives a structural reason that fidelity-only QSS refinements cannot add probabilistic content (Remark˜30). It is a second theoretical anchor—statistical physics alongside the numerical analysis of Section˜2.3—and a tighter tie to LLM computation (attention as a Markov transition operating around the manifold’s stationary law). It does not, by itself, make coupling load-bearing; that remains the empirical question of §9 and §12. 9 Preliminary Experiments We report a single-seed comparison on a 222222k-character multi-domain corpus, byte-level tokeniser (V=320V=320), 650650 steps (∼500 500k tokens seen), AdamW lr=10−4lr=10^-4, batch 44, T=256T=256. Our model uses D=128D=128, P=4P=4, (npre,npost,nslow,nrounds)=(1,1,2,2)(n_pre,n_post,n_slow,n_rounds)=(1,1,2,2) (0.780.78M parameters). All “Ours” rows use a held-out validation split (last 10%10\% of packed windows). 9.1 Baseline comparison Model Params Step-650 VAL ms/step Ours (frozen, γ≡0γ≡ 0) 0.78M 2.238 28 Ours (coupled, γ learned) 0.78M 2.231 28 MiniMind-style (4L, D=128D=128)† 1.03M 2.16 29 Gemma 4-style (4L, D=96D=96)† 0.59M 2.39 24 Gate stays closed: |tanh(γ)|<0.01| (γ)|<0.01. †Baseline numbers from training-loss runs; to be re-measured on the held-out validation split. Table 2: Single-seed validation loss (nats/byte). Frozen vs. coupled differ by 0.0070.007 nat—within noise. Table˜2 shows: (1) at 0.780.78M parameters, the frozen variant is within ∼0.08 0.08 nat of a 1.031.03M dense baseline; (2) the coupling gate does not open, so coupling is neutral at this scale; (3) wall-clock is comparable to the same-width dense baseline (≈0.97×≈ 0.97×; see §3.4). 10 Discussion Horizontal vs. vertical coupling. Attention itself is a horizontal coupling mechanism (across positions, within one timescale): xt←xt+∑s<tαtsvsx_t← x_t+ _s<t _tsv_s. The Euler forcing is a vertical coupling (between timescales): the update direction comes from the slow average y. The title’s question is structural: attention is coupling, but only of one kind; whether a second, vertical coupling provides benefits beyond what the horizontal coupling already captures is the open empirical question. At 500500k tokens, the answer is “not yet”: the coupling is neutral in absolute terms at this scale, and whether the small gate drift observed here (see below) generalises under a multi-seed protocol is left to future work. Why the gate drifts to ±0.01± 0.01 rather than staying at 0. Three conditions are present at 500500k tokens: (a) the slow path’s gradient is non-zero, so γ is pulled in some direction; (b) T/P=64T/P=64 pooled positions are a non-trivial function of the input, so the slow path has some signal; (c) the gate’s gradient |∂ℒ/∂γ|| /∂γ| is non-zero in expectation, so the gate moves. Three conditions are absent for a large movement: (a) the slow path’s gradient is attenuated through nslown_slow attention layers and the upsample; (b) the fast path’s own attention already handles 6464 pooled positions trivially; (c) the byte-level 222222k-character corpus lacks the long-range dependencies that would reward a 6464-position slow attention span. The net effect is a small drift in tanh(γ) (γ), not an opening. The ODE mapping as a conceptual tool. The value of the ODE framing is not in claiming the architecture is a fast–slow system—it is in providing a precise language for what is structural (ε , HMM-style aggregation, Euler forcing, causality) and what is missing (QSS manifold). Table˜1 is a core contribution: a vocabulary that makes design choices, negative results, and future directions semantically clear. The master-equation reading sharpens the representation. Theorem 28 gives the dynamical manifold ℳ3M_3 a concrete identity (pst(y)p_st(y)) and, in the generator regime, reframes the learned projector ϕθ _θ from a heuristic into a variational approximation of the ME stationary distribution (Corollary˜29). The reading does not, by itself, make coupling load-bearing; that remains an empirical question. 11 Approximating the QSS Manifold The QSS reduction requires f(x,y)=0f(x,y)=0, i.e. the fast variable must reach equilibrium given the slow context. Our architecture does one fast pass per round—a single forward-Euler step, not a fixed-point solve. This section describes four candidate approximations, ordered by increasing conceptual distance from the Euler baseline; a controlled comparison of their effect on loss is left to a multi-seed study (§12), as the single-seed differences we observed are within run-to-run noise. 11.1 Single pass = one Euler step (baseline) One pass of the post layers, x←Post(x)x← Post(x) with y fixed, is a single forward-Euler step of x˙=f(x,y) x=f(x,y). The residual ‖x−Post(x)‖\|x- Post(x)\| measures the distance to equilibrium. 11.2 Truncated fixed-point iteration (k passes) Apply the post layers k>1k>1 times with the same yupy_up held constant, sharing weights: x(i+1)=Post(x(i))x^(i+1)= Post(x^(i)), i=0,…,k−1i=0,…,k-1. If Post is a contraction (plausible under RMSNorm), x(k)→x⋆x^(k)→ x as k→∞k→∞. The cost scales linearly with k. 11.3 Anderson acceleration Anderson(m) uses the last m iterates to extrapolate. The special case m=1m=1 used here is the two-point secant method: α=⟨Δr,rcurr⟩/‖Δr‖2α= r,r_curr /\| r\|^2, xacc=xcurr+rcurr−α(Δx+Δr)x_acc=x_curr+r_curr-α( x+ r), where ri=Post(xi)−xir_i= Post(x_i)-x_i. 11.4 Auxiliary fixed-point loss Add ℒfp=λ⋅‖x−Post(x)‖22L_fp=λ·\|x- Post(x)\|_2^2 to the LM loss. This does not enforce x=ϕ(y)x=φ(y) during the forward pass but pushes parameters toward a regime where a single pass is a good approximation. 11.5 Learned equilibrium projector ϕθ _θ Train a small MLP ϕθ(yup) _θ(y_up) that predicts the equilibrium displacement directly from the slow context, bypassing the iterative solve: x←x+tanh(γ)⋅Norm(ϕθ(yup)),x← x+ (γ)· Norm( _θ(y_up)), (33) where ϕθ _θ is a 2-layer GELU MLP (D→H→D→ H→ D). This is trained jointly with the LM loss. Unlike the previous three methods, which modify how closely x approaches ϕ(y)φ(y), this method modifies how information is extracted from y: the nonlinear ϕθ _θ can learn representations that the linear WsfW_sf cannot. In the generator regime of Theorem 28, ϕ(y)=pst(y)φ(y)=p_st(y) is a structured target, so there ϕθ _θ is a variational approximation of a stationary distribution rather than a generic function of y; in the trained network (not a generator) it is the corresponding learned coupling (Corollary˜29). At larger training scale (≥10≥ 10M tokens, T≥1024T≥ 1024, BPE tokeniser), y will be better-trained and ϕθ _θ may extract a clean signal. 12 Conclusion and Future Work We described a multirate architecture with an explicit, status-marked mapping to the fast–slow ODE formalism (Table˜1). The architecture is cheap, causal, and enforces genuine timescale separation (ε=1/P =1/P) structurally. At 500500k-token scale, coupling is neutral—the single-seed gap of 0.0070.007 nat between the coupled and frozen variants (Table˜2) is within run-to-run noise. We also proved that, under the linear generator-structure assumption f(x,y)=A(y)xf(x,y)=A(y)x, the QSS fixed point ϕ(y)φ(y) is exactly the master equation’s stationary distribution pst(y)p_st(y) (Theorem 28). In that regime the learned projector ϕθ _θ is a variational approximation of a known, structured quantity rather than a generic function of y. The theorem’s boundary is sharp: a trained block is not a generator, which gives a structural reason that fidelity-only QSS refinements cannot add probabilistic content. At larger training scale, y will be better-trained and ϕθ _θ may extract a clean signal. This is the most promising next experiment, and is affordable. It is natural to make the fast path a DEQ solve [8] for x⋆=ϕ(y)x =φ(y). Then x=ϕ(y)x=φ(y) holds by construction, the slow attention drifts on the genuine equilibrium manifold ℳ3M_3, and the QSS manifold—currently the one ODE property the architecture does not instantiate (it performs a single Euler step, not a fixed-point solve)—would become instantiated by construction. References [1] A. Vaswani et al., “Attention is all you need,” in NeurIPS, 2017. [2] N. Fenichel, “Geometric singular perturbation theory for ordinary differential equations,” J. Differential Equations, vol. 31, no. 1, p. 53–98, 1979. [3] E. Hairer and G. Wanner, Solving Ordinary Differential Equations I. Springer, 1996. [4] W. E and B. Engquist, “The heterogeneous multiscale methods,” Commun. Math. Sci., vol. 1, no. 1, p. 87–132, 2003. [5] G. Haag, Modelling with the Master Equation. Springer, 2017. [6] D. Kulasiri and R. Kosarwal, Chemical Master Equation for Large Biological Networks. Springer, 2021. [7] J. R. Norris, Markov Chains. Cambridge University Press, 1997. [8] S. Bai, J. Z. Kolter, and V. Koltun, “Deep equilibrium models,” in NeurIPS, 2019. [9] G. Wang et al., “HRM: A hierarchical recurrent model for structured reasoning,” 2025. [10] S. Yang et al., “Gated linear attention transformers with hardware-efficient training,” in ICML, 2024. [11] P. J. Liu et al., “Generating Wikipedia by summarizing long sequences,” in ICLR, 2018. [12] M. Zaheer et al., “Big bird: Transformers for longer sequences,” in NeurIPS, 2020. [13] Z. Xie et al., “mHC: Manifold-constrained hyper-connections,” DeepSeek-AI, 2025.