Paper deep dive
World Modeling through Spectral Alignment
Holger Molin, William Peng, Marco Bagatella, Randall Balestriero
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control.
Tags
Links
- Source: https://arxiv.org/abs/2610.15429v1
- Canonical: https://arxiv.org/abs/2610.15429v1
Trouble viewing inline? Open PDF directly →
Full Text
91,174 characters extracted from source content.
Expand or collapse full text
World Modeling through Spectral Alignment Holger Molin †thanks: Equal contribution. Affiliation: Stanford University William Peng11footnotemark: 1 Affiliation: Stanford University Marco Bagatella Affiliation: ETH Zürich Randall Balestriero Affiliation: Brown University Abstract How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control. Website Code 1 Introduction The goal of a world model is to encode sufficient information to predict the consequences of an agent’s actions (Ha and Schmidhuber, 2018). At the same time, complex real-world observations contain large amounts of task-irrelevant information (Zhang et al., 2021). Learning a compact representation therefore hinges on deciding which aspects of an observation to retain. Joint Embedding Predictive Architectures (JEPAs; LeCun, 2022) predict outcomes in a latent space, rather than reconstructing high-dimensional observations in pixel space, allowing learned representations to abstract away details of the input. Due to the lack of explicit supervision or grounding in raw observation space, training JEPAs is generally non-trivial. The training objective must implicitly guide which distinctions the model preserves without requiring it to reproduce every detail of an observation. Recent work has made substantial progress toward stable JEPA training through regularizing the embedding distribution (Balestriero and LeCun, 2025; Maes et al., 2026). However, avoiding collapse alone does not ensure that a representation preserves information needed for control, and the degree to which this happens is dictated implicitly by self-prediction (Tang et al., 2023). For instance, an encoder may distinguish robot configurations while ignoring the objects the robot must manipulate. We seek a direct method to express these relationships and understand their consequences. We approach this problem through the lens of spectral representation learning (Belkin and Niyogi, 2003; Bengio et al., 2004), which historically focuses on extracting representations from pairwise relationships between samples. Prior work has shown that several contrastive and non-contrastive objectives generally admit spectral interpretations (HaoChen et al., 2021; Balestriero and LeCun, 2022). Building on this perspective, we make the pairwise notion of similarity between two observations the center of a new training objective for latent dynamics modeling. We introduce a spectral alignment framework for training world models to preserve a chosen similarity structure. Leveraging a notion of distances between observations (the teacher), we define a target similarity kernel. We then train an encoder (the student) to learn embeddings whose pairwise similarities match the target and an action-conditioned predictor to output future latents. This framework can be instantiated through different teachers requiring varying degrees of supervision: when some information about the underlying system is known, encoders can be trained based on physical distances, but the natural ordering of states within a trajectory is also sufficient to provide a training signal. We name the resulting world model SpecWM. Figure 1: Spectral alignment framework. Pairwise distances (target) between samples define a kernel that an encoder is trained to reconstruct through embedding similarities. Theoretically, we first study how the choice of teacher shapes the learned representation. When the teacher is specified for all pairs of states, we characterize the global optima as the kernel’s spectral embeddings. This connection allows us to identify which quantities can be recovered from the representation via a linear probe. When teacher supervision is only available within a trajectory, we establish conditions under which matching the teacher removes observation noise from the embeddings. Finally, for the specific choice of temporal kernels, we connect the expected similarity target to the behavioral successor measure, which describes how likely the data-collection policy is to visit a given state in the future. Practically, we instantiate this framework across visual manipulation environments spanning OGBench Scene and Cube (Park et al., 2025) as well as CALVIN (Mees et al., 2022). Empirically, we confirm that kernel choice substantially impacts which physical quantities are linearly recoverable. Further, when leveraging a self-supervised temporal teacher, whose only source of supervision is the order of observations in a trajectory, the resulting world models outperform existing JEPA world models (Maes et al., 2026) in planning performance. Our contributions can be summarized as follows: 1. We introduce a novel world modeling objective based on spectral alignment. 2. We formally study this framework through the lens of spectral representation learning, with a particular focus on temporal kernels. 3. We present a broad empirical evaluation on visual control tasks and measure the ability of different kernels to recover physical quantities and drive planning performance. 2 Related Work World models learn how environments respond to actions, enabling agents to plan and learn from predicted outcomes (Ha and Schmidhuber, 2018). Generative approaches predict future observations, often by modeling dynamics in a latent space and decoding predictions into images. PlaNet uses learned dynamics to evaluate candidate action sequences, while the Dreamer series trains policies on imagined trajectories (Hafner et al., 2019; Hafner et al., 2025). More broadly, generative models support interactive environments and spatial world generation (Parker-Holder and Fruchter, 2025; Team HY-World, 2026; World Labs Team, 2026), with world-action models such as Cosmos Policy, DreamZero, and LingBotVA extending these capabilities to action generation for robot control (Kim et al., 2026; Ye et al., 2026; Li et al., 2026). In contrast, reconstruction-free world models focus prediction on representations without requiring them to reproduce observations. DINO-WM predicts features from a pretrained visual encoder (Zhou et al., 2025) while JEPAs such as LeWorldModel learn the representations alongside the dynamics (LeCun, 2022; Maes et al., 2026). JEPA as proposed in LeCun (2022) features the central question of what anti-collapse mechanism is used to guide which information is retained and which is abstracted away. In the cases of I-JEPA (Assran et al., 2023) and V-JEPA 2.1 (Mur-Labadia et al., 2026), the target encoder is an exponential moving average of the context encoder with a stop-grad, making collapse difficult to attain but not actually a non-optimal solution. LeJEPA (Balestriero and LeCun, 2025) solves anti-collapse by enforcing the distribution of latents to be an isotropic Gaussian, which prevents trivial collapse but does not require any particular information to be retained. Spectral representation learning is intimately tied to self-supervised learning. HaoChen et al. (2021) interpret contrastive learning as spectral decomposition of an augmentation graph. Balestriero and LeCun (2022) extend this perspective, showing that several objectives including InfoNCE (van den Oord et al., 2019), VICReg (Bardes et al., 2022), and Barlow Twins (Zbontar et al., 2021) recover classical spectral embedding methods where the graph is defined implicitly using the choice of augmentations. Our spectral alignment objective extends this work by making previously implicit relations between samples the central object of study. We cover additional families of related work in Appendix A. 3 Preliminaries Let xii=1n\x_i\^n_i=1 be a collection of inputs and let fθ:→ℝdf_θ:X ^d be an encoder parameterized by θ. We denote its representations by hi=fθ(xi)h_i=f_θ(x_i), which we stack as rows of H∈ℝn×dH ^n× d. A symmetric matrix K∈ℝn×nK ^n× n can be used to specify the pairwise relationships between inputs. For instance, in an augmentation-based method, KijK_ij may be designated as 11 for two views produced from the same observation, and as 00 otherwise. More generally, K can be interpreted as the result of a symmetric similarity function — i.e., a kernel — k:×→ℝk:X×X , which defines relationships through Kij=k(xi,xj)K_ij=k(x_i,x_j). Spectral embeddings. Given a teacher kernel K specifying the desired inner product between every pair of inputs in a collection, we can construct a coordinate vector for each input that realizes these pairwise relationships. Such a construction is called a spectral embedding. Suppose that K is symmetric and positive semidefinite, meaning that a valid spectral embedding exists. K then has eigendecomposition K=∑ℓ=1rλℓuℓuℓ⊤=UrΛrUr⊤,K= _ =1^r _ u_ u_ =U_r _rU_r , (1) where r=rank(K)r=rank(K), λℓ>0 _ >0 is the ℓ -th positive eigenvalue of K, uℓ∈ℝnu_ ^n is its corresponding unit eigenvector, Ur=[u1,…,ur]U_r=[u_1,…,u_r], and Λr=diag(λ1,…,λr) _r=diag( _1,…, _r). A spectral embedding of K is given by Hspec=UrΛr1/2H_spec=U_r _r^1/2 where the i-th row hispech_i^spec provides the coordinates of sample i. Exactly satisfying all the inner products of K requires an embedding dimension d≥r.d≥ r. When d<rd<r, we get Hd=UdΛd1/2H_d=U_d _d^1/2, for which HdHd⊤=UdΛdUd⊤H_dH_d =U_d _dU_d is the best rank-d approximation of K in Frobenius norm. Each eigenvalue determines the relative contribution of its corresponding eigenvector to the kernel, as seen in Equation 1, making eigenvalue rescaling a natural reweighting operation. To construct a particular spectral embedding, one selects d eigenvectors and assigns each a nonnegative scale sℓs_ yielding Hspec=Uddiag(s1,…,sd),hispec=(s1u1(i),…,sdud(i)).H_spec=U_ddiag(s_1,…,s_d), h_i^spec= (s_1u_1(i),…,s_du_d(i) ). (2) The canonical embedding of K is recovered through sℓ=λℓs_ = _ ; other choices reweight the spectrum. Linear probing. To assess which physical quantities are linearly accessible from a representation, we fit a linear map from fixed embeddings to target labels. Given targets Y∈ℝn×pY ^n× p, such as object positions, the optimal unregularized probing error is minW∈ℝd×p‖Y−HW‖F2 _W ^d× p\|Y-HW\|_F^2. This error is zero precisely when every column of Y lies in the column space of H, denoted col(H)col(H). 4 Method Figure 2: Spectral alignment on a single trajectory. Top: sampled observations from OGBench Scene (Park et al., 2025). Bottom: the temporal target kernel and encoder Gram matrix (before centering), the spectral embedding of K (top three eigenvectors), and the encoder rows hih_i (top three principal components), colored by frame index. Matching pairwise similarities encourages the encoder to recover the target’s geometry. We will state the general framework for arbitrary kernels, and later specialize it to practical kernels. Building a target. We start from a distance d between pairs of observations (which could be derived from ground-truth measurements, if available, or simply by counting the number of observations that separate two observations in the data). Considering a sequence of observations xii=1n\x_i\_i=1^n, we leverage the distance d to build a target kernel KijK_ij. In practice, we adopt a Laplacian kernel to convert distances into similarities: Kij=exp(−d(xi,xj)/σ),K_ij= (-d(x_i,x_j)/σ ), (3) with values in (0,1](0,1]. Its slope at d=0d=0 is −1/σ-1/σ, so that observations that are close according to d remain distinguishable, while pairs more than a few σ apart have similarity near 00. The bandwidth σ therefore sets the range of distances the target resolves; we set it to the median teacher distance over the pairs the kernel is defined on. We then double center K to form K~ K: K~ij=Kij−1n∑kKik−1n∑kKkj+1n2∑k,lKkl. K_ij\;=\;K_ij\;-\; 1n _kK_ik\;-\; 1n _kK_kj\;+\; 1n^2 _k,lK_kl. (4) Without centering, the kernel would be dominated by a positive baseline, since kernel values concentrate above zero, and regressing this target would favor matching that shared baseline rather than the variation around it. Still, if a few eigenvalues dominate the kernel, the encoder can achieve a small squared error by matching those directions while largely ignoring the others. We therefore temper the kernel by reshaping its spectrum before it is used as a target. We write the eigendecomposition of the centered kernel as K~=UΛUT K=U U^T with eigenvalues in decreasing order, and define Λα ^α by raising the leading m positive eigenvalues to the power α∈[0,1]α∈[0,1] and setting all remaining entries to zero. The target is then Kα=nUΛαUTtrΛα.K^α=n U ^αU^Ttr ^α. (5) At α=1α=1, each eigenvector occupies the target’s variance according to its eigenvalue fraction. At α=0α=0, each eigenvector’s share is uniform. In practice, we will use α=1/2α=1/2, thus using the square root of the kernel matrix as target, and truncate the spectrum at m=128m=128. Training the encoder. We implement the encoder fθf_θ as a deep neural network that maps each observation xix_i to an embedding hi∈ℝdh_i ^d. It will act as a student, and regress the kernel designed from the teacher distance. We first rescale each embedding to the length the target prescribes h¯i=Kiiαhi‖hi‖, h_i= K^α_i h_i h_i , (6) and then compute the student’s Gram matrix — the symmetric matrix of inner products of its rows — S=H¯H¯TS= H H^T. Similarly to the kernel, we double center the Gram matrix as S~ S, and regress the target through a spectral loss: ℒspec=‖S~−Kα‖F2.L_spec= S-K^α ^2_F. (7) Prediction anchor. While ℒspecL_spec is sufficient for training an encoder, planning in latent space requires learning a predictor, i.e., a neural network gϕ:ℝd×→ℝdg_φ:R^d×A ^d parameterized by ϕφ and trained to predict the latent outcome of applying some action from an action space A after receiving a given observation. For stability, we apply a parameter-free RMS normalization to the encoder output, so that every embedding has norm d d. While the spectral loss is invariant to this step, it fixes the scale of the embeddings seen by the predictor and the planner. We train a predictor by a simple regression objective in latent space, which is also backpropagated through the encoder, such that the total loss is ℒ=ℒspec−λ∑i(cos(G,H′))i,L=L_spec-λ _i ( (G,H ) )_i, (8) where G stacks predictions gϕ(fθ(xi),ai)g_φ(f_θ(x_i),a_i), H′H stacks representations of next observations fθ(xi′)f_θ(x_i ) for each training transition (xi,ai,xi′)(x_i,a_i,x_i ), and cos(⋅) (·) extracts the vector of row-wise cosine similarities from the two matrices. Planning. Once a spectral world model is trained through the objective in Equation 8, it can be directly deployed in the standard planning pipeline for JEPA world models (Maes et al., 2026). In practice, we use the Cross-Entropy Method (CEM) (Rubinstein, 1999) to optimize action sequences from the current observation x0x_0: candidate action sequences are sampled from a Gaussian distribution, and the world model predicts the representation of observations T steps ahead for each. Candidates are then scored according to similarity to the embedding of a goal observation g: cos(gϕT(fθ(x0),a0T−1),fθ(g)), (g_φ^T(f_θ(x_0),a_0^T-1),f_θ(g) ), (9) where gϕTg_φ^T denotes recursive application of the predictor over T steps, and a0T−1a_0^T-1 is a sequence of actions. The action distribution is refit on the best candidates for a fixed number of iterations. We then execute a prefix of the selected action sequence before replanning from the new observation. We detail hyperparameters in Appendix C. Kernel design. Often, robotics data is labeled by informative metadata such as proprioception or depth masks. In such cases, that metadata can be used to construct a supervised kernel. We evaluate three such kernels: state, depth, and proprioception. For each, we use the Euclidean distance between the corresponding feature vectors as the teacher distance. The state kernel uses the simulator state sis_i underlying each observation xix_i as its feature vector. Although this information is generally unavailable in real-world settings, it lets us evaluate a teacher with access to the full physical state. The proprioception and depth kernels use the robot’s proprioceptive state and the flattened depth mask as feature vectors, respectively. When metadata is not available, the ordering of observations in trajectories itself can form an unsupervised kernel. We use a temporal kernel, which relies on absolute difference between indices for observations ordered in a trajectory: d(xi,xj)=|i−j|d(x_i,x_j)=|i-j|. We also evaluate a quasimetric kernel as a close relative to the temporal kernel. This kernel relies on a distance dqd_q learned through quasimetric reinforcement learning (Wang et al., 2023; Wang and Isola, 2022), which estimates temporal distances between observations according to the optimal goal-reaching policy. While this potentially introduces estimation errors, it labels state pairs by the minimum number of steps between states, instead of expressing the temporal separations observed in the data, and can thus compensate for poor data quality. Since the student’s Gram matrix is symmetric, we symmetrize the learned quasimetric d(xi,xj)=dq(xi→xj)+dq(xj→xi)d(x_i,x_j)=d_q(x_i→ x_j)+d_q(x_j→ x_i). While supervised kernels can describe relationships between all pairs of inputs, i.e., are fully specified, the unsupervised kernels we use only express relationships for pairs of inputs in the same trajectory. In this underspecified setting, cross-trajectory similarities are excluded from the training objective. We also use a different normalization of the spectral loss: we omit tempering and the target-diagonal rescaling in Equation 6. Each clip’s squared error is normalized by the mean squared magnitude of its centered target before averaging across clips. While representations are fixed up to an orthogonal transformation in the fully specified case, underspecification allows the model to form its own abstractions (HaoChen and Ma, 2023). 5 Theoretical Analysis In this section, we build upon previous analyses (Bengio et al., 2004; Balestriero and LeCun, 2022; Blier et al., 2021) and provide formal statements regarding the spectral alignment objective and some intuition on their practical relevance. Full proofs of each theorem are provided in Appendix D. 5.1 The spectral loss has a known optimum A foundational property of our objective is that in the fully specified setting, an embedding table that minimizes ℒspecL_spec recovers the row-normalized spectral embeddings of KαK^α (Bengio et al., 2004). Theorem 1 (Spectral recovery). Suppose m≤dm≤ d and Kiiα>0K^α_i>0 for all i. Then minHℒspec(H)=0 _HL_spec(H)=0, which is attained exactly at H⋆=diag(c)HspecQ,H =diag(c)\,H_spec\,Q, (10) with c∈ℝ>0nc ^n_>0, Q∈ℝd×dQ ^d× d an orthogonal matrix, and HspecH_spec the spectral embedding of KαK^α with square-root eigenvalue scaling. This result is useful as the spectral embeddings of the kernel can be calculated through eigendecomposition as described in Section 3. We can thus evaluate the probing qualities of spectral embeddings derived from arbitrary kernels without ever having to train a model. 5.2 Downstream probing guarantees The ability to probe a given quantity from spectral embeddings depends on its relationship with the chosen kernel. Following prior work (Balestriero and LeCun, 2022), we show that a kernel constructed from label relationships can make the labels themselves linearly recoverable, without directly regressing encoder outputs onto them. Intuitively, this suggests that the choice of a certain kernel encourages embeddings to extract the information used to construct it; this motivates the supervised kernels evaluated in Section 6.3. Theorem 2 (Linear recoverability). Let Y∈ℝn×pY ^n× p satisfy ⊤Y=01 Y=0, row norms ‖yi‖=1\|y_i\|=1, and rank(Y)≤drank(Y)≤ d. For K=YY⊤K=Y , α=1α=1, and all positive spectral directions retained, every global minimizer H⋆H of ℒspecL_spec subject to ‖hi‖=d\|h_i\|= d satisfies minW∈ℝd×p‖Y−H⋆W‖F2=0. _W ^d× p\|Y-H W\|_F^2=0. (11) 5.3 Task-irrelevant information A central goal of a JEPA is selective invariance— dropping irrelevant parts of an observation (LeCun, 2022). We define task-irrelevant information as any aspect of an observation the teacher kernel is invariant to. For example, a cost-to-go kernel for robotic manipulation may ignore the exact patterns of leaves rustling in the background. We ask whether matching such a teacher also makes the encoder ignore these details. Formally, consider a sequence of observations xii=1n=(si,ni)i=1n,\x_i\^n_i=1=\(s_i,n_i)\^n_i=1, where sis_i is the task-relevant component and nin_i is the task-irrelevant component. Let ν(⋅∣si)ν(· s_i) denote the distribution of task-irrelevant values at state sis_i, and let Z collect the task-relevant trajectory. We establish invariance under a coverage assumption: nuisance values that are individually possible at two states must also be possible together in a trajectory visiting both states. This excludes a background color that varies across clips but is always identical between frames of a clip. Coverage. (A1) Fix a task-relevant trajectory Z and two distinct observation indices i and j. Draw n∼ν(⋅∣si)n ν(· s_i) and n′∼ν(⋅∣sj)n ν(· s_j) independently. For almost every Z, all i≠ji≠ j, and every measurable set E of nuisance pairs, assume Pr((n,n′)∈E∣si,sj)>0⟹Pr((ni,nj)∈E∣Z)>0. ((n,n )∈ E s_i,s_j )>0 ((n_i,n_j)∈ E Z )>0. Conditioning on the trajectory may change which nuisance pairs are likely, but cannot make an otherwise possible set of pairs impossible. This assumption covers independently sampled sensor noise and allows correlated disturbances, such as stochastic foliage motion or rainfall, when all otherwise possible combinations remain possible. It excludes persistent trajectory-level attributes, such as a background identity or lighting that varies across trajectories but is fixed within each one. Theorem 3 (Abstraction of task-irrelevant information). Under the coverage assumption, fixed-norm encoder outputs, and the target-diagonal rescaling in Equation 6, if each trajectory’s target matrix KαK^α has strictly positive diagonal entries almost surely and [ℒspec]=0E\,[L_spec]=0, then for almost every clip and every frame i, fθ((si,n))=fθ((si,n′))f_θ((s_i,n))=f_θ((s_i,n )) (12) for almost every pair (n,n′)(n,n ) drawn independently from ν(⋅∣si)ν(· s_i). At zero expected loss, varying the task-irrelevant component while holding the task-relevant component fixed leaves the representation unchanged. 5.4 Temporal kernels and the successor measure Having analyzed what information spectral alignment keeps or discards for arbitrary kernels, we now focus on the temporal kernel, and show that the representations it retrieves can be desirable for planning with a world model. This analysis replaces the free embedding tables above with a state-conditioned encoder h:→z∈ℝd:‖z‖=1h:S→\z ^d: z =1\. We will show that ℒspecL_spec corresponds to a regression problem, whose target can be related to the (behavioral) successor measure (Blier et al., 2021): the discounted cumulative probability of the data generation process to visit a given state in the future. This connects the embedding similarity used in planning to how readily states lead to one another under the behavior represented in the data. We assume that each trajectory in the dataset is generated by a stationary Markov policy πβ _β in a Markov decision process with a finite state space S, and that the induced Markov chain P is irreducible and aperiodic, admitting a stationary distribution μ, from which the initial state x0x_0 is sampled. We can introduce a discount factor derived from the Laplacian kernel and its temperature: γ:=e−1/σ∈(0,1)γ:=e^-1/σ∈(0,1), and then define the behavioral successor measure as Mβ(x,x′)=∑t=0∞γtPr(xt=x′∣x0=x,πβ).M^β(x,x )\;=\; _t=0^∞γ^t\, \! (x_t=x x_0=x, _β ). (13) The training data is generated by drawing trajectories (x0,…,xT−1)(x_0,…,x_T-1) from the induced Markov chain, and indices (i,j)(i,j) uniformly from 0,…,T−12\0,…,T-1\^2. For simplicity, we finally assume that the student and the teacher are not centered and that the latter does not undergo tempering: α=1α=1, and no truncation is applied. Let s(x,x′)=h(x)⊤h(x′)s(x,x )=h(x) h(x ) denote embedding similarity. Under these assumptions, we consider the population-level objective: ℒspecpop(h)=τ,i,j[(s(xi,xj)−γ|i−j|)2],L_spec^pop(h)\;=\;E_τ,i,j [ (s(x_i,x_j)-γ^|i-j| )^2 ], (14) with τ:=(x0,…,xT−1)∼τ:=(x_0,…,x_T-1) and (i,j)∼(0,…,T−12)(i,j) (\0,…,T-1\^2). We also define the expected teacher similarity of the state pair as RT(x,x′)=τ,i,j[γ|i−j||xi=x,xj=x′].R_T(x,x )=E_τ,i,j[γ^|i-j||x_i=x,x_j=x ]. (15) In Appendix D, we demonstrate that ℒspecpop(h)L_spec^pop(h) is equivalent to minimizing τ,i,j[(s(xi,xj)−RT(xi,xj))2]E_τ,i,j [(s(x_i,x_j)-R_T(x_i,x_j))^2 ], with the embedding similarity s(xi,xj)=h(xi)⊤h(xj)s(x_i,x_j)=h(x_i) h(x_j). We can then study the target RTR_T in the limit of long trajectories, and connect it to the successor measure MβM^β of the behavioral policy πβ _β. Theorem 4 (Population target). Under the setup above, with a finite state space, an irreducible, aperiodic Markov chain P induced by πβ _β with stationary distribution μ and x0∼μx_0 μ, for all x,x′∈x,x , as T→∞T→∞, TRT(x,x′)⟶Mβ(x,x′)μ(x′)+Mβ(x′,x)μ(x)−[x=x′]μ(x).T\,R_T(x,x )\; \; M^β(x,x )μ(x )+ M^β(x ,x)μ(x)- 1[x=x ]μ(x). (16) Moreover, if the induced Markov chain is reversible, i.e., μ(x)P(x′|x)=μ(x′)P(x|x′)μ(x)P(x |x)=μ(x )P(x|x ), TRT(x,x′)⟶2Mβ(x,x′)μ(x′)−[x=x′]μ(x′).T\,R_T(x,x )\; \; 2M^β(x,x )μ(x )- 1[x=x ]μ(x ). (17) Embedding similarity is thus encouraged to regress a target that tracks a symmetrized likelihood ratio, holding the behavioral successor measure as its numerator. This relates the planning score in Equation 9 to discounted visitation between predicted states and the goal, in both temporal directions and relative to their stationary frequencies. 6 Experiments Our empirical evaluation revolves around three visual manipulation environments: OGBench Cube and Scene (Park et al., 2025) and CALVIN (Mees et al., 2022). Scene and CALVIN extend the evaluation beyond the simpler manipulation settings considered in LeWorldModel (Maes et al., 2026), introducing multiple objects and articulated fixtures whose states must be represented for control. These environments directly test whether the learned representation preserves information about the surrounding scene as well as the robot itself. We primarily compare SpecWM to LeWM (Maes et al., 2026), and we instantiate SpecWM for each of the kernels introduced in Section 4. A generally observed phenomenon in world modeling is that predictors tend to generalize poorly to random actions when trained on only expert data (Fu et al., 2021; Kidambi et al., 2020; Yu et al., 2020). Motivated by this, we generate failure data for each environment and append it to the canonical datasets; this decision is ablated in Appendix B.3. Our main points of comparison are success rate with CEM planning (Section 6.1) and linear recoverability of physical quantities from embeddings (Section 6.2). Further, we provide an ablation of various kernels in both fully specified and underspecified settings in Section 6.3. We extend these results with additional ablations in Appendix B. For CALVIN, we augment training clips with temporally distant observations and use a multiscale kernel to capture block movements that short clips often miss. Appendix C details this construction, failure data generation, and evaluation settings. 6.1 Planning With temporal and quasimetric kernels, SpecWM improves upon LeWM by an average of 21 percentage points in planning success rate (Figure 3). Intuitively, the planner ranks predicted outcomes by their embedding similarity to the goal, which spectral alignment shapes to reflect temporal or quasimetric proximity. Overall, we observe that a learned quasimetric kernel is generally on par with a readily available temporal kernel, but either may be preferable depending on the environment. Figure 3: Average CEM planning success rate on 100 fixed tasks per environment. Bars report the mean over the 20k, 24k and 28k checkpoints and 3 seeds, with whiskers showing one cross-seed standard deviation. SpecWM consistently outperforms LeWM. 6.2 Probing To isolate representation quality from prediction accuracy, we now evaluate linear probing performance for the position of objects across environments. We consider a held-out set of observations that the encoder was never trained on for each dataset, and define 55 train-evaluation splits. For each split, we fit a ridge regressor for each state variable, choose regularization by cross-validation, and report linear recoverability on the evaluation set. We report R2R^2 between predicted and ground-truth values, averaged over variables and splits. As reported in Figure 4, we find that SpecWM does not always beat LeWM on probing, and probe quality does not directly correlate with planning performance. Figure 4: Linear probe R2R^2 of the manipulated object’s position from each model’s final checkpoint embedding. Bars report the mean over 3 seeds, with whiskers showing one cross-seed standard deviation. Random is an untrained encoder of the same architecture. 6.3 Kernel Ablations While the previous evaluations focus on the temporal and quasimetric targets, we now extend the investigation to the full set of kernels we introduced in Section 4, including supervised, fully specified variants (i.e., state, depth, and proprioception), and unsupervised, underspecified variants (i.e., temporal, quasimetric). The latter will help in quantifying the information lost due to lack of ground-truth supervision. Table 1 reports linear probes on all of the fixtures in the Scene environment. The fully specified state kernel achieves the highest average R2R^2 (0.93). Under underspecified supervision, the temporal kernel nearly matches the state kernel and outperforms the quasimetric kernel. Proprioceptive kernels preserve the robot’s configuration but largely omit object states, supporting Theorem 3. For the fully specified kernels, we note that kernel eigenvectors are indeed strongly recoverable from the learned embeddings (Appendix B.1). Table 1: Linear probe R2R^2 on held-out OGBench Scene episodes. Bold marks the best result in each row and results within 0.01 of it. Underspecified Fully specified Temporal Quasimetric State Proprio Depth State Proprio Depth Scene (avg) 0.84 0.78 0.85 0.16 0.15 0.93 0.16 0.50 cube 0.79 0.51 0.57 0.10 0.09 0.96 0.08 0.09 arm 0.67 0.66 0.69 0.69 0.60 0.73 0.67 0.59 drawer 0.91 0.90 0.98 0.01 0.01 0.99 0.01 0.87 window 0.87 0.88 0.99 0.00 0.01 0.99 0.00 0.91 buttons 0.96 0.94 1.00 0.02 0.01 0.99 0.02 0.04 7 Conclusion We present world modeling through spectral alignment as a framework for designing the spectral methods that have so far been implicit in self-supervised learning. The core idea is to directly specify the pairwise relationships that representations should preserve through a teacher kernel. We explore theoretical connections to fundamental results in spectral manifold learning and, for a specific choice of kernel, to the likelihood of reaching goals expressed in the data distribution. Further, we validate the method empirically and analyze its abstraction properties. We find that readily available kernels result in strong planning performance, which is not entirely correlated with probing accuracy. Pursuing a better understanding of the mechanisms enabling effective planning would be valuable. We view the study of which kernels are most conducive to large-scale self-supervised training as the next primary direction of work. We further view kernels involving language for semantic distances as an interesting future direction. AI use statement In this work, we used generative AI tools to implement research methods and develop and revise code. We also used these tools to discuss research methodology and theoretical results, identify relevant literature, assist in proving theoretical statements, and draft and edit portions of the manuscript. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. Reproducibility statement To ensure effective reproducibility, all code necessary for this paper is available at (https://anonymous.4open.science/r/SpecWM/). Full proofs of presented theorems can be found in Appendix D. The dataset generation process along with evaluation protocols and training settings are described in Appendix C as well as available in the code. References Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15619–15629. Cited by: §2. Bagatella et al. (2026) M. Bagatella, M. Pirotta, A. Touati, A. Lazaric, and A. Tirinzoni TD-JEPA: latent-predictive representations for zero-shot reinforcement learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Balestriero and LeCun (2022) R. Balestriero and Y. LeCun Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Advances in Neural Information Processing Systems, NIPS ’22, Vol. 35, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §1, §2, §5.2, §5. Balestriero and LeCun (2025) R. Balestriero and Y. LeCun LeJEPA: provable and scalable self-supervised learning without the heuristics. External Links: 2511.08544, Link Cited by: §1, §2. Bardes et al. (2022) A. Bardes, J. Ponce, and Y. LeCun VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, External Links: Link Cited by: §2. Belkin and Niyogi (2003) M. Belkin and P. Niyogi Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15 (6), p. 1373–1396. External Links: Document Cited by: §1. Bengio et al. (2004) Y. Bengio, O. Delalleau, N. Le Roux, J. Paiement, P. Vincent, and M. Ouimet Learning eigenfunctions links spectral embedding and kernel pca. Neural Computation 16 (10), p. 2197–2219. External Links: Document Cited by: §1, §5.1, §5. Blier et al. (2021) L. Blier, C. Tallec, and Y. Ollivier Learning successor states and goal-dependent values: a mathematical viewpoint. External Links: 2101.07123, Link Cited by: §5.4, §5. Cristianini et al. (2001) N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola On kernel-target alignment. In Advances in Neural Information Processing Systems, T. Dietterich, S. Becker, and Z. Ghahramani (Eds.), Vol. 14, p. . External Links: Link Cited by: Appendix A. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §B.5. Eysenbach et al. (2022) B. Eysenbach, T. Zhang, S. Levine, and R. Salakhutdinov Contrastive learning as goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: Appendix A. Fu et al. (2021) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4RL: datasets for deep data-driven reinforcement learning. External Links: 2004.07219, Link Cited by: §B.3, §6. Ghosh et al. (2023) D. Ghosh, C. A. Bhateja, and S. Levine Reinforcement learning from passive data via latent intentions. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 11321–11339. External Links: Link Cited by: Appendix A. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber World models. Zenodo. External Links: Document, Link Cited by: §1, §2. Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2555–2565. External Links: Link Cited by: §2. Hafner et al. (2025) D. Hafner, W. Yan, and T. Lillicrap Training agents inside of scalable world models. External Links: 2509.24527, Link Cited by: §2. Hansen et al. (2024) N. Hansen, H. Su, and X. Wang TD-MPC2: scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. HaoChen and Ma (2023) J. Z. HaoChen and T. Ma A theoretical study of inductive biases in contrastive learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §4. HaoChen et al. (2021) J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. External Links: ISBN 9781713845393 Cited by: §1, §2. Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei (Eds.), Proceedings of Machine Learning Research, Vol. 37, Lille, France, p. 448–456. External Links: Link Cited by: §B.5. Kampffmeyer et al. (2018) M. Kampffmeyer, S. Løkse, F. M. Bianchi, R. Jenssen, and L. Livi The deep kernelized autoencoder. Applied Soft Computing 71, p. 816–825. External Links: ISSN 1568-4946, Document, Link Cited by: Appendix A. Kidambi et al. (2020) R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims MOReL: model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 21810–21823. External Links: Link Cited by: §B.3, §6. Kim et al. (2026) M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, and J. Gu Cosmos policy: fine-tuning video models for visuomotor control and planning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Laskin et al. (2021) M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel URLB: unsupervised reinforcement learning benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1, p. . External Links: Link Cited by: Appendix A. Lawson et al. (2026) D. Lawson, A. Hugessen, C. Cloutier, G. Berseth, and K. Khetarpal Self-predictive representations for combinatorial generalization in behavioral cloning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence. Note: Open ReviewVersion 0.9.2 External Links: Link Cited by: §1, §2, §2, §5.3. Li et al. (2026) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, L. Zhang, M. Yu, Z. Gao, N. Xue, B. Zhou, X. Zhu, M. Ding, Y. Shen, and Y. Xu Causal World Modeling for Robot Control. In Proceedings of Robotics: Science and Systems, Sydney, Australia. External Links: Document Cited by: §2. Maes et al. (2026) L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, and R. Balestriero LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. External Links: 2603.19312, Link Cited by: §B.5, §C.4, §1, §1, §2, §4, §6. Mees et al. (2022) O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard CALVIN: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), p. 7327–7334. External Links: Document Cited by: §1, §6. Mur-Labadia et al. (2026) L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, and N. Ballas V-JEPA 2.1: unlocking dense features in video self-supervised learning. In Computer Vision – ECCV 2026, Lecture Notes in Computer Science, Vol. 17005, Cham, p. 671–689. External Links: Document, Link Cited by: §2. Park et al. (2025) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned RL. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, Figure 2, §6. Parker-Holder and Fruchter (2025) J. Parker-Holder and S. Fruchter Genie 3: a new frontier for world models. Note: Google DeepMind Blog External Links: Link Cited by: §2. Rubinstein (1999) R. Y. Rubinstein The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability 1 (2), p. 127–190. External Links: Document Cited by: §4. Schrittwieser et al. (2020) J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver Mastering atari, go, chess and shogi by planning with a learned model. Nature 588 (7839), p. 604–609. External Links: ISSN 1476-4687, Link, Document Cited by: Appendix A. Schwarzer et al. (2021) M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Tang et al. (2023) Y. Tang, Z. D. Guo, P. H. Richemond, B. Avila Pires, Y. Chandak, R. Munos, M. Rowland, M. Gheshlaghi Azar, C. Le Lan, C. Lyle, A. György, S. Thakoor, W. Dabney, B. Piot, D. Calandriello, and M. Valko Understanding self-predictive learning for reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 33632–33656. External Links: Link Cited by: §1. Team HY-World (2026) Team HY-World HY-world 2.0: a multi-modal world model for reconstructing, generating, and simulating 3d worlds. arXiv preprint arXiv:2604.14268. External Links: Document, Link Cited by: §2. Tian et al. (2021) S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine Model-based visual planning with self-supervised functional distances. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Touati and Ollivier (2021) A. Touati and Y. Ollivier Learning one representation to optimize all rewards. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: Appendix A. Touati et al. (2023) A. Touati, J. Rapin, and Y. Ollivier Does zero-shot reinforcement learning exist?. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix A. van den Oord et al. (2019) A. van den Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. External Links: 1807.03748, Link Cited by: §2. Wang and Isola (2022) T. Wang and P. Isola Improved representation of asymmetrical distances with interval quasimetric embeddings. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations, External Links: Link Cited by: §4. Wang et al. (2023) T. Wang, A. Torralba, P. Isola, and A. Zhang Optimal goal-reaching reinforcement learning via quasimetric learning. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 36411–36430. External Links: Link Cited by: §C.4, §4. World Labs Team (2026) World Labs Team Atlas: a world model for spatial intelligence. World Labs Blog. Note: https://w.worldlabs.ai/blog/atlas Cited by: §2. Ye et al. (2026) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, A. N. Malik, K. Lee, W. Liang, N. R. Arachchige, J. Gu, Y. Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y. Xie, J. Wu, Q. Wang, D. Xu, Y. Du, R. Julian, Y. Chebotar, S. Reed, J. Kautz, Y. Zhu, L. Fan, and J. Jang World action models are zero-shot policies. In ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling, External Links: Link Cited by: §2. Yu et al. (2020) T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma MOPO: model-based offline policy optimization. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 14129–14142. External Links: Link Cited by: §B.3, §6. Zbontar et al. (2021) J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny Barlow twins: self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 12310–12320. External Links: Link Cited by: §2. Zhang et al. (2021) A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. Levine Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §1. Zheng et al. (2026) C. Zheng, R. K. Jayanth, and B. Eysenbach Can we really learn one representation to optimize all rewards?. In Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning, External Links: Link Cited by: Appendix A. Zhou et al. (2025) G. Zhou, H. Pan, Y. LeCun, and L. Pinto DINO-WM: world models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Appendix A Extended Related Work Kernel alignment, the normalized Frobenius inner product between a candidate kernel and a target similarity matrix, was introduced by Cristianini et al. (2001) as a criterion for kernel selection. Kampffmeyer et al. (2018) later introduced alignment of an autoencoder’s embedding Gram matrix to a provided kernel. We extend this paradigm by removing the reconstruction term and use action conditioned prediction as an additional shaping force for the latent. Their unsupervised kernel derives similarity from multiscale clustering of input features, which need not reflect the dynamical relationships relevant to control. We instead study both supervised and unsupervised kernels with a more direct relevance for control and world modeling. Furthermore, our trajectory-local formulation leaves cross-trajectory relationships unspecified, allowing flexibility in the learned geometry. Self-Supervised Representations in RL have been explored both as auxiliary objectives in model-free reinforcement learning (Schwarzer et al., 2021), and as part of latent model-based approaches (Schrittwieser et al., 2020; Hansen et al., 2024). A further line of work has focused on representation learning as a central component of practical methods, for instance enabling generalization in behavior cloning (Lawson et al., 2026), or robustness in value-based learning (Zhang et al., 2021). Self-supervised representations are also a core mechanism in unsupervised (Laskin et al., 2021) and zero-shot reinforcement learning (Touati et al., 2023). In particular, contrastive RL (Eysenbach et al., 2022), forward-backward representation (Touati and Ollivier, 2021; Zheng et al., 2026), ICVF (Ghosh et al., 2023) and TD-JEPA (Bagatella et al., 2026) recover representations which can be connected to the successor measure; we highlight a similar connection for spectral alignment with a temporal kernel. Finally, this class of kernels shares the motivation of early works on temporal distance regression (Tian et al., 2021), which however do not extract representations explicitly. Appendix B Additional Experiments B.1 Spectral recovery Theorem 1 characterizes optimal representations as row-normalized spectral embeddings of the teacher kernel, up to an orthogonal transformation. To examine this connection empirically, we measure how well linear probes recover individual kernel eigenvectors from the learned representations (Figure 5). We observe that recovery is strongest for the leading eigenvectors and decreases for those associated with smaller eigenvalues. The lower correlations observed in the state kernel are largely the result of a less even distribution of eigenvalue size, concentrating most of the energy in the lower eigenvector indices. In terms of mean R2R^2 weighted by each eigenvector’s normalized eigenvalue, state gets a 0.965, proprio a 0.987, and depth a 0.992. Figure 5: Linear probe R2R^2 between eigenvectors and embeddings from encoders trained on fully specified state, proprioception, and depth. Underspecified variants and an untrained ViT are included for comparison. B.2 Prediction gradients and downstream performance In Figure 6, we additionally evaluate a variant of SpecWM which does not backpropagate the predictor regression objective through the target (i.e., the next state representations): ℒ=ℒspec−λ∑i(cos(G,sg[H′]))iL=L_spec-λ _i ( (G,sg[H ]) )_i (18) The prediction loss still updates the encoder through the current observation, but no longer through the next observation used as its target. We observe that planning success generally decreases, while probing results do not follow an obvious trend. Figure 6: Effect of a stop-gradient on the prediction target. Bars report the mean over 3 seeds, with whiskers showing one cross-seed standard deviation. The stop-gradient generally lowers planning success and raises object recoverability in some cases while diminishing recoverability in others. B.3 Data ablations It is known that predictors tend to generalize poorly to random actions when trained on expert-only data (Fu et al., 2021; Kidambi et al., 2020; Yu et al., 2020). Here we ablate the failure data (Appendix C.5) by training LeWM and SpecWM with the temporal kernel on the play data alone, at matched updates, frames per update and schedule. As Figure 7 shows, the failure data raises planning success for both models in every environment, but does not explain the difference between them. Figure 7: Effect of the failure data. CEM success (top) and object position probe R2R^2 (bottom) for LeWM and SpecWM (temporal), trained with the failure rollouts (solid, 3 seeds) or on the play data alone (hatched, 1 seed), at matched updates and frames per update. Failure data raises planning success for both models; SpecWM outperforms LeWM in both cases. B.4 CEM cost ablation The two compared world models have different inherent distance metrics between samples in their latent distribution. For the purposes of the main paper, we use the one that fits each latent most naturally (L2 for LeWM and cosine similarity for SpecWM). Figure 8 demonstrates that the choice of measure is not the cause of the planning gains. Figure 8: Effect of the choice of distance measure on CEM planning results. Cosine similarity and L2 distance between current embedding and target embedding are evaluated on both LeWM and SpecWM. The choice of distance measure appears to not be a major contributor to planning ability. Three training seeds are evaluated at their respective final three checkpoints for each setting. B.5 Normalization layer ablation To be faithful to the original LeWorldModel (Maes et al., 2026) implementation, we use BatchNorm (Ioffe and Szegedy, 2015) in the MLP producing the embedding from the ViT (Dosovitskiy et al., 2021) encoder’s CLS token. As SpecWM does not require BatchNorm to effectively optimize its objective, we use a LayerNorm for simplicity. As demonstrated in Figure 9, the choice of norm does not have a substantial impact on planning performance. Figure 9: Effect of choice of normalization layer in the MLP converting the CLS token into an embedding on downstream CEM planning. One seed is used per run and CEM is reported as the average of the final three checkpoints of each run. Appendix C Reproducibility This appendix contains the necessary details to reproduce the results of the paper. For code, see: https://anonymous.4open.science/r/SpecWM/. C.1 Probing To measure to what extent the embeddings contain control-relevant objects, we perform linear probing of position coordinates. We sample 5,000 frames from the validation episodes, which the encoder never trained on, and split full episodes into training and testing sets of 80% and 20% respectively, repeated over five splits. Linear readouts use ridge regression, with all preprocessing and regularization selection performed using training episodes only. Hyperparameters are selected using episode-grouped subsets of the training split. Ridge coefficients are selected from a logarithmic grid of 13 values between 10−210^-2 and 10410^4. We then report held-out R2R^2 of the best ridge coefficient, using the full 80% to fit the linear probe, averaged over the five splits. C.2 Long-range pairs on CALVIN In CALVIN, blocks move only when manipulated, so their positions often remain constant within a short clip. Within-clip temporal supervision therefore provides little signal to distinguish block positions. We augment each clip with 8 observations from the same episode, sampled 300300 to 3,0003,000 raw frames away (roughly 20 seconds to 3 minutes), to capture changes over longer periods. These observations contribute only to the spectral loss, using their original time indices, and require no associated actions. To capture both short and long separations, we average two centered, trace-normalized Laplacian kernels with bandwidths σ1=7 _1=7 and σ2=45 _2=45, measured in sampled observations. We use the same construction for the quasimetric kernel: Kij=12(e−|i−j|/σ1+e−|i−j|/σ2).K_ij= 12 (e^-|i-j|/ _1+e^-|i-j|/ _2 ). (19) Scene and Cube clips already capture the relevant changes, so we retain a single-scale kernel. C.3 Planning via CEM CEM hyperparameters are detailed in Table 2. The base action sampling distribution is a Gaussian with starting mean and variance set according to expert data statistics. SpecWM cost-to-go is evaluated using cosine similarity and LeWM uses L2 distance. Table 2: CEM planning configuration. Parameter Scene Cube CALVIN Candidate sequences 300 300 300 CEM iterations 30 30 30 Elite sequences 30 30 30 Initial variance scale 1.0 1.0 1.0 Planning horizon (blocks) 5 5 5 Environment steps per block 5 5 10 Steps executed before replanning 15 15 30 Number of evaluation tasks 100 100 100 Goal offset (steps) 25 25 50 Evaluation budget (steps) 60 60 100 C.4 Training Architecture. The JEPA architecture follows that of LeWorldModel and is consistent across objectives aside from normalization scheme. LeWorldModel uses BatchNorm in its projection head in line with Maes et al. (2026), whereas SpecWM uses LayerNorm. Their remaining shared architectural settings are summarized in Table 3. Table 3: Shared visual-encoder and action-conditioned dynamics architecture. Component Configuration Visual encoder Randomly initialized ViT-T/14 with output dimension 192 Projection head MLP 192→2048→192192→ 2048→ 192 Action encoder 1×11× 1 Conv1D followed by an MLP with hidden dimension 768 and output dimension 192 Dynamics predictor 6-layer action-conditioned transformer Attention 16 heads with head dimension 64 Transformer MLP dimension 2048 Dropout 0.1 Observation-history length 3 sampled observations Table 4: Environment-dependent action representation. Each action block contains all low-level actions between two consecutive sampled observations. Environment Frame skip Raw action dim. Action-block dim. OGBench Scene 5 5 25 OGBench Cube 5 5 25 CALVIN 10 7 70 Optimization and training. Optimization hyperparameters are detailed in Table 5. The unusual learning-rate values result from square-root batch-size scaling of base rates 5×10−55× 10^-5 for AdamW and 10−310^-3 for Muon. For CALVIN, SpecWM additionally encodes 8 distant observations per clip for the spectral loss, giving 24×(23+8)=74424×(23+8)=744 total images per update. LeWorldModel uses 552 images per update in all environments. Table 5: Optimization and student-training settings. Setting Configuration Peak learning rates 1.461×10−31.461× 10^-3 for matrices, 2.922×10−42.922× 10^-4 for convolutions, and 7.30×10−57.30× 10^-5 for biases and normalization parameters MuonAdamW (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), ϵ=10−8ε=10^-8, Muon momentum 0.950.95, Nesterov updates, and RMS matching Regularization Weight decay 10−310^-3 on weights; no decay on biases or normalization parameters; gradient clipping at 1.01.0 SpecWM 33,375 updates; 552 base clip frames per update (24×2324× 23); λspec=1 _spec=1 for Cube and Scene and 22 for CALVIN; cosine prediction-anchor weight 22. LeWorldModel 33,375 updates; 552 frames per update (138×4138× 4); prediction-MSE weight 11 and SIGReg weight 0.090.09 Quasimetric training. We train the quasimetric teachers using a separate ViT-T/14 followed by a projection and an IQE head. The IQE head’s 384-dimensional output is divided into 24 IQE groups of dimension 16, producing an asymmetric directed distance. The training objective expands distances between distinct observations while constraining consecutive observations (Wang et al., 2023): ℒpush _push =−i≠j[d(zi,zj)], =-E_i≠ j\! [d(z_i,z_j) ], ℒstep _step =t[max(0,d(zt,zt+1)−1)2], =E_t\! [ \! (0,d(z_t,z_t+1)-1 )^2 ], ℒQRL _QRL =ℒpush+λ(ℒstep−ϵ2). =L_push+λ (L_step-ε^2 ). The dual variable is initialized to 11 and updated with learning rate 0.050.05 toward a constraint value of ϵ=0.25ε=0.25, while being clipped to [0,50][0,50]. Teachers are trained for 9,500 updates using 552 frames per update (2424 clips of 2323 frames). C.5 Data For all environments, we supplement the official datasets with failure data to aid in predictor training using noised actions. For each episode or segment i of length LiL_i, we sample an onset ti∼Uniform(⌊0.05Li⌋,…,⌊0.90Li⌋)t_i ( \ 0.05L_i ,…, 0.90L_i \ ) and alternate the noise scale between σi=0.3 _i=0.3 and σi=0.6 _i=0.6. The replayed action is a~t=at,t<ti,clip(at+σiϵt,−1,1),t≥ti,ϵt∼(0,I). a_t= casesa_t,&t<t_i,\\ clip(a_t+ _i _t,-1,1),&t≥ t_i, cases _t (0,I). Each replay is initialized from its recorded simulator state and follows the recorded open-loop actions exactly before the noise onset. For Scene and Cube, noise is applied to all five action dimensions. For CALVIN, noise is applied only to the six continuous motion dimensions, while the binary gripper command is retained from the recorded trajectory. The resulting Scene and Cube datasets each contain 1,000,000 frames from the official dataset and 1,000,000 frames from our failure rollouts. The CALVIN dataset contains 1,795,045 official frames and 1,793,322 failure frames. Appendix D Full Proofs The proofs below follow the order of the stated theorems in Section 5. D.1 Proof of Theorem 1 Let H∈ℝn×dH ^n× d be a free embedding table with nonzero rows h1,…,hnh_1,…,h_n, and let KαK^α be the centered, tempered target. The loss rescales each row to the length the target prescribes, forms the Gram matrix, and compares it to the target after double centering: h¯i=Kiiαhi‖hi‖,S=H¯H¯⊤,ℒspec(H)=‖CSC−Kα‖F2, h_i= K^α_i\, h_i h_i , S= H H , _spec(H)= CSC-K^α _F^2, (20) with C=I−1n⊤C=I- 1n11 . Note that the rescaling pins the diagonal of S to that of the target, Sii=‖h¯i‖2=Kiiα,S_i= h_i ^2=K^α_i, (21) for every H. Let K~=CKC K=CKC have eigenvalues λ1≥⋯≥λn _1≥…≥ _n with unit eigenvectors u1,…,unu_1,…,u_n, and let m≥1m≥ 1 be the number of directions the tempering keeps (the minimum of the count of positive λℓ _ and the truncation count). Since the tempered spectrum vanishes outside these m directions, the target becomes Kα=∑ℓ=1msℓ2uℓuℓ⊤,sℓ=(nλℓα∑j≤mλjα)1/2.K^α= _ =1^ms_ ^2\,u_ u_ , s_ = ( n _ ^α _j≤ m _j^α )^1/2. (22) Three properties of the target are used below: Kα⪰0,rankKα=m,Kα=0.K^α 0, ^α=m, K^α1=0. (23) The first two come from equation 22, whose m terms have orthogonal directions uℓu_ and positive weights sℓ2s_ ^2. For the third, C=0C1=0 gives K~=0 K1=0, and every uℓu_ in equation 22 has λℓ>0 _ >0, so each is orthogonal to 1. As assumed in Theorem 1, the target has strictly positive diagonal entries. Lemma 0.1. ℒspec(H)=0L_spec(H)=0 if and only if S=KαS=K^α. Proof. If S=KαS=K^α the loss is ‖CKαC−Kα‖F2 CK^αC-K^α _F^2. Since KαK^α is symmetric and Kα=0K^α1=0, we have ⊤Kα=01 K^α=0. Thus, CKα=Kα−1n(Kα)⊤=KαCK^α=K^α- 1n1(K^α1) =K^α and likewise KαC=Kα−1n(Kα)⊤=KαK^αC=K^α- 1n(K^α1)1 =K^α. Hence CKαC=KαCK^αC=K^α and the loss is 00. Conversely, suppose CSC=KαCSC=K^α. Centering is not injective, so this alone does not give S=KαS=K^α. Accordingly, we use the diagonal constraint equation 21. Since C is symmetric and idempotent, trKα ^α =tr(CSC) =tr(CSC) =tr(SC2) =tr(SC^2) =tr(SC) =tr(SC) =trS−1n 1⊤S. =trS- 1n\,1 S1. Since trS=trKαtrS=trK^α by equation 21, we have that ⊤S=01 S1=0. Expanding, we see that ⊤S=⊤H¯H¯⊤=‖H¯⊤‖2=01 S1=1 H H 1= H 1 ^2=0, which forces H¯⊤=0 H 1=0. Thus, we have that H¯H¯⊤=S=0 H H 1=S1=0. Since S is symmetric, S=0S1=0 also gives ⊤S=01 S=0. Hence CS=SC=SCS=SC=S, and therefore S=CSC=KαS=CSC=K^α. ∎ Lemma 0.2. Assume m≤dm≤ d. Let Um=[u1,…,um]U_m=[u_1,…,u_m] contain the retained orthonormal eigenvectors of the centered kernel K~=CKC K=CKC. Define Hspec∈ℝn×dH_spec ^n× d by assigning sample i the row (Hspec)i,:=(s1u1(i),…,smum(i),0,…,0⏟d−m),(H_spec)_i,:= (s_1u_1(i),…,s_mu_m(i), 0,…,0_d-m ), where uℓ(i)u_ (i) is the i-th entry of eigenvector uℓu_ and the scales sℓs_ are given by equation 22. Then S=KαS=K^α if and only if H=diag(c)HspecQH=diag(c)H_specQ for some c∈ℝ>0nc _>0^n and orthogonal Q∈ℝd×dQ ^d× d. Proof. First, we will establish that HspecHspec⊤=KαH_specH_spec =K^α. By equation 2 we have that Hspec=Uddiag(s1,…,sd)H_spec=U_ddiag(s_1,…,s_d). Thus, HspecHspec⊤ H_specH_spec =Umdiag(s12,…,sm2)Um⊤ =U_mdiag(s_1^2,…,s_m^2)\,U_m =∑ℓ=1msℓ2uℓuℓ⊤ = _ =1^ms_ ^2\,u_ u_ =Kα, =K^α, where the last equality is equation 22. The diagonal of a Gram matrix lists the squared row norms, so ‖(Hspec)i‖2=Kiiα (H_spec)_i ^2=K^α_i. Second, we use a standard fact about Gram matrices: if BB⊤=B′B′⊤B =B B for B,B′∈ℝn×dB,B ^n× d, then B=B′QB=B Q for some orthogonal Q∈ℝd×dQ ^d× d. Suppose S=KαS=K^α. Then H¯H¯⊤=HspecHspec⊤ H H =H_specH_spec , so H¯=HspecQ H=H_specQ for some orthogonal Q. By equation 20, hi=cih¯ih_i=c_i h_i with ci=‖hi‖/Kiiα>0c_i= h_i / K^α_i>0, so H=diag(c)H¯=diag(c)HspecQH=diag(c)\, H=diag(c)\,H_specQ. Suppose instead H=diag(c)HspecQH=diag(c)\,H_specQ. Since Q preserves lengths, row i of HspecQH_specQ has the same length as row i of HspecH_spec, namely Kiiα>0 K^α_i>0. The rows of H are therefore nonzero, and rescaling them to length Kiiα K^α_i removes the factor cic_i, so H¯=HspecQ H=H_specQ. Hence S=H¯H¯⊤=HspecQQ⊤Hspec⊤=HspecHspec⊤=KαS= H H =H_specQ H_spec =H_specH_spec =K^α. ∎ Suppose m≤dm≤ d. Then minHℒspec(H)=0 _HL_spec(H)=0, attained exactly at H⋆=diag(c)HspecQ,H =diag(c)\,H_spec\,Q, (24) with c∈ℝ>0nc ^n_>0, Q∈ℝd×d orthogonalQ ^d× d orthogonal, and HspecH_spec the truncated spectral embedding of equation 2. Proof of Theorem 1. By Lemma 0.1, ℒspec(H)=0L_spec(H)=0 if and only if S=KαS=K^α, and by Lemma 0.2 this holds if and only if H has the form equation 10. The set of such H is nonempty, since it contains HspecH_spec itself (c=c=1, Q=IQ=I), so the minimum is attained and equals 00. ∎ D.2 Proof of Theorem 2 We next connect this embedding geometry to linear probing performance. In particular, a kernel constructed from label relationships can make the labels themselves linearly recoverable from the embeddings, without directly regressing encoder outputs onto them. We formalize this below for row-normalized and centered vector-valued labels Y. Let Y∈ℝn×pY ^n× p satisfy ⊤Y=01 Y=0, row norms ‖yi‖=1\|y_i\|=1, and rank(Y)≤drank(Y)≤ d. For K=YY⊤K=Y , α=1α=1, and all positive spectral directions retained, every global minimizer H⋆H of ℒspecL_spec subject to ‖hi‖=d\|h_i\|= d satisfies minW∈ℝd×p‖Y−H⋆W‖F2=0. _W ^d× p\|Y-H W\|_F^2=0. (25) Proof of Theorem 2. We will construct W that satisfies 25. Since Y is centered and its rows have unit norm, we have that K~=CYYTC=YYT K=CYY^TC=Y^T with tr(K~)=ntr( K)=n. With α=1α=1 and all positive directions retained, the target is therefore Kα=YYTK^α=Y^T, with Kiiα=1K^α_i=1. Write the compact SVD of Y and YYTYY^T as Y=UmΣmVT,YYT=UmΣm2UmT.Y=U_m _mV^T, Y^T=U_m ^2_mU^T_m. To pad the m-dimensional spectral embeddings to the encoder’s d dimensions, let Em=[Im 0]∈ℝm×dE_m=[I_m\;\;0] ^m× d. The spectral embedding is Hspec=UmΣmEm.H_spec=U_m _mE_m. The rows of HspecH_spec have unit norm because HspecHspec⊤=KαH_specH_spec =K^α and Kiiα=1K^α_i=1. Multiplying by d d makes each row satisfy the required norm d d, while preserving zero loss by Theorem 1. Since the loss is nonnegative, every global minimizer under this row-norm constraint must therefore also achieve zero loss. Further by Theorem 1, each such minimizer has the form H⋆=diag(c)HspecQ.H =diag(c)\,H_specQ. Since Q preserves lengths, the RMS constraint forces ci=dc_i= d for every i. Hence, we have that H⋆=dUmΣmEmQ.H = dU_m _mE_mQ. We can now define W⋆W : W⋆=1dQTEmTVT.W = 1 dQ^TE^T_mV^T. (26) Using QQT=IdQQ^T=I_d and EmEmT=ImE_mE_m^T=I_m, we have that H⋆W⋆ H W =UmΣmEmQQTEmTVT =U_m _mE_mQ^TE^T_mV^T =UmΣmVT =U_m _mV^T =Y. =Y. Therefore minW‖Y−H⋆W‖F2=0 _W\|Y-H W\|_F^2=0. ∎ D.3 Proof of Theorem 3 Under the coverage assumption, fixed-norm encoder outputs, and the target-diagonal rescaling in Equation 6, if each trajectory’s target matrix KαK^α has strictly positive diagonal entries almost surely and [ℒspec]=0E\,[L_spec]=0, then for almost every clip and every frame i, fθ((si,n))=fθ((si,n′))f_θ((s_i,n))=f_θ((s_i,n )) for almost every pair (n,n′)(n,n ) drawn independently from ν(⋅∣si)ν(· s_i). Proof of Theorem 3. We will show that coverage allows independent nuisance draws while preserving the target inner products. Centering then forces their rescaled embeddings to sum to zero, so changing one nuisance draw cannot change its embedding. Since the loss is nonnegative, ℒspec=0E\,L_spec=0 implies zero loss almost surely. Lemma 0.1 therefore gives S=KαS=K^α almost surely. Fix a task-relevant trajectory and teacher metadata Z for which this equality holds with conditional probability one and the coverage assumption 5.3 holds. Let q be the number of frames. For this fixed Z, define bi(n)=Kiiαdfθ((si,n)).b_i(n)= K^α_id\,f_θ((s_i,n)). These are the rescaled embeddings used in the spectral loss. Since the encoder outputs have norm d d, we have ‖bi(n)‖2=Kiiα\|b_i(n)\|^2=K^α_i. Now draw auxiliary nuisance values n^1,…,n^q n_1,…, n_q independently, with n^i∼ν(⋅∣si) n_i ν(· s_i). For each i≠ji≠ j, coverage implies bi(n^i)⊤bj(n^j)=Kijαalmost surely.b_i( n_i) b_j( n_j)=K^α_ij surely. If the set of pairs violating this equality had positive probability under the independent draws, assumption (A1) would assign it positive conditional probability in the trajectory, causing non-zero loss. For diagonal entries, we also have bi(n^i)⊤bi(n^i)=Kiiαb_i( n_i) b_i( n_i)=K^α_i by the definition of bi.b_i. Together with the off-diagonal equalities above, this gives bi(n^i)⊤bj(n^j)=Kijαfor all i,jalmost surely.b_i( n_i) b_j( n_j)=K^α_ij all i,j surely. Using this equality together with the fact that KαK^α is centered, ‖∑j=1qbj(n^j)‖2=∑j,k=1qbj(n^j)⊤bk(n^k)=∑j,k=1qKjkα=⊤Kα=0almost surely. \| _j=1^qb_j( n_j) \|^2= _j,k=1^qb_j( n_j) b_k( n_k)= _j,k=1^qK^α_jk=1 K^α1=0 surely. Fix a frame i and replace only n^i n_i by an independent draw n^i′∼ν(⋅∣si) n_i ν(· s_i). The new tuple has the same distribution, so both sums below vanish almost surely: bi(n^i)+∑j≠ibj(n^j)=0,bi(n^i′)+∑j≠ibj(n^j)=0.b_i( n_i)+ _j≠ ib_j( n_j)=0, b_i( n_i )+ _j≠ ib_j( n_j)=0. Subtracting gives bi(n^i)=bi(n^i′)b_i( n_i)=b_i( n_i ) almost surely. Since Kiiα>0K^α_i>0, the rescaling can be cancelled, yielding fθ((si,n^i))=fθ((si,n^i′))almost surely.f_θ((s_i, n_i))=f_θ((s_i, n_i )) surely. The two nuisance draws are independent with distribution ν(⋅∣si)ν(· s_i), which is the claimed invariance. The argument applies to every frame and almost every Z. ∎ D.4 Proof of Theorem 4 We resume from the assumptions stated in Section 5.4, from the population-level variant of the spectral alignment objective ℒspecpop(h)=τ,i,j[(s(xi,xj)−γ|i−j|)2],L_spec^pop(h)\;=\;E_τ,i,j [ (s(x_i,x_j)-γ^|i-j| )^2 ], (27) with τ:=(x0,…,xT−1)∼τ:=(x_0,…,x_T-1) and (i,j)∼(0,…,T−12)(i,j) (\0,…,T-1\^2), and from the expected teacher similarity of the state pair RT(x,x′)=τ,i,j[γ|i−j||xi=x,xj=x′].R_T(x,x )=E_τ,i,j[γ^|i-j||x_i=x,x_j=x ]. (28) We can then apply a bias-variance decomposition and rewrite the loss as: ℒspecpop(h) _spec^pop(h) =τ,i,j[(s(xi,xj)−γ|i−j|)2] =E_τ,i,j [(s(x_i,x_j)-γ^|i-j|)^2 ] (29) =τ,i,j[((s(xi,xj)−RT(xi,xj))+(RT(xi,xj)−γ|i−j|))2] =E_τ,i,j [ ((s(x_i,x_j)-R_T(x_i,x_j) )+ (R_T(x_i,x_j)-γ^|i-j|) )^2 ] (30) =x,x′[(τ,i,j)∼p(⋅∣xi=x,xj=x′)[(s(xi,xj)−RT(xi,xj))2] =E_x,x [E_(τ,i,j) p(· x_i=x,\,x_j=x ) [ (s(x_i,x_j)-R_T(x_i,x_j) )^2 ] (31) +(τ,i,j)∼p(⋅∣xi=x,xj=x′)[(RT(xi,xj)−γ|i−j|)2] +E_(τ,i,j) p(· x_i=x,\,x_j=x ) [ (R_T(x_i,x_j)-γ^|i-j| )^2 ] (32) +2(τ,i,j)∼p(⋅∣xi=x,xj=x′)[(s(xi,xj)−RT(xi,xj))(RT(xi,xj)−γ|i−j|)]], +2E_(τ,i,j) p(· x_i=x,\,x_j=x ) [ (s(x_i,x_j)-R_T(x_i,x_j) ) (R_T(x_i,x_j)-γ^|i-j| ) ] ], (33) where x,x′E_x,x is over pairs of observations sampled from the dataset. The second term is not dependent on h and the third term fades as (s(xi,xj)−RT(xi,xj))(s(x_i,x_j)-R_T(x_i,x_j)) is constant upon conditioning on (xi=x,xj=x′)(x_i=x,x_j=x ), and (τ,i,j)∼p(⋅∣xi=x,xj=x′)[RT(xi,xj)−γ|i−j|]=0E_(τ,i,j) p(· x_i=x,\,x_j=x )[R_T(x_i,x_j)-γ^|i-j|]=0; thus minimizing ℒspecpop(h)L_spec^pop(h) is equivalent to minimizing τ,i,j[(s(xi,xj)−RT(xi,xj))2]E_τ,i,j [(s(x_i,x_j)-R_T(x_i,x_j))^2 ]. We will now study the target RTR_T in the limit of long trajectories, and connect it to the successor measure MβM^β of the behavioral policy πβ _β. Under the setup above, with a finite state space, an irreducible, aperiodic Markov chain P induced by πβ _β with stationary distribution μ and x0∼μx_0 μ, for all x,x′∈x,x , as T→∞T→∞, TRT(x,x′)⟶Mβ(x,x′)μ(x′)+Mβ(x′,x)μ(x)−[x=x′]μ(x).T\,R_T(x,x )\; \; M^β(x,x )μ(x )+ M^β(x ,x)μ(x)- 1[x=x ]μ(x). Moreover, if the induced Markov chain is reversible, i.e., μ(x)P(x′|x)=μ(x′)P(x|x′)μ(x)P(x |x)=μ(x )P(x|x ), TRT(x,x′)⟶2Mβ(x,x′)μ(x′)−[x=x′]μ(x′).T\,R_T(x,x )\; \; 2M^β(x,x )μ(x )- 1[x=x ]μ(x ). Proof of Theorem 4. By definition, TRT(x,x′)=N(x,x′)D(x,x′)TR_T(x,x )= N(x,x )D(x,x ) with N(x,x′):=Tτ,i,j[γ|i−j| 1[xi=x,xj=x′]],D(x,x′):=Eτ,i,j[[xi=x,xj=x′]].N(x,x ):=T\;E_τ,i,j [γ^|i-j|\,1[x_i=x,x_j=x ] ], D(x,x ):=E_τ,i,j [1[x_i=x,x_j=x ] ]. (34) We will study the numerator and the denominator separately. Numerator. Let us consider the index difference k:=j−ik:=j-i, which is distributed as p(k)=(T−|k|)/T2p(k)=(T-|k|)/T^2 for |k|≤T−1|k|≤ T-1, and denote the probability that xi=x_i=x and xj=x′x_j=x as qk(x,x′):=Pr(xi=x,xj=x′∣j−i=k)=μ(x)Pk(x,x′),k≥0,μ(x′)P|k|(x′,x),k<0.q_k(x,x ):= (x_i=x,x_j=x j-i=k)= casesμ(x)P^k(x,x ),&k≥ 0,\\ μ(x )P^|k|(x ,x),&k<0. cases We then have N(x,x′) N(x,x ) =Tτ,i,j[γ|i−j| 1[xi=x,xj=x′]] =T\;E_τ,i,j [γ^|i-j|\,1[x_i=x,x_j=x ] ] (35) =Ti,j[γ|i−j|Pr(xi=x,xj=x′|i,j)] =T\;E_i,j [γ^|i-j| (x_i=x,x_j=x |i,j) ] (36) =Ti,j[γ|i−j|qj−i(x,x′)] =T\;E_i,j [γ^|i-j|q_j-i(x,x ) ] (37) =k=j−iTk[γ|k|qk(x,x′)] k=j-i=T\;E_k [γ^|k|q_k(x,x ) ] (38) =T∑k=−T+1T−1p(k)γ|k|qk(x,x′) =T\; _k=-T+1^T-1p(k)γ^|k|q_k(x,x ) (39) =T∑k=−T+1−1p(k)γ|k|qk(x,x′)+T∑k=0T−1p(k)γ|k|qk(x,x′) =T\; _k=-T+1^-1p(k)γ^|k|q_k(x,x )+T _k=0^T-1p(k)γ^|k|q_k(x,x ) (40) =T∑k=−T+1−1T+kT2γ|k|μ(x′)P|k|(x′,x)+T∑k=0T−1T−kT2γkμ(x)Pk(x,x′) =T\; _k=-T+1^-1 T+kT^2γ^|k|μ(x )P^|k|(x ,x)+T _k=0^T-1 T-kT^2γ^kμ(x)P^k(x,x ) (41) =∑k=−T+1−1T+kTγ|k|μ(x′)P|k|(x′,x)+∑k=0T−1T−kTγkμ(x)Pk(x,x′) = _k=-T+1^-1 T+kTγ^|k|μ(x )P^|k|(x ,x)+ _k=0^T-1 T-kTγ^kμ(x)P^k(x,x ) (42) =k′=−k∑k′=1T−1T−k′Tγk′μ(x′)Pk′(x′,x)+∑k=0T−1T−kTγkμ(x)Pk(x,x′) k =-k= _k =1^T-1 T-k Tγ^k μ(x )P^k (x ,x)+ _k=0^T-1 T-kTγ^kμ(x)P^k(x,x ) (43) =μ(x′)fT(x′,x)+μ(x)fT(x,x′)−μ(x)[x=x′], =μ(x )f_T(x ,x)+μ(x)f_T(x,x )-μ(x)1[x=x ], (44) where fT(x,x′)=∑k=0T−1T−kTγkPk(x,x′)f_T(x,x )= _k=0^T-1 T-kTγ^kP^k(x,x ). Each summand is bounded by γkγ^k and ∑k=0∞γk<∞ _k=0^∞γ^k<∞, so by dominated convergence fT(x,x′)→∑k=0∞γkPk(x,x′)=Mβ(x,x′).f_T(x,x )→ _k=0^∞γ^kP^k(x,x )=M^β(x,x ). (45) Thus, we conclude that N(x,x′)→μ(x′)Mβ(x′,x)+μ(x)Mβ(x,x′)−μ(x)[x=x′].N(x,x )→μ(x )M^β(x ,x)+μ(x)M^β(x,x )-μ(x)1[x=x ]. (46) Denominator. We can deal with the denominator in a similar fashion to the numerator. D(x,x′) D(x,x ) =τ,i,j[[xi=x,xj=x′]] =E_τ,i,j [1[x_i=x,x_j=x ] ] (47) =i,jqj−i(x,x′) =E_i,jq_j-i(x,x ) (48) =k=j−ikqk(x,x′) k=j-i=\;E_kq_k(x,x ) (49) =∑k=−T+1−1p(k)qk(x,x′)+∑k=0T−1p(k)qk(x,x′) = _k=-T+1^-1p(k)q_k(x,x )+ _k=0^T-1p(k)q_k(x,x ) (50) =∑k=−T+1−1T+kT2μ(x′)P|k|(x′,x)+∑k=0T−1T−kT2μ(x)Pk(x,x′) = _k=-T+1^-1 T+kT^2μ(x )P^|k|(x ,x)+ _k=0^T-1 T-kT^2μ(x)P^k(x,x ) (51) =k′=−k∑k′=1T−1T−k′T2μ(x′)Pk′(x′,x)+∑k=0T−1T−kT2μ(x)Pk(x,x′) k =-k= _k =1^T-1 T-k T^2μ(x )P^k (x ,x)+ _k=0^T-1 T-kT^2μ(x)P^k(x,x ) (52) =μ(x′)gT(x′,x)+μ(x)gT(x,x′)−1Tμ(x)[x=x′], =μ(x )g_T(x ,x)+μ(x)g_T(x,x )- 1Tμ(x)1[x=x ], (53) with gT(x,x′):=∑k=0T−1T−kT2Pk(x,x′)g_T(x,x ):= _k=0^T-1 T-kT^2P^k(x,x ). We have that 0≤T−kT2≤1T0≤ T-kT^2≤ 1T summing to T+12T→12 T+12T→ 12; moreover, due to irreducibility and aperiodicity, Pk(x,x′)→μ(x′)P^k(x,x )→μ(x ). Hence, by the Toeplitz lemma, gT(x,x′)→μ(x′)2g_T(x,x )→ μ(x )2, and finally D(x,x′)→μ(x)μ(x′)D(x,x )→μ(x)μ(x ). Dividing the limits of N(x,x′)N(x,x ) and D(x,x′)D(x,x ) concludes the proof. This concludes our analysis: embedding similarity is encouraged to regress a target that contains a symmetrized likelihood ratio, holding the behavioral successor measure as its numerator. D.5 Additional result: kernel dominance We have established that a label kernel can make a particular target linearly recoverable from embeddings. We now ask when one kernel produces a representation that is at least as useful as another for every linear probe task. This provides a notion of superiority of kernels. Recall that KαK^α is the centered, tempered target. At the optimum, the rescaled embeddings satisfy H¯H¯⊤=Kα H H =K^α. However, a downstream probe reads the encoder output H⋆H , whose rows have fixed norm d d. We therefore first express the geometry of H⋆H directly in terms of KαK^α. Since KαK^α is the Gram matrix of H¯ H, its diagonal entry KiiαK^α_i is the squared length of row h¯i h_i. Dividing each inner product by the lengths of the two rows gives their cosine similarity. Define Kikcos=KikαKiiαKkkα.K^cos_ik= K^α_ik K^α_i K^α_k. Thus KcosK^cos records the pairwise similarities after removing the row lengths prescribed by KαK^α. In particular, Kiicos=1K^cos_i=1. Recall also that the loss rescales each encoder output according to h¯i=Kiiαhi⋆d h_i= K^α_i h _i d. At a global minimizer, it follows that Kikα=⟨h¯i,h¯k⟩=KiiαKkkαd⟨hi⋆,hk⋆⟩.K^α_ik= h_i, h_k = K^α_i K^α_kd h _i,h _k . Consequently, 1dH⋆(H⋆)⊤=Kcos 1dH (H ) =K^cos. This identifies KcosK^cos as the cosine Gram matrix of the encoder output. A scalar linear probe predicts y as H⋆wH w, a linear combination of the columns of H⋆H . Thus the exactly recoverable targets form col(H⋆)col(H ). Using the identity col(HH⊤)=col(H)col(H )=col(H) and the relation H⋆(H⋆)⊤=dKcosH (H ) =dK^cos, we obtain col(H⋆)=col(Kcos).col(H )=col(K^cos). For j∈1,2j∈\1,2\, let KjαK^α_j be the centered, tempered target constructed from KjK_j, with positive diagonal entries and rank at most d. Let Hj⋆H _j be any corresponding global minimizer subject to ‖hi‖=d h_i = d. We say that K1K_1 dominates K2K_2 for linear probing if minW‖Y−H1⋆W‖F2≤minW‖Y−H2⋆W‖F2 _W Y-H _1W ^2_F≤ _W Y-H _2W ^2_F (54) for every vector-valued target Y. Dominance is strict if the inequality is strict for at least one target. Theorem 5 (Kernel dominance). Under the assumptions above, K1K_1 dominates K2K_2 if and only if col(K2cos)⊆col(K1cos).col(K^cos_2) (K^cos_1). (55) Dominance is strict if and only if this inclusion is strict. Proof. Suppose first that col(K2cos)⊆col(K1cos).col(K^cos_2) (K^cos_1). Since col(Hj⋆)=col(Kjcos)col(H _j)=col(K^cos_j), every column of H2⋆H _2 is a linear combination of the columns of H1⋆H _1. Hence there exists T∈ℝd×dT ^d× d such that H2⋆=H1⋆TH _2=H _1T. In particular, any prediction made from H2⋆H _2 can also be made from H1⋆H _1. To see this explicitly, let W2⋆W _2 minimize the probe error for a target Y. We have that minW‖Y−H1⋆W‖F2 _W Y-H _1W ^2_F ≤‖Y−H1⋆TW2⋆‖F2 ≤ Y-H _1TW _2 ^2_F =‖Y−H2⋆W2⋆‖F2 = Y-H _2W _2 ^2_F =minW‖Y−H2⋆W‖F2. = _W Y-H _2W ^2_F. Thus K1K_1 dominates K2K_2. Conversely, suppose K1K_1 dominates K2K_2. Take any scalar target y∈col(K2cos)y (K^cos_2). This target is exactly recoverable from H2⋆H _2, so dominance implies 0≤minw‖y−H1⋆w‖2≤minw‖y−H2⋆w‖2=0.0≤ _w y-H _1w ^2≤ _w y-H _2w ^2=0. It is therefore also exactly recoverable from H1⋆H _1, giving y∈col(H1⋆)=col(K1cos)y (H _1)=col(K^cos_1). Since this holds for every y∈col(K2cos),y (K^cos_2), the required inclusion follows. If the inclusion is strict, choose a target y∈col(K1cos)∖col(K2cos)y (K^cos_1) (K^cos_2). Its optimal probe error is zero under K1K_1 and positive under K2K_2, establishing strict dominance. If the column spaces coincide, their optimal probe errors agree for every target. Thus strict dominance requires strict inclusion. ∎