Paper deep dive
Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Models
Pranaya Jajoo, Harshit Sikchi, Siddhant Agarwal, Amy Zhang, Scott Niekum, Martha White
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:26:11 AM
Summary
The paper introduces Regularized Latent Dynamics Prediction (RLDP), a method for learning state representations for Behavioral Foundation Models (BFMs) in zero-shot reinforcement learning. RLDP addresses the issue of feature collapse in latent dynamics prediction by incorporating an orthogonality regularization term, enabling the model to maintain feature diversity and perform effectively in low-coverage scenarios.
Entities (5)
Relation Signals (3)
Regularized Latent Dynamics Prediction → uses → Orthogonality Regularization
confidence 100% · RLDP adds a simple orthogonality regularization to maintain feature diversity.
Behavioral Foundation Models → enables → Zero-shot Reinforcement Learning
confidence 95% · Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task.
Regularized Latent Dynamics Prediction → improves → Behavioral Foundation Models
confidence 90% · RLDP can match or surpass state-of-the-art complex representation learning methods for zero-shot RL.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task. These methods, however, are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing state features, making the choice of state features crucial to the expressivity of the BFM. As a result, BFMs are trained using a variety of complex objectives and require sufficient dataset coverage, to train task-useful spanning features. In this work, we examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone is prone to increasing state-feature similarity, and subsequently reducing span. We propose an approach, Regularized Latent Dynamics Prediction (RLDP), that adds a simple orthogonality regularization to maintain feature diversity and can match or surpass state-of-the-art complex representation learning methods for zero-shot RL. Furthermore, we empirically show that prior approaches perform poorly in low-coverage scenarios where RLDP still succeeds.
Tags
Links
- Source: https://arxiv.org/abs/2603.15857v1
- Canonical: https://arxiv.org/abs/2603.15857v1
Trouble viewing inline? Open PDF directly →
Full Text
83,198 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2026 REGULARIZEDLATENTDYNAMICSPREDICTION IS ASTRONGBASELINE FOR BEHAVIORALFOUNDATIONMODELS Pranaya Jajoo 1,2∗ Harshit Sikchi 4,† Siddhant Agarwal 4,† Amy Zhang 4 Scott Niekum 5 Martha White 1,2,3 1 Department of Computing Science, University of Alberta, Canada 2 Alberta Machine Intelligence Institute (Amii) 3 Canada CIFAR AI Chair 4 The University of Texas at Austin 5 University of Massachusetts Amherst † Equal contribution ABSTRACT Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task. These methods, however, are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existingstate features, making the choice of state features crucial to the expressivity of the BFM. As a result, BFMs are trained using a variety of complex objectives and require sufficient dataset coverage, to train task useful spanning features. In this work, we examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone is prone to increasing state-feature similarity, and subsequently reducing span. We propose an approach, Regularized Latent Dynamics Prediction (RLDP), that adds a simple orthogonality regularization to maintain feature diversity and can match or surpass state-of-the- art complex representation learning methods for zero-shot RL. Furthermore, we empirically show that prior approaches perform poorly in low-coverage scenarios whereRLDPstill succeeds. 1INTRODUCTION Zero-shot reinforcement learning (RL) (Touati et al., 2022) is a problem setting where we learn an agent that can solveanytask in the environment without any additional training or planning, after an initial pretraining phase. Zero-shot RL has significant practical potential in developing generalist agents with wide applicability. For instance, robotics applications, like robotic manipulation or drone navigation, often require agents to solve a wide variety of unknown tasks. A general-purpose household robot needs to possess the capability to flexibly adapt to various household chores without explicit training for each new task. Behavioral Foundation Models (BFMs) have been shown to be promising for zero-shot RL (Touati et al., 2022; Agarwal et al., 2024). BFMs are trained on a dataset of reward-free interactions, with the aim to provide a near-optimal policy for a wide class of reward functions without additional learning or training during test-time. BFMs are trained by a) learning a state representationφ:s→R d , and b) learning a policyπconditioned on a latent vectorz∈R d , where thezcan be seen as a task embedding for rewardr(s) =φ(s) ⊤ z. In this way, the BFM consists of a space of policies, where different policies can be extracted by querying the learned policy using differentz. At test time, given any reward functionr test (s), the near-optimal policyπ z r test is obtained zero-shot by solving forz r test such thatr test (s)≈φ(s) ⊤ z r test . This assumption on the reward is also used for successor ∗ Correspondence topranayajajoo@ualberta.ca 1 arXiv:2603.15857v1 [cs.AI] 16 Mar 2026 Published as a conference paper at ICLR 2026 features (Barreto et al. (2016)), which consist of the discounted cumulative sum of feature vectors under a policy. Successor features zero-shot produce the action-values for a new reward, given by vectorz, and BFM approaches often use successor features to learn the policies. The performance of the BFM relies heavily on the state representation, which is both used to extractzfor the reward and for the policy. State-of-the-art methods Touati & Ollivier (2021); Agarwal et al. (2024); Park et al. (2024) usually learn state representations that retain information to representsuccessor measuresunder a wide class of policies. A successor measure captures the (discounted) state visitation of a policy, given any starting state. They are the generalization of successor representations Dayan (1993) to continuous states, and have a simple linear relationship to successor features Touati & Ollivier (2021). They can therefore be used to both encourage learning a generalizable state representation as well as simultaneously learning the successor features for the BFM. Successor measures are usually learned for an explicitly defined class of policies (Agarwal et al., 2024) or implicitly by first defining a class of reward functions (Touati et al., 2022; Park et al., 2024) and considering optimal policies for those reward functions as the set of policies. The main intuition behind predicting successor measures as a target for state representation learning is that representations sufficient to explain future state-visitation for a wide range of policies capture features that are relevant for sequential decision making under various reward functions. Unfortunately, learning state representations by estimating successor measures requires iteratively applying Bellman evaluation backups or Bellman optimality backups, both of which are known to result in a variety of learning difficulties. They can suffer from various forms of bias Thrun & Schwartz (2014); Fujimoto et al. (2018); Lu et al. (2018); Fu et al. (2019) and can suffer from feature collapse due to the instability inherent in bootstrapping in the function approximation regime (Kumar et al., 2021). Using Bellman backups to learn a representation requires choosing a class of policies or a class of reward functions a priori. Further, because the state representation is trained from a batch of offline data, unless chosen carefully, the policies may select out-of- distribution actions, leading to incorrect generalization and degenerate representations. A simple alternative that sidesteps these issues is to use latent dynamics learning: predicting future latent states given the current state and the sequence of actions. Learning the state representation by predicting the latent dynamics has the benefit of being independent of the policy and thus avoids using Bellman backups and these out-of-distribution issues. This work investigates the following question: Does latent next-state prediction produce state features that enable performant zero-shot RL? Our investigation is inspired by the work of Fujimoto et al. (2025), which showed that using dynamics prediction losses as auxiliary losses boosted performance of a single-task RL agent. Unlike the single task RL setting examined by Fujimoto et al. (2025), we find that in its naive form, this objective leads to a mild form of feature collapse where the representation of different states increase in similarity over training. This collapse results in poor zero-shot RL performance when evaluated on a number of downstream tasks. With a simple orthogonality regularization to prevent collapse, we show that the representations learned are competitive and present a scalable alternative to representations learned via complex successor measure estimation methods for zero-shot RL. In summary, the contributions of this paper are as follows. 1. We propose regularized latent- dynamics prediction (RLDP) as a simple alternative to learn state features for zero-shot RL. We identify as well as mitigate feature collapse plaguing latent dynamics prediction. 2. We show that our method remains competitive through an extensive empirical evaluation of representations for task generalization across a variety of domains, in online and offline RL settings, including in humanoid with a large state-action space. 3. We show that theRLDPobjective can learn performant policies in low-coverage settings where other methods fail. 2RELATEDWORK Unsupervised RLencompasses the class of algorithms that enable learning general-purpose skills and representations without relying on reward signal in the data. Works that have focused on intent or skill discovery have used diversity-driven objectives (Eysenbach et al., 2018; Achiam et al., 2018), maximizing mutual information (Warde-Farley et al. (2018), Eysenbach et al. (2018), 2 Published as a conference paper at ICLR 2026 Achiam et al. (2018), Eysenbach et al. (2021)) or minimizing the Wasserstein distance (Park et al. (2023)) between latents and the induced state-visitation distribution. These discovered skills can be used to compose optimal policies for several rewards. Our work, on the other hand, focuses on learning representations capable of producing optimal value functions for any arbitrary function reward specification. There are also a variety of pre-training approaches for representations that can be fine-tuned for downstream control. Recent pre-training approaches (e.g., Ma et al. (2022); Nair et al. (2022)) borrow self-supervised techniques such as temporal contrastive objectives to extract embeddings from large-scale datasets (Grauman et al. (2021)). HILP (Park et al. (2024)) goes beyond standard masked autoencoding approaches by using Hilbert-space representations to preserve temporal dynamics. Auxiliary objectives involve complementary predictive tasks to get richer semantic or temporal structures (Agarwal et al., 2021; Schwarzer et al., 2020). Although representations from auxiliary objectives can accelerate policy learning, a new policy still needs to be learned from scratch for each new reward function. Behavioral Foundation Modelsare obtained by training an RL agent in an unsupervised manner using task-agnostic reward-free offline transitions. Forward-Backward representations (Touati & Ollivier (2021)) and PSM (Agarwal et al. (2024)) provide one such framework for training BFMs by learning representations that capture a set of successor measures, on which several successive works are based. Fast Imitation with BFMs (Pirotta et al. (2023)) demonstrates the ability of successor- measure–based BFMs to imitate new behaviors from just a few demonstrations, while Sikchi et al. (2025) builds upon this by fine-tuning the BFM’s latent embedding space, yielding 10-40% improvement over their zero-shot performance. Recent progress in imitation learning has led to the development of BFMs tailored for humanoid control tasks (Peng et al. (2022), Won et al. (2022), Luo et al. (2023), Tirinzoni et al. (2025)) which can produce diverse behaviors trained using human demonstration data. Our work differs from these in that it provides a new, simpler state-representation learning objective for training BFMs. 3PRELIMINARIES We consider a reward-free Markov Decision Process (MDP) (Puterman, 2014) which is defined as a tupleM= (S,A,P,d 0 ,γ), whereSandArespectively denote the state and action spaces,P denotes the transition dynamics withP(s ′ |s,a)indicating the probability of transitioning fromsto s ′ by taking actiona,d 0 denotes the initial state distribution andγ∈(0,1)specifies the discount factor. A policyπis a functionπ:S →∆(A)mapping a state s to probabilities of action inA. We denote by Pr(· |s,a,π)andE[· |s,a,π]the probability and expectation operators under state- action sequences(s t ,a t ) t≥0 starting at(s,a)and following policyπwiths t ∼P(· |s t−1 ,a t−1 ) anda t ∼π(·|s t ). Given any reward functionr:S →R, the Q-function ofπforrisQ π r (s,a) := P t≥0 γ t E[r(s t+1 )|s,a,π]. A Behavioral Foundation Model (BFM) using Successor Featuresis a tuple(φ,ψ,π z )for state features,φ:S → Z, successor features defined asψ(s,a,π) =E π [ P t γ t φ(s t )|s,a]and the task- conditioned learned policyπ z that inputs any task embeddingzthat corresponds to a reward function r z =φ ⊤ z. To define this policyπ z , we use action-values produced by the successor featuresψ. The action-value function for rewardr z and a fixed policyπcan be written as Q π z (s,a) =E π h X t γ t φ(s t ) ⊤ z|s,a i =E π h X t γ t φ(s t ) ⊤ |s,a i z=ψ(s,a,π) ⊤ z The policyπ z is the optimal policy for rewardr z , obtained by iteratively greedifying overQ π z (s,a). Using the successor feature notation, we can iteratively updateπ z for all states until it satisfies the following fixed-point equation π z (·|s) = arg max a ψ(s,a,π z ) ⊤ zfor alls.(1) We overload notation and writeψ(s,a,z)to meanψ(s,a,π z ), because later we will directly input zinto a network to learn these successor features for a near-optimal policy forr z . The BFM can be used for any new reward function as long as we can obtain thezcorresponding to that reward. This is straightforward to do if we are given a datasetρ, as the correspondingz 3 Published as a conference paper at ICLR 2026 can be extracted by solving the linear regression problem,min z E ρ [(φ ⊤ z−r) 2 ] = (φ ⊤ φ) −1 φ ⊤ r. Naturally, BFMs depend heavily on the choice of the state representationφ; how to learn an effective φis the subject of this work. Learning the successor featurescan be done using successor measures, instead of directly estimating the discounted sum of featuresφ. Thesuccessor measure(Dayan, 1993; Blier et al., 2021) of state-action(s,a)under a policyπis the (discounted) distribution over future states obtained by taking actionain statesand following policyπthereafter: M π (s,a,X) := X t≥0 γ t Pr(s t+1 ∈X|s,a,π)∀X⊂S.(2) The action-value can be represented as,Q π (s,a) = P s + M π (s,a,s + )r(s + ). This simple linear relationship between action-value functions and successor measures is similar to that of successor features and has been exploited by recent works (Touati & Ollivier, 2021; Agarwal et al., 2024; Park et al., 2024) to train BFMs. It has been shown by Touati & Ollivier (2021) that parameterizing the successor measures asM π z (s,a,s + ) =ψ π (s,a,z) ⊤ φ(s + )yieldsψ(s,a,z)as successor features for the state featureφ(s) = (φ ⊤ ) −1 φ(s)(Theorem 12 of Touati & Ollivier (2021)). Since, the closed form solution forzfor any reward functionrwas(φ ⊤ ) −1 φr, using the parameterization of M π impliesz=φ ⊤ r. To train the BFM, we alternate between a successor measure learning phase to get and a policy improvement phase. The successor measure learning phase learns to model densitiesM π z (s,a,s + ) using the contrastive objective (Blier et al., 2021): Successor-measure estimation:L SM (M π z ) =−E s,a,s ′ ∼ρ [M π z (s,a,s ′ )] + 1 2 E s,a,s ′ ∼ρ,s + ∼ρ [(M π z (s,a,s + )−γ ̄ M π z (s ′ ,π z (s ′ ),s + )) 2 ].(3) This objective can be used assuming a fixed state representationφ, only training the successor featuresψ(as we will do for our approach), or allows for bothφandψto be jointly optimized, as was done in the Forward-Backward (FB) algorithm (Touati & Ollivier, 2021), PSM (Agarwal et al., 2024) and HILP (Park et al., 2024). The policy improvement step greedily optimizes the action-value function given by this successor measure π z (s) = arg max a Q π z (s,a) = arg max a X s + M π z (s,a,s + )·r(s + ) = arg max a X s + [ψ(s,a,z) ⊤ (φ(s + )·r(s + ))] = arg max a ψ(s,a,z) ⊤ X s + φ(s + )·r(s + ) = arg max a ψ(s,a,z) ⊤ z(4) In practice, we cannot directly set the policy to this argmax. Instead, we optimize the following loss. Policy Improvement:L P (π z ) =−E a∼π z (s) [ψ(s,a,z) ⊤ z](5) Appendix A.2 provides a further overview of approaches to train BFMs. In this work, we leverage this machinery for BFMs and focus on a new approach to estimate the state representationsφ. 4METHOD This method can be broadly divided into two parts: representation learning and zero-shot RL using successor features. The state representation encoder is trained using latent dynamics prediction with diversity regularization. We will show that these representations lead to a reduction in the prediction error for successor measures for any policy. Leveraging these robust state embeddings, we then pretrain a Behavioral Foundation Model (BFM) to predict successor measures, enabling zero-shot inference of near-optimal policies for unseen reward functions. We refer to this method asRLDP (RegularizedLatentDynamicsPrediction based Behavioral Foundation Policies). 4 Published as a conference paper at ICLR 2026 4.1LEARNINGREPRESENTATIONS WITHREGULARIZEDLATENTDYNAMICSPREDICTION Zero-shot RL based on successor features relies on learning a state representation denoted byφ(s). This state representation will define the span of reward functions that the zero-shot RL method is guaranteed to output optimal policies for. The primary representation learning objective is unrolled latent dynamics prediction. We learn a state representation encoderφ:S →R d , (Z=R d ) and a latent state-action representation encoderg:R d ×A →R d such that latent dynamics can be expressed asφ(s ′ ) =g(φ(s),a) ⊤ w with some learned weightsw, informing our loss function for representation learning.A sub-sequence of horizonHis sampled from the offline interaction datasetρgiven byτ i = s i 0 ,a i 0 ,s i 1 ,a i 1 ,...,s i H−1 ,a i H−1 ,s i H . A sequence of future latent statesh i 1:H are obtained by encoding the initial stateh i 0 =φ(s i 0 )and unrolling using the defined dynamics modelh i t+1 = g(h i t ,a t ) ⊤ w. Then the objective is to predict the encoded future latent states: L d (φ,g,w) =E τ i ∼d O H X t=1 h i t − ̄ φ(s i t ) 2 ,(6) whereτ i =s i 0 ,a i 0 ,s i 1 ,a i 1 ,...,s i H−1 ,a i H−1 ,s i H ,h i 0 =φ(s i 0 ), and ̄ φis the slowly moving encoder target, which is periodically set toφ. Figure 1:Average Cosine similarity between state- representations sampled uniformly from the training dataset. Feature similarity increases over the course of training; once adding our orthogonality regularizer (withλ= 1), we obtain more diverse representations. Shaded region shows standard deviation over 4 seeds. Latent dynamics models have been shown to significantly improve sample efficiency for single task RL when models are used for planning (Hansen et al., 2022), learning (Hafner et al., 2019), or as representations (Fujimoto et al., 2025) for model-free RL, but their suitability as general-purpose representations for multi-task and zero-shot RL remains understudied. Most successful methods (Touati & Ollivier, 2021; Agarwal et al., 2024) for zero-shot RL train representations to predict successor measures. However, directly estimating successor measures requires learning future state-occupancies under a predefined set of policies.This poses a problem in the low-coverage setting as Bellman backups with policies that choose out of distribution action will result in incorrect predictions and negatively affect representation learning. In contrast, latent dynamics prediction is a policy-independent representation learning objective. However, solely learning with the latent dynamics objective can lead to convergence to a collapsed solution. This is unsurprising as trivial solutions of predicting a constant zero vector achieves a perfect loss in equation 6. To combat this, prior works (Grill et al., 2020) have proposed the use of a semi-gradient update where a stop-gradient is used for targeth t+1 in equation 6 along with a slowly updating target. However, we find these techniques insufficient to maintain representation diversity. We investigate this by computing the cosine similarity of state representations as a function of gradient steps trained via minimizing equation 6 on an offline dataset collected by an exploration algorithm RND (Burda et al., 2018). Figure 1 shows that while the solutions do not collapse, there is an increase in feature similarity over the course of learning, which we refer to as amildform of collapse. As the space of reward functions is spanned by state features, such an increase in feature similarity directly reduce the class of reward functions for which we can learn optimal policies and negatively impact task generalization. Mitigating collapse in latent dynamics prediction:In order to prevent the mild form of feature collapse discussed earlier, we propose to add an auxiliary regularization objective that encourages diversity. Orthogonal regularization has been also studied in self-supervised learning (He et al., 2024; Bansal et al., 2018) as a way to mitigate collapse. We project all state representations φas well as predicted latent next-stateg(φ(s),a) ⊤ win a hypersphere:S d−1 =x∈R d : ∥x∥ 2 = √ dand regularize by minimizing cosine similarity between any two states.We 5 Published as a conference paper at ICLR 2026 ablate the choice of hyperspherical normalization ongin Appendix A.4.2 and observe it to give consistent improvements. We note that a similar regularization was applied to state features in the implementation for Forward-Backward representations (Touati et al., 2022) to encourage solution identifiability and uniqueness. In the case of latent dynamics prediction this step becomes crucial to mitigate the increase in representation similarity. Figure 2:RLDPcombines latent next state prediction + regularization for diversity (an orthogonality regularizer) to learn representations for BFMs. The orthogonal regularization loss takes the following form: L r (φ) =E s,s ′ ∼ρ [φ(s) ⊤ φ(s ′ )](7) whereφ∈S d−1 . Our final loss is a weighted combination of dynamics prediction combined with orthogonal diversity regularization L RLDP (φ,g,w) =L d (φ,g,w) +λL r (φ)(8) whereλcontrols the regularization strength. We visualize this loss in figure 2, where the encoder is given byφ, the dynamics by gandwand the diversity across encoded states is encouraged withL r . We find that adding this regularization prevents collapse, shown in figure 1, even with a relatively small regularization coefficientλ= 0.01. We evaluate the impact of orthogonality regularization further with other coefficientsλ in Appendix section A.4.2. RLDPleads to representations capable of predicting successor measures:The representation learning objective is simply latent dynamics prediction with an orthogonal regularization. Through this objective, we are enforcing the representations to be good for predicting successor measures, which forms the basis of the BFMs that will be constructed using these. While prior work (Agarwal et al., 2025) already indicate that these representations are suitable for predicting successor measures, we further formalize this intuition. Let’s begin by looking at the latent space abstract MDP ̄ Mdefined using the state representationφ. Definition 4.1.Let MDP ̄ Mcorresponding to the state abstractionφ:S → Zbe defined as < φ(s),A,P(·|φ(s),a),γ,r >. Apart from facilitating the construction of BFMs and zero-shot RL, one of the utilities of the state-representations is to compress the state space to a smaller space. MDP ̄ Mrepresents this compression. We will assume that ̄ Mis Lipschitz. Formally, Assumption 4.2. ̄ Mis(K R ,K P )−Lipschitz.(Gelada et al., 2019) We now have all the components to show in the Lemma 4.3 that minimizingL RLDP will lead to a reduction in the prediction error of successor measure for anyK V -Lipschitz valued policy. Lemma 4.3.Given MDP ̄ M, letπbe anyK V -Lipschitz valued policy,M π be the successor measure forπand ̄ M π be the corresponding successor measure on ̄ M,L RLDP (φ,g,w)upper bounds the prediction error in successor measure, E s,a∼d π ,s + ∼ρ [|M π (s,a,s + )− ̄ M π (φ(s),a,φ(s + ))|]≤ L RLDP (φ,g,w) 1−γ (9) 4.2ZERO-SHOTRLWITHRLDPREPRESENTATIONS The first step to useRLDPis to train the representations from a batch of reward-free offline environment transitions. TheRLDPrepresentation loss given in equation 8 does not rely on reward, because it only uses the latent dynamics prediction loss and the orthogonality regularizer. 6 Published as a conference paper at ICLR 2026 It is straightforward to optimize this loss on the batch of offline data, to obtain a learned state representationφ. Key choices to be made include the regularization coefficientλ, the length of the rolloutHfor the latent dynamics prediction loss and the update frequency for encoder target ̄ φ. The next step is to train the BFM, by alternating successor measure estimation and policy improvement. TheRLDPrepresentations are kept frozen in the successor measure parameterization M π z (s,a,s + ) =ψ π (s,a,z) ⊤ φ(s + )andψ(s,a,z)andπ z are trained using losses 10 and 5 respectively. L zsrl (ψ) =−E s,a,s ′ ∼ρ [ψ(s,a,z)φ(s ′ )] + 1 2 E s,a,s ′ ∼ρ,s + ∼ρ [(ψ(s,a,z)φ(s + )−γ ̄ ψ(s ′ ,π z (s ′ ),z)φ(s + )) 2 ](10) Following prior work, in our experiments we consider variations of the policy improvement step (Eq 5) where we use an expert regularization in the policy update (Tirinzoni et al., 2025) to guide exploration during online RL for high-dimensional state-action spaces or use a behavior cloning regularization (Fujimoto & Gu, 2021) when learning offline for low-coverage datasets. These modifications are discussed in detail in the next section. We provide the full representation and policy learning pipeline forRLDPin Appendix section A.6. 5EXPERIMENTS The goal of our experiments is to perform an extensive empirical study of the suitability of state representations learned by a regularized latent next-state prediction objective when compared to other methods that employ more complex strategies. In particular, we aim to answer the following questions: (a) Keeping all other learning factors similar, how does our method compare to baselines in enabling generalization to unseen reward functions? We compare the representations learned by training multi-task policies with zero-shot RL both in the offline setting and the online setting. (b) By avoiding querying actions out of distribution doesRLDPprovide a robust choice for learning representations in low coverage datasets? (c) What design decisions are crucial to the success of our method? We perform extensive ablation studies to understand our design choices. For all datasets, we pretrain a BFM using the successor feature approach outlined in our method section 4. Each algorithm is given the same budget of gradient steps during pretraining, controlling the state representation dimension, and the final performance is obtained by taking the pre-trained model at the end and querying it for different task-rewards for 50 episodes. 5.1BENCHMARKINGZERO-SHOTRLFORCONTINUOUSCONTROL TaskRandom FeaturesFBPSMRLDP Walker Stand392.40±58.03918.29±28.83899.54±30.73890.40±27.33 Run75.39±20.97381.31±17.32450.57±28.95334.26±49.69 Walk193.84±112.98779.29±63.60875.61±33.44779.77±137.16 Flip132.02±67.85977.08±2.76621.36±75.62492.94±22.79 Cheetah Run31.82±36.88129.39±37.63181.85±54.17157.12±29.92 Run Backward60.08±12.82142.41±36.77158.64±18.56170.52±15.30 Walk147.52±155.66604.54±80.51576.98±209.45592.92±104.66 Walk Backward272.77±42.40630.40±144.23817.92±98.86821.51±50.62 Quadruped Stand240.01±66.06732.59±101.33708.03±34.99794.94±43.25 Run114.19±30.22425.15±52.02404.32±23.26457.41±74.70 Walk137.65±47.57492.91±17.55523.94±52.13465.40±185.29 Jump190.62±46.63567.27±48.90549.57±15.86733.32±55.30 Pointmass Top Left258.59±183.56943.85±17.31924.20±10.64890.41±60.79 Top Right216.30±189.05550.84±282.41666.00±133.15795.47±21.10 Bottom Left193.32±90.37672.28±153.06800.93±15.62805.17±20.44 Bottom Right64.08±72.21272.97±274.99123.44±138.82193.38±167.63 Table 1:Comparison (over 4 seeds) of zero-shot RL performance between using an untrained initialized encoder, FB, PSM, andRLDPwith representation sized= 512. Bold indicates the best mean and any method whose mean plus one standard deviation overlaps with the best mean. Baselines:We broadly compareRLDPagainst commonlyusedstate- of-the-artbaselinesfor zero-shot RL such as: FB, PSM, and HILP. These baselines represent a set ofdiverseandstrong approaches in the area of zero-shot RL. 5.1.1OFFLINE ZERO-SHOTRL Setup:Weconsider continuous control tasks from DeepMind control suite (Tassa et al., 2018) –Pointmass,Cheetah, Walker, Quadruped under a similar setup considered 7 Published as a conference paper at ICLR 2026 by prior works in zero-shot RL. We use datasets from the ExoRL suite (Yarats et al. (2022)) that are obtained by an exploratory algorithm RND (Burda et al. (2018)). Random features use representations from a randomly initialized N encoder. Evaluation:To evaluate the different zero-shot RL methods we take the pretrained policies and query them on a variety of tasks. For each environment, we consider 4 tasks similar to prior works (Touati et al., 2022; Park et al., 2024; Agarwal et al., 2024). We conduct our experiments across two axes: a) Table 1 pretrains all the BFMs on same number of representation dimensions (512) and gradient steps. ForRLDP, we use an encoding horizon of 5. We train representations for 2 million steps and train policy for additional 3 million steps. b) Table 6 in the Appendix compares against representation dimension forφfound to be best for prior methods andRLDPwith the same number of gradient updates for pretraining each BFM. Results:Overall, using learned representations (FB, PSM,RLDP) outperforms random features, confirming that representation learning is crucial for zero-shot RL. Among learned methods, PSM andRLDPgenerally achieve the strongest performance.Furthermore, training FB and PSM baselines is sensitive to hyperparameters and we rely on author’s implementation to tune hyperparameters. 5.1.2ONLINEZERO-SHOTRL Previous section validated thatRLDPrepresentations lead to competitive zero-shot RL when the learning policies use offline interaction data. We explore if the learned representation enable competitive multi-task learning when agent is allowed interaction with the environment. Figure 3:Pair-wise comparison ofRLDPagainst prior offline representation learning methods using per-task oracle normalized performance differences (∆=RLDP– Baseline) in SMPL Humanoid environment.The gray diamond represents the IQM (Interquartile Mean). Setup:We consider the SMPL (Loper et al. (2015)) Humanoid environment that aims to mimic real human embodiment andprovidesacomplexlearning challengewitha358dimensional observation space and a 69 dimensional action space.Due to the exploratory challenge of the environment, Tirinzoni et al. (2025) presented a new approach, ConditionalPolicyRegularization (CPR), to guide RL learning regularized with expert real-human trajectories. CPR trains successor measures in a similar way as equation 3 but adds a regularization objective to policy encouraging it to jointlymaximizeQ-functionwhile staying close to expert. This allows for better exploration and more realistic motions. Further implementation details can be found in appendix sections A.3.2, A.5.2. Evaluation:Our representation learning phase is offline and we use the metamotivo 5M transition dataset 1 collected from replay buffer of an online RL agent to learn state-representations and then use the CPR approach to train zero-shot policies. We train representations for 2 million gradient steps and policy for 20 million environment steps. The offline phase of representation learning helps us remove the exploration confounder and test the quality of representations obtained by different approaches. The evaluation is performed on the full suite of 45 tasks provided by Tirinzoni et al. (2025). For each task, we present the normalized scores with respect to fully-online trained representations and policy in table 8 and we present the aggregates results across tasks in figure 3 over 4 seeds. Results:Figure 3 summarizes the results ofRLDPrepresentations with respect to baseline methods across all 45 tasks. Positive values indicate tasks whereRLDPachieves higher normalized returns. These results suggest that overallRLDPfares competitively to the baselines. Complete results for this evaluation are provided in table 8. Further analysis shows that the performance is task dependent - on some tasks (such as raisearms and lieonground),RLDPoutperforms the baselines, even beating 1 https://huggingface.co/facebook/metamotivo-M-1 8 Published as a conference paper at ICLR 2026 the oracle performance for some tasks (shown in table 8). In others (like crawl or rotate tasks), all methods perform subpar to oracle. 5.2LEARNINGREPRESENTATIONS WITHLOWCOVERAGEDATASETS RLDPlearns a policy-independent representation through latent dynamics prediction.Prior approaches assume a class of policies to learn representations predictive of successor measures, and this strategy can lead to poor out-of-distribution generalization when actions proposed by the policy are not covered by the dataset. Setup.To evaluate this hypothesis concretely, we consider the D4RL benchmark of OpenAI Gym MuJoCo tasks (Fu et al. (2020), Todorov et al. (2012), Brockman et al. (2016)). This dataset has been widely used to examine the effects of value estimation error from out-of-distribution actions due to low coverage, which many offline RL algorithms struggle with (Kostrikov et al. (2021); Fujimoto & Gu (2021); Kumar et al. (2020); Wu et al. (2019); Sikchi et al. (2023)). We consider halfcheetah, hopper, and walker2d domains, and medium and medium-expert datasets. Figure 4:Pair-wise comparison ofRLDPagainst baseline representation learning methods in low-coverage D4RL dataset. Each point represents∆ =R RLDP −R baseline for a singletask,seedpair. The gray diamond represents the IQM (Interquartile Mean). Evaluation:To evaluate the different zero-shot RL methods, we first pretrain the representation learning methods on these datasets for 1 million gradient steps.We use a modified zero-shot policy learning approach that alternates between equation 3 and equation 5 that is additionally augmented policy improvement loss with a behavioral regularization inspired by Fujimoto & Gu (2021). This regularization allow the RL approach to learn without overestimation bias and enabling us to establish a fair comparison among representations learned by different approaches.We use the corresponding reward function provided by each dataset to do reward inference and evaluate the zero-shot policy.Further details are provided in Appendix sections A.3.3 and A.5.3. Results:Figure 4 shows one-to-one comparison of normalized returns ofRLDPagainst baseline methods using paired per-seed performance differences across 6 low-coverage D4RL tasks over 10 seeds.RLDPoutperforms all baseline methods in 5 out of 6 tasks. The overall mass of the violin lies above zero and IQM is positive, indicating thatRLDPachieves higher normalized returns compared to the baseline. Overall, the results suggest thatRLDPis a reliable choice for feature learning in low coverage datasets while providing a simpler alternative to otherwise complex representation learning approaches. Per task normalized scores and statistical significance testing is reported in the Appendix section A.9 and table 9. We further bisect the individual and combined impact of using Bellman backups (similar to FB which may query out-of-distribution actions), latent next-state prediction, and orthogonality regularization for representation learning in section A.11 and find explicit Bellman backups to hurt performance of learned policy. 5.3WHAT MATTERS FOR SUPERVISING REPRESENTATIONS SUITABLE FOR CONTROL? In section 4, we introducedRLDPmethod of representation learning with the loss used (equation 8) and the encoder training process. In this section, we aim to ablate components of this loss and the architecture of the encoder. Orthogonality regularization:Keeping the encoding horizon constant (H= 5), we change the orthogonality regularization coefficient. The results, presented in Figure 5, show that for zero regularization (λ= 0), the average return decreases compared toλ >0. This shows that 9 Published as a conference paper at ICLR 2026 diversity regularization is critical to the representation loss. For fixed encoding horizon, we see that orthogonality regularizerλ= 1performs best. To further understand the role of the orthogonality regularizer in representation learning and how it helps prevent feature collapse, we refer to section 4 and section A.4.2, where we show that the learned representations increase in cosine similarity without regularization. Figure 5:Evaluating the impact of Orthogonality Regularization: We ran one-sided Mann–Whitney U tests on the per-seed returns over 4 seeds to compare different values of the orthogonality regularization, and we observe that adding small orthogonality regularization coefficientλ= 0.01gives a statistically significant improvement over coefficientλ= 0.0. Encoderarchitecture:In section 4.1, we introduce encoder training, where we project the latent next state representationg(φ(s),a).w to a hypersphere.Here, we ablate the importance of this projection. The results are presented in table 2. We observe thatRLDPconsistently outperformsitsvariantwithout spherical normalization on most tasks. The standard deviation is also higher for most results on the variant without hypersphere projection. This indicates that spherical normalization isanimportantdesignchoice forstabilizationandimproving performance. Results for all environments are reported in table 5.We provide additional discussion of the complete encoder architecture in section A.4, where we further ablate the encoder architecture. 6CONCLUSION TaskRLDP RLDPw/o SN Quadruped Stand794.94±43.25661.73±95.75 Run457.41±74.70378.97±148.47 Walk465.40±185.29 519.39±251.11 Jump733.32±55.30495.98±133.81 Average(*)612.77±83.17514.02±86.94 Pointmass Top Left890.41±60.79892.13±41.74 Top Right795.47±21.10728.72±122.99 Bottom Left805.17±20.44683.12±76.22 Bottom Right 193.38±167.6322.54±39.04 Average(*)671.11±292.58 581.63±341.00 Table 2:Study of encoder architecture (subset). Table shows mean±std;RLDPsignificantly outperformsRLDP w/o SN on Pointmass, Quadruped, and pooled.SN: Spherical Normalization ong. This paper introducesRLDP, a representation learningobjectiveforeffectivetask generalizationenablingperformant behavioral foundation models. Our objective takes the simple form of regularized latent- dynamics prediction, an objective that does not require any reconstruction, making it able to handle high-dimensional observation space and does not require explicit Bellman backups, making it more amenable to optimization.We identify that simply using latent-dynamics prediction leads to a mild form of feature collapse where the state-representation similarity increases over time. To combat this issue, we propose using orthogonal regularization as a way to maintain feature diversity and prevent collapse. Using our method enables learning generalizable, stable, and robust representations that can achieve competitive performance compared to prior zero-shot RL techniques without relying on reinforcement-driven signals. Importantly, we show that prior approaches struggle in low coverage setting andRLDPworks robustly across different dataset types, making it a practical unsupervised learning approach. This work, thus, paves the way for simpler yet effective approaches to learn zero-shot policies in behavioral foundation models. 10 Published as a conference paper at ICLR 2026 7ACKNOWLEDGEMENTS We thank Siddarth Chandrasekar, Dikshant Shehmar, and Diego Gomez for enlightening discussions on unsupervised RL. This work has been conducted at the Reinforcement Learning and Artificial Intelligence (RLAI) lab at the University of Alberta, the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at the University of Massachusetts Amherst, and Machine Intelligence through Decision-making and Interaction (MIDI) Lab at The University of Texas at Austin. Support for this work was provided by the Canada CIFAR AI Chair Program, the Alberta Machine Intelligence Institute, and the Natural Sciences and Engineering Research Council of Canada (NSERC). HS, SA, and AZ are supported by NSF 2340651, NSF 2402650, DARPA HR00112490431, and ARO W911NF-24-1-0193. We are also grateful for the computational resources provided by the Digital Research Alliance of Canada. REFERENCES Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms.arXiv preprint arXiv:1807.10299, 2018. Siddhant Agarwal, Aaron Courville, and Rishabh Agarwal. Behavior predictive representations for generalization in reinforcement learning. InDeep RL Workshop Neural Information Processing Systems 2021, 2021. URLhttps://openreview.net/forum?id=b5PJaxS6Jxg. Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang.Proto successor measure: Representing the space of all possible solutions of reinforcement learning.arXiv preprint arXiv:2411.19418, 2024. Siddhant Agarwal, Caleb Chuck, Harshit Sikchi, Jiaheng Hu, Max Rudolph, Scott Niekum, Peter Stone, and Amy Zhang. A unified framework for unsupervised reinforcement learning algorithms. InWorkshop on Reinforcement Learning Beyond Rewards@ Reinforcement Learning Conference 2025, 2025. Nitin Bansal, Xiaohan Chen, and Zhangyang Wang.Can we gain more from orthogonality regularizations in training deep cnns?arXiv preprint arXiv:1810.09102, 2018. Andr ́ e Barreto, R ́ emi Munos, Tom Schaul, and David Silver. Successor features for transfer in reinforcement learning.arXiv preprint arXiv:1606.05312, 2016. L ́ eonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint.arXiv preprint arXiv:2101.07123, 2021. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym.arXiv preprint arXiv:1606.01540, 2016. Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.arXiv preprint arXiv:1810.12894, 2018. Peter Dayan.Improving generalization for temporal difference learning:The successor representation.Neural computation, 5(4):613–624, 1993. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070, 2018. Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning.arXiv preprint arXiv:2110.02719, 2021. Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms.arXiv preprint arXiv:1902.10250, 2019. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020. Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06860, 2021. 11 Published as a conference paper at ICLR 2026 Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. corr abs/1812.02900 (2018).arXiv preprint arXiv:1812.02900, 2018. Scott Fujimoto, Pierluca D’Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning.arXiv preprint arXiv:2501.16142, 2025. Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. InInternational conference on machine learning, p. 2170–2179. PMLR, 2019. K Grauman, A Westbury, E Byrne, Z Chavis, A Furnari, R Girdhar, J Hamburger, H Jiang, M Liu, X Liu, et al. Ego4d: around the world in 3,000 hours of egocentric video. arxiv 2021.arXiv preprint arXiv:2110.07058, 2021. Jean-Bastien Grill, Florian Strub, Florent Altch ́ e, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. eprint arXiv: 2006.07733, 2020. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor.arXiv preprint arXiv:1801.01290, 2018. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019. Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control.arXiv preprint arXiv:2203.04955, 2022. Junlin He, Jinxiao Du, and Wei Ma. Preventing dimensional collapse in self-supervised learning via orthogonality regularization.arXiv preprint arXiv:2411.00392, 2024. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning.arXiv preprint arXiv:2110.06169, 2021. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning.arXiv preprint arXiv:2006.04779, 2020. Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization.arXiv preprint arXiv:2112.04716, 2021. Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, et al. Bfm-zero: A promptable behavioral foundation model for humanoid control using unsupervised reinforcement learning.arXiv preprint arXiv:2511.04131, 2025. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model.ACM Transactions on Graphics, 34(6), 2015. Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional q-learning and value-iteration. Advances in neural information processing systems, 31, 2018. Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu.Universal humanoid motion representations for physics-based control.arXiv preprint arXiv:2310.04582, 2023. Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022. Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022. 12 Published as a conference paper at ICLR 2026 Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction.arXiv preprint arXiv:2310.08887, 2023. Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15567, 2024. Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.ACM Transactions On Graphics (TOG), 41(4):1–17, 2022. Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imitation via behavior foundation models. InNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman.Data-efficient reinforcement learning with self-predictive representations.arXiv preprint arXiv:2007.05929, 2020. Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning.arXiv preprint arXiv:2302.08560, 2023. Harshit Sikchi, Siddhant Agarwal, Pranaya Jajoo, Samyak Parajuli, Caleb Chuck, Max Rudolph, Peter Stone, Amy Zhang, and Scott Niekum. Rlzero: Direct policy inference from language without in-domain supervision.arXiv preprint arXiv:2412.05718, 2024. Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, Scott Niekum, Amy Zhang, Alessandro Lazaric, and Matteo Pirotta. Fast adaptation with behavioral foundation models.arXiv preprint arXiv:2504.07896, 2025. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite.arXiv preprint arXiv:1801.00690, 2018. Chen Tessler, Yifeng Jiang, Erwin Coumans, Zhengyi Luo, Gal Chechik, and Xue Bin Peng. Maskedmanipulator: Versatile whole-body control for loco-manipulation.arXiv preprint arXiv:2505.19086, 2025. Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning.InProceedings of the 1993 connectionist models summer school, p. 255–263. Psychology Press, 2014. Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta.Zero-shot whole-body humanoid control via behavioral foundation models.arXiv preprint arXiv:2504.11054, 2025. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, p. 5026–5033. IEEE, 2012. Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards.arXiv preprint arXiv:2103.07945, 2021. Ahmed Touati, J ́ er ́ emy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209.14935, 2022. David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards.arXiv preprint arXiv:1811.11359, 2018. Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes.ACM Transactions on Graphics (TOG), 41(4):1–12, 2022. 13 Published as a conference paper at ICLR 2026 Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations.arXiv preprint arXiv:1810.04586, 2018. Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning.arXiv preprint arXiv:2201.13425, 2022. 14 Published as a conference paper at ICLR 2026 AAPPENDIX A.1PROOF OFLEMMA4.3 Lemma A.1.Given MDP ̄ M, letπbe anyK V -Lipschitz valued policy,M π be the successor measure forπand ̄ M π be the corresponding successor measure on ̄ M,L RLDP (φ,g,w)upper bounds the prediction error in successor measure, E s,a∼d π ,s + ∼ρ [|M π (s,a,s + )− ̄ M π (φ(s),a,φ(s + ))|]≤ L RLDP (φ,g,w) 1−γ (9) Proof.Let’s begin withE s,a∼d π [|M π (s,a,s + )− ̄ M π (s,a,s + )]. For a fixeds + E s,a∼d π [|M π (s,a,s + )− ̄ M π (s,a,s + )]≤E s,a∼d π |p(s ′ =s + )−p(φ(s ′ ) =φ(s + ))|+ γE s,a∼d π |E s ′ ∼P(·|s,a) V π (s ′ )−E φ(s ′ )∼P(·|φ(s),a) V π (φ(s ′ )| ≤L R +γE s,a∼d π |E s ′ ∼P(·|s,a) [V π (s ′ )−V π (φ(s ′ )]|+ γE s,a∼d π |E s ′ ∼P(·|s,a), φ(s ′ )∼P(·|φ(s),a) [V π (s ′ )−V π (φ(s ′ ))]| ≤L R +γE s,a∼d π |E s ′ ∼P(·|s,a) [V π (s ′ )−V π (φ(s ′ )]|+ γK V E s,a∼d π D(φP(·|s,a),P(·|φ(s),a)) (Dis the distance metric used byK V -Lipschitz policy) =L R +γE s,a∼d π |E s ′ ∼P(·|s,a) [V π (s ′ )−V π (φ(s ′ )]|+ γK V E s,a∼d π L d ≤L R +γE s,a∼d π E s ′ ∼P(·|s,a) |V π (s ′ )−V π (φ(s ′ )|+ γK V E s,a∼d π L d ≤L R +γE s,a∼d π |V π (s)−V π (φ(s))|+ γK V L d =L R +γE s,a∼d π |M π (s,a,s + )−M π (φ(s),a,φ(s + ))|+ γK V L d This implies,(1−γ)E s,a∼d π [|M π (s,a,s + )− ̄ M π (s,a,s + )]≤ L R +γK V L d whereL R = E s,a∼d π |p(s ′ =s + )−p(φ(s ′ ) =φ(s + ))|. Taking expectation unders + ∼ρ(s + ), (1−γ)E s,a∼d π ,s + ∼ρ [|M π (s,a,s + )− ̄ M π (s,a,s + )]≤E s,a∼d π ,s + ∼ρ |p(s ′ =s + )−p(φ(s ′ ) =φ(s + ))|+γK V L d =L R +γK V L d This implies,E s,a∼d π ,s + ∼ρ [|M π (s,a,s + )− ̄ M π (s,a,s + )]≤ L r +γK V L d 1−γ = L RLDP 1−γ A.2PRIORAPPROACHES FOR REPRESENTATION LEARNING INBFM’S Prior work has often relied on complex objectives to enable learning ofφandψfor BFMs. Forward- Backward (FB) (Touati et al., 2022) combine learning the state representation,φwith successor features,ψand the policy.φandψare jointly learned to represent successor measures for a class of reward-optimal policies. FB alternates minimizing the successor measure loss below jointly forψ,φ alongside policy improvement by optimizing Eq 5. FB uses the following loss minimizing Bellman residuals to learn representations: L(φ,ψ) =−E s,a,s ′ ∼ρ [ψ(s,a,z) T φ(s ′ )] + 1 2 E s,a,s ′ ∼ρ,s + ∼ρ [(ψ(s,a,z) T φ(s + )−γ ̄ ψ(s ′ ,π z (s ′ ),z) T ̄ φ(s + )) 2 ](11) 15 Published as a conference paper at ICLR 2026 HILP (Park et al., 2024) learns state representationφthat are suitable to predict value function for goal-reaching which is subsequently used for zero-shot RL in the same way asRLDP. HILP parameterizes the value function to beV(s,g) =∥ ̄ φ(s)− ̄ φ(g)∥and then minimizes: L(φ) =E s,s ′ ,g∼ρ [ℓ 2 τ (−1(s̸=g)−γV(s ′ ,g) +V(s,g)](12) whereℓ 2 τ is an expectile loss (Kostrikov et al. (2021)). PSM (Agarwal et al., 2024) learns state representation to represent the successor measures for a class of policies defined with a discrete codebook. The loss used is as follows - L(φ,ψ,w) =−E s,a,s ′ ∼ρ [ψ(s,a)w(c)φ(s ′ )] + 1 2 E s,a,s ′ ∼ρ,s + ∼ρ [(ψ(s,a)w(c)φ(s + )−γ ̄ ψ(s ′ ,π c (s ′ )) ̄w(c) ̄ φ(s + )) 2 ](13) wherecis a discrete code defining a policy andwmaps the discrete code to a continuous space. The above loss is minimized averaged over a pre-determined distribution of discrete codes. The Laplacian approach (Wu et al., 2018) is action-independent and learns state representation using eigenvectors of graph-Laplacian induced by a random-walk operator. The representation objective for Laplacian approach takes the following form: L(φ) = 1 2 E s∼ρ, s ′ ∼P π (·|s) ∥φ(s)−φ(s ′ )∥ 2 2 +βE s,s ′ ∈ρ [φ(s) T φ(s ′ )](14) A.3EXPERIMENTAL DETAILS A.3.1EXORL ExoRL (Exploratory Offline Reinforcement Learning) is a benchmark suite that provides large, diverse offline datasets generated by exploratory policies across multiple domains (e.g., locomotion, manipulation, navigation).We consider three locomotion and one goal-based navigation environments – Walker, Quadruped, Cheetah, Pointmass – from the Deepmind Control Suite (Tassa et al. (2018)). For offline training, we use data provided from the EXORL benchmark trained using RND agent. These domains are explained further in table 3. All DM control tasks have an episode length of 1000. DomainDescriptionType Observation/Action Dimension TasksReward Walkertwo-legged robotLocomotion24/6 stand walk run flip Dense Quadrupedfour-legged robotLocomotion78/12 jump walk run stand Dense Cheetahplanar, 2D robotLocomotion17/6 walk run walk backward run backward Sparse Pointmassnavigation in 2D planeGoal-reaching4/2 reach top left reach top right reach bottom right reach bottom left Sparse Table 3: ExoRL dataset summary.Domainis the environment name in the ExoRL benchmark.Descriptionis a natural language description of the agent embodiment and environment.Typerefers to the broad task category. Observation/Action Dimensionrefers to the size of observation and action vectors from the environment.Tasks refers to the suite of evaluation tasks provided in the ExoRL benchmark.Rewardrefers to the density of non- zero reward signals from the environment. 16 Published as a conference paper at ICLR 2026 A.3.2SMPL 3D HUMANOID SMPL (Skinned Multi-Person Linear Model) is a 3D parametric model of the human body that is widely used for character animation. It has a 358 dimensional proprioceptive observation space that includes body pose, rotation, and velocities. The action space is 69 dimensional where each action dimension lies in [-1,1]. All episodes are of length 300. A.3.3D4RL DomainTask Name# Samples Gym-MuJoCo hopper-medium10 6 hopper-medium-expert2×10 6 halfcheetah-medium10 6 halfcheetah-medium-expert2×10 6 walker2d-medium10 6 walker2d-medium-expert2×10 6 Table 4:Gym-MuJoCo tasks from D4RL. D4RL (Datasets for Deep Data- DrivenReinforcementLearning) (Fu et al. (2020)) is an offline RL benchmark suite built on the v2 Open AI Gym (Brockman et al. (2016)) that provides standardized datasets and evaluation protocols across simulated and real-world tasks. We consider three simulated locomotion tasks–Hopper,HalfCheetah, Walker2D – and two datasets – medium and medium-expert. As described in Fu et al. (2020), the medium dataset is generated by online training a Soft-Actor Critic (Haarnoja et al. (2018)) agent, early-stopping the training, and collecting 1 million samples from this partially-trained policy. The “medium-expert” dataset is generated by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a uniform-at-random policy. Further details about these tasks have been provided in table 4. Episodes have inconsistent length depending on termination/truncation with a maximum of 1000. A.4REGULARIZEDLATENTDYNAMICSPREDICTION RLDPaims to learn a state representation encoderφsuch that latent state dynamics can be expressed asφ(s ′ ) =g(φ(s),a) ⊤ wwheregis a latent-state action encoder and w are some constant weights. A.4.1ARCHITECTURE The architecture of theRLDPlatent next-state prediction network is as pictured in figure 6 Figure 6:Architecture of latent next-state prediction network inRLDP. The state representation networkφis a feedforward MLP with two hidden layers of 256 units that maps a statesto ad-dimensional embedding. In our defaultRLDParchitecture, the actionais mapped to 256-dimensional space using linear networkA. In this section, we make this distinction clear and useato denote raw action input to the network andA to denote a projection of action as input to network. The outputs of these two networks are concatenated and passed through a feedforward neural networkgthat has two hidden layers of 512 units and ad-dimensional output. The output of thegnetwork is passed through a linear layerw.The finald-dimensional representations are spherically normalized. During encoder training, the encoder map is unrolled to perform next latent state prediction from current latent state and action asφ(s ′ ) = g(φ(s),A) ⊤ w.After encoder training, the encoder network is frozen. To obtain latent state embeddings, the states are passed through the 17 Published as a conference paper at ICLR 2026 state representation network to getφ(s).The encoder architecture forRLDPis kept consistent across all methods and datasets. A.4.2ABLATIONS In this section, we aim to examine the components of theRLDPstate encoder to understand which parts of the method are crucial to learn representations that can maximize the span of reward functions we can represent optimal policies for. We pretrain state representation networkφand policy using the ExoRL dataset generated with RND exploration policy and evaluate the performance in DMC environments cheetah, pointmass, quadruped, and walker. Does orthogonality regularization matter? Figure 7:Evaluating the impact of Orthogonality Regularization on representations learned across four environments: Cheetah (top left), Pointmass (top right), Quadruped (bottom left), and Walker (bottom right). Figure 7 shows the impact of changing orthogonality regularization while keeping a constant encoding horizon (H= 5). The figure shows how the cosine similarity between latent states changes during encoder training for different regularization coefficients. For a regularization coefficientλ= 0, the cosine similarity increases, indicating that all states are getting mapped to similar representations. For any regularization coefficientλ >0, we observe that the cosine similarity follows a steep descent, indicating that the states are being mapped to diverse representations. These results indicate that adding even small orthogonality regularization can reduce representation collapse significantly. How does encoding horizon impact performance? As discussed in section 4,RLDPis trained with the objective to do latent next state prediction from latent current state and action. This prediction can be done multiple steps into the future latent states (equation 6), depending on the choice of encoding horizonH. In this section, we examine if the choice of encoding horizon impacts performance. To this end, we set the orthogonality regularization coefficientλ= 1.0and sweep over encoding horizon (1,5,10,20). The results are presented in figure 8. The average performance across environments is relatively stable with a small dip atH= 10, indicating that encoding horizon does not significantly impact performance.For our experiments, we use encoding horizonH= 1orH= 5 18 Published as a conference paper at ICLR 2026 depending on the setting. Specific encoding horizon setting for each experiment is discussed in section A.5. We do not choose higher encoding horizonH= 20despite comparable performance in figure 8 because higher encoding horizon can result in slower encoder training. This is because each additional future state prediction involves a forward pass through the encoder network. Figure 8:Evaluating the impact of Encoding Horizon What is important for the encoder architecture? In this section, we aim to ablate components of the encoder map to understand which factors contribute toRLDP’s performance.For this setting, we fix encoding horizonH= 5and orthogonality regularization coefficientλ= 1.0. We focus on two components of the encoder architecture – linear layerA and spherical normalizationSNong. We compare the completeRLDP encoder network with its variations – a)RLDPw/o SNwhere spherical normalization is replaced with an identity mapping; b)RLDPw/o Awhere A is replaced with an identity mapping; c)RLDPw/o SN & Awhere A is also replaced with an identity mapping. The results are shown in table 5. Although per-task results are variable, the fullRLDPencoder delivered the strongest average performance on all four domains. Removing spherical normalization lowers returns and increases variance on most tasks and removing A also degrades performance. There are isolated wins for all variants, but these do not impact the domain-level results that favor the fullRLDPencoder network. Thus, bothSNandAcontribute meaningfully to representation learning. A.4.3WHAT DO THESE REPRESENTATIONS LOOK LIKE? TaskRLDP RLDPw/o SNRLDPw/o ARLDPw/o SN & A Walker Stand890.40±27.33860.74±62.47810.79±100.90881.69±6.92 Run334.26±49.69324.04±6.73290.78±26.52276.30±47.21 Walk779.77±137.16728.29±43.09715.83±92.43583.60±28.26 Flip492.94±22.79501.59±45.04477.95±37.88447.73±33.59 Average(*)624.34603.66573.84547.33 Cheetah Run157.12±29.9284.99±67.31115.25±14.13118.67±32.67 Run Backward170.52±15.30193.69±40.10192.20±42.07156.56±45.98 Walk592.92±104.66387.50±244.76526.02±52.89559.82±177.29 Walk Backward821.51±50.62838.12±145.37836.29±173.10668.46±186.17 Average(*)435.52376.08417.44375.88 Quadruped Stand794.94±43.25661.73±95.75518.61±69.24687.43±155.33 Run457.41±74.70378.97±148.47358.55±53.61475.07±45.66 Walk465.40±185.29519.39±251.11384.92±119.49575.32±120.82 Jump733.32±55.30495.98±133.81319.18±55.16510.55±151.18 Average(*)612.77514.02395.34562.09 Pointmass Top Left890.41±60.79892.13±41.74886.19±10.07890.89±13.06 Top Right795.47±21.10728.72±122.99809.64±11.23797.59±19.44 Bottom Left805.17±20.44683.12±76.22730.74±63.72735.42±61.83 Bottom Right193.38±167.6322.54±39.04206.59±214.98178.77±130.17 Average(*)671.11515.02547.62583.78 Table 5:Study of encoder architecture. Cells show mean±std over 4 seeds; boldface indicates the highest mean per task. Toqualitativelyassess thelearnedstate representations,we usethePointmass environment,where weuniformlysampled 10,000 equidistant states from the underlying state space (figure 9 (a)). Weinitializeastate representationencoder φand pass these states through the encoder to get latent embedding before training (figure 9 (b)). We then train two encoders with different losses: a. we setλ=0.0in equation 8 and train using only latent state prediction loss (figure 9 (c)); b. we set λ= 1.0in equation 8 and train using latent state prediction loss and orthogonality regularization. 19 Published as a conference paper at ICLR 2026 (a)Pointmass observations(b)Latent states before encoder training (c)Latent states after training (prediction error) (d)Latent states after training (prediction error + orthogonality) Figure 9:t-SNE visualizations of state features in Pointmass. Each panel shows the 2D projection of 10,000 uniformly sampled states. We project all these embeddings into two dimensions t-distributed Stochastic Neighbor Embedding (t-SNE). This visualization highlights the geometric structure captured by the representation and provides intuition about how the encoder organizes states in latent space. The results in figure 9 show that training an encoder using only latent state prediction loss (figure (c)) is ineffective at capturing the layout of the environment and maps different states to similar latent representations. Using both latent state prediction loss and orthogonality regularization enables the encoder to better capture the environment layout (figure(d)). A.4.4FUTURE DIRECTIONS: EXTENSION TO REAL-WORLD EMBODIMENTS RLDPpresents a simple, stable and performant approach to train behavior foundation models in applications like robotics. An agent can be promptable to obtain low-level actions with such a BFM. Recent works (Tessler et al., 2025; Li et al., 2025) have made promising attempts to extend BFM algorithms to real-world domains and prior works have made it possible to prompt BFMs with language and videos (Sikchi et al., 2024) which can be more intuitive interface for humans than reward functions. We believe the simplicity of this method and stability across hyperparameter choices, as demonstrated in table 5, makes it a promising candidate for real-world embodiments. A.5IMPLEMENTATION DETAILS In this section, we discuss the implementation details of all the methods and experiments described in the paper. A.5.1OFFLINEZERO-SHOTRL We use the same architecture for forward and policy networks as presented in (Touati et al., 2022) for all representation learning methods. 20 Published as a conference paper at ICLR 2026 The forward networkF(s,a,z)has two parallel embedding layers that take in(s,a)and(s,z) independently using feedforward networks with a single hidden layer of 1024 units, projecting to 512 dimensions. Their outputs are concatenated and passed into two separate feedforward heads (each with one hidden layer of 1024 units), which output ad-dimensional vector. The policy networkπ(s,z)has two parallel embedding layers that take inputssand(s,z)and embeds them similar to the forward network (one hidden layer of 1024 units mapping to 512 dimensions). The outputs of the embedding layers are concatenated, and then passed into another single-hidden-layer feedforward network (1024 units) to produce an action vector of dimensiond A . A final Tanh activation ensures that actions lie in the space[−1,1] d A . For results in table 1: For all methods, the backward representation networkB(s)is implemented as a feedforward neural network with two hidden layers of 512 units each, mapping a statesto a512-dimensional embedding. For results in table 6: ForRLDP, we sweep over representation dimensions(64,128,256,512,1024)and report the results for the dimension that achieves the highest average performance across all tasks within each environment. ForFB,PSM,HILP, andLaplacian, we use the representation dimensions previously identified as optimal for each respective method. A.5.2ONLINEZERO-SHOTRL Results for oracle baseline, FB-CPR, are taken from Tirinzoni et al. (2025), where the model was trained for 30M environment steps and averaged across five seeds. For the offline representation learning methods (HILP, PSM, FB,RLDP), the backward representation networkB(s)follows the architecture of the backward network of FB-CPR. It is a 2-layer MLP with 256 hidden dimension that maps a statesto a 256-dimensional embedding. We train this for 2 million timesteps on a dataset provided by Tirinzoni et al. (2025), which is generated by online training an FB-CPR agent for 30 million environment steps and saving the final 5 million steps. TheRLDPrepresentations are trained with encoding horizon 1. We integrate the learned representation network into an FB-CPR agent to train the forward and policy networks. This training is performed online for 20 million environment steps where no updates are performed on the representation network. A.5.3LOWCOVERAGEDATASETS For all offline representation learning methods (HILP, PSM, FB,RLDP), the backward representation networkB(s)is a feedforward neural network with two hidden layers of 256 dimension that maps a statesto a 512-dimensional embedding. TheRLDPrepresentations are trained with encoding horizon 1. The forward networkF(s,a,z)and policy networkπ(s,z)follow the same architecture as FB. We introduce an additional loss term for training the policy network that resembles TD3+BC (Fujimoto & Gu, 2021). The policy improvement loss is defined as L P (π z ) =−λψ(s,a,z) ⊤ z+ π z (s)−a 2 (15) where λ= α 1 N P (s i ,a i ) Q(s i ,a i ) . Following Fujimoto & Gu (2021), we setα= 2.5 21 Published as a conference paper at ICLR 2026 A.6ALGORITHM In algorithm 1, we present the full algorithm for pre-training and inference of BFMs. In pretraining, we present RLDP representation learning as well as successor-measure estimation and policy learning. Algorithm 1 Require:Offline dataset of trajectoriesD. Require:Randomly initialized encoderφ, successor-measure modelψ, actorπ. Require:Representation-learning stepsN repr , total stepsN. 1:Part I: Pretraining (offline) 2:forlearning stepn= 1,2,...,Ndo 3:ifn≤N repr then 4:Sample segment batchτ=s i 0:H ,a i 0:H B i=1 ∼D 5:h i 0 =φ(s i 0 ),h i t+1 =g(h i t ,a i t ) ⊤ w 6:L d (φ,g,w) =E τ i ∼d O P H t=1 h i t − ̄ φ(s i t ) 2 7:L r (φ) =E s,s ′ ∼ρ φ(s) ⊤ φ(s ′ ) 8:L(φ,g,w)←L d (φ,g,w) +λL r (φ) 9:Updateφ,g,w 10:else 11:Sample transitions(s,a,s ′ ,done)∼D 12:Samplez∼Uniform Mixrandom prior + goal-encoded 13:Policy Evaluation: 14:L zsrl (ψ)from Equation 10 15:ψ←ψ−α ψ L zsrl (ψ) 16:Policy update: 17:a∼π(s,z) 18:Q=ψ(s,a,z)·z 19:π←π+α π ∇ π Q(s,π(s,z)) 20:end if 21:end for 22: 23:Part I: Inference (reward-based task embedding) Require:Task specification for the test task (e.g., name, parameters). 24:Set up the task-specific reward functionr task (s)using the environment’s reward routine 25:Sample transitions(s i ,a i ,s ′ i ) N i=1 ∼D 26:z← 1 N P i φ(s ′ i )r task (s ′ i ) A.7ADDITIONAL RESULTS This section details additional experiments we conducted to evaluateRLDPagainst baseline methods, visualize the successor measures learned by RLDP, and study the effect of different loss components on representation learning. A.7.1OFFLINEZERO-SHOTRLINDMC In table 6, we compare the returns for the representation dimension found to be the best for the baseline methods and theRLDP. Across all methods,RLDPfares competitively to baselines that employ complex strategies such as FB, PSM to learn representation optimizing for successor measures across the environments despite its simplicity. 22 Published as a conference paper at ICLR 2026 TaskLaplaceFBHILPPSMRLDP Walker Stand243.70±151.40902.63±38.94607.07±165.28872.61±38.81877.69±45.03 Run63.65±31.02392.76±31.29107.84±34.24351.50±19.46324.85±54.57 Walk190.53±168.45877.10±81.05399.67±39.31891.44±46.81790.94±67.55 Flip48.73±17.66206.22±162.27277.95±59.63640.75±31.88491.64±37.30 Cheetah Run96.32±35.69257.59±58.5168.22±47.08244.38±80.00236.31±20.75 Run Backward106.38±29.40307.07±14.9137.99±25.16296.44±20.14322.08±39.28 Walk409.15±56.08799.83±67.51318.30±168.42984.21±0.49895.31±49.84 Walk Backward654.29±219.81980.76±2.32349.61±236.29979.01±7.73984.76±0.85 Quadruped Stand854.50±41.47740.05±107.15409.54±97.59842.86±82.18794.94±43.25 Run412.98±54.03386.67±32.53205.44±47.89431.77±44.69457.41±74.70 Walk494.56±62.49566.57±53.22218.54±86.67603.97±73.67465.40±185.29 Jump642.84±114.15581.28±107.38325.51±93.06596.37±94.23733.32±55.30 Pointmass Top Left713.46±58.90897.83±35.79944.46±12.94831.43±69.51890.41±60.79 Top Right581.14±214.79274.95±197.9096.04±166.34730.27±58.10795.47±21.10 Bottom Left689.05±37.08517.23±302.63192.34±177.48451.38±73.46805.17±20.44 Bottom Right21.29±42.5419.37±33.540.17±0.2943.29±38.40193.38±167.63 Table 6:Comparison of zero-shot offline RL performance between different methods. Entries inboldare within one standard deviation of the per-task best mean (i.e.,μ i ≥μ ∗ −σ ∗ ) aggregated over 4 seeds. A.8TRAININGUSFAS ON TOP OFRLDPREPRESENTATIONS In table 7, we examine the impact of directly learning Universal Successor Features on top ofRLDP representations. Typically for offline zero-shot RL onRLDPrepresentations, we use loss equation 10 to update the critic network. To train USFAs, we use the following loss: L USFA (ψ) =E s,a,s ′ ∼ρ,s + ∼ρ [(ψ(s,a,z)−[φ(s) +γ ̄ ψ(s ′ ,π z (s ′ ),z)]) 2 ](16) TaskRLDP Learning USFAs on RLDPrepresentations Walker Stand890.40±27.33854.84±64.50 Run334.26±49.69304.89±63.15 Walk779.77±137.16665.98±117.64 Flip492.94±22.79497.77±70.80 Cheetah Run157.12±29.92139.00±6.72 Run Backward170.52±15.30172.52±25.72 Walk592.92±104.66647.35±66.05 Walk Backward821.51±50.62800.02±82.94 Quadruped Stand794.94±43.25294.09±165.70 Run457.41±74.70238.42±121.54 Walk465.40±185.29326.36±111.55 Jump733.32±55.30220.66±67.59 Pointmass Top Left890.41±60.79723.80±55.33 Top Right795.47±21.10723.79±57.89 Bottom Left805.17±20.44753.66±16.59 Bottom Right193.38±167.6394.39±77.81 Table 7:Comparison (over 4 seeds) of zero-shot RL performance when equation 10 is used to train the critic and when Universal Successor Features are trained on top of the state features. For fair comparison, we setRLDPrepresentation dimensiond= 512for both methods. Wefindthatacrossawide range of control tasks, training a USFA module on top ofRLDP’s staterepresentationsdoesnot consistentlyoutperformdirectly using successor measure loss 10 for policy evaluation.Critic learned using successor measure lossachievesoverlapping-best performanceonmostWalker, Quadruped, and Pointmass tasks, while USFAs occasionally match orslightlyexceedoncertain Cheetah behaviors. Overall, these results indicate thatRLDP’s learned representations capture most of the structure required for effective zero- shot generalization using either loss. The critic trained with successor measure loss typically achieves the strongest overall performance. A.8.1ONLINEZERO-SHOTRL FB-CPR is an off-policy online unsupervised RL algorithm that introduces a latent conditional- discriminator in the form of Conditional-Policy Regularization to output policies close to an unlabeled demonstration datasetM. The results for FB-CPR are as reported in Tirinzoni et al. (2025). In table 8 and figure 10, we provide the full suite of results on 45 SMPL Humanoid task for all baseline methods,RLDP, and the oracle method FB-CPR. 23 Published as a conference paper at ICLR 2026 metricFB-CPR (Oracle)FBPSMHILPRLDP crawl-0.4-0-d191.75±43.6026.06±39.4338.38±14.7352.31±23.1586.48±45.87 crawl-0.4-0-u101.76±15.908.38±9.014.52±7.4718.59±20.4225.00±27.32 crawl-0.4-2-d19.00±4.003.05±3.656.52±1.278.97±5.6211.21±4.72 crawl-0.4-2-u15.02±6.030.79±1.070.64±0.822.95±1.372.76±3.73 crawl-0.5-0-d131.13±64.9743.27±34.6646.17±13.5652.41±27.8255.82±18.40 crawl-0.5-0-u101.92±16.394.04±5.874.18±4.8321.14±24.2620.22±30.91 crawl-0.5-2-d22.93±5.314.14±4.105.64±1.798.64±4.215.69±2.18 crawl-0.5-2-u15.81±6.100.94±0.990.77±0.702.67±1.032.95±3.28 crouch-0226.28±28.1755.12±47.0992.70±60.8672.94±76.254.83±5.28 headstand41.27±10.200.00±0.000.00±0.010.11±0.162.63±1.99 jump-234.88±3.5229.08±3.7621.21±11.6012.25±14.0527.89±1.66 lieonground-down193.50±18.8935.41±26.0863.87±26.6869.79±27.0274.69±30.03 lieonground-up193.66±33.1820.83±12.7013.92±5.5430.81±1.3754.38±31.06 move-ego–90-2210.99±6.55207.47±9.92179.67±49.64196.81±40.36178.82±45.10 move-ego–90-4202.99±9.33161.84±12.65102.35±35.15102.98±40.4799.96±32.58 move-ego-0-0274.68±1.48261.63±1.76264.32±1.95267.57±0.95178.92±92.57 move-ego-0-2260.93±5.2187.46±21.99252.75±15.03260.35±2.58250.92±6.75 move-ego-0-4235.44±29.42133.47±33.86234.14±8.81233.02±14.75201.90±38.55 move-ego-180-2227.34±27.01232.14±20.35141.56±32.41139.12±83.74222.83±28.29 move-ego-180-4205.54±14.40109.04±27.8971.42±19.9853.37±25.6581.92±29.90 move-ego-90-2210.99±6.55217.16±26.35214.64±37.08178.96±45.28221.43±33.90 move-ego-90-4202.99±9.33154.20±41.82104.73±20.95102.51±63.37160.31±37.02 move-ego-low–90-2221.37±35.3575.28±29.8076.96±49.83126.80±80.7630.15±26.04 move-ego-low-0-0215.61±27.63168.33±5.95150.34±62.07188.29±49.41133.68±52.10 move-ego-low-0-2207.27±58.0182.66±20.5573.60±49.86104.77±23.0066.84±44.92 move-ego-low-180-265.20±32.6452.38±27.6746.28±22.2843.90±39.8628.71±12.52 move-ego-low-90-2222.81±21.94100.75±39.2753.20±21.2685.54±82.0463.19±42.99 raisearms-h-h199.88±42.03192.49±101.9194.64±94.26171.41±71.90217.09±34.35 raisearms-h-l167.98±82.03226.33±35.5590.57±68.3782.42±43.38201.33±87.51 raisearms-h-m104.26±81.69100.49±76.1261.82±20.38112.16±76.75155.36±85.85 raisearms-l-h243.16±19.18255.41±1.55128.56±63.06136.49±85.25233.82±27.26 raisearms-l-l270.43±0.37251.82±9.70260.48±3.52258.50±6.0739.87±34.16 raisearms-l-m97.66±81.17135.05±80.31254.91±3.7891.49±46.58217.42±39.30 raisearms-m-h75.05±69.3279.25±31.9941.58±13.58126.62±80.07107.70±79.18 raisearms-m-l134.83±70.28218.22±46.82173.28±72.83155.21±71.93220.67±50.89 raisearms-m-m87.25±98.42179.60±74.63109.47±91.6282.36±38.59211.30±48.89 rotate-x–5-0.82.29±1.781.69±2.321.49±1.430.29±0.182.44±2.02 rotate-x-5-0.87.42±5.692.55±1.290.53±0.430.32±0.286.43±3.15 rotate-y–5-0.8199.08±51.785.87±3.632.13±2.171.04±0.118.18±4.71 rotate-y-5-0.8217.70±43.674.86±1.441.58±0.440.89±0.1314.03±12.12 rotate-z–5-0.8124.95±17.610.72±0.790.42±0.300.31±0.2317.09±9.10 rotate-z-5-0.895.23±15.751.71±1.670.39±0.370.38±0.220.66±0.76 sitonground199.44±22.155.88±4.7527.39±22.1926.12±21.6997.88±34.91 split-0.5232.18±20.2612.64±14.4834.31±32.9887.22±5.9255.50±33.46 split-1117.67±61.276.80±9.146.12±7.176.13±5.7213.02±16.90 Table 8:Comparing (over 4 seeds) FB, PSM, HILP,RLDPperformance on SMPL Humanoid. FB-CPR (online oracle baseline) results are from Tirinzoni et al. (2025). Bold indicates the best mean across methods. Figure 10:Evaluating offline representation learning methods using an online oracle policy in high dimensional 3D humanoid. Solid lines shows mean performance across tasks for each of the method. A.9FULL RESULTS OND4RL Table 9 shows the normalized average returns ofRLDPand baseline methods on six low-coverage D4RL environments over 10 seeds.RLDPachieves the best mean performance on 5/6 tasks. Using Welch’s t-tests with Holm correction,RLDPsignificantly outperforms FB on all tasks and 24 Published as a conference paper at ICLR 2026 TaskFBPSMHILPRLDP halfcheetah-medium-expert-v252.17±11.3749.92±26.8968.47±8.2086.03±8.36 halfcheetah-medium-v239.27±8.7142.64±0.6443.85±1.4949.08±1.93 hopper-medium-expert-v254.64±19.4714.59±26.3568.18±18.1977.21±16.77 hopper-medium-v243.75±6.6533.49±26.7152.19±4.0244.93±13.08 walker2d-medium-expert-v260.17±28.9479.32±41.1693.72±21.12103.87±3.31 walker2d-medium-v243.72±24.9555.70±22.0156.34±15.0883.83±2.66 Table 9:Normalized returns comparing FB, PSM, HILP, andRLDPin low-coverage setting.RLDPshows significant gains over approaches that rely on explicit Bellman backups for representation learning. Table shows mean±std over 10 seeds;Boldfaceindicates the highest mean return per environment. Statistical comparisons use per-seed returns with Welch’s t-test and Holm correction; cases where the bolded method is not significantly better than the runner-up are discussed in the text. significantly outperforms PSM on 5/6 tasks.RLDP’s gains over HILP are statistically significant on 3/6 tasks, while differences on Hopper-medium-expert and Walker2d-medium-expert are not statistically decisive with 10 seeds due to high variance. On Hopper-medium, HILP has the highest mean but the HILP–RLDPdifference is not significant under Welch’s test, indicating comparable performance under seed variability. A.10VISUALIZATIONS OFLEARNEDSUCCESSORMEASURES (a)(b) (c)(d) Figure 11:Visualization of successor measuresM π z (s 0 ,a 0 ,s + )for randomly sampledz(a) and (b); and goal-conditionedz(c) and (d). 25 Published as a conference paper at ICLR 2026 We used a four room gridworld (as used in Touati & Ollivier (2021); Agarwal et al. (2024)) to plot the successor measures learned byRLDP. We collect a dataset of all transitions and run RLDPwith horizon1to learn representationsφ, successor featuresψand policyπ. Note that any policy parameterized by latentzproduces a successor measure parameterized byM π z (s,a,s + ) = ψ(s,a,z) T φ(s + ). We have plotted the observed successor measures:M π z (s 0 ,a 0 ,s + ), where we fixs 0 anda 0 for a few differentzin figure 11. We have fixeds 0 to the corner state anda 0 to action: right. We have plotted the policy for visualizing the policy represent byz. A.11EVALUATING THE EFFECT OF DIFFERENT LOSSES ON REPRESENTATION LEARNING In table 10, we ablate the objectives used to learn representations and compare (a) Bellman-style loss used in FB, (b) latent prediction loss used inRLDP, and c) orthogonality regularization, that both FB and RLDP use. We evaluate these loss objectives individually and combined with unit scaling. L Bellman (φ,ψ) =−E s,a,s ′ ∼ρ [ψ(s,a,z) T φ(s ′ )] + 1 2 E s,a,s ′ ∼ρ,s + ∼ρ [(ψ(s,a,z) T φ(s + )−γ ̄ ψ(s ′ ,π z (s ′ ),z) T ̄ φ(s + )) 2 ](17) L Ortho (φ) =E s,s ′ ∼ρ [φ(s) ⊤ φ(s ′ )](18) L Prediction (φ,g,w) =E τ i ∼d O " H−1 X t=0 h i t+1 − ̄ φ(s i t+1 ) 2 2 # , h i 0 =φ(s i 0 ),h i t+1 =g(h i t ,a i t )w (19) EnvironmentReturns (mean±std) halfcheetah-medium-expert L Ortho 51.36±6.93 L Bellman 64.42±7.59 L Prediction 49.00±4.34 L Bellman +L Prediction +L Ortho 89.45±3.81 L Bellman +L Ortho (FB)55.46±7.75 L Prediction +L Ortho (RLDP)88.55±6.31 hopper-medium-expert L Ortho 56.13±3.40 L Bellman 59.02±12.19 L Prediction 54.63±3.51 L Bellman +L Prediction +L Ortho 49.25±11.90 L Bellman +L Ortho (FB)49.93±28.78 L Prediction +L Ortho (RLDP)75.53±12.70 walker2d-medium-expert L Ortho 95.10±12.28 L Bellman 89.12±19.36 L Prediction 97.52±14.27 L Bellman +L Prediction +L Ortho 27.10±22.51 L Bellman +L Ortho (FB)36.85±14.89 L Prediction +L Ortho (RLDP)101.30±3.92 Table 10:Results in D4RL expert environments across different loss combinations aggregated over 4 seeds. The results in table 10 highlight that the interaction between different objectives matters much more than any individual loss in isolation. Across all three expert datasets, we find that a combined objective of prediction loss and orthogonality regularizationyieldthelargest returns. In addition, adding Bellman backup to this objective with unit scaling results in inconsistent final returns. Overall, these results support the designchoiceofusinglatent predictionwithorthogonal regularizationastheprimary representationlearningobjective in low-coverage settings due to its high-performance and robustness across multiple settings. We leave a more exhaustive study of this setting to future work, including evaluating these objectives on a wider set of environments and exploring principled scaling for the loss components. 26