Paper deep dive
The Geometry of Empowerment
Catherine Ji, Vivek Myers, Sergey Levine, Benjamin Eysenbach
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/13/2026, 4:34:10 AM
Summary
This paper investigates the theoretical relationship between empowerment (an information-theoretic measure of controllability) and structural centrality in Markov Decision Processes (MDPs). It distinguishes between potential empowerment (maximal control capacity) and effective empowerment (actualized control via skills). The authors prove that high potential empowerment states correspond to central states in both information geometry and temporal distance geometry, providing a lower bound on an agent's ability to adapt to downstream tasks in tabular settings. However, they also demonstrate that in continuous-state MDPs or with anisotropic reward priors, information and reward geometries diverge, meaning empowered states do not guarantee adaptability.
Entities (10)
Relation Signals (7)
Skill Learning Methods → maximizes → Effective Empowerment
confidence 96% · mutual information skill learning (MISL)... maximize effective empowerment
Potential Empowerment → correspondsto → Temporal Distance Centrality
confidence 95% · The empowerment objective... capture centrality in a temporal distance geometry
Potential Empowerment → correspondsto → Information Geometry Centrality
confidence 95% · High potential empowerment states lie at centers of reachable regions in the state occupancy simplex... empowerment is closely related to a probabilistic notion of centrality in information geometry
Continuous-State MDPs → exhibitsdivergencebetween → Information and Reward Geometries
confidence 94% · Attempting to extend the adaptation result to continuous-state MDPs reveals that information and reward geometries can diverge
Anisotropic Reward Priors → causesmisalignment → Empowerment and Adaptation
confidence 93% · The same misalignment appears in discrete settings once the reward prior is anisotropic... empowered agents are not generally adaptive agents
Effective Empowerment → quantifies → Adaptation
confidence 92% · effective empowerment quantifies the adaptation.
Potential Empowerment → lowerbounds → Adaptation to Downstream Tasks
confidence 90% · these notions of centrality directly lower bound an agent’s ability to adapt to downstream tasks in tabular settings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Empowerment captures the capacity for an agent to actively control its environment. While conceptually appealing as an information-theoretic quantity, the connection between empowerment and structurally central states that provide broad access to future outcomes has remained an open question. In this work, we link empowerment maximization and skill-learning methods to provide new geometries for interpreting and analyzing empowerment. Our analyses answer longstanding open questions on the connections between empowerment and structural centrality. Our analyses also reveal distinctions between information and reward geometries, highlighting important theoretical implications to build scalable empowerment-maximization methods. Website and code can be found at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.07796v1
- Canonical: https://arxiv.org/abs/2610.07796v1
Trouble viewing inline? Open PDF directly →
Full Text
126,518 characters extracted from source content.
Expand or collapse full text
The Geometry of Empowerment Catherine Ji Department of Physics Princeton University cj7280@princeton.edu Vivek Myers Department of Computer Science UC Berkeley vmyers@berkeley.edu Sergey Levine Department of Computer Science UC Berkeley Benjamin Eysenbach Department of Computer Science Princeton University Abstract Empowerment captures the capacity for an agent to actively control its environment. While conceptually appealing as an information-theoretic quantity, the connection between empowerment and structurally central states that provide broad access to future outcomes has remained an open question. In this work, we link empowerment maximization and skill-learning methods to provide new geometries for interpreting and analyzing empowerment. Our analyses answer longstanding open questions on the connections between empowerment and structural centrality. Our analyses also reveal distinctions between information and reward geometries, highlighting important theoretical implications to build scalable empowerment-maximization methods.11 1 Website & Code: https://empowerment-geometry.github.io/ 1 Introduction max ℰpotE_pot (potential) max ℰeffE_eff (effective) (a)(b) πpot(a∣s) _pot(a s) πeff(a∣s,z;s0) _eff(a s,z;s_0) Figure 1: Two trained empowerment-maximizing policies characterize central states in an MDP: Consider an agent (robot) on a hill that is easier to descend than to ascend. In Fig. 1(a), the potential policy ascends the empowerment landscape to the top of the hill, using E pot (s) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s) as an intrinsic reward. In Fig. 1(b), The effective policy π eff maximizes Iπ(Z;S+∣S0)I^π(Z;S_+ S_0), associating each skill z with a downhill route. An intelligent agent should be in control of its future. For example, a well-trained hiker at a mountain can (1) choose and traverse down one of many trails from a mountain peak (Fig. 1(b)) or (2) navigate to and stay at the peak (Fig. 1(a)). While both useful, these behaviors are distinct: the first behavior characterizes “actively exerting control” while the latter characterizes “being in control.” Cleanly parameterizing, optimizing, and separating these behaviors requires a tractable mathematical formulation of controllability. Empowerment is an information-theoretic object that quantifies controllability in terms of bits. As originally defined in [1], potential empowerment measures the maximal bits an agent’s actions can inform its future state from some current state. In other words, the potential empowerment measures how much control an agent could command from a given state. An alternative measure of empowerment, termed here as effective empowerment, is a mutual information that quantifies how readily an agent’s policy actively informs its future state [2]. While sometimes conflated, these two objects are different. In the example of the mountain in Fig. 1, a potential empowerment policy ascends the mountain to a state from which the agent has options. Meanwhile, the effective empowerment policies actively exercise options and descend the mountain in different directions. Empowered States have Broad Access. Prior works conjecture that empowerment is a generic way for agents to do goal-oriented tasks [3, 4]. Salge et al., [3, §4.3.3] call this the AI Empowerment Hypothesis: Empowerment provides a task-independent motivation that generate AI behaviour which is beneficial for a range of goal-oriented behaviour. However, empowerment does not have immediate obvious connections to downstream task performance. Indeed, while early works in empowerment suggest a link between empowerment and preferred or central states [5, 6, 3], to the best of our knowledge, formally characterizing notions of MDP centrality and adaptation in relation to empowerment has remained a longstanding open problem. In this work, we formally show high potential empowerment states lie at centers of reachable regions in the state occupancy simplex (). The empowerment objective and long horizon generalizations also admit exact hitting time interpretation (Lemma 4.2) and capture centrality in a temporal distance geometry [7]. In addition to providing intuition, these notions of centrality directly lower bound an agent’s ability to adapt to downstream tasks in tabular settings (), providing a partial answer to Salge’s longstanding hypothesis. Thus, potentially empowered agents can reach central states from which they can adapt, where effective empowerment quantifies the adaptation. Information Geometries and Reward Geometries can Diverge in Continuous-State Markov decision process (MDP) s. The above results theoretically answer questions in the empowerment literature: (1) empowered states are central states and (2) in discrete settings, empowered states are states from which agents can adapt to downstream tasks. Attempting to extend the adaptation result to continuous-state MDP s reveals that information and reward geometries can diverge: an arbitrarily empowered state can provide arbitrarily small adaptation in a continuous MDP with continuous rewards (Proposition G.2). The same misalignment appears in discrete settings once the reward prior is anisotropic (). Thus, empowered agents are not generally adaptive agents in the continuous-state MDP s. Scalable formulations of empowerment maximization must address this discrepancy between information and reward adaptation geometries. Taken together, our results give an information-geometric centrality theorem for the empowerment objective and a complementary temporal interpretation over skill rollouts. These theoretical interpretations provide an answer to longstanding questions about the relationship between empowerment, centrality and access, and adaptation to downstream reward functions: in tabular settings, empowered agents navigate to states centrally-located in an information geometry from which they can adapt. When considering continuous-state MDP s and/or anisotropic reward priors, this intuition no longer holds. Thus, our theoretical results show that scalably unifying empowerment and adaptation requires alternative formulations that align information and reward geometries. 2 Related Work As originally introduced, the empowerment at a state is the maximum possible information that can be communicated from open-loop action sequences to future states [8], though closed-loop variants have also been discussed [3, 9]. An empowerment maximizing agent should maximize the empowerment of its occupied states. Capdepuy, [2] distinguishes this channel capacity form of empowerment, termed potential empowerment, from the mutual information between the action sequences and a future state realized by a specific policy, termed actualized empowerment. In this work, we refer to this form of empowerment as effective empowerment. These quantities are formalized in 3. Where should an agent go if told that it will be asked to solve some soon-to-be-specified task? One approach is to consider the geometry of the state space and navigate to central, connected states or bottleneck states within the MDP: states from which the agent can easily visit many other states [10]. Such notions of MDP centrality have been studied in both reinforcement learning [6, 5, 11, 12] and neuroscience [13] community, often in the context of exploration. A related line of work studies whether optimal behavior under task uncertainty implies being at central states and actions. In particular, Turner et al., [12] show that, under symmetry assumptions on the MDP and uncertainty in the reward function, optimal agents prefer state-actions with many future options. While these notions and definitions of centrality and bottlenecks are appealing, these objects are only well-defined in settings with discrete states and actions. Prior empowerment literature has conjectured that potential empowerment measures the degree of centrality of a state in an MDP [5, 6]. For example, Salge et al., [3] note that potential empowerment exactly corresponds to the log of the total reachable states in discrete, deterministic settings. In stochastic settings, prior works discuss the relationship between empowerment and closeness centrality and show correlations between these quantities in simple didactic experiments; however, no formal link between centrality and empowerment is provided. Jung et al., [6] mention the possibility that empowerment might identify “gateway” states, but do not explicitly test nor quantify this observation in experiments. Our work provides a rigorous grounding to these observations, formally connecting potential and effective empowerment to two different notions of centrality: centrality in temporal distance geometry and centrality in the information geometry. One important consequence of our theoretical results is a notion of centrality that remains well defined in continuous state spaces. A natural generalization of discrete measures of reachability and closeness is the temporal distance [7]. In discrete settings, the temporal distance corresponds to a simple function of the hitting time from state-to-state. Previous work has shown temporal distances can effectively guide exploration [14] and goal-reaching [15, 16] in continuous, high-dimensional tasks. To our knowledge, there has not been prior work utilizing temporal distances to identify and characterize centralized or bottleneck states. Here, we show that empowerment corresponds to the reduction in time/temporal distance to future states. Estimating and Maximizing Empowerment in Practice. There are two different forms of empowerment: potential empowerment and effective empowerment [2]. While prior works discuss maximizing both quantities and their variants [17, 18, 19, 4], to the best of our knowledge, there have been no prior works that scalably maximize potential empowerment in high-dimensional, continuous-state settings. Meanwhile, there has been substantially more prior work in RL estimating and maximizing effective empowerment (or similar) quantities in high-dimensional, continuous settings, in a line of work termed mutual information skill learning (MISL). Prior works use mutual information objectives to learn a latent z-conditioned policy π(a∣s,z)π(a s,z) that transmits bits about an input vector Z to future states S+S_+ via its actions (i.e., closed-loop control [18, 20, 21, 22]). These methods learn skills π(a∣s,z)π(a s,z) that actively transmit information to the future, which are high effective empowerment policies. By making use of contrastive and variational estimators of mutual information, prior works scale these methods to learn reactive policies in high-dimensional (e.g., with images) continuous-control tasks [21]. While some of these skill-learning methods mention empowerment [18], they do not directly optimize potential empowerment as defined in [1] and instead only optimize effective empowerment. Recently, empowerment and its variants have also served as objectives for assistance [23, 24, 25]. While we do not present experiments in high-dimensional or continuous settings, we highlight these lines of work to emphasize that empowerment and related objectives underpin state-of-the-art intrinsic rewards for unsupervised reinforcement learning (RL) and new assistance methods. Understanding the theoretical reasons to maximize or modify empowerment therefore directly informs the design of these important methods. 3 Preliminaries We will define notions of empowerment in terms of occupancy measures over MDP s, following a long line of work that casts decision making as probabilistic inference [26, 27, 28, 18, 29]. 3.1 Discounted State Occupancy MDP problem setting. Our analysis focuses on applying information theory to a fully-observed MDP with states s∈s , actions a∈a , and environment dynamics p(s′∣s,a)p(s s,a). Let the skill and starting state-conditioned policy be the function π(a∣s,z;s0)π(a s,z;s_0) with skill z∈z . The latent skill z lives in a continuous skill space ⊆ℝdZ ^d (i.e. the surface of a hypersphere), including in tabular MDPs. The policy π(a∣s,z;s0)π(a s,z;s_0) is a distribution over actions given the current state s, the latent z, and the start state s0s_0. We write Π _Z for this class of skill- and starting-state conditioned policies, and Π for the class of ordinary policies π(a∣s)π(a s) that carry no skill. Fixing a skill in a policy of Π _Z leaves an ordinary policy in Π . The start-state s0s_0 is the context of a skill [8]. Reachable state-occupancy polytope. For some initial state s0s_0, skill latent z∈z , and skill and start-state-conditioned policy π(a∣s,z;s0)π(a s,z;s_0), the discounted state occupancy measure (DSOM) takes the form pπ(s+=s∣s0,z)≜(1−γ)∑t=0∞γtpπ(st=s∣s0,z),p^π(s_+=s s_0,z) (1-γ) _t=0^∞γ^tp^π(s_t=s s_0,z), llasection q:dsom (1) where pπ(st=s∣s0,z)p^π(s_t=s s_0,z) is the probability, under policy π and the MDP dynamics, visits state s after t time steps when initialized at state s0s_0 under skill z. Equivalently, the DSOM is the probability measure of the occupied future state S+S_+ at a geometric termination time τ∼Geom(1−γ)τ (1-γ). Averaging over the skill distribution p(z∣s0)p(z s_0) gives the skill-marginal DSOM, pπ(s+∣s0)=p(z∣s0)[pπ(s+∣s0,z)],p^π(s_+ s_0)=E_p(z s_0) [p^π(s_+ s_0,z) ], llasection q:dsom_marginal (2) the future-state distribution induced by the policy when the skill is unknown. Every DSOM is a distribution over states within the simplex Δ() (S). For a fixed MDP, an agent can only realize a strict subset of DSOM in the simplex. For a fixed start state s0s_0, the realizable DSOM s form the reachable state-occupancy polytope (s0)≜pπ(s+∣s0):π∈Π⊆Δ(),P(s_0) \\,p^π(s_+ s_0)\;:\;π∈ \, \\; \; (S), llasection q:polytope (3) a convex subset of the simplex whose linear constraints are the Bellman flow equations (see the polytope inside the simplex in Fig. 3(a)) [30]. We write (s)P(s) for the polytope reachable from a generic start state s. Our analysis in §. 4 studies the geometry of exactly this polytope. 3.2 Empowerment Preliminaries We consider two forms of empowerment: potential (the maximal empowerment that could be achieved at a state) and effective (the actualized influence of a particular skill-conditioned policy (i.e., collection of policies) at a given state on the future state distribution). Potential and Effective Empowerment. Our work highlights the difference between empowerment, as originally defined in Klyubin et al., 2005b [31] and the mutual information (MI) objects directly maximized in MISL methods [9, 20, 22, 21]. We term these forms of empowerment as potential and effective empowerment, respectively. The distinction between these forms of empowerment is not new and has been pointed out in prior work [2], though (to our knowledge) modern works do not always differentiate between these objects [9]. Klyubin et al., 2005b [31] originally defines (potential) empowerment as the degree to which action sequences of length H from a current state (i.e. input of the information channel) control the future state (i.e. output). Formally, E pot H(s)=maxp(a0H−1)I(A0H−1;SH∣S0=s) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot_H(s)= _p(a_0^H-1)I(A_0^H-1;S_H S_0=s) where A0H−1≜(A0,…,AH−1)A_0^H-1 (A_0,…,A_H-1) denotes the action sequence. In our work, we make two changes. First, rather than considering entire sequences of actions as inputs (i.e. open-loop controls), we instead consider latents z that parameterize a reactive policy. We refer to these skill-conditioned, closed-loop controls as “skills”. Potential empowerment then measures the maximal influence these skills exert on a future state. Second, rather than measure these skills’ influence on a state H steps in the future, we measure how much skills control states at a geometric stopping time, so the discount factor sets the timescale of the empowerment. Definition 3.1 (Potential empowerment). Let policies π∈Ππ∈ _Z condition on skill latents z∈z and let S+S_+ denote the random state s+∼pπ(s+∣s0,z)s_+ p^π(s_+ s_0,z) sampled from the DSOM. Potential empowerment is the channel capacity over a family of policies dependent on an initial state [1, 2] E pot (s0) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0) =maxπ∈ΠIπ(Z;S+∣S0=s0). = _π∈ _ZI^π(Z;S_+ S_0=s_0). Definition 3.2 (Potential empowerment policy). A potential empowerment policy π pot ∈Π @glossaries@gls@linkmainpi_mathrmpot glo:pi_mathrmpot _pot∈ that takes the form π pot (a∣s) @glossaries@gls@linkmainpi_mathrmpot glo:pi_mathrmpot _pot(a s) maximizes the potential empowerment as a reward: π pot ∗=argmaxπ∈Ππ[∑tγtE pot (st)]. @glossaries@gls@linkmainpi_mathrmpot glo:pi_mathrmpot _pot^*= _π∈ E_π [ _tγ^t @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_t) ]. While conceptually similar, mutual information skill learning methods do not maximize potential empowerment, and maximize effective empowerment. Effective empowerment captures the MI itself: Definition 3.3 (Effective empowerment). Consider skill and start state-conditioned policies π∈Ππ∈ _Z that take the form π(a∣s,z;s0)π(a s,z;s_0). Effective empowerment is the mutual information realized by a particular skill and start state-conditioned policy at a particular starting state s0s_0 under the uniform skill source p(z∣s0)=Unif()p(z s_0)=Unif(Z): E eff (π,s0) @glossaries@gls@linkmainge_mathrmeff glo:ge_mathrmeffE_eff(π,s_0) =Iπ(Z;S+∣S0=s0)=p(z∣s0)DKL[pπ(s+∣s0,z)∥pπ(s+∣s0)] =I^π(Z;S_+ S_0=s_0)=E_p( [rgb]0.5,0.5,0.5z s_0)D_KL[p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|p^π( [rgb]0.5,0.5,0.5s_+ s_0)] Correspondingly, we refer to the skill and start state-conditioned policy π(a∣s,z;s0)π(a s,z;s_0) as the effective empowerment policy. This objective is similar to the objectives directly maximized in modern skill-learning algorithms [18, 20, 21]; note that MISL methods generally assume a fixed starting state s0s_0, so the learned skills are specific to s0s_0. Thus, we also refer to the effective empowerment policies as skills. Importantly, effective empowerment involves no maximization and measures the MI achieved by a particular skill-conditioned policy π∈Ππ∈ _Z that begins from a particular starting state s0s_0. This MI depends on both the channel z↦pπ(s+∣s0,z)z p^π( [rgb]0.5,0.5,0.5s_+ s_0,z), determined by the policy, and the source p(z∣s0)p(z s_0) set to uniform. Potential empowerment is the maximum-achievable effective empowerment for a given starting state s0s_0, so that E pot (s0)=maxπ∈Πmaxp(z∣s0)Iπ(Z;S+∣S0=s0). @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _π∈ _Z\; _p(z s_0)I^π(Z;S_+ S_0=s_0). (4) Under a uniform skill prior, the policy can represent any skill source by simply relabeling skills (Lemma D.1). Fixing the source p(z∣s0)p(z s_0) to uniform therefore loses no generality. We include a full statement of this result in Appendix D. Thus, a learned skill set (effective empowerment policies) acts as a probe of the potential empowerment at a starting state. If skills initialized at s0s_0 induce easily distinguishable future occupancies pπ(s+∣s0,z)p^π(s_+ s_0,z), then s0s_0 affords many controllable futures and has high potential empowerment E pot (s0) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0). Notation convention. Throughout the paper, we write π for the effective policy when context is clear and π eff only when both policies appear in the same expression. The potential policy is always written as π pot . The object ℰ(s)E(s) always refers to potential empowerment measured at starting-state s. 4 The Geometry of Empowerment Prior work has empirically observed a close connection between geometry and empowerment [5, 6], wherein states that are more centralized in an MDP have higher empowerment. However, to the best of our knowledge, such connections have not been proven theoretically. In this section, we show that empowerment is closely related to a probabilistic notion of centrality in information geometry [32] and temporal distance geometry [7]: • §. 4.1: Effective empowerment corresponds to the information-theoretic distance between states a specific skill-conditioned policy may reach in the future. Potential empowerment is the maximum over these distances from a starting state and thus measures the radius of a reachable space. • §. 4.2: High empowerment states are central in temporal distance geometry. From a current state, the potential empowerment measures the average time saved to reach future states when committing to a skill. We then connect empowerment to adaptation on downstream tasks (§. 4.3). We show a positive result, a lower bound on adaptation, but also prove important limitations that highlight the fundamental difference between information and reward geometries: • §. 4.3, : The potential empowerment of a state lower bounds adaptation to unknown reward functions drawn from an isotropic distribution. • §. 4.3, Proposition G.2: In continuous settings, a state can have unbounded potential empowerment while providing arbitrarily small adaptation to a continuous reward. • §. 4.3, Proposition H.2: For anisotropic reward distributions, an empowered state can provide no adaptation. 4.1 Information Geometry of Potential and Effective Empowerment Policies. Figure 2: Skills measure the centrality of a state. We visualize the potential empowerment field ℰ(s)E(s) (a) for each state in a tabular GridWorld. The central state has high potential empowerment, because (b) the central state gives an agent access to a maximal set of distinguishable skills. A potential empowerment policy ascends the potential empowerment landscape to the center state in (c), from which it can execute different skills. See for the formal statement. See Appendix I for tabular experiment details. Geometric intuition. Prior work has found empirically that high empowerment states are central states in an MDP, relating empowerment to the volume of reachable states in discrete, deterministic settings [3]. Here, we extend definitions of centrality to the full MDP setting with stochasticity and continuous states. Effective empowerment is a mutual information. In the dual form, effective empowerment maximization finds state occupancy distributions that lie at the center of extremized state distributions. Then, potential empowerment-seeking policies search for the states that lie at the centers of the largest radii reachable polytopes. Before presenting formal results, we visualize an example of the information geometry of empowerment in Fig. 3. Let the triangle visualize the state occupancy simplex reachable from different states s1s_1 and s2s_2. Maximizing the mutual information (effective empowerment) for a given starting state corresponds to learning skills that reach vertices of the set of reachable state occupancies. The agent maximizes effective empowerment separately at every possible starting state (orange for s1s_1, teal for s2s_2) and learns skills (orange and teal arrows) that span reachable regions. Then, to maximize potential empowerment, an agent searches among these starting states for the largest reachable region. In this case, the teal state s2s_2 corresponds to the state with the highest potential empowerment and occupies a more centralized point within the information geometry. High Empowerment States are Central in Information Geometry. Formally, consider the state probability simplex Δ() (S), where DSOM are elements of the simplex pπ(s+∣s0,z)∈Δ()p^π(s_+ s_0,z)∈ (S). A straightforward extension of prior work [30] shows that effective empowerment maximization is a center localization problem in the initial state s0s_0-conditioned state probability simplex, where distance is given by the Kullback–Leibler divergence [33]. Then, potential empowerment measures the min-max distance to the reachable state-occupancy polytope: Theorem 4.1. Let (s0)⊂Δ()P(s_0)⊂ (S) be the reachable state-occupancy polytope of Eq. 3. Jointly maximizing the mutual information over the skill source and the skill-conditioned policy solves the KL 1-center localization problem for (s0)P(s_0): E pot (s0)=maxπ∈Π,p(z∣s0)Iπ(Z;S+∣S0=s0) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _π∈ _Z,\,p(z s_0)I^π(Z;S_+ S_0=s_0) =minq(s+)∈Δ()maxp∈(s0)DKL(p(s+)∥q(s+)). = _q(s_+)∈ (S) _p (s_0)\;D_KL (p( [rgb]0.5,0.5,0.5s_+)\|q( [rgb]0.5,0.5,0.5s_+) ). (5) Moreover, every joint optimum (π,p(z∣s0))(π,p(z s_0)) places every skill on the boundary of the optimal KL ball measured from the optimal central occupancy q⋆q : DKL(pπ(s+∣s0,z)∥q⋆(s+)) D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q ( [rgb]0.5,0.5,0.5s_+) ) =maxp∈(s0)DKL(p(s+)∥q⋆(s+))for every skill z. = _p (s_0)D_KL (p( [rgb]0.5,0.5,0.5s_+)\|q ( [rgb]0.5,0.5,0.5s_+) ) every skill z. (6) In other words, the optimized skills span the maximal information radius of the reachable polytope given starting state s0s_0. We prove a general version of in Appendix B. This result shows that an exploration agent maximizing potential empowerment will seek central states in the information geometry. Prior work on empowerment was correct in conjecturing that empowerment maximization has a formal connection to centrality. However, the right geometry for studying this centrality is not (say) the Euclidean metric on states, but rather the Kullback-Leibler (KL) divergence between state distributions induced by different skills. (a) State occupancy simplex (b) Potential policy finds central states. Figure 3: Potentially empowered states are central states in the polytope. Tabular intuition for . (a) We can use a probability simplex Δ() (S) to visualize the discounted state occupancy measures of each skill (circles). Orange circles indicate the best set of skills starting from state s1s_1, and teal circles indicate the best skills starting from state s2s_2. from starting states s1s_1 and s2s_2 in arbitrary MDP. The skills starting from s1s_1 define a smaller polytope (orange points) in the simplex than s2s_2 (teal points) as a result of their transition dynamics, so the potential empowerment is higher at s2s_2 than s1s_1. (b) We use a GridWorld to illustrate the connection between centrality and tool use. An agent in the GridWorld begins at the bottom of the hallway. The agent can set down and pick up a key at the top of the hallway. The key provides the agent access to shaded regions of the MDP. (b, i) A potentially empowered agent picks up the key then navigates to the center of the room, correctly identifying the middle state with the key as the most central state. (b, i) From the center, the agent can execute the maximal number of distinguishable skills. See §. I.3 for experimental details. A simple tabular experiment demonstrates this notion of centrality (Fig. 3(b)). In this MDP, the agent can pick up and drop off a key at the top of the hallway in addition to Left, Right, Up, Down, Stay\Left, Right, Up, Down, Stay\ actions. Picking up this key allows access to the blue-shaded regions of the MDP; without the key, the agent cannot enter these spaces in the grid. The potential empowerment policy π pot that ascends the potential empowerment landscape correctly reaches the true central state: the center of the MDP with the key. This experiment shows that states with maximal potential empowerment capture an information-geometric notion of centrality, rather than navigating towards the physically central state. 4.2 Empowerment, Bottlenecks, and Temporal Distance Beyond information geometry, the empowerment objective has a direct hitting time interpretation in discrete state spaces. Suppose that the agent has multiple trial trajectories to reach a future state, where each trajectory rolls out a skill from the same starting state until a geometric termination time. The empowerment measures the reduction in hitting time when the agent commits to a skill: Lemma 4.2 (Bottleneck states are high potential empowerment states). In a discrete MDP, let the hitting time random variable Hπ eff (s0→s+)H @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_0→ s_+) be the number of rollouts from s0s_0 until a rollout hits s+s_+, for skills Z∼p(z∣s0)Z p( [rgb]0.5,0.5,0.5z s_0). Let Hπ eff (s0→s+∣z)H @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_0→ s_+ z) be the corresponding hitting time when rolling out z every time. Then, E pot (s0)=maxπ eff ∈Πpπ eff (z,s+∣s0)[log[Hπ eff (s0→s+)]−log[Hπ eff (s0→s+∣z)]]. @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _ @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff∈ _ZE_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(z,s_+ s_0) [ [H @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_0→ s_+)]- [H @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_0→ s_+ z)] ]. We prove Lemma 4.2 in Appendix C. In tabular settings, potential empowerment measures the reduction in time to reach future states over independent rollouts when committing to an option/skill. However, hitting times are not well-defined in continuous geometries, where hitting a particular state is a measure-zero event. The temporal distance is an object that remains well-defined in continuous settings and reduces to hitting times in the discrete limit [7]. In the appendix, we show that a closely related empowerment objective over contiguous (versus repeated) skill rollouts, where the agent re-samples a skill at each geometric termination in a single trajectory, admits an exact temporal distance formulation [7]. In that construction, the empowerment object measures the reduction in temporal distance to future states after committing to a skill (Proposition C.1), recovering a hitting-time interpretation in the discrete, deterministic limit. Conceptually, these temporal formulations capture the bottleneck effect. From a bottleneck, an agent must commit to different skills in order to quickly reach different downstream regions which is conceptually the reduction in temporal distance and hitting times considered in our results. Tabular experiments in Fig. 4 illustrate this effect [13, 34]. Adding slim openings (“bottlenecks”) to the grid shifts the high empowerment states from the original central state to the bottleneck openings, prioritizing states where committing to a skill (e.g. move left or move right) quickly reaches different regions. This interpretation also connects empowerment to closeness centrality [5]. Figure 4: Bottleneck states are high potential empowerment states. Consider GridWorlds with a door, or a “bottleneck”. (a) For short horizon behaviors where γ=0.5γ=0.5, the high empowerment regions are the central door (a, left) and the center of the two sides of the room (a, right). (b) Lifting γ=0.95γ=0.95 increases the horizon of possible behaviors, such that the empowerment tracks the location of the door. In §. 4.2 we show that empowerment measures the reduction in temporal distance to future states, which quantifies a bottleneck effect. See §. I.2 for experimental details. 4.3 Is Empowerment Optimal for Downstream Adaptation? Potential empowerment lower bounds skill adaptation under an uninformed reward prior. While empowerment is often introduced as an appealing “universal” [31] objective, is there a precise sense in which empowerment measures the ability to adapt in the future? In Appendix F, we formally prove that empowerment lower bounds skill adaptation to rewards drawn from an uninformed prior. To formalize adaptation when rewards are not available during training, we define an adaptation objective with respect to a reward prior distribution ℛR. A simple reward prior is a Gaussian distribution centered at a fixed baseline distribution, set to the uniform occupancy over the state space. Prior works use similar isotropic reward distributions when proving general downstream adaptation results [35]. The reward prior treats all states the same and does not prefer particular states. Here, we define adaptation as a skill adaptation to a sampled reward. At test time, an agent must select a skill z∈z to adapt to a sampled reward function r∼ℛr . For example, in the case of the 5×55× 5 grid, an agent should select the skill “go left” if the reward provided at test time is on the left. In the best case, the agent selects the skill z with the highest expected return; note that the agent can do no better within the space of learned skills. This type of policy selection for adaptation/transfer has precedent in prior successor feature literature [26, 36]. For the following result, we consider linear adaptation objective J for the skill set learned by an empowerment-maximizing agent, which is exactly the best skill’s mean return averaged over the reward prior. This adaptation objective is notably different from prior results connecting MI and adaptation in Eysenbach et al., [30], which considers regularized adaptation directly in the state occupancy. Here, the adaptation is the selection of a pre-trained skill behavior, which does not assume the ability to directly adapt state occupancy distributions: Theorem 4.3. Consider the start state s0s_0 in state space S. Let the skill-marginal DSOM be pπ eff (s+∣s0)=p(z∣s0)[pπ eff (s+∣s0,z)]p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0)=E_p( [rgb]0.5,0.5,0.5z s_0) [p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z) ] of Eq. 2. Fix a reference distribution b on S with full support (e.g. the uniform baseline b(s+)=1/||b(s_+)=1/|S|), and sample rewards from the Gaussian r(s+)∼(0,1b(s+)).r(s_+)\ \ N\! (0,\ 1b(s_+) ). Define the linear adaptation objective J(s0)≜r[supz∈pπ eff (s+∣s0,z)[r(s+)]],J(s_0) _r [ _z E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)[r(s_+)] ], which calculates the average adapted mean reward. Then, for any skill space Z and any skill prior p(z∣s0)p(z s_0), J(s0)≥I(Z;S+∣S0=s0)c2π,c≜supx≥0ln(1+x)x≈0.8.J(s_0)\ ≥\ I(Z;S_+ S_0=s_0)c 2π, c _x≥ 0 (1+x) x≈ 0.8. We give the formal statement and proof in (Appendix F), where J(s0)J(s_0) turns out to be the Gaussian width of the skill offsets uzz∈\u_z\_z (Eq. 12). Intuitively, for the simple 5×55× 5 grid (Fig. 2), the central maximal potential empowerment state is exactly the state from which an agent can easily adapt to the left/right/upper/lower rewarding states in the grid. Thus, maximizing potential empowerment provides a new form of provable adaptation where agents navigate to central states from which they can later adapt, which existing skill-learning or effective empowerment methods do not capture on their own [18]. Two didactic tabular experiments demonstrate the behavior of the bound across MDP parameters. The first set of MDP s varies the degree of commitment within an MDP, which measures how much probability mass an action places on a target state above uniform. Commitment measures the controllability of the MDP. As controllability increases, the bound tightens (Fig. 9). The second set of MDP s varies the number of possible distinct skills/actions while holding commitment fixed. As the number of possible skills grows, the bound loosens (Fig. 10). The bound is therefore most informative in MDP s where an agent exerts strong control over comparatively few distinguishable regions. Full experimental details are in Appendix F. The First Limitation of Empowered Adaptation: Gaussian widths versus KL divergence. While the above result demonstrates that potential empowerment lower bounds adaptation in discrete MDPs, there remain open questions: (1) is the bound tight? and, correspondingly, (2) does the bound extend to continuous state geometries? The answer to both of these questions is generally no: without additional assumptions, the geometry of reward adaptation and that of information geometry are different. First, we must define the geometry of reward adaptation. Gaussian width measures the spread of the skills’ future occupancies and governs the adaptation geometry [32]. Because the cumulative reward is linear in the occupancy, the width is large when different skills reach futures with clearly different values (Fig. 5, (b,e)). This quantity differs from MI, where distances are given by the Kullback–Leibler divergence. Our first observation is that the adaptation result is a bound, not a ranking result. Consider the simple example of two starting states in a discrete MDP (Fig. 5, (a) and (d)). The first starting state in (a) can reach two distinct absorbing states. The second starting state (d), on the other hand, can reach three states. Under an isotropic reward prior, an agent optimizing adaptation should choose the second starting state (Fig. 5, (e)) according to the Gaussian-width geometry that characterizes reward adaptation. Meanwhile, an agent optimizing the information radius will choose the first starting state (Fig. 5, (c)). This discrepancy is a direct consequence of the difference between information and reward adaptation geometry. Thus, while the lower bound holds, the bound does not guarantee a ranking result with provable counterexamples. Figure 5: Discrepancy between information geometry and reward adaptation geometry in discrete settings. (a) By going left/right from s, an agent can reach node s1s_1 or s2s_2 with high likelihood, so there are two easily separable skills from s. (d) Targeting nodes s1s_1, s2s_2, or s3s_3 can make the agent inadvertently land in other nodes, so there are three somewhat separable skills from s. (b,e) The MDP s in (a) and (d) admit polytopes of feasible state occupancies in the simplex, visualized by the segment and triangle. (b) Axis u1u_1 trades off occupancy of s1s_1 vs. s2s_2. (e) Axis u1u_1 trades off occupying s2s_2 vs. s3s_3; axis u2u_2 trades off occupying s1s_1 vs. s2/s3 [rgb]0.8672,0.5195,0.3203s_2/ [rgb]0.5781,0.4688,0.375s_3. Return is linear in the state occupancy, so we can also represent reward functions as vectors in state-occupancy space corresponding to the reward weights. The color indicates the best adapted skill for that reward coordinate. The polytope in (e) has a larger Gaussian Width than that in (b), supporting better adaptation. (c,f) However, increased skill separability in (a) gives larger information radius and MI, where the black contour indicates the maximal MI. Thus, even though empowerment is higher for the 2 state system (a) versus the 3 state system (d), the 2 state system has worse adapted returns. See §. I.4 for details. In the continuous state limit, the difference between empowerment and adaptation geometry becomes more salient: Theorem 4.4 (Informal statement that large MI does not necessarily imply adaptation to smooth rewards in continuous state MDP s). For every skill count K and ϵε, there exists a continuous state MDP admitting K distinct skills that induce DSOM s within an ϵε-ball in arbitrary continuous state space. Taking K→∞K→∞, the MDP admits a set of skills that induce unbounded empowerment while adaptation to a Lipschitz-continuous reward can be arbitrarily small. We give the formal statements and proofs in Propositions G.1 and G.2. Our result shows that skills can have unbounded distinguishability but fail to adapt to smooth rewards, inducing identical returns across the skill set Fig. 6 illustrates how information geometries can lead to collapsed returns in a continuous 2D state space, where the potential empowerment grows from left to right while returns homogenize. Thus, larger potential empowerment does not imply better adaptation to smooth rewards in continuous state MDP s. We fully discuss this collapse in Appendix G. Figure 6: The information ball can collapse in the continuous limit (Proposition G.2). Consider an arbitrary smooth reward function (level curves in gray) and some MDP that admits Gaussian skill state occupancy measures (colored distributions). While the potential empowerment of the starting state s∈s increases from subfigure (a) → (c), the skills have increasingly identical returns and, thus, provide progressively smaller adaptation benefit. Therefore, large channel capacities do not imply adaptation for continuous MDP s and reward functions. See Proposition G.2 for the theoretical result. The Second Limitation of Empowered Adaptation: Controllable Degrees of Freedom versus Rewarding Degrees of Freedom. The second limitation of empowered adaptation is the misalignment between controllable degrees of freedom and preferred degrees of freedom. While the adaptation result of requires an isotropic reward prior, naive empowerment maximization cannot generically provide adaptation guarantees for anisotropic reward priors. Without external information, empowerment can only break symmetry across states via controllability. Then, the reward prior is free to vary in directions in the state occupancy simplex that empowerment does not naturally select: Theorem 4.5 (Informal statement that large MI need not imply adaptation to anisotropic reward function priors in discrete settings). A state can have arbitrarily large MI while adaptation with respect to an anisotropic reward prior is arbitrarily close to zero. The formal statements and proofs are in Appendix H ( and H.2). We can reinterpret Fig. 5 as an example where the maximizing MI selects for skills that do not control degrees of freedom relevant to the anisotropic return: if the variation in an anisotropic reward prior concentrates on three absorbing states in Fig. 5(d), an empowerment-maximizing agent will ignore those reward-relevant states and, instead, navigate towards the state with maximal channel capacity. Therefore, the skill set is suboptimal for adaptation to the anisotropic reward prior. 5 Discussion, Limitations, & Future Work In this work, we analyze the geometry of empowerment. Our theoretical results show that high potential empowerment states are central states in information geometry (). The empowerment objective and a multistep skill rollout variant also admit exact hitting time interpretations and capture centrality in a temporal distance geometry (Lemmas 4.2 and C.1). Thus, policies that maximize potential empowerment naturally seek out central states within an MDP. Furthermore, effective empowerment policies, or skills, measure the degree of centrality (Fig. 2). Finally, in the tabular setting, this notion of centrality achieved by a set of skills directly lower bounds an agent’s ability to adapt to unknown rewards (sampled from an uninformed prior). Our results formally link empowerment and the ability to complete downstream tasks (). Limitations & Future Work. That being said, while the above results provide support for empowerment as a generic objective, information geometry and return geometries are fundamentally different: reward adaptation geometry is typically that of Gaussian widths, while information geometry is that of KL-divergences. This observation has a few downstream implications. First, a skill set that obtains a larger MI does not necessarily exhibit better adaptation; our adaptation result is a lower bound, not a ranking result (Fig. 5). Second, large empowerment does not imply adaptation in continuous MDP s with Lipschitz-continuous rewards: there exist MDP s where skill sets can obtain unbounded empowerment, while providing no adaptation benefit to a continuous reward function (Propositions G.2 and 6). Third, empowerment alone does not provide theoretical guarantees on downstream adaptation to anisotropic reward priors () that prefer certain states over others. Thus, while empowerment has attractive properties in addition to a straightforward information-theoretic interpretation, there exist theoretical gaps between the empowerment object and downstream task adaptation. Our work provides a partial answer to Salge’s hypothesis [3]: while empowerment, a task-agnostic quantity, does provably lower bound goal-oriented behavior in tabular settings, this hypothesis is not generally true for current formulations of empowerment maximization and return maximization. Possible future directions include developing scalable formulations of continuous empowerment maximization studied in prior works [6, 37, 38] and modifying recent scalable skill-learning algorithms for empowerment estimation [17, 21, 22]. Until we consider exploration and adaptation methods that appropriately unify these geometries, there will remain a disconnect between such intrinsic reward approaches and the broader RL problem setting. Indeed, our analysis can generically apply to any method that combines extrinsic and information-theoretic intrinsic rewards [39]. We believe that scalably addressing these limitations is key in order to make information-based intrinsic rewards, including empowerment, relevant to the broader RL community and is the natural next step for future work. Acknowledgments. We would like to thank David Abel, André Barreto, Mahsa Bastankhah, Raphaël Bauer, Max Kleiman-Weiner, Andrew Levy, Sikata Sangupta, and anonymous reviewers for their insight and helpful feedback. This material is based upon work supported by Google and the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2444107 and Award No. 2441665. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Generative AI Disclosure Statement. We acknowledge the use of generative AI tools for figure generation, setting up the paper website, code implementation for the tabular experiments, and minor edits to improve readability. References [1] A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: A Universal Agent-Centric Measure of Control. IEEE Congress on Evolutionary Computation, volume 1, p. 128–135 Vol.1, 2005a. [2] Philippe Capdepuy. Informational Principles of Perception-Action Loops and Collective Behaviours. Ph.D. thesis, University of Hertfordshire, 2011. [3] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment–an Introduction. Guided Self-Organization: Inception, p. 67–114. Springer, 2014. [4] Moritz Schneider, Robert Krug, Narunas Vaskevicius, Luigi Palmieri, Michael Volpp, and Joschka Boedecker. Information-Theoretic Policy Pre-Training With Empowerment. arXiv:2510.05996, 2025. [5] Tom Anthony, Daniel Polani, and Chrystopher L Nehaniv. On Preferred States of Agents: How Global Structure Is Reflected in Local Structure. 2008. [6] Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for Continuous Agent—Environment Systems. Adaptive Behavior, 19(1):16–39, 2011. [7] Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making. International Conference on Machine Learning, 2024a. [8] Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Keep Your Options Open: An Information-Based Driving Principle for Sensorimotor Systems. PloS One, 3(12):e4018, 2008. [9] Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. Neural Information Processing Systems, 28, 2015. [10] Gert Sabidussi. The Centrality Index of a Graph. Psychometrika, 31(4):581–603, 1966. [11] Özgür Şimşek and Andrew G Barto. Betweenness Centrality as a Basis for Forming Skills. 2007. [12] Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal Policies Tend to Seek Power. arXiv:1912.01683, 2023. [13] Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. Design Principles of the Hippocampal Cognitive Map. Neural Information Processing Systems, 27, 2014. [14] Yuhua Jiang, Qihan Liu, Yiqin Yang, Xiaoteng Ma, Dianyu Zhong, Hao Hu, Jun Yang, Bin Liang, Bo Xu, Chongjie Zhang, and Qianchuan Zhao. Episodic Novelty Through Temporal Distance. arXiv:2501.15418, 2025. [15] Vivek Myers, Bill Chunyuan Zheng, Benjamin Eysenbach, and Sergey Levine. Offline Goal-Conditioned Reinforcement Learning With Quasimetric Representations. arXiv:2509.20478, 2025. [16] Seohong Park, Deepinder Mann, and Sergey Levine. Dual Goal Representations. arXiv:2510.06714, 2026. [17] Andrew Levy, Alessandro G Allievi, and George Konidaris. Representation Learning and Skill Discovery With Empowerment. Reinforcement Learning Conference, 2025. [18] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational Intrinsic Control. International Conference on Learning Representations, 2017. [19] Thomas J Ringstrom. Reward is not necessary: how to create a modular & compositional self-preserving agent for life-long learning. arXiv preprint arXiv:2211.10851, 2022. [20] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity Is All You Need: Learning Skills Without a Reward Function. International Conference on Learning Representations, 2019. [21] Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Eysenbach. Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning. ICLR 2025, 2025. [22] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable Unsupervised RL With Metric-Aware Abstraction. International Conference on Learning Representations, 2024. [23] Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, and Anca Dragan. Learning to Assist Humans Without Inferring Rewards. Neural Information Processing Systems, 2024b. [24] Evan Ellis, Vivek Myers, Jens Tuyls, Sergey Levine, Anca Dragan, and Benjamin Eysenbach. Training LLM Agents to Empower Humans. International Conference on Machine Learning, 2026. [25] Claire Yang, Claire Jie Zhang, Maya Cakmak, and Max Kleiman-Weiner. When Assisting One Disempowers Another. arXiv:2511.04177, 2026. [26] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor Features for Transfer in Reinforcement Learning. Neural Information Processing Systems, 30, 2017. [27] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-Learning: Learning to Achieve Goals via Recursive Classification. International Conference on Learning Representations, 2021. [28] Peter Dayan. Improving Generalisation for Temporal Difference Learning: The Successor Representation. Neural Computation, 1993. [29] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational Inverse Control With Events: A General Framework for Data-Driven Reward Definition. Neural Information Processing Systems, 2018. [30] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The Information Geometry of Unsupervised Reinforcement Learning. International Conference on Learning Representations, 2022. [31] Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. All Else Being Equal Be Empowered. European Conference on Artificial Life, p. 744–753, 2005b. [32] Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, first edition, 2022. [33] Nimrod Megiddo. The Weighted Euclidean 1-Center Problem. Mathematics of Operations Research, 8(4):498–504, 1983. [34] Özgür Şimşek, Alicia P Wolfe, and Andrew G Barto. Identifying Useful Subgoals in Reinforcement Learning by Local Graph Partitioning. International Conference on Machine Learning, p. 816–823, 2005. [35] Yann Ollivier. Which Features Are Best for Successor Features? arXiv:2502.10790, 2025. [36] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. International conference on machine learning, p. 501–510, 2018. [37] Stas Tiomkin, Daniel Polani, and Naftali Tishby. Control Capacity of Partially Observable Dynamic Systems in Continuous Time. arXiv:1701.04984, 2017. [38] Ruihan Zhao, Kevin Lu, Pieter Abbeel, and Stas Tiomkin. Efficient Empowerment Estimation for Unsupervised Stabilization. aXiv:2007.07356, 2021. [39] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. Neural Information Processing Systems, 2016. [40] Thomas M Cover and Joy Thomas. Elements of Information Theory. John Wiley & Sons, second edition, 2006. [41] Alison L Gibbs and Francis Edward Su. On Choosing and Bounding Probability Metrics. International Statistical Review, 70(3):419–435, 2002. [42] Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic Symmetries and Invariant Neural Networks. Journal of Machine Learning Research, 21(90):1–61, 2020. [43] Olav Kallenberg. Foundations of Modern Probability, volume 99. Springer, Probability Theory and Stochastic Modelling, 2021. [44] Christoph Salge, Cornelius Glackin, and Daniel Polani. Approximation of Empowerment in the Continuous Domain. Complex Systems, 16(02n03):1250079, 2013. [45] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014. [46] Suguru Arimoto. An Algorithm for Computing the Capacity of Arbitrary Discrete Memoryless Channels. IEEE Transactions on Information Theory, 18(1):14–20, 1972. [47] Richard Blahut. Computation of Channel Capacity and Rate-Distortion Functions. IEEE Transactions on Information Theory, 18(4):460–473, 1972. [48] Paul J. Goulart and Yuwen Chen. Clarabel: An Interior-Point Solver for Conic Programs With Quadratic Objectives. arXiv:2405.12762, 2024. Glossary dsom gaussian_width kl mdp mi misl pi_mathrmeff pi_mathrmpot rl Appendix A Preliminaries for Proofs We restate four standard results in forms used in our statements. We cite referenced restatements of these results, which are more general versions of the results below. The first result gives the standard concavity and convexity properties of mutual information [40, Thm. 2.7.4] [32, Thm. 5.3]. Theorem A.1 (Concavity and convexity of mutual information). Let PXP_X be a source distribution and let PY|XP_Y X be a channel between standard Borel spaces. For a fixed channel, I(X,Y)I(X;Y) is concave in PXP_X. For a fixed source, I(X,Y)I(X;Y) is convex in PY|XP_Y X. In our setting, the source is the skill prior p(z∣s0)p(z s_0) and the channel maps each skill z to a DSOM pπ(s+∣s0,z)p^π( [rgb]0.5,0.5,0.5s_+ s_0,z). Thus, the MI Iπ(Z;S+∣S0=s0)I^π(Z;S_+ S_0=s_0) is concave in the skill prior for fixed skill occupancies and convex in the occupancies for a fixed skill prior. The second result expresses the channel capacity in information radius form as the smallest worst-case divergence from the channel outputs to a central distribution [32, Sec. 5.3]. Theorem A.2 (Dual form of channel capacity). Let PY|XP_Y X be a channel between standard Borel spaces X and Y, and write PY|X=xP_Y X=x for the output distribution at input x∈x . Then supPX∈Δ()I(X;Y)=infq(y)supx∈DKL(PY|X=x(y)∥q(y)). _P_X∈ (X)I(X;Y)= _q( [rgb]0.5,0.5,0.5y) _x D_KL (P_Y X=x( [rgb]0.5,0.5,0.5y)\,\|\,q( [rgb]0.5,0.5,0.5y) ). If this value is finite, the infimum is attained at a unique center q⋆q . In our setting, the input is the skill, X=ZX=Z, and the output is the future state, Y=S+Y=S_+, so each output distribution PY|X=xP_Y X=x is a skill occupancy pπ(s+∣s0,z)p^π( [rgb]0.5,0.5,0.5s_+ s_0,z). We utilize this dual form to prove Theorem B.2. The third result compares KL and χ2χ^2 divergence [32, Eq. (7.31)]; see also Gibbs and Su, [41] for the general statement. Theorem A.3 (KL and χ2χ^2 comparison). For any distributions P≪QP Q, DKL(P∥Q)≤log(1+χ2(P∥Q)).D_KL(P\|Q)≤ (1+χ^2(P\|Q) ). We apply this bound with P=pπ eff (z∣s0,s+)andQ=p(z∣s0).P=p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5z s_0,s_+) Q=p( [rgb]0.5,0.5,0.5z s_0). Averaging the bound over s+s_+ gives the χ2χ^2 control of mutual information that appears in . The fourth result shows that a measurable transformation of noise can induce any distribution on a standard Borel space [42, Lem. 5] [43, Thm. 8.17]. Theorem A.4 (Noise outsourcing). Let μ be a probability distribution on a standard Borel space Z. Let U be the uniform distribution on a standard Borel space V. If Z~∼U Z U, then there exists a measurable map T:→T:V such that T(Z~)∼μ.T( Z) μ. We utilize this statement to prove that a uniform skill prior does not lose generality in Lemma D.1. Appendix B Information Geometry of Empowerment We prove a general theorem of the information radius result from §. 4.1 that applies to both continuous and discrete state space MDP s. We include the reduction of the result to the finite-state statement found in §. 4.1. We start with intuition about the information channel, then explain potential simplications of this information channel. First, note that skill latents z∈z can only influence future state distributions via a skill-conditioned policy. The information channel PS+|ZP_S_+ Z is a function of the skill-conditioned policy π∈Ππ∈ _Z, which sends each skill to the induced occupancy: →(s0),z⟼pπ(s+∣s0,z). \ π\ P(s_0), z p^π( [rgb]0.5,0.5,0.5s_+ s_0,z). If this mapping is measurable, we can directly reduce the skills to their induced DSOM s. Specifically, there would exist a pushforward map from the skill random variable Z to a random state occupancy P drawn from a distribution over (s0)P(s_0) (e.g. a distribution over state occupancy distributions) which, by the data-processing inequality from the Markov chain Z→PZ→S+Z→ P_Z→ S_+ and the sufficiency of P, would imply I(Z;S+∣S0)=I(PZ;S+∣S0). I(Z;S_+ S_0)=I(P_Z;S_+ S_0). Note, however, that not all policies admit a measurable mapping from skill space to occupancy space. We require the additional assumption that the stochastic policies π(a∣s,z)π(a s,z) are jointly measurable over the state and skill space: Assumption B.1 (Policies measurable in skill and state space). Let S, A, and Z denote the state space, action space, and skill spaces respectively, which we take to be standard Borel. Take the skill-conditioned policy space Π _Z to be the space of policies measurable in the state space S and skill space Z. That is, (s,z)↦π(⋅∣s,z)∈Δ()(s,z) π(· s,z)∈ (A) is a measurable function. Accordingly, the state occupancy set (s0)P(s_0) is now occupancies reachable by measurable policies. From this assumption, there exists a measurable mapping from →(s0)Z \ π\ P(s_0). Thus, we can effectively alias the skill latent Z and the random state occupancy measure PZP_Z. For the information geometry result, this association allows direct application of Theorem A.2 to the information channel from skills to occupancies: Theorem B.2 (General information geometry of empowerment). Fix a start state s0s_0. Then, with both sides taking values in [0,∞][0,∞], E pot (s0)=infq(s+)supp∈(s0)DKL(p(s+)∥q(s+)). @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _q( [rgb]0.5,0.5,0.5s_+) _p (s_0)D_KL (p( [rgb]0.5,0.5,0.5s_+)\|q( [rgb]0.5,0.5,0.5s_+) ). If the empowerment is bounded, the infimum is attainable. Let the skill-conditioned policy π∈Ππ∈ _Z and the skill source p(z∣s0)p(z s_0) over skills z∈z attain E pot (s0) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0). Then, the skill marginal is the unique center: q⋆=pπ(s+∣s0).q =p^π( [rgb]0.5,0.5,0.5s_+ s_0). Moreover, for p(z∣s0)p(z s_0)-almost every z, DKL(pπ(s+∣s0,z)∥q⋆(s+))=E pot (s0),D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q ( [rgb]0.5,0.5,0.5s_+) )=E @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0), where pπ(s+∣s0,z)p^π( [rgb]0.5,0.5,0.5s_+ s_0,z) is an extreme point of (s0)P(s_0). Thus, an optimal policy places every skill occupancy at the maximal KL radius, while an optimal source induces the average occupancy q⋆q . We now prove Theorem B.2 with the information-radius identity of Theorem A.2. Proof. Fix S0=s0S_0=s_0. Each skill z induces a future state occupancy pπ(s+∣s0,z)∈(s0),p^π( [rgb]0.5,0.5,0.5s_+ s_0,z) (s_0), and the conditional distribution of S+S_+ depends on z only through this occupancy. From the measurability assumption Assumption B.1, we may alias skills that induce the same occupancy. Thus, jointly optimizing the policy and skill source is equivalent to choosing a source over (s0)P(s_0) for the channel p∈(s0),S+∼p.p (s_0), S_+ p. In other words, the source is a distribution over a set of state-occupancy distributions that give the channel output S+S_+. Applying Theorem A.2 with the random occupancy PZ∈(s0)P_Z (s_0) as the input and the future state S+S_+ as the output gives E pot (s0)=infq(s+)supp∈(s0)DKL(p(s+)∥q(s+)). @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _q( [rgb]0.5,0.5,0.5s_+) _p (s_0)D_KL (p( [rgb]0.5,0.5,0.5s_+)\|q( [rgb]0.5,0.5,0.5s_+) ). Thus, assuming the potential empowerment is finite, a unique center q⋆q attains the infimum. Now, suppose that the skill policy and the prior attains the maximal capacity C=E pot (s0)C= @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0). For any distribution q, we have that p(z∣s0)[DKL(pπ(s+∣s0,z)∥q(s+))]=Iπ(Z;S+∣S0=s0)+DKL(pπ(s+∣s0)∥q(s+))E_p( [rgb]0.5,0.5,0.5z s_0) [D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q( [rgb]0.5,0.5,0.5s_+) ) ]=I^π(Z;S_+ S_0=s_0)+D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0)\|q( [rgb]0.5,0.5,0.5s_+) ) following the Golden Formula (32, Thm. 4.1). Taking q=q⋆q=q , every reachable occupancy has divergence at most C. Hence, C C ≥p(z∣s0)[DKL(pπ(s+∣s0,z)∥q⋆(s+))] _p( [rgb]0.5,0.5,0.5z s_0) [D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q ( [rgb]0.5,0.5,0.5s_+) ) ] (7) =C+DKL(pπ(s+∣s0)∥q⋆(s+)) =C+D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0)\|q ( [rgb]0.5,0.5,0.5s_+) ) (Golden formula) ≥C. ≥ C. (8) Thus, the KL between the marginal state occupancy and q⋆q is 00, which implies q⋆=pπ(s+∣s0).q =p^π( [rgb]0.5,0.5,0.5s_+ s_0). and that every individual skill divergence is at most C: DKL(pπ(s+∣s0,z)∥q⋆(s+))=C D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q ( [rgb]0.5,0.5,0.5s_+) )=C (9) for p(z∣s0)p(z s_0)-almost every z. Thus, an optimal source places skill occupancy mass on the boundary of the optimal KL ball, with the KL measured from the central marginal distribution q⋆q . Finally, suppose there exists an active skill occupancy that is not an extreme point of (s0)P(s_0). Then, for distinct p1,p2∈(s0)p_1,p_2 (s_0) and some λ∈(0,1)λ∈(0,1), pπ(s+∣s0,z)=λp1+(1−λ)p2.p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)=λ p_1+(1-λ)p_2. Convexity of the KL divergence in the first argument gives DKL(pπ(s+∣s0,z)∥q⋆(s+)) D_KL (p^π( [rgb]0.5,0.5,0.5s_+ s_0,z)\|q ( [rgb]0.5,0.5,0.5s_+) ) <λDKL(p1(s+)∥q⋆(s+))+(1−λ)DKL(p2(s+)∥q⋆(s+)) <λ D_KL (p_1( [rgb]0.5,0.5,0.5s_+)\|q ( [rgb]0.5,0.5,0.5s_+) )+(1-λ)D_KL (p_2( [rgb]0.5,0.5,0.5s_+)\|q ( [rgb]0.5,0.5,0.5s_+) ) ≤C, ≤ C, which contradicts the equality in Eq. 9. Thus, p(z∣s0)p(z s_0)-almost every skill induces an extreme point of (s0)P(s_0). ∎ The reduction to the finite tabular case directly follows: Proof of . In the finite-state setting, (s0)P(s_0) is a polytope and E pot (s0)≤log|| @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)≤ |S|, so the channel capacity is finite. Theorem B.2 shows that the center distribution is the skill-marginal where active skill occupancies lie on the boundary of the optimal KL ball at extreme points of (s0)P(s_0). The extreme points of a polytope are vertices, which proves the result. ∎ B.1 Log reachability interpretation of potential empowerment in discrete deterministic settings For completeness, we include the log reachability interpretation of the action sequence potential empowerment (as originally defined in [1]) in discrete deterministic settings. Salge et al., [3] originally present this result. We begin by defining the action sequence potential empowerment. Let A0H−1=(A0,…,AH−1)∈HA_0^H-1=(A_0,…,A_H-1) ^H be an open-loop action sequence over the fixed horizon H. Let StS_t denote the random variable of the state after t timesteps. Klyubin et al., 2005a [1] define (potential) empowerment as the capacity of the channel from this action sequence to the resulting state: E pot H(s)=maxp(a0H−1)I(A0H−1;SH∣S0=s). @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot_H(s)= _p(a_0^H-1)I(A_0^H-1;S_H S_0=s). This expression is equivalent to Definition 3.1, replacing Z with the action sequence A0H−1A_0^H-1 and S+S_+ with SHS_H. Lemma B.3 (Discrete deterministic potential empowerment is log reachability, adapted from Salge et al., [3]). Fix a discrete, deterministic MDP and horizon H. Because dynamics are deterministic, the future state SHS_H is a deterministic function of S0S_0 and A0H−1A_0^H-1. Then, the reachable set of states in H timesteps takes the form ℛH(s)=s+∈:∃a0H−1∈H that reaches SH=s+ from S0=s,R_H(s)=\s_+ :\ ∃\,a_0^H-1 ^H that reaches S_H=s_+ from S_0=s\, and the empowerment at s is the log of the size of this set: E pot H(s)=log|ℛH(s)|. @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot_H(s)= |R_H(s)|. Proof. Because SHS_H is a deterministic function of A0H−1A_0^H-1 given S0=sS_0=s, the conditional entropy H(SH∣A0H−1,S0=s)H(S_H A_0^H-1,S_0=s) is 00. Then, I(A0H−1;SH∣S0=s)=H(SH∣S0=s).I(A_0^H-1;S_H S_0=s)=H(S_H S_0=s). Because the support of SHS_H is in the reachable set ℛH(s)R_H(s), the entropy is at most log|ℛH(s)| |R_H(s)|. By construction, we can choose one action sequence for each reachable s+∈ℛH(s)s_+ _H(s) and place uniform mass p(a0H−1)p(a_0^H-1) over the chosen action sequences. Thus, the random variable SHS_H is uniform on ℛH(s)R_H(s) with entropy log|ℛH(s)| |R_H(s)|. ∎ Appendix C Empowerment and Bottleneck States We state and prove the hitting-time and temporal-distance statements from §. 4.2. C.1 Bottleneck states are high-empowerment states We first define the repeated rollout hitting time from the DSOM. A single rollout from s0s_0 samples a skill z∼p(z∣s0)z p( [rgb]0.5,0.5,0.5z s_0) and a geometric termination time τ∼Geom(1−γ)τ (1-γ). The agent executes skill z until τ, where the rollout terminates at S+≜SτS_+ S_τ. Writing S+(k)S_+^(k) for the termination state of the kkth rollout from s0s_0, the repeated-rollout hitting time measures the first rollout count that hits s+s_+: Hπ(s0→s+)≜infk≥1:S+(k)=s+,H^π(s_0→ s_+) \k≥ 1:S_+^(k)=s_+ \, where Hπ(s0→s+∣z)H^π(s_0→ s_+ z) is the corresponding count when every rollout only uses skill z. Then, potential empowerment is exactly the largest expected reduction in log hitting time after committing to a skill: See 4.2 Proof. The repeated-rollout hitting times are geometric random variables with success probabilities pπ(s+∣s0)p^π(s_+ s_0) and pπ(s+∣s0,z)p^π(s_+ s_0,z), respectively. Hence, [Hπ(s0→s+)]=1pπ(s+∣s0),[Hπ(s0→s+∣z)]=1pπ(s+∣s0,z).E[H^π(s_0→ s_+)]= 1p^π(s_+ s_0), [H^π(s_0→ s_+ z)]= 1p^π(s_+ s_0,z). The MI is the expected log ratio of the skill-conditioned to the marginal DSOM, so Iπ(Z;S+∣S0=s0) I^π(Z;S_+ S_0=s_0) =pπ(z,s+∣s0)logpπ(s+∣s0,z)pπ(s+∣s0) =E_p^π(z,s_+ s_0) p^π(s_+ s_0,z)p^π(s_+ s_0) =pπ(z,s+∣s0)[log[Hπ(s0→s+)]−log[Hπ(s0→s+∣z)]]. =E_p^π(z,s_+ s_0) [ [H^π(s_0→ s_+)]- [H^π(s_0→ s_+ z)] ]. (10) This identity holds for any policy. Maximizing both sides over effective empowerment policies π eff gives the result. ∎ C.2 Temporal distances over contiguous skill rollouts The above result considers the empowerment objective used throughout the rest of the paper, which measures the controllability of a single skill on the termination state/discounted future observation. We now define a closely related objective over contiguous rollouts where the temporal distance interpretation from Myers et al., 2024a [7] is exact. Rather selecting a single skill and rolling out until termination, the agent can execute contiguous skill rollouts in a single trajectory. An agent can begin by selecting a skill z. At some geometric termination time τ∼Geom(1−γ)τ (1-γ), the agent resamples a random skill and continues forth from the terminated state. This process repeats forever. Intuitively, this rollout scheme is a generalization of standard MDP dynamics, where skill Z now takes the role of the action A, and the β discount factor over units of rollouts controls the fundamental timescale, in addition to the standard γ. We can mathematically write out the state occupancy measure of such a process. Let the random variable XkX_k be the state occupied at the k-th geometric termination time. Let the trajectory begin at state X0=s0X_0=s_0. At the k-th termination, the agent re-draws a skill Zk∼p(z∣Xk)Z_k p( [rgb]0.5,0.5,0.5z X_k) and continues the trajectory. Then, the state occupancy of the process takes the form Mβπ(g∣s)≜(1−β)∑k=0∞βkpπ(Xk=g∣X0=s).M_β^π(g s) (1-β) _k=0^∞β^kp^π(X_k=g X_0=s). We can correspondingly define the state and skill-conditioned DSOM of the contiguous process as Mβπ(g∣s,z)≜(1−β)∑k=0∞βkpπ(Xk=g∣X0=s,Z0=z),M_β^π(g s,z) (1-β) _k=0^∞β^kp^π(X_k=g X_0=s,Z_0=z), where the agent rolls out skill z until the first geometric time, then resamples skills from the prior p(z∣sk)p(z s_k) thereafter. Note that averaging the skill-conditioned DSOM gives the state-conditioned DSOM: z∼p(z∣s)[Mβπ(g∣s,z)]=Mβπ(g∣s).E_z p( [rgb]0.5,0.5,0.5z s) [M_β^π(g s,z) ]=M_β^π(g s). llasection q:rollout_successor_consistency (11) The effective contiguous empowerment measures the degree to which the first skill controls the occupancy of the entire process. Let Z∼p(z∣s0)Z p( [rgb]0.5,0.5,0.5z s_0) be the first skill and let G∼Mβπ(g∣s0,Z)G M_β^π( [rgb]0.5,0.5,0.5g s_0,Z) be the state the process occupies at a geometrically sampled termination time. Taking the supremum over the policy and source gives the channel capacity form, or the potential contiguous empowerment: ℰeffcontig(π,s0) E^contig_eff(π,s_0) ≜Iπ(Z;G∣X0=s0), I^π(Z;G X_0=s_0), ℰpotcontig(s0) E^contig_pot(s_0) ≜supπ∈Π,p(z∣s)ℰeffcontig(π,s0). _π∈ _Z,\,p(z s)E^contig_eff(π,s_0). An agent is effectively contiguously empowered if their intial skill/generalized action exerts control on their future state for the contiguous process. An agent is potentially contiguously empowered when occupying a state from which immediate skills provide long-term control over the future. We highlight that this modification of empowerment is conceptually similar to the definitions of empowerment considered in the rest of the paper, despite different mathematical forms. The potential contiguous empowerment admits a temporal distance interpretation, following Myers et al., 2024a [7]. The temporal distance captures a notion of the time to traverse between states that generalizes to continuous state settings. Let β be the discount factor. Let Mβπ(g∣s)M^π_β(g s) denote the DSOM of state g for a process that begins at s. Then, the temporal distance from a state s to a future state g is defined as dβπ(s,g)≜−logMβπ(g∣s)Mβπ(g∣g)d^π_β(s,g) - M_β^π(g s)M_β^π(g g) where the denominator measures the self-reachability probability for a process that begins at g. Conditioning on the initial skill gives the skill-conditioned temporal distance dβπ(s,g∣z)≜−logMβπ(g∣s,z)Mβπ(g∣g),d^π_β(s,g z) - M_β^π(g s,z)M_β^π(g g), again normalized by the self-reaching probability. Let Mβπ(z,g∣s0)=p(z∣s0)Mβπ(g∣s0,z)M_β^π( [rgb]0.5,0.5,0.5z, [rgb]0.5,0.5,0.5g s_0)=p( [rgb]0.5,0.5,0.5z s_0)\,M_β^π( [rgb]0.5,0.5,0.5g s_0,z) be the joint law of the initial skill Z and future state G. Adding and subtracting the self-reaching term unifies the contiguous empowerment and the temporal distance: Proposition C.1 (Contiguous empowerment measures temporal distance reduction). For any skill policy π∈Ππ∈ _Z and skill source p(z∣s)p(z s), ℰeffcontig(π,s0)=Mβπ(z,g∣s0)[dβπ(s0,g)−dβπ(s0,g∣z)].E^contig_eff(π,s_0)=E_M_β^π( [rgb]0.5,0.5,0.5z, [rgb]0.5,0.5,0.5g s_0) [d_β^π(s_0,g)-d_β^π(s_0,g z) ]. Consequently, ℰpotcontig(s0)=supπ∈Π,p(z∣s)Mβπ(z,g∣s0)[dβπ(s0,g)−dβπ(s0,g∣z)].E^contig_pot(s_0)= _π∈ _Z,\,p(z s)E_M_β^π( [rgb]0.5,0.5,0.5z, [rgb]0.5,0.5,0.5g s_0) [d_β^π(s_0,g)-d_β^π(s_0,g z) ]. Thus, the potential contiguous empowerment is the maximum expected reduction in temporal distance to future states when committing to an initial skill Z. Proof. The result follows directly from definitions. The self-reaching term cancels: dβπ(s0,g)−dβπ(s0,g∣z)=logMβπ(g∣s0,z)Mβπ(g∣s0).d_β^π(s_0,g)-d_β^π(s_0,g z)= M_β^π(g s_0,z)M_β^π(g s_0). Taking the expectation over Mβπ(z,g∣s0)M_β^π( [rgb]0.5,0.5,0.5z, [rgb]0.5,0.5,0.5g s_0) gives the equality between the temporal distance reduction and effective contiguous empowerment. Taking the supremum of both sides over the skill-conditioned policy and source gives the potential contiguous empowerment expression. ∎ Unlike the hitting time expression, this form of the contiguous potential empowerment holds in continuous settings. In discrete spaces, the temporal distance reduces to a typical hitting time interpretation [7]. Let Hcontigπ(s→g)≜infk≥0:Xk=gH_contig^π(s→ g) \k≥ 0:X_k=g\ be the time to hit g from s, counted in units of contiguous skill rollouts. Let Hcontigπ(s→g∣z)H_contig^π(s→ g z) be the corresponding skill-conditioned hitting time. Following Myers et al., 2024a [7], we can rewrite the temporal distance in terms of these hitting times: dβπ(s,g)=−log[βHcontigπ(s→g)],dβπ(s,g∣z)=−log[βHcontigπ(s→g∣z)]for g≠s.d_β^π(s,g)=- [β^H_contig^π(s→ g) ], d_β^π(s,g z)=- [β^H_contig^π(s→ g z) ] g≠ s. Thus, there is a clear hitting time interpretation of the temporal distance object in the discrete limit. Note on different empowerment expressions. The contiguous empowerment object differs from the empowerment objective used elsewhere in the paper. All other statements in the paper characterize the original potential empowerment; this result is an extension that provides a neat connection to temporal distance as defined in [7], which (unlike hitting time) remains well-defined in continuous settings.Estimating the contiguous empowerment in tabular settings requires a methodology different from estimating empowerment considered elsewhere in the paper, as the channel mapping from skills to outcomes is also a function of the source p(z∣s)p(z s). In addition to doing the randomized vertex search, we estimate contiguous empowerment by doing gradient descent on a parameterized source. Further implementation details are in the codebase. Figure 7 shows the estimated MI of the contiguous process from each start state for the Fig. 4 MDP s. The resulting potential contiguous empowerment field closely matches the potential empowerment field considered elsewhere in the paper in Fig. 4. In both experiments, the bottleneck states have high empowerment. Figure 7: Empowerment over contiguous rollouts concentrates at bottlenecks (Proposition C.1). The figures show the MI of the contiguous process at each start state on the 8×88× 8 grids from Fig. 4, with (a) a center opening and (b) a corner opening. The bottleneck openings have high potential contiguous empowerment. Appendix D Skill Selection: Fixed Continuous Uniform Priors are General Lemma D.1 (A continuous uniform latent can represent any source distribution, discrete or continuous). Fix a start state s0s_0 and let the skill alphabet Z be standard Borel. Consider the mutual information Iπ(Z;S+∣S0=s0)I^π(Z;S_+ S_0=s_0) under a skill-conditioned policy π∈Ππ∈ _Z, where Z∼p(z∣s0)Z p(z s_0) is sampled once at s0s_0 and held fixed for the rollout. Let the reference source U be a continuous distribution on Z, placing zero mass on every single point, e.g. the uniform distribution on Z. Then supπ∈Π,p(z∣s0)∈Δ()Iπ(Z;S+∣S0=s0)=supπ~∈Π,Z~∼UIπ~(Z~;S+∣S0=s0), _π∈ _Z,\,p(z s_0)∈ (Z)I^π(Z;S_+ S_0=s_0)= _ π∈ _Z,\, Z UI π( Z;S_+ S_0=s_0), where the maximization on the right ranges over all skill-conditioned policies. Proof. Fix any pair (π,p)(π,p) with p∈Δ()p∈ (Z). By the transfer theorem (Theorem A.4; 43), there exists a measurable map Ts0:→T_s_0:Z such that Ts0(Z~)∼p(z∣s0)T_s_0( Z) p( [rgb]0.5,0.5,0.5z s_0) for Z~∼U Z U.11 1 For U=Unif[0,1]U=Unif[0,1] this can be the inverse-CDF construction. Set Z=Ts0(Z~)Z=T_s_0( Z). Define π~(a∣s,z~;s0)=π(a∣s,Ts0(z~);s0). π(a s, z;s_0)=π(a s,T_s_0( z);s_0). Conditioned on Z=Ts0(Z~)Z=T_s_0( Z), the action kernels of π and π~ π agree at every timestep and the environment dynamics are identical. Therefore, the joint law of (Z,S+)(Z,S_+) under (π,p)(π,p) is the same as the joint law of (Ts0(Z~),S+)(T_s_0( Z),S_+) under (π~,U)( π,U). Moreover, π~ π only receives information from z~ z through Ts0(z~)T_s_0( z), so S+S_+ is conditionally independent of Z~ Z given Ts0(Z~)T_s_0( Z), and Iπ~(Z~;S+∣S0=s0)=Iπ~(Ts0(Z~);S+∣S0=s0)=Iπ(Z;S+∣S0=s0).I π( Z;S_+ S_0=s_0)=I π (T_s_0( Z);S_+ S_0=s_0 )=I^π(Z;S_+ S_0=s_0). Taking the supremum over (π,p)(π,p) shows that the RHS upper bounds the LHS. Conversely, any policy π~ π with source U∈Δ()U∈ (Z) is within the domain of maximization on the LHS, so we have an equality and are done. ∎ Appendix E Continuous versus Discrete Skill Spaces Lemma E.1 (Uniform skill priors are not generally optimal with finitely many skills). With a finite skill alphabet, fixing the skill prior to be uniform can strictly reduce channel capacity. Proof. Consider an MDP consisting of a start state s0s_0 and two terminal states s1,s2\s_1,s_2\. Let the skill set be the discrete skill set =0,1Z=\0,1\. Let the action space be a continuous action a∈[0,1]a∈[0,1]. The transition is p(s1∣s0,a)=βa,p(s2∣s0,a)=1−βa,0<β<1.p(s_1 s_0,a)=β a, p(s_2 s_0,a)=1-β a, 0<β<1. We can fully represent any deterministic, skill-conditioned policy by assigning each skill z to an action az∈[0,1]a_z∈[0,1] that completely determines the transition probability p(s1∣s0,z)=βazp(s_1 s_0,z)=β a_z. First optimize the channel for a fixed source. By convexity of MI in the channel (Theorem A.1; 40, 32), the maximum over (a0,a1)∈[0,1]2(a_0,a_1)∈[0,1]^2 occurs at a vertex. The vertices (0,0)(0,0) and (1,1)(1,1) transmit no information. The informative vertices are (0,1)(0,1) and (1,0)(1,0), which are equivalent up to relabeling. Without loss of generality, set (a0,a1)=(0,1)(a_0,a_1)=(0,1), so that p(s1∣s0,z=0)=0p(s_1 s_0,z=0)=0 and p(s1∣s0,z=1)=βp(s_1 s_0,z=1)=β. Let h(x)=−xlogx−(1−x)log(1−x)h(x)=-x x-(1-x) (1-x) denote the binary entropy. Under this channel, the output marginal is p(s1∣s0)=p(Z=1∣s0)βp(s_1 s_0)=p(Z=1 s_0)\,β, and the conditional output entropies are h(0)=0h(0)=0 and h(β)h(β). The MI is therefore I(Z;S+∣S0=s0)=h(p(Z=1∣s0)β)−p(Z=1∣s0)h(β).I(Z;S_+ S_0=s_0)=h (p(Z=1 s_0)\,β )-p(Z=1 s_0)\,h(β). Differentiating in the source probability p(Z=1∣s0)p(Z=1 s_0) gives ∂I(Z;S+∣S0=s0)∂p(Z=1∣s0)=βlog1−p(Z=1∣s0)βp(Z=1∣s0)β−h(β), ∂\,I(Z;S_+ S_0=s_0)∂\,p(Z=1 s_0)=β 1-p(Z=1 s_0)\,βp(Z=1 s_0)\,β-h(β), so the capacity-achieving source is p⋆(Z=1∣s0)=1β(1+exp(h(β)/β)).p (Z=1 s_0)= 1β(1+ (h(β)/β)). For β=1/2β=1/2, this gives p⋆(Z=1∣s0)=2/5p (Z=1 s_0)=2/5, not 1/21/2. To see that the uniform prior is strictly worse, maximize over channels with p(Z=1∣s0)=1/2p(Z=1 s_0)=1/2. The same convexity argument places the optimum at vertex (a0,a1)=(0,1)(a_0,a_1)=(0,1) or (1,0)(1,0). For β=1/2β=1/2, the best uniform-prior value is h(1/4)−12h(1/2),h(1/4)- 12h(1/2), whereas the best arbitrary source value is h(1/5)−25h(1/2).h(1/5)- 25h(1/2). The latter is strictly larger. Thus a uniform prior over two discrete skills cannot generally achieve the channel capacity. ∎ Comment on VIC [18] and DIAYN [20]. A fixed continuous uniform prior is fully general for discrete and continuous MDP s, while a discrete prior is not. This result has implications for skill learning algorithms. VIC optimizes both a source distribution and channel [18], with options (termed here as “skills”) that may be finite or continuous. However, the authors note that the joint optimization is difficult and practically implement a uniform prior over a discrete skill set. DIAYN similarly fixes the source to a uniform categorical over a discrete skill set [20]. While neither work claims that fixing the prior preserves the objective, our result Lemma E.1 is a concrete counterexample that shows uniform discrete priors lose generality. Finally, Lemma D.1 shows that utilizing a continuous uniform skill prior recovers full generality without the need for source learning. Combined, our results theoretically suggest that skill learning methods should utilize continuous uniform sources; note that SOTA skill learning methods already utilize such priors [21, 22]. Appendix F Empowerment Lower Bounds Tabular Adaptation We now prove that empowerment lower bounds zero-shot adaptation in the tabular setting under an uninformed reward prior. Define the skill offset uz≜(pπ eff (s+∣s0,z)−pπ eff (s+∣s0))/bu_z (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)-p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0) )/ b. The whitened reward g≜brg b\,r is standard normal under the prior, and the marginal’s return ⟨g,pπ eff (s+∣s0)/b⟩ g,\ p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0)/ b is mean-zero and z-independent, so J(s0)=gsupz∈⟨g,uz⟩,J(s_0)=E_g _z \, g,\ u_z , llasection q:J_width (12) is the Gaussian width of uzz∈\u_z\_z . Before presenting the proof, we discuss the chosen reward prior and motivate the variance scaling. Under the 1/b1/b scaling, splitting a state into substates leaves the distribution of the returns unchanged. Example F.1 (Splitting a two-state MDP into a four-state MDP Fig. 8). Let =1,2S=\1,2\ with uniform reference b(1)=b(2)=12b(1)=b(2)= 12. The prior samples r(1),r(2)∼(0,2)r(1),r(2) (0,2) independently (Fig. 8). For a skill z with DSOM pπ eff (s+∣s0,z)=(p1,p2)p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)=(p_1,p_2), the mean reward is a centered Gaussian: pπ eff (s+∣s0,z)[r(s+)]=p1r(1)+p2r(2)∼(0, 2p12+2p22).E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)[r(s_+)]=p_1\,r(1)+p_2\,r(2)\ \ N\! (0,\ 2p_1^2+2p_2^2 ). Splitting each state into two substates gives ′=1a,1b,2a,2bS =\1a,1b,2a,2b\ where the refined prior samples each coordinate reward r′∼(0,4)r (0,4). Furthermore, split every skill’s occupancy in the same proportions, so the occupancy of z becomes (p12,p12,p22,p22) ( p_12, p_12, p_22, p_22 ). The mean reward on the fine-grained space is p12(r′(1a)+r′(1b))+p22(r′(2a)+r′(2b))∼(0,p124⋅8+p224⋅8)=(0, 2p12+2p22), p_12 (r (1a)+r (1b) )+ p_22 (r (2a)+r (2b) )\ \ N\! (0,\ p_1^24· 8+ p_2^24· 8 )=N\! (0,\ 2p_1^2+2p_2^2 ), since each sum r′(⋅a)+r′(⋅b)r (· a)+r (· b) has variance 88. Thus, the distribution of the mean reward is the same before and after splitting the state. In this example, the 1/b1/b variance scaling is key. Under a unit-variance prior r(s+)∼(0,1)r(s_+) (0,1), the same state splitting shrinks the return variance from p12+p22p_1^2+p_2^2 to 12(p12+p22) 12 (p_1^2+p_2^2 ). Thus, the constraint that the return distribution is scale invariant implies the above scaling of the reward prior variance. (a)=1,2S=\1,2\p1p_1p2p_21122b≡12,r(s+)∼(0,2)b≡ 12,\ \ r(s_+) (0,2)p[r]∼(0, 2p12+2p22)E_p[r]\ \ N\! (0,\,2p_1^2+2p_2^2 )refine: b′=b2b = b2occupancy splitsin proportion= d=(b)′=1a,1b,2a,2bS =\1a,1b,2a,2b\p12 p_12p12 p_12p22 p_22p22 p_221a1a1b1b2a2a2b2b′≡14,r′(s+)∼(0,4)b ≡ 14,\ \ r (s_+) (0,4)p′[r′]∼(0, 2p12+2p22)E_p [r ]\ \ N\! (0,\,2p_1^2+2p_2^2 ) Figure 8: State refinement leaves return distribution unchanged (Example F.1). Bars display the occupancy pπ eff (s+∣s0,z)=(p1,p2)p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)=(p_1,p_2) of a skill over the coarse states (a) and the occupancy over the split states (b). Each panel lists the reference mass and per-state reward prior r(s+)∼(0,1/b(s+))r(s_+) (0,1/b(s_+)). Splitting states halves the reference mass and doubles the reward variance. Thus, the return obeys the same distribution (0, 2p12+2p22)N(0,\,2p_1^2+2p_2^2) in both MDP s. Now that we have built intuition for the choice of reward prior, we present the proof: Proof. By Eq. 12, it suffices to lower bound the Gaussian width. The proof consists of two inequalities: J(s0)≥step 112π(p(z∣s0)‖uz‖2)1/2≥step 2I(Z;S+∣S0=s0)c2π.J(s_0)\ step 1≥\ 1 2π (E_p( [rgb]0.5,0.5,0.5z s_0) \|u_z \|^2 )^1/2\ step 2≥\ I(Z;S_+ S_0=s_0)c 2π. llasection q:linear_rate_pivot (13) Fix the reward r and a skill z∈z . For the first inequality (step 1), we observe that the best average return obtainable by a skill dominates the marginal skill return: supz′∈pπ eff (s+∣s0,z′)[r(s+)]≥max(pπ eff (s+∣s0,z)[r(s+)],pπ eff (s+∣s0)[r(s+)]). _z E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z )[r(s_+)]\ ≥\ (E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)[r(s_+)],\ E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0)[r(s_+)] ). Take the expectation over rewards on both sides using the identity max(x,y)=12(x+y)+12|x−y| (x,y)= 12(x+y)+ 12|x-y|. Both returns are linear in r, so the 12(x+y) 12(x+y) term vanishes and J(s0)≥12r|pπ eff (s+∣s0,z)[r(s+)]−pπ eff (s+∣s0)[r(s+)]|.J(s_0)\ ≥\ 12\,E_r |E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)[r(s_+)]-E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0)[r(s_+)] |. The gap in returns is an inner product of the whitened quantities: pπ eff (s+∣s0,z)[r(s+)]−pπ eff (s+∣s0)[r(s+)]=⟨g,uz⟩.E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0,z)[r(s_+)]-E_p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s_0)[r(s_+)]= g,\ u_z . Because the coordinates of g are i.i.d. standard Gaussians, we know that the gap in returns is a Gaussian with variance ‖uz‖2\|u_z\|^2. Hence, for every z∈z , J(s0) J(s_0)\ ≥12r|⟨g,uz⟩| ≥\ 12\,E_r | g,\ u_z | =12π‖uz‖. =\ 1 2π \|u_z \|. Averaging the square of this bound over z∼p(z∣s0)z p( [rgb]0.5,0.5,0.5z s_0) (LHS is independent of the skill z) gives step 1 of Eq. 13. For step 2, express the MI as the expected KL divergence between the skill prior and the skill posterior after observing s+s_+: I(Z;S+∣S0=s0)=∑s+pπ eff (s+∣s0)DKL(pπ eff (z∣s0,s+)∥p(z∣s0)).I(Z;S_+ S_0=s_0)= _s_+p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0)\,D_KL (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5z s_0,s_+)\, \|\,p( [rgb]0.5,0.5,0.5z s_0) ). Applying Bayes’ rule, the χ2χ^2 divergence takes the form of a mean squared deviation of the skill DSOM s from the marginal: χ2(pπ eff (z∣s0,s+)∥p(z∣s0)) χ^2 (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5z s_0,s_+)\, \|\,p( [rgb]0.5,0.5,0.5z s_0) ) =p(z∣s0)[(pπ eff (s+∣s0,z)pπ eff (s+∣s0)−1)2] =E_p( [rgb]0.5,0.5,0.5z s_0) [ ( p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z)p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0)-1 )^2 ] =p(z∣s0)[(pπ eff (s+∣s0,z)−pπ eff (s+∣s0))2]pπ eff (s+∣s0)2. = E_p( [rgb]0.5,0.5,0.5z s_0) [ (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z)-p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0) )^2 ]p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0)^2. Then, through χ2χ^2, we can upper bound the MI with the norm of the occupancy gap vector: I(Z;S+∣S0=s0) I(Z;S_+ S_0=s_0) ≤c∑s+pπ eff (s+∣s0)χ2(pπ eff (z∣s0,s+)∥p(z∣s0)) ≤ c _s_+p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0) χ^2 (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5z s_0,s_+)\, \|\,p( [rgb]0.5,0.5,0.5z s_0) ) (Theorem A.3) =c∑s+p(z∣s0)[(pπ eff (s+∣s0,z)−pπ eff (s+∣s0))2] =c _s_+ E_p( [rgb]0.5,0.5,0.5z s_0) [ (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z)-p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0) )^2 ] =c∑s+b(s+)p(z∣s0)[(pπ eff (s+∣s0,z)−pπ eff (s+∣s0))2]b(s+) =c _s_+ b(s_+)\, E_p( [rgb]0.5,0.5,0.5z s_0) [ (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z)-p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0) )^2 ]b(s_+) ≤c(∑s+p(z∣s0)[(pπ eff (s+∣s0,z)−pπ eff (s+∣s0))2]b(s+))1/2 ≤ c ( _s_+ E_p( [rgb]0.5,0.5,0.5z s_0) [ (p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0,z)-p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(s_+ s_0) )^2 ]b(s_+) )^1/2 (Cauchy-Schwarz) =c(p(z∣s0)‖uz‖2)1/2. =c (E_p( [rgb]0.5,0.5,0.5z s_0) \|u_z \|^2 )^1/2. The first line applies Theorem A.3 with the bound ln(1+x)≤cx (1+x)≤ c x to each divergence. The second line expands χ2χ^2 and cancels the marginal weight. The third and fourth lines apply Cauchy–Schwarz. The last equality is the definition of uzu_z, proving step 2 of Eq. 13 which finishes the proof. ∎ We provide some didactic tabular experiments that demonstrate the behavior and scaling of the bound over different MDP s. In Fig. 9, we demonstrate the relationship between the degree of commitment within an MDP– the controllability of the MDP– and the bound for a fixed set of states. As controllability of the MDP increases, the bound tightens. In Fig. 10, we demonstrate the relationship between the number of absorbing target states in an MDP and the bound under fixed commitment. As the number of absorbing target states grows, the bound loosens. Figure 9: MI bound as a function of skill commitment (). Consider the MDP consisting of 5 states with a central starting state, where any left/right/up/down action sends probability mass to absorbing left/right/up/down states. Actions left/right/up/down have a level of commitment t, which measures the additional mass above uniform assigned to the corresponding state. When commitment is small, probability is almost spread equally over all four states regardless of action so there is little control. In this regime, the adaptation J dominates the lower bound. As the commitment increases, actions/skills commit to states and the bound tightens. Figure 10: MI bound as a function of absorbing states (). Consider an MDP with one starting state and K absorbing target states. Each of the K actions transitions deterministically to a distinct target. We normalize occupancies over the targets (neglecting the occupancy of the starting state). Thus, K counts the maximum possible number of total skills. The scaled MI (orange) correctly lower bounds the adaptation J (purple). The bound weakens as the number of absorbing target states in the MDP grows. Appendix G When Information Geometry is Insufficient for Adaptation The adaptation result above is in the tabular setting. For the following results, we also consider MDP s whose state space S is a metric space with arbitrary metric d. Importantly, these state spaces may be continuous. Proposition G.1 (MDP s can admit arbitrarily many distinct skills). Fix a discount γ∈(0,1)γ∈(0,1) and a skill space =[K]Z=[K] with the uniform prior Z∼Unif([K])Z ([K]). There exists an MDP with a start state s∈s and K skills that attains the potential empowerment E pot (s)=maxπ eff Iπ eff (Z;S+∣S0=s)=γlogK. @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s)= _ @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _effI @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(Z;S_+ S_0=s)=γ K. If S is continuous, there exists an MDP where a continuum of these skills gives unbounded potential empowerment E pot (s)=+∞ @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s)=+∞. Proof. Discrete case. We construct MDP s that admit E pot (s)≥γlogK @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s)≥γ K for all K. Consider an MDP that admits a large number of distinguishable occupancies. Then, we can assign each skill a distinct occupancy at some target future state. In this limit, the skill becomes readily identifiable from the future state. For finite K, consider a discretized state space S with states s,s1,…,sK∈s,s_1,…,s_K where all states s1,…,sKs_1,…,s_K are absorbing. Let skill z immediately transition from start state s to another target state. Then, the induced DSOM is pπ eff (s+∣s,z)=(1−γ)δs+γδsz,p @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff( [rgb]0.5,0.5,0.5s_+ s,z)=(1-γ)\, _s+γ\, _s_z, where δ is the Kronecker delta. Here, the skills share the start state s with probability mass 1−γ1-γ and are otherwise disjoint over ∖sS \s\. Observing S+≠sS_+≠ s exactly determines the skill while observing S+=sS_+=s gives no information. Thus, the MI is Iπ eff (Z;S+∣S0=s)=γH(Z)=γlogK,I @glossaries@gls@linkmainpi_mathrmeff glo:pi_mathrmeff _eff(Z;S_+ S_0=s)=γ\,H(Z)=γ K, which obtains the potential empowerment. Continuous case. We similarly construct MDP s that admit unbounded E pot (s) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s). When S is continuous, we can index skills by a continuous parameter rather than a finite set and map z∈[0,1]z∈[0,1] to S∖sS \s\. The induced DSOM is p(s+∣s,z)=(1−γ)δs+γδϕ(z)p( [rgb]0.5,0.5,0.5s_+ s,z)=(1-γ)\, _s+γ\, _φ(z) where δ is now the Dirac delta. As before, any final state that is not the start state identifies the skill. If we cut [0,1][0,1] into n equal segments, the future state S+S_+ reveals at least γlognγ n bits about the skill. Since this construction holds for all n, we have that I(Z;S+∣S0=s)≥supnγlogn=+∞.I(Z;S_+ S_0=s)≥ _nγ n=+∞. Thus, we have constructed an MDP where the MI is unbounded for every γ∈(0,1)γ∈(0,1) for continuous state spaces. ∎ We can utilize this proposition to prove that potentially infinite bits can fit within a tiny metric ball: Proposition G.2 (Many bits can fit in a tiny metric ball). Fix a start state s and a ball B(s¯,ϵ)B( s,ε) of radius ϵε with infinitely many states, centered at an arbitrary state s¯∈ s . For every K, there exists an MDP with K skills as in Proposition G.1 whose target states szs_z all lie inside the ball, so I(Z;S+∣S0=s)=γlogK,I(Z;S_+ S_0=s)=γ K, yet every bounded 1-Lipschitz reward r gives all skills nearly the same return: supz,z′|p(s+∣s,z)[r(S+)]−p(s+∣s,z′)[r(S+)]|≤2γϵ≤2ϵ. _z,z |E_p( [rgb]0.5,0.5,0.5s_+ s,z)[r(S_+)]-E_p( [rgb]0.5,0.5,0.5s_+ s,z )[r(S_+)] |≤ 2γε≤ 2ε. With a continuum of skills inside the ball, the MI can be unbounded while the upper bound on the return difference between skills is unchanged. Proof. We can apply Proposition G.1 to the ball B(s¯,ϵ)B( s,ε) and construct valid MDP s with the above MI s. For any 1-Lipschitz reward, the skill returns of z≠z′z≠ z in these MDP s differ by |p(s+∣s,z)[r(S+)]−p(s+∣s,z′)[r(S+)]| |E_p( [rgb]0.5,0.5,0.5s_+ s,z)[r(S_+)]-E_p( [rgb]0.5,0.5,0.5s_+ s,z )[r(S_+)] | =γ|r(sz)−r(sz′)| =γ\,|r(s_z)-r(s_z )| ≤γ(d(sz,s¯)+d(s¯,sz′)) ≤γ\, (d(s_z, s)+d( s,s_z ) ) ≤2γϵ ≤ 2γε which proves the result. ∎ For any 1-Lipschitz reward, it is always possible for a set of skills to have high MI (potentially unbounded) while providing no adaptation advantage for continuous rewards beyond a single policy. Thus, we should not extrapolate the tabular adaptation bound () to continuous spaces with smooth rewards. We show an example of this collapse in Fig. 6. The reachable future state occupancies pack closer together in continuous space from left to right. In the limit, the difference in values between skills shrinks to zero while the MI diverges, so the discrete adaptation bound becomes trivial. We note that prior works explicitly point out and propose solutions to the problem of diverging channel capacities in continuous spaces [37, 44], though the works do not formally connect empowerment and adaptation. Appendix H Anisotropic Reward Priors This section relaxes the reward isotropy assumption in the adaptation result . Unfortunately, using an anisotropic reward prior with covariance Σ scales the lower bound by λmin(Σ) _ ( ) (). In the limit, skills with arbitrarily large mutual information can exhibit no adaptation even with fixed reward power constraints (e.g. fixed TrΣ ; see Proposition H.2). We keep the discrete setting of and only change the reward prior. We sample rewards with br∼(0,Σ) b\,r (0, ), so the law of the uniform baseline-whitened reward g in Eq. 12 becomes (0,Σ)N(0, ). An arbitrary anisotropic prior may concentrate variance on directions that the empowerment-maximizing skills choose to ignore. The marginal’s return remains mean zero for any Σ , so the adaptation objective takes the form JΣ(s0)≜g∼(0,Σ)[supz∈⟨g,uz⟩].J (s_0) _g (0, ) [ _z g,\ u_z ]. Proposition H.1. In the setting of , sample rewards with br∼(0,Σ) b\,r (0, ) instead. Then, the MI bound rescales by λmin(Σ) _ ( ): JΣ(s0)≥λmin(Σ)I(Z;S+∣S0=s0)c2π.J (s_0)\;≥\; _ ( )\; I(Z;S_+ S_0=s_0)c 2π. The isotropic prior Σ=Id =Id recovers exactly. Proof. The proof of uses the reward prior to express a lower bound on J(s0)J(s_0) in terms of the norm of the occupancy gap vector. Under the (0,Σ)N(0, ) prior, the occupancy gap vector variance scales by the anisotropic covariance. Then, the result from suggests the following modified bound: JΣ(s0)≥step 112π(p(z∣s0)‖Σ1/2uz‖2)1/2≥step 2λmin(Σ)I(Z;S+∣S0=s0)c2π.J (s_0)\ step 1≥\ 1 2π (E_p( [rgb]0.5,0.5,0.5z s_0) \| ^1/2u_z \|^2 )^1/2\ step 2≥\ _ ( )\; I(Z;S_+ S_0=s_0)c 2π. llasection q:linear_rate_pivot_anisotropic (14) Now, we carefully derive this bound. The reduction of J to the gap in returns in only depends on a centered reward. Thus, as before, we have that JΣ(s0)≥12g|⟨g,uz⟩|.J (s_0)\ ≥\ 12\,E_g | g,\ u_z |. For g∼(0,Σ)g (0, ), the return gap ⟨g,uz⟩ g,\ u_z is Gaussian with variance ‖Σ1/2uz‖2 \| ^1/2u_z \|^2. Hence, for every z∈z , JΣ(s0)≥12π‖Σ1/2uz‖.J (s_0)\ ≥\ 1 2π \| ^1/2u_z \|. Averaging the square of this bound over z∼p(z∣s0)z p( [rgb]0.5,0.5,0.5z s_0) (the left side does not depend on z) gives step 1. We can lower bound the norm: ‖Σ1/2uz‖2=⟨uz,Σuz⟩≥λmin(Σ)‖uz‖2.\| ^1/2u_z\|^2= u_z, u_z \;≥\; _ ( )\,\|u_z\|^2. Combining with the step 2 bound (does not change with the reward prior) gives the desired result. ∎ Indeed, we can construct an example where a state and set of policies with high potential empowerment exhibit zero adapation to anisotropic rewards: Proposition H.2 (Large MI with zero anisotropic adaptation). Fix a power budget T>0T>0. For every K, there exists a discrete MDP with skills achieving I(Z;S+∣S0=s0)=γlogKI(Z;S_+ S_0=s_0)=γ K and a reward prior br∼(0,Σ) b\,r (0, ) with TrΣ=T =T such that JΣ(s0)=0J (s_0)=0. Proof. The discrete construction in Proposition G.1 attains I(Z;S+∣S0=s0)=γlogKI(Z;S_+ S_0=s_0)=γ K with K occupancy vector offsets uzu_z that span a subspace of dimension at most K−1K-1 of the (K+1)(K+1)-dimensional whitened reward space. Choose a unit vector v orthogonal to every uzu_z and set the reward variance Σ=Tvv⊤ =T\,v . This choice places all variance of the reward prior in the space orthogonal to the span of the occupancy gap vectors. By the proof of , each return gap ⟨g,uz⟩ g,u_z is Gaussian with variance ‖Σ1/2uz‖2=0\| ^1/2u_z\|^2=0, so JΣ(s0)=0J (s_0)=0 while the MI is unchanged. ∎ Thus, while the MI alone can provide nontrivial lower bounds on adaptation in tabular settings for an isotropic reward prior, the same no longer holds true for anisotropic reward priors. An empowerment-maximizing agent could learn skills with high control over observations and ignore other learnable skills that control the rewarding degrees of freedom. Appendix I Tabular Experiment Details I.1 Tabular 5×55× 5 grid experiments Table 1 details the 5×55× 5 grid MDP parameters used for Fig. 2. Skill-learning via convex optimization. In the state centrality experiments, we used a convex solver to enumerate vertices of the reachable discounted state occupancy measure (DSOM) polytope for all possible starting states. An MDP with |||S| states and |||A| actions can have up to |||||A|^|S| vertices. Thus, enumerating all vertices is challenging [30]. However, we can use the fact that reward-maximizing DSOM always sit at the vertices of the convex polytope to conduct vertex search. Specifically, we (1) sample a random reward vector θ∈Δ()θ∈ (S) in the state simplex and (2) use a convex solver to maximize DSOM alignment with the reward vector subject to probability flow constraints [45]. Since a linear objective over the reachable state polytope is maximized at an extreme point, and a generic random objective exposes a vertex almost surely, this procedure retrieves valid vertices of the polytope conditioned on specific starting states. This procedure generally does not retrieve the complete set of candidate vertices over which to specify skills, but it provides a good approximation. Policies that optimize maxπ(a∣s,z;s0),p(z∣s0)I(Z;S+∣S0=s0), _π(a s,z;s_0),\,p(z s_0)I(Z;S_+ S_0=s_0), seek out the vertices of the polytope (see ). Thus, we fix the approximate feasible set of optimal policies to the set of found vertices and break symmetries among policies with equivalent vertex DSOM s by selecting the policy with minimal entropy. This procedure constructs a one-to-one mapping between policies and vertex DSOM s and fixes the encoder in the information channel. Then, getting the empowerment-maximizing policy with known dynamics is simple: it is well known that for a fixed channel encoder pπ(s+∣s0,z)p^π(s_+ s_0,z), the MI is concave in the source distribution p(z∣s0)p(z s_0) and convex in the channel (Theorem A.1; 40). Thus, finding the optimal source distribution p(z∣s0)p(z s_0) is reduced to a convex optimization problem solvable with standard convex optimization packages (see the Blahut–Arimoto algorithm; 46, 47). The results in Fig. 2 show that the latent z-conditioned policies that start at s0=(2,2)s_0=(2,2) cover all states within the MDP. Learning empowerment-maximizing policies. The above procedure optimizes the channel capacity over the sampled vertex set, yielding a lower bound on E pot (s0)=maxπ(a∣s,z;s0),p(z∣s0)I(Z;S+∣S0=s0) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_0)= _π(a s,z;s_0),\,p(z s_0)I(Z;S_+ S_0=s_0) that tightens as the candidate vertex set grows. We use standard Q-learning methods to obtain the greedy policy that maximizes Qπ pot (s,a)=[∑t=0∞γtE pot (st)|s0=s,a0=a],Q @glossaries@gls@linkmainpi_mathrmpot glo:pi_mathrmpot _pot(s,a)=E [ _t=0^∞γ^t @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s_t) |s_0=s,a_0=a ], which we refer to as the potential empowerment Q-value. We anneal the epsilon greedy action rate from 1.0→0.011.0→ 0.01 from the start to the end of Q-learning. The optimal policy shown in Fig. 2(c) moves to the center of the room, maximizing reachability to all other states in the MDP. Table 1: Hyperparameters for the 5×55× 5 gridworld experiments (Fig. 2) Parameters Value Environment size (n × m) 5 × 5 Environment actions Left, Right, Up, Down, Stay Periodic boundary conditions No P(random action)P(random action) 0.0 Gamma for DSOM 0.95 # uniformly sampled vertex directions per s0s_0 500 Convex solver Clarabel [48] # max iterations 50_000 Error tolerance 11e−6-6 Vertex seed 0 Epsilon schedule Linear interpolation from 1→0.011→ 0.01 Gamma for potential policy 0.95 # Q-learning iterations 2×1052× 10^5 I.2 Bottleneck grid experiments Figure 4 uses the same vertex search, source-distribution optimization, and Q-learning procedure as §. I.1. The only difference is that in this GridWorld, the map is now 8×88× 8 and has impenetrable walls within the state space. We consider γ∈0.5,0.95γ∈\0.5,0.95\. This MDP demonstrates the bottleneck effect discussed in §. 4.2. In the open grid, the center state has maximal potential empowerment. With the added walls, the bottleneck openings at the gaps in the walls have high empowerment. At these states, the agent’s choice in skill substantially changes its temporal distance to future states. I.3 Key-finding experiments Table 2 gives the key-finding experiment hyperparameters. We use a numerical experimental setup identical to that of the 5×55× 5 room with walls. In this MDP, the agent can pick up and drop off a key at the square (0,2)(0,2). Picking up this key allows access to the blue-shaded regions of the MDP; without the key, the agent cannot enter these spaces in the grid. Fig. 11 shows the resulting empowerment-maximizing policy that ascends the empowerment landscape. Starting at state s0=(4,2)s_0=(4,2) without a key, the agent learns to navigate to the key, pick it up, and navigate to the center square (2,2)(2,2). This experiment shows that states with maximal empowerment capture an information-geometric notion of centrality—rather than simply navigating to and staying at the physically center state, as in the 5×55× 5 grid, the agent recognizes that the most central state is the one where it has obtained the key. Table 2: Hyperparameters for the 5×55× 5 gridworld with walls and a key Parameters Value Environment size (n × m) 5 × 5 Environment actions Left, Right, Up, Down, Stay, Pick up key at (0,2)(0,2), Drop key at (0,2)(0,2) Periodic boundary conditions No P(random action)P(random action) 0.0 Gamma for skill-learning 0.99 # uniformly sampled vertex directions per s0s_0 1000 Convex solver Clarabel [48] # max iterations 50_000 Error tolerance 10−610^-6 Vertex seed 0 Epsilon schedule Linear interpolation from 1→0.011→ 0.01 Gamma for potential policy 0.99 # Q-learning iterations 2×1052× 10^5 Figure 11: High empowerment finds central states, where skills are diverse from the central states (). Potential policy navigates to central states. Learned skills spread out from the central state and measure the degree of centrality of the state (in information geometry). The dotted line denotes the potential empowerment policy’s trajectory for 11−γ 11-γ steps. Diamond denotes the handoff state s0s_0, where the agent randomly samples z∼p(z∣s0)z p(z s_0) and rolls out skill policy π(a∣s,z;s0)π(a s,z;s_0). I.4 Adaptation geometry examples In the example MDP of Fig. 5, every skill transitions in one step from the starting state to an absorbing target state. Thus, a skill’s DSOM, normalized over the targets, is the transition distribution. Table 3 lists the resulting occupancy distributions. In Example X (Fig. 5(a)), there are NT=2N_T=2 targets, and each of two skills transitions to its target with probability one. In Example Y (Fig. 5(d)), all three skills reach NT=3N_T=3 targets and no others. Each skill transitions to its target with probability 2330 2330 and to each of the other two targets with probability 760 760. Table 3: Skill occupancy distributions of the example MDP s. (a) Example X with mass on the two reachable targets. (b) Example Y with mass on the three reachable targets. (a) Figure 5(a) MDP s1s_1 s2s_2 z=1z=1 11 00 z=2z=2 00 11 (b) Figure 5(d) MDP s1s_1 s2s_2 s3s_3 z=1z=1 23/3023/30 7/607/60 7/607/60 z=2z=2 7/607/60 23/3023/30 7/607/60 z=3z=3 7/607/60 7/607/60 23/3023/30 Figure 5(b, e) plot the polytope of the whitened state occupancy offsets uzu_z in Eq. 12 with the uniform baseline b=1/NTb=1/N_T over the targets. The overlap between Example Y’s skills leads to low channel capacity. However, the spread over the target states gives a larger Gaussian width. The orderings of the adaptation and the obtained MI therefore flip: X attains the higher mutual information and Y the larger width. In the Gaussian width panels (Fig. 5(b, e)), the offsets in Eq. 12 span at most two dimensions, and the axes (u1,u2)(u_1,u_2) give the coordinates of u in an orthonormal basis of this span. At most two offsets are linearly independent in each example in Fig. 5. The occupancy offsets in the top row are negatives of each other under the uniform skill prior and the bottom row’s three offsets satisfy u1+u2+u3=0u_1+u_2+u_3=0. Thus, we can draw a 2D diagram of these geometries (Fig. 5(b, c, e, f)); the insets in Fig. 5(f) visualize the occupancy distributions. Example X’s antipodal pair spans only one degree of freedom, so Fig. 5(b) shows a segment. In addition to the possible occupancies, the coordinates in this space can also denote reward vectors, and the background color indicates the skill that adapts to that reward. In the information panels, each colored endpoint is a possible skill’s DSOM, at the vertices of a reachable polytope in Fig. 5. The skill DSOM s and their mixture are shown explicitly in the inset bar charts. Arrows point from the capacity-achieving mixture q⋆=bq =b (black dot) to every possible skill’s DSOM, with equal positive source weights. The solid black contour in Fig. 5(f) traces the KL level set at the capacity. In Fig. 5(c), this level set is just two endpoints. I.5 Worked example of centrality in information geometry Figure 12: Worked Example of Empowerment. We illustrate the geometry of empowerment in Fig. 3(a). To start, consider learning skills that start at state s2s_2. From this state, there are three actions which deterministically transition to s3s_3, s4s_4, or s5s_5. We consider learning three skills, so z∈1,2,3z∈\1,2,3\. The problem of optimizing mutual information at state s2s_2 corresponds to learning the policy π(a∣s=s2,z)π(a s=s_2,z) so that the skill index z conveys as many bits as possible about the next state. A poor choice for this policy would be the random policy, wherein each skill selects randomly among the three actions. Each of these three skills induces the same distribution over future states s3,s4,s5s_3,s_4,s_5, conveying zero bits of information. We visualize the induced state occupancies in Fig. 3(a). Conversely, if we assign each skill a unique action (e.g., z=1z=1 always takes the leftmost action), then the state distributions the skills induce become markedly different. As visualized in Fig. 3(a), this choice of skills covers a much larger region. In state s1s_1, the transitions are stochastic. If we similarly learned a set of skills to maximize mutual information, they would arrive at occupancy measures denoted by the orange dots in Fig. 3(a). We can now interpret this example through the lens of empowerment. Is the empowerment E pot (s) @glossaries@gls@linkmainge_mathrmpot glo:ge_mathrmpotE_pot(s) higher in state s1s_1 or s2s_2? Looking at Fig. 3(a), we see that the skills at state s2s_2 are more “spread out”; formally, they convey more bits to the future. Thus, an empowerment-maximizing policy would prefer to navigate towards and stay at state s2s_2.