Paper deep dive
Generalised Bellman recurrence and three dualities in sequential decision-making
Fernando E. Rosas, David Hyland, Daniel Polani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/21/2026, 6:15:15 AM
Summary
This paper establishes that the Bellman equation arises from the mutual consistency of three conditions: dynamics decomposing through sufficient statistics (Condition D), returns decomposing recursively (Condition R), and aggregation of uncertainty being compatible with both (Condition A). The framework reveals three dualities—object, assessment, and representation—that unify methods across reinforcement learning, control, and decision theory by showing how relaxing these conditions leads to various generalizations like robust or risk-sensitive control.
Entities (13)
Relation Signals (13)
Daniel Polani → authorof → Generalised Bellman recurrence and three dualities in sequential decision-making
confidence 99% · Fernando E. Rosas 1-3 , David Hyland 4 , and Daniel Polani 5
Fernando E. Rosas → authorof → Generalised Bellman recurrence and three dualities in sequential decision-making
confidence 99% · Fernando E. Rosas 1-3 , David Hyland 4 , and Daniel Polani 5
David Hyland → authorof → Generalised Bellman recurrence and three dualities in sequential decision-making
confidence 99% · Fernando E. Rosas 1-3 , David Hyland 4 , and Daniel Polani 5
Bellman Equation → derivedfrom → Condition A
confidence 95% · When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency
Bellman Equation → derivedfrom → Condition D
confidence 95% · When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency
Bellman Equation → derivedfrom → Condition R
confidence 95% · When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency
Assessment duality → relates → return
confidence 90% · one between return and aggregation
Object duality → relates →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.
Tags
Links
- Source: https://arxiv.org/abs/2607.18077v1
- Canonical: https://arxiv.org/abs/2607.18077v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
79,945 characters extracted from source content.
Expand or collapse full text
Generalised Bellman recurrence and three dualities in sequential decision-making Generalised Bellman recurrence and three dualities in sequential decision-making Fernando E. Rosas 1-3 , David Hyland 4 , and Daniel Polani 5 f.rosas@sussex.ac.uk, david.hyland@cs.ox.ac.uk, d.polani@herts.ac.uk 1 Department of Informatics, University of Sussex 2 Department of Brain Sciences and Centre for Complexity Science, Imperial College London 3 Centre for Eudaimonia and Human Flourishing, University of Oxford 4 Department of Computer Science, University of Oxford 5 Department of Computer Science, University of Hertfordshire Abstract What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the ag- gregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming re- turn or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learn- ing, control, and decision theory. 1 Introduction The recurrence relations satisfied by value functions and the fixed-point properties of optimal solu- tions lie at the heart of reinforcement learning. These properties are usually traced to the principle of optimality, which states that optimal plans are made of optimal sub-plans (Bellman, 1957). These ideas are so central, and their applicability so broad, that Bellman recursion is often treated as the canonical expression of optimal decision-making in sequential settings. In this paper, we investigate the conditions that make the principle of optimality hold. We frame the answer as three requirements: that dynamics decompose through sufficient statistics (Condi- tion D), that returns decompose recursively (Condition R), and that the aggregation of uncertainty is compatible with both (Condition A). We then show that the Bellman equations arise when all three conditions are satisfied on a common state, and that many generalisations correspond to relaxing one condition and re-establishing it through state augmentation or by deforming returns or dynamics. These three conditions are found to give rise to three dualities, each holding one ingredient fixed while exchanging the roles of the other two. Specific instances of these dualities have been in- vestigated by separate literatures — control-as-inference, robust and regularised control, and risk- sensitive control — yet the connections between these dualities remain unexplored. Our framework shows how these symmetries arise from a single construction, and how Bellman recursion results from their mutual compatibility. This leads to a unifying language that organises multiple frame- works across the reinforcement learning, control, and decision theory literatures, revealing connec- tions between seemingly unrelated approaches to sequential decision-making. arXiv:2607.18077v1 [cs.LG] 20 Jul 2026 Finding the Frame Workshop at RLC 20262026 2 A general framework for sequential decision problems An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. (Bellman, 1957). Bellman’s principle of optimality is often taken as a starting point for deriving optimality relations. Instead, the aim of this paper is to formalise the conditions under which this principle holds and to explore the implications of those conditions. To do this, let us consider an agent taking actions a∈A in a scenario with states s∈S. 1 We denote decision histories as h t = (s 0 ,a 0 ,...,a t−1 ,s t ) in H := S × (A×S) ∗ , and infinite trajectories as h = (s 0 ,a 0 ,s 1 ,a 1 ,...) in T := (S ×A) N . Let Π be the class of all policies π of the form π : H → ∆(A), so that A t ∼ π(h t ) maps histories into distributions over actions. 2 Each policy π, together with the environment’s dynamics and initial distribution over S 0 , induces uncertainty over trajectories described by an uncertainty functional P π :T → R. In this setting, we consider the following decision problem: π ∗ ∈ arg max π∈Π F h J(H) ; H ∼ P π i .(1) Here, J : T → U is a return functional encoding preferences by assigning returns u ∈ U to each trajectory, where U is a totally or partially ordered set. Additionally, F : ∆(U ) → R is an aggregation functional that specifies how uncertainty is treated, mapping the distribution J(H) under P π to a real number. Thus, the triple⟨P π ,J,F⟩ encodes how trajectories are generated, how each trajectory is evaluated, and how those evaluations are aggregated, respectively. On its own, Eq. (1) imposes no conditions on P π , J, or F. It subsumes finite- and infinite- horizon problems, deterministic and stochastic dynamics, additive and non-additive utilities, and risk-neutral, risk-sensitive, and robust aggregations. The price of this generality is computational: to evaluate the objective in Eq. (1) requires aggregating J over the entire trajectory distribution, and optimising it requires doing so for every policy π ∈ Π. The rest of this section identifies structural assumptions under which this problem reduces to a local recursion, leading to Bellman’s principle. 2.1 Three conditions for Bellman recursion To make Eq. (1) tractable, we now formulate conditions on the statistics P π , returns J, and aggre- gation F under which the optimisation admits a Bellman recursion. We state each in turn before combining them in Sec. 2.2. Below, we consider functions over histories φ(h t ) that are unifilar (i.e., recursively updatable), so that φ(h t+1 ) is determined from φ(h t ), a t , and s t+1 . Condition D (history sufficiency). There exists a unifilar map φ D : H → Z D into a dynamics latent spaceZ D and a suitable kernel K :Z D ×A→ ∆(S) such that the distribution of trajectories under a policy π factorises as P π (h t ) = ρ(s 0 ) t−1 Y τ =0 π(a τ h τ ) K s τ +1 φ D (h τ ),a τ ,(2) with ρ a distribution over S 0 . This implies that the one-step transition depends on the history only through the sufficient statistic φ D (h t ) = z t . The Markov property is the special case where z t = s t . In partially observed settings, this condition still holds by setting z t to be a belief state (Kaelbling et al., 1998). Thus, Condition D is always met (even in non-stationary settings), but it becomes non-trivial when considering non-probabilistic notions of uncertainty (see App. B). 1 Uppercase letters (e.g. X,Y ) are used to denote random variables and lowercase (e.g. x,y) their realisations, calli- graphic letters (e.g.X,Y ) denote the sets over which they take values, the symbol ∆ (as in ∆(X )) denotes the collection of all distributions over those sets, and boldface letters (e.g. F) denote operators. 2 By doing this, we follow a time-indexing convention such that s t happens before a t . Generalised Bellman recurrence and three dualities in sequential decision-making Condition R (return compositionality). There exist a unifilar map φ R : H → Z R into a return latent spaceZ R , a local return function ℓ : Z R ×A×S → U , and an associative binary operator ⊕ :U ×U →U such that the return functional admits the decomposition J(h) = M τ≥0 ℓ φ R (h τ ), a τ , s τ +1 ,(3) where L τ≥0 A τ := A 0 ⊕ A 1 ⊕ ... . This condition implies that the one-step return depends on the history only through the sufficient statistic φ R (h t ). For example, if ℓ depends only on (s t ,a t ) and there is no time dependence, φ R can be taken to be a constant and Eq. (3) leads to memoryless rewards. In contrast, ℓ t = γ t r(s t ,a t ) requires φ R to track the running discount γ t . Condition A (aggregation compatibility). The aggregation functional F must be compatible with the compositions induced by Condition D and Condition R: • Kernel-composition compatibility (A1): For any kernels K 1 :X → ∆(Y) and K 2 :Y → ∆(Z), F Z∼K 2 ◦K 1 (·|x) Z = F Y∼K 1 (·|x) h F Z∼K 2 (·|Y ) Z i .(4) Thus, the map K 7→ F K satisfies F K 2 ◦K 1 = F K 1 ◦ F K 2 . 3 • Return compatibility (A2): There exists an binary operator⊙ :U × R→ R such that F Y∼K u⊕ Y = u⊙ F Y∼K [Y ], ∀u∈U.(5) Thus, the aggregation functional respects return composition for non-random terms. If F = E is the expectation and ⊕ = ⊙ = +, then (A1) is satisfied by the tower property of expectation and (A2) is satisfied by linearity. More generally, Condition A is also satisfied by the log-sum-exp aggregation ˆ F β [Z] = β −1 log E[exp(βZ)] (Morales et al., 2023). It is useful to note that Condition D and Condition R can be expressed recursively. Eq. (2) can be expressed as the following two-stage factorisation P π (h| h t ) = h π(a t | h t )K(s t+1 | φ D (h t ),a t ) i × P π (h| h t+1 ).(6) Similarly, Eq. (3) can be expressed as J (t) (h) = ℓ φ R (h t ), a t , s t+1 ⊕ J (t+1) (h),(7) where J (t) (h) := L τ≥t ℓ(φ R (h τ ),a τ ,s τ +1 ) is the tail return from step t, with J (0) = J. 2.2 Value function and Bellman recursion Let us now introduce value functions for this general setting. Definition 1. The value of history h t ∈H under policy π ∈ Π is V π h t := F h J (t) H ; H ∼ P π (·| h t ) i ,(8) where P π (h|h t ) is the distribution defined in Eq. (6) and J (t) is the tail return as defined in Eq. (7). We now show how the above conditions jointly give rise to Bellman recursion. For this, let us define the Bellman state z t ∈Z :=Z D ×Z R as z D t := φ D (h t ), z R t := φ R (h t ), z t := (z D t ,z R t ).(9) 3 For convenience, we sometimes use the shorthand notation F X∼K [X] := F[X ; X ∼ K], where K ∈ ∆(X ). Finding the Frame Workshop at RLC 20262026 Let us also introduce φ := (φ D ,φ R ) with φ : H → Z , so that z t = φ(h t ). The unifilarity of φ D and φ R (assumed in Condition D and Condition R) imply that φ itself is unifilar, so that there exists a map ν :Z ×A×S →Z such that φ(h t+1 ) = ν φ(h t ), a t , s t+1 , ∀h t ∈H, a t ∈A, s t+1 ∈S.(10) We consider the subclass of policies of the form Π φ := n π ∈ Π : φ(h t ) = φ(h ′ t ) =⇒ π(·| h t ) = π(·| h ′ t ),∀h t ,h ′ t ,t≥ 0 o .(11) Each π ∈ Π φ factors through φ as π(·|h t ) = ̄π(·|z t ) for a unique ̄π : Z → ∆(A). If Condition D and Condition R hold, then φ(h t ) = φ(h ′ t ) implies V π (h t ) = V π (h ′ t ) for all π ∈ Π φ (see Lemma 1, proved in App. A.3). This implies that V π also factors through φ as V π (h t ) = ̄ V π (φ(h t )) for a unique ̄ V π :Z → R. This leads to a corresponding Q-function B ̄ V (z,a) := F S∼K(·|z D ,a) h ℓ(z R ,a,S)⊙ ̄ V ν(z,a,S) i ,(12) with ν as defined in Eq. (10). One can use this to define (T π ̄ V )(z) := F A∼ ̄π(·|z) B ̄ V (z,A) as a generalised policy evaluation operator for π ∈ Π φ . Theorem 1 (Generalised Bellman recursion). Assume Condition D, Condition R, and Condition A hold. Then, ̄ V π for π ∈ Π φ satisfies the following fixed-point condition: ̄ V π (z) = (T π ̄ V π )(z), ∀z ∈Z.(13) Corollary 1 (Bellman expectation equation). Applying Theorem 1 to F = E, ⊕ = ⊙ = +, φ D (h t ) = s t , φ R (h t ) = (s t ,γ t ), and ℓ (s,γ t ),a,s ′ = γ t r(s,a) reduces Eq. (13) to ̄ V π (s) = E A∼π(·|s), S ′ ∼K(·|s,A) h r(s,A) + γ ̄ V π (S ′ ) i , ∀s∈S.(14) 2.3 Generalised Bellman optimality To go from recursion to optimality, let us define the generalised Bellman optimality operator (T ∗ ̄ V )(z) := sup μ∈∆(A) F A∼μ h B ̄ V (z,A) i .(15) We now focus on aggregator operators that are monotonic, which satisfy f (u)≤ g(u), ∀u∈U =⇒ F U∼K f (U ) ≤ F U∼K g(U ) (16) for functions f,g :U →U . This condition is satisfied by many aggregators of interest (expectation, log-sum-exp, sup, inf, robust min-max). We also focus on monotonic operations⊙ satisfying x≤ y =⇒ u⊙ x≤ u⊙ y, ∀u∈U.(17) Define the optimal value function V ∗ (h t ) := sup π∈Π V π (h t ). By Lemma 2, there exists a function ̄ V ∗ :Z → R such that V ∗ (h t ) = ̄ V ∗ φ(h t ) . Theorem 2 (Generalised Bellman optimality). Assume Condition D, Condition R, and Condition A hold, and F and⊙ are monotonic. If ̄ V ∗ is attained, then ̄ V ∗ (z) = (T ∗ ̄ V ∗ )(z), ∀z ∈ Z.(18) Corollary 2 (Bellman optimality equation). If F = E, ⊕ = +, φ D (h t ) = s t , φ R (h t ) = (s t ,γ t ), and ℓ (s,γ t ),a,s ′ = γ t r(s,a), then Eq. (18) reduces to ̄ V ∗ (s) = sup a∈A n r(s,a) + γ E S ′ ∼K(·|s,a) ̄ V ∗ (S ′ ) o .(19) Generalised Bellman recurrence and three dualities in sequential decision-making P π JF Condition D history sufficiency Condition R return compositionality Condition A aggregation compatibility Object duality P π ↔J (F = E fixed) Representation duality P ′ π ↔ ̃ F g (J fixed) Assessment duality u◦ J↔ ˆ F u J + Reg↔ ˆ F β (P π fixed) Figure 1: Three dualities in reinforcement learning. These can be arranged as a triangle, where each edge is a duality holding the opposite vertex fixed and exchanging the other two. Object duality relates the dynamics and return using the bilinearity of the expectation; assessment duality exchanges non-linearity of aggregation and return curvature or regularisation; and representation duality exchanges deformations of the probabilities. 3 Three dualities in the Bellman structure After clarifying how the principle of optimality arises from conditions over dynamics, returns, and aggregation, a natural next question is whether these conditions play comparable roles in the result- ing recursion. Here we investigate three dualities between these elements, which are illustrated in Figure 1. For simplicity, we focus on the case ofU = R and⊕ =⊙ = +. 3.1 Object duality: statistics–vs–returns Classical decision theory describes preferences as arising from two distinct entities: probabilities representing beliefs and returns representing desires (Von Neumann & Morgenstern, 1944; Savage, 1954). Despite these objects being of different kinds, they are both treated linearly by the expectation operator. This symmetry allows us to reparameterise the roles of probabilities and returns in Eq. (1) while keeping the preferences unchanged — a degree of freedom we call object duality. To study this, let us first note that using F = E as aggregator results in a bilinear pairing of the form E H∼P π h J(H) i = X h∈T J(h)P π (h) = X i∈I u i p i = u,p ,(20) whereI indexes the possible trajectories. Moreover, u,p = p,u , which leverages the natural duality between ∆(T ) and R T . In particular, if u i ≥ 0 and κ = P i u i ∈ (0,∞), then maximising the expected return J under P π is equivalent to maximising the return κ· P π under J/κ. A richer way to intermix probabilities and returns is by applying the following rotation: p i → p ′ i = g(u i )p i P i g(u i )p i , u i → u ′ i = u i g(u i ) ,(21) where it is assumed that g(u i ) > 0 for all u i . For an affine map g(u) = α + βu, one finds that u ′ ,p ′ = u,p α + β u,p .(22) As the mapping x→ x/(α + βx) is increasing for α > 0, this implies that an agent maximising u i under p i is indistinguishable from one maximising u ′ i under p ′ i . For α = 1 this corresponds to the so-called Jeffrey–Bolker rotations (Jeffrey, 1965; Bolker, 1966; Demski, 2018). Finding the Frame Workshop at RLC 20262026 The symmetry between returns and probabilities can also be exploited by interpreting the return as arising from a distribution. This can be done by introducing the distribution q i := e u i /Z with Z = P j e u j , which allows us to rewrite Eq. (20) as (Ortega & Tishby, 2016; Wentworth, 2021) E H∼P π h J(H) i = X i p i logq i + logZ =− H(p;q) |z cross-entropy + logZ | z free energy .(23) Thus, re-expressing the return as a log-probability turns a problem of expected return maximisation into a minimisation of description length (Grünwald, 2007). This serves as the foundation for active inference (Friston et al., 2015; Da Costa et al., 2020), which replaces the Helmholtz free energy in Eq. (23) with variational free energy to pursue approximate Bayesian inference (see App. C). 3.2 Assessment duality: return–vs–aggregation An agent’s attitude towards uncertainty is reflected by the aggregation functional F, which may operate non-linearly on the return. Here we study how such non-linear effects can be relocated from the aggregation into the returns — a degree of freedom that we call assessment duality. To investigate this, consider an increasing utility transformation u : R → R. Given a return func- tional J(h), one may either transform the return itself and aggregate linearly, E H∼P π u J(H) , or leave the return unchanged and replace expectation by the certainty-equivalent aggregator ˆ F u [Z] := u −1 E u(Z) .(24) For example, if u is concave then Jensen’s inequality gives ˆ F u [Z] ≤ E[Z], so ˆ F u penalises uncer- tainty. Since u −1 is also strictly increasing, the two objectives induce the same policy ordering: arg max π E h∼P π h u J(h) i = arg max π ˆ F u h∼P π h J(h) i .(25) Regarding Condition A, ˆ F u satisfies A1 but only satisfies A2 for the log-sum-exp family given by u(x) = e βx (Morales et al., 2023). If u(x) is such that A2 does not hold (e.g. for the geometric mean), assessment duality can be used to recover Bellman recursion (MacLean et al., 2011). Thus, the same risk attitude can be represented either by risk-sensitive aggregation functional ˆ F u , or by modifying the curvature of returns via functional composition u◦ J and aggregating them in a risk-neutral manner. This idea is exploited by risk-sensitive control (Howard & Matheson, 1972; Whittle, 1981) and multiplicative dynamic programming (Bellman, 1957; Puterman, 1994). A second facet of the assessment duality turns the effects of non-linear aggregation into reward regularisation. To study this, note first that the Bellman optimality operator features two distinct aggregations: an expectation over next-state transitions (Eq. (12)) and a maximisation over actions (Eq. (15)). These two operations can be unified by seeing them as particular cases of the log-sum- exp family given by ˆ F β := ˆ F u for u(x) = e βx , which interpolates between expectation (β → 0 + ) and supremum (β →∞) (Ortega et al., 2015). Then, one can generalise Eq. (19) as V ∗ β a ,β s (s) = ˆ F β a A∼π 0 (·|s) h r(s,A) + γ ˆ F β s S ′ ∼K(·|s,A) V ∗ β a ,β s (S ′ ) i , β a ,β s ∈ R∪±∞, (26) where π 0 is a reference policy. For different values of (β a ,β s ) this formulation recovers several well-known formulations: • For (+∞,−∞), it becomes the robust Bellman equation V ∗ (s) = max a∈A n r(s,a) + γmin q∈Q(s,a) E S ′ ∼q [V ∗ (S ′ )] o , Q(s,a)⊂ ∆(S),(27) whose fixed-point corresponds to the best policy against an adversarial environment with value V ∗ rob (s) = sup π inf Q π ∈Q V π Q π (s) (Iyengar, 2005; Nilim & El Ghaoui, 2005). 4 4 Formulating robust RL within our framework requires a more general notion of uncertainty (see App. B.2). Generalised Bellman recurrence and three dualities in sequential decision-making • For (+∞,β s ), it becomes the risk-sensitive Bellman equation V ∗ (s) = max a∈A n r(s,a) + 1 β s log E S ′ ∼K(·|s,a) exp β s V ∗ (S ′ ) o ,(28) whose fixed point is V ∗ rsc (s) = sup π β −1 s log E π [exp(β s J (t) )|S t = s], being a non-linear average of the return (Howard & Matheson, 1972; Whittle, 1981). 5 • For (β a , 0), it becomes the soft Bellman equation V ∗ (s) = 1 β a log X a∈A π 0 (a| s) exp β a r(s,a) + E S ′ ∼K(·|s,a) V ∗ (S ′ ) ,(29) whose fixed-point satisfies V ∗ soft (s) = sup π E π [J|S t = s]−β −1 a D KL P π ∥P π 0 (Ziebart, 2010; Haarnoja et al., 2017). For π 0 uniform this reduces to the objective in control-as-inference and maximum-entropy RL (Todorov, 2007; Kappen et al., 2012; Levine, 2018), being closely related to active inference (Millidge et al., 2020; Da Costa et al., 2023). Thanks to assessment duality, these three formulations can also be obtained by regularising the return as J(h)→ J(h) + (Reg) (Petersen et al., 2000; Osogami, 2012; Husain et al., 2021; Derman et al., 2021; Brekelmans et al., 2022). For the robust and risk-sensitive settings one can find that V ∗ (s) = max a∈A ext q∈∆(S) E S ′ ∼q h ˆ ℓ(s,a,S ′ ;q) + γV ∗ (S ′ ) i (30) where ext = inf for robust and risk-averse cases and ext = sup for risk-seeking, with ˆ ℓ rob (s,a,s ′ ;q) = r(s,a) + I Q(s,a) (q),(robust Bellman)(31a) ˆ ℓ risk (s,a,s ′ ;q) = r(s,a) + 1 β s log K(s ′ |s,a) q(s ′ ) ,(risk-sensitive Bellman)(31b) where I Q(s,a) (q) =∞ if q /∈Q(s,a) and zero otherwise. The soft setting becomes V ∗ (s) = sup μ∈∆(A) E A∼μ,S ′ ∼K(·|s,A) h ˆ ℓ(s,A;π) + γV ∗ (S ′ ) i with(32) ˆ ℓ soft (s,a;π) = r(s,a)− 1 β a log μ(a|s) π 0 (a|s) .(33) Proofs are provided in App. D. 3.3 Representation duality: dynamics–vs–aggregation While the previous section considered aggregation functionals that are non-linear over the return, here we consider aggregations that may be non-linear over distributions. In particular, we study how such non-linearities can be relocated into a deformation of the probabilities — a degree of freedom we call representation duality. Concretely, we are looking for a deformation of the trajectory law P π 7→ P ′ π such that F H∼P π J(H) = E H∼P ′ π J(H) ,(34) thus turning a non-linear aggregator into ordinary expectation. It can be shown that such a deforma- tion exists for aggregators of the form (see App. E) ̃ F g h J(H) ; H ∼ P π i := Z ∞ 0 g S(x) dx− Z 0 −∞ h 1− g S(x) i dx,(35) 5 For small β s the value becomes V ∗ rsc (s) ≈ E π [J|S t = s] + β s /2· Var[J|S t = s], so that variance is rewarded or penalised depending on the sign of β s . Finding the Frame Workshop at RLC 20262026 where S(x) = P π J(H) > x and g : [0, 1] → [0, 1] is an increasing function with g(0) = 0 and g(1) = 1 (Föllmer & Schied, 2025). Representation duality is closely related to three classic decision-theoretic constructions: Yaari’s dual theory of choice (Yaari, 1987), Choquet expected utility (Schmeidler, 1989), and max-min expected utility (Gilboa & Schmeidler, 1989). For example, consider the conditional value at risk aggregator CVaR α [J(H)] = ̃ F g [J(H);H ∼ P π ] for g(s) = α −1 maxs− (1− α), 0 (Rockafellar & Uryasev, 2000). It can be shown that CVaR α [J(H)] = E H∼P ′ π J(H) , dP ′ π dP π (h) = 1 α 1J(h)≤ q α (J).(36) where q α (J) is the α-quantile of J under P π . Thus, the non-standard aggregator has been absorbed into a reweighting of P π that concentrates mass on the worst α-fraction of outcomes. With respect to Condition A, ̃ F g always satisfies A2 but generally does not satisfy A1. Luckily, representation duality can recover Bellman recursion for a deformed distribution P ′ π . The trans- formation P π → P ′ π may depend on J — for instance, the reweighting dP ′ /dP π in Eq. (36) is a function of the rank ordering of J. Thus, this is not a fixed deformation that could be applied without knowledge of J. 4 Discussion We have shown that Bellman equations and the optimality principle are consequences of the struc- tural compatibility of dynamics, return, and aggregation, which can be stated as three fundamental conditions. Many reinforcement learning frameworks can be understood as relaxing one of these conditions, and restoring it by augmenting the state or by deforming returns or probabilities. Our framework provides insight into how such strategies work and the limits of their applicability. These three conditions were found to give rise to three dualities, which synthesise and extend a substantial body of prior work across reinforcement learning, optimal control, statistics, theoretical neuroscience, and category theory. • Object duality explores symmetries induced by the expectation operator, which treats both re- turns and probabilities linearly. This enables exchangeability between them, underlying the util- ity–probability conjugacy of Jeffrey (1965) and Bolker (1966) and the usage of priors instead of returns in active inference (Friston et al., 2015). • Assessment duality explores how non-linearities in how returns are aggregated can be turned into either return curvature or reward regularisation, being the foundation behind the regularisation observed in control-as-inference (Levine, 2018), maximum-entropy (Ziebart, 2010) and robust reinforcement learning (Husain et al., 2021). • Representation duality explores how non-linearities in how probabilities are aggregated can be turned into deformations of the distribution, being closely related to Choquet (Schmeidler, 1989) and max-min expected utility (Gilboa & Schmeidler, 1989). The mathematical foundation is the categorical and coalgebraic treatment of dynamical systems (Feys et al., 2018; Hedges & Rodríguez Sakamoto, 2025; Fritz, 2020), which already formalises some of these dualities in isolation. For details regarding related work, see App. F. Thus, while each of these dualities was previously known, our contributions are: (i) the formulation of a general objective and conditions on dynamics, return, and aggregation such that the Bellman equation is the result of a structural compatibility, (i) the treatment of state augmentation and return or dynamics deformation as a recourse when a condition fails; (i) a synthesis identifying the three dualities as pairwise symmetries of the transition, return, and aggregation functionals, each holding one of the conditions fixed; and (iv) the derivation of existing formulations in various subfields as special cases of these dualities. Generalised Bellman recurrence and three dualities in sequential decision-making Acknowledgments The authors thank Cameron Allen, Artemy Kolchinsky, Alexander Gietelink Oldenziel, and Pedro Ortega for insightful discussions and useful feedback. References Nicole Bäuerle and Ulrich Rieder. More risk-sensitive Markov decision processes. Mathematics of Operations Research, 39(1):105–120, 2014. Richard Bellman. Dynamic Programming. Princeton University Press, 1957. Philippe Bich, Jean-Pierre Drugeon, and Lisa Morhaim. On aggregators and dynamic programming. Technical report, Université Panthéon-Sorbonne, Centre d’Economie de la Sorbonne, 2015. Martin Biehl, Felix Pollock, and Ryota Kanai. A technical critique of some parts of the free energy principle. Entropy, 23(3):293, 2021. Ethan Bolker. Functions resembling quotients of measures. Transactions of the American Mathe- matical Society, 124(2):292–312, 1966. Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, and Pedro Ortega.Your policy regularizer is secretly an adversary.arXiv preprint arXiv:2203.12592, 2022. Bryce Clarke, Derek Elkins, Jeremy Gibbons, Fosco Loregian, Bartosz Milewski, Emily Pillmore, and Mario Román. Profunctor optics, a categorical update. Compositionality, 6(1), 2024. Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston. Active inference on discrete state-spaces: A synthesis. Journal of Mathematical Psychology, 99: 102447, 2020. Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith. Reward maximization through discrete active inference. Neural Computation, 35(5):807–852, 04 2023. AbramDemski.Probabilityisreal,andvalueiscomplex,2018.URL https://w.lesswrong.com/posts/oheKfWA7SsvpK7SGp/ probability-is-real-and-value-is-complex. Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34: 22274–22287, 2021. Jan Dhaene, Steven Vanduffel, Marc Goovaerts, Rob Kaas, Qihe Tang, and David Vyncke. Risk measures and comonotonicity: a review. Stochastic models, 22(4):573–606, 2006. Michael Duff. Optimal Learning: Computational Procedures for Bayes-Adaptive Markov Decision Processes. PhD thesis, University of Massachusetts Amherst, 2002. Paul Dupuis, Matthew James, and Ian Petersen. Robust properties of risk-sensitive control. Mathe- matics of Control, Signals and Systems, 13(4):318–332, 2000. Frank Feys, Helle Hansen, and Lawrence Moss. Long-term values in Markov decision processes, (co)algebraically. In Coalgebraic Methods in Computer Science: 14th IFIP WG 1.3 International Workshop (CMCS 2018), p. 78–99, Berlin, Heidelberg, 2018. Springer-Verlag. Hans Föllmer and Alexander Schied. Stochastic finance: an introduction in discrete time. Walter de Gruyter GmbH & Co KG, 2025. Finding the Frame Workshop at RLC 20262026 Karl Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11 (2):127–138, 2010. Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Gio- vanni Pezzulo. Active inference and epistemic value. Cognitive Neuroscience, 6(4):187–214, 2015. Karl Friston, Lancelot Da Costa, Danijar Hafner, Casper Hesp, and Thomas Parr. Sophisticated inference. Neural Computation, 33(3):713–763, 03 2021. Tobias Fritz. A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mathematics, 370:107239, 2020. Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In International conference on machine learning, p. 2160–2169. PMLR, 2019. Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5–6):359–483, 2015. Itzhak Gilboa and David Schmeidler. Maxmin expected utility with non-unique prior. Journal of mathematical economics, 18(2):141–153, 1989. Peter Grünwald. The minimum description length principle. MIT press, 2007. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning, p. 1352–1361, 2017. Lars Hansen and Thomas Sargent. Robustness. Princeton university press, 2011. Jules Hedges and Riu Rodríguez Sakamoto. Value iteration is optic composition. In Proceedings of Applied Category Theory 2022, volume 380 of Electronic Proceedings in Theoretical Computer Science, p. 417–432, 2023. Jules Hedges and Riu Rodríguez Sakamoto. Reinforcement learning in categorical cybernetics. In Proceedings of Applied Category Theory 2024, volume 429 of Electronic Proceedings in Theo- retical Computer Science, p. 270–286, 2025. Ronald Howard and James Matheson. Risk-sensitive Markov decision processes. Management Science, 18(7):356–369, 1972. Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics, p. 64–72. PMLR, 2021. Rodrigo Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, p. 2107–2116. PMLR, 2018. Garud Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257– 280, 2005. Richard Jeffrey. The Logic of Decision. McGraw-Hill, 1965. Leslie Kaelbling, Michael Littman, and Anthony Cassandra. Planning and acting in partially ob- servable stochastic domains. Artificial Intelligence, 101(1–2):99–134, 1998. Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine Learning, 87(2):159–182, 2012. Generalised Bellman recurrence and three dualities in sequential decision-making Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, p. 128–135, 2005. Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018. Leonard MacLean, Edward Thorp, and William Ziemba. The Kelly capital growth investment crite- rion: Theory and practice, volume 3. World Scientific, 2011. Beren Millidge. A retrospective on active inference. Blog post, 2024. URL https://w. beren.io/2024-07-27-A-Retrospective-on-Active-Inference/. Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher Buckley. On the relationship be- tween active inference and control as inference. arXiv preprint arXiv:2006.12964, 2020. Beren Millidge, Alexander Tschantz, and Christopher Buckley. Whence the expected free energy? Neural Computation, 33(2):447–482, 2021. Pablo Morales, Jan Korbel, and Fernando Rosas. Thermodynamics of exponential Kolmogorov– Nagumo averages. New Journal of Physics, 25(7):073011, 2023. Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing sys- tems, 30, 2017. Cleo Nardo. Uncertainty in all its flavours, 2024. URL https://w.lesswrong.com/ posts/JhhFokDRkAfg5Tnu9/uncertainty-in-all-its-flavours. Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005. Junhyuk Oh, Matteo Hessel, Wojciech Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Informa- tion Processing Systems, 33:1060–1070, 2020. Pedro Ortega and Daniel Braun. Thermodynamics as a theory of decision-making with information- processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469(2153):20120683, 2013. Pedro Ortega and Naftali Tishby. Memory shapes time perception and intertemporal choices. arXiv preprint arXiv:1604.05129, 2016. Pedro Ortega, Daniel Braun, Justin Dyer, Kee-Eung Kim, and Naftali Tishby. Information-theoretic bounded rationality. arXiv preprint arXiv:1512.06789, 2015. Takayuki Osogami. Robustness and risk-sensitivity in Markov decision processes. Advances in neural information processing systems, 25, 2012. Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in Neurorobotics, 1:6, 2007. Thomas Parr, Giovanni Pezzulo, and Karl Friston. Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. The MIT Press, 03 2022. Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, p. 2778–2787, 2017. Finding the Frame Workshop at RLC 20262026 Ian Petersen, Matthew James, and Paul Dupuis. Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Transactions on Automatic Control, 45(3):398– 412, 2000. Martin Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994. Jad Rahme and Ryan Adams. A theoretical connection between statistical physics and reinforcement learning. arXiv preprint arXiv:1906.10228, 2019. Mitchell Riley. Categories of optics. arXiv preprint arXiv:1809.00738, 2018. R Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of risk, 2: 21–42, 2000. Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems 27, p. 1583–1591, 2014. Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. Operations Research, 66(1):230–252, 2018. Noor Sajid, Philip Ball, Thomas Parr, and Karl Friston. Active inference: Demystified and com- pared. Neural Computation, 33(3):674–712, 2021. Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment: An introduction. In Guided Self-Organization: Inception, p. 67–114. Springer, 2014. Leonard Savage. The foundations of statistics. John Wiley & Sons, 1954. David Schmeidler. Subjective probability and expected utility without additivity. Econometrica: Journal of the Econometric Society, p. 571–587, 1989. John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint arXiv:1704.06440, 2017. Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th international conference on autonomous agents and multiagent systems-volume 2, p. 761–768, 2011. Yuting Tang, Yivan Zhang, Johannes Ackermann, Yu-Jie Zhang, Soichiro Nishimori, and Masashi Sugiyama. Recursive reward aggregation. In Reinforcement Learning Conference, 2025. Emanuel Todorov. Linearly-solvable Markov decision problems. In Advances in Neural Information Processing Systems 19, p. 1369–1376, 2007. John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton University Press, 1944. John Wentworth.Utility maximization = description length minimization.Less- Wrong, 2021. URL https://w.lesswrong.com/posts/voLHQgNncnjjgAPH7/ utility-maximization-description-length-minimization. Peter Whittle. Risk-sensitive linear/quadratic/Gaussian control. Advances in Applied Probability, 13(4):764–777, 1981. Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathe- matics of Operations Research, 38(1):153–183, 2013. Menahem Yaari. The dual theory of choice under risk. Econometrica: Journal of the Econometric Society, p. 95–115, 1987. Generalised Bellman recurrence and three dualities in sequential decision-making Brian Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: A very good method for Bayes-adaptive deep RL via meta- learning. In International Conference on Learning Representations (ICLR), 2020. Finding the Frame Workshop at RLC 20262026 Supplementary Materials The following content was not necessarily subject to peer review. A Proofs For notational convenience, the proofs will denote the combined action-transition kernel as e K(a,s| z) := ̄π(a| z) K(s| z D ,a), e K :Z → ∆(A×S).(37) A.1 Proof of Theorem 1 Theorem 1 (Generalised Bellman recursion). Assume Condition D, Condition R, and Condition A hold. Then, ̄ V π for π ∈ Π φ satisfies the following fixed-point condition: ̄ V π (z) = (T π ̄ V π )(z), ∀z ∈Z.(13) Proof. To start, let us note that Condition R’s one-step recursion (Eq. (7)) implies that J (t) (h) = ℓ z R t ,a t ,s t+1 ⊕ J (t+1) (h).(38) Additionally, by Condition D and π ∈ Π φ , the conditional trajectory law admits the two-stage factorisation P π (·| h t ) = e K(·| z t )× P π (·| h t+1 )(39) with z t = φ(h t ), where the first stage samples (a t ,s t+1 ) from e K(· | z D t ) and the second samples the remaining suffix from P π (· | h t+1 ) with h t+1 = (h t ,a t ,s t+1 ). By Lemma 1 applied at time t + 1, P π (· | h t+1 ) depends on h t+1 only through φ(h t+1 ) = ν(z t ,a t ,s t+1 ), so the second factor is a genuine kernel on (A×S)→ ∆(T ). With all these pieces in place, we can prove the result as follows: ̄ V π (z t ) = F H∼P π (·|h t ) J (t) (H) (def. + L1) = F H∼P π (·|h t ) ℓ(z R t ,A t ,S t+1 )⊕ J (t+1) (H) (Eq. (38)) = F A t ,S t+1 ∼ e K(·|z t ) ◦ F H∼P π (·|h t+1 ) ℓ(z R t ,A t ,S t+1 )⊕ J (t+1) (H) (A1 + Eq. (39)) = F A t ,S t+1 ∼ e K(·|z t ) h ℓ(z R t ,A t ,S t+1 ) ⊙ F H∼P π (·|h t+1 ) J (t+1) (H) i (A2) = F A t ,S t+1 ∼ e K(·|z t ) h ℓ(z R t ,A t ,S t+1 ) ⊙ ̄ V π ν(z t ,A t ,S t+1 ) i (def. + L1) = F A t ∼ ̄π(·|z t ) ◦ F S t+1 ∼K(·|z D t ,A t ) h ℓ(z R t ,A t ,S t+1 ) ⊙ ̄ V π ν(z t ,A t ,S t+1 ) i (A1 + Eq. (37)). Renaming the dummy variables z ↔ z t gives Eq. (13). A.2 Proof of Theorem 2 Theorem 2 (Generalised Bellman optimality). Assume Condition D, Condition R, and Condition A hold, and F and⊙ are monotonic. If ̄ V ∗ is attained, then ̄ V ∗ (z) = (T ∗ ̄ V ∗ )(z), ∀z ∈ Z.(18) Proof. By Lemma 2, V ∗ (h t ) = ̄ V ∗ φ(h t ) . We prove the history-level optimality identity V ∗ (h t ) = sup μ∈∆(A) F A∼μ,S∼K(·|φ D (h t ),A) h ℓ φ R (h t ),A,S ⊙ V ∗ (h t ,A,S) i ,(40) Generalised Bellman recurrence and three dualities in sequential decision-making Condition D History sufficiency Condition R Return compositionality Condition A Aggregation compatibility Bellman recursion ̄ V π (z) = (T π ̄ V π )(z) Theorem 1 Bellman optimality ̄ V ∗ (z) = (T ∗ ̄ V ∗ )(z) Theorem 2 F,⊙ monotonic Figure 2: Conditions giving rise to generalised Bellman recursion and optimality. The three conditions D, R, A yield a generalised Bellman recursion (Theorem 1). Adding an additional monotonicity assumption onF and⊙ yields the Bellman optimality equation (Theorem 2). which equals (T ∗ ̄ V ∗ )(z) after factorisation. For notational convenience, the proof will use e K π (a,s|h t ) := π(a| h t )K(s| φ D (h t ),a).(41) Upper bound. Let π ∈ Π and write μ π := π(· | h t ). Since V π ≤ V ∗ pointwise, monotonicity of⊙ and of F applied to Lemma 3 give V π (h t ) = F e K μ π ℓ⊙ V π (·) ≤ F e K μ π ℓ⊙ V ∗ (·) ≤ sup μ∈∆(A) F e K μ ℓ⊙ V ∗ (·) . Taking the supremum over π ∈ Π on the left yields V ∗ (h t )≤ RHS of Eq. (40). Lower bound (≥). Fix μ∈ ∆(A). For each successor (h t ,a,s) choose an attaining policy π a,s ∈ Π with V π a,s (h t ,a,s) = V ∗ (h t ,a,s) . Define π μ ∈ Π to play μ at h t and, after observing (a,s), to follow π a,s . Because Π is the full class of history-dependent policies, it is closed under this one-step concatenation, so π μ ∈ Π. Applying Lemma 3 to π μ gives V π μ (h t ) = F e K μ ℓ⊙ V π μ (·) = F e K μ ℓ⊙ V ∗ (·) , the second equality holding because V π μ (h t ,a,s) = V π a,s (h t ,a,s) = V ∗ (h t ,a,s) . Since π μ ∈ Π we have V π μ (h t )≤ V ∗ (h t ), hence F e K μ [ℓ⊙ V ∗ (·)]≤ V ∗ (h t ). Taking the supremum over μ∈ ∆(A) gives V ∗ (h t )≥ RHS of Eq. (40). The two bounds establish Eq. (40), and factorising through φ, gives ̄ V ∗ = T ∗ ̄ V ∗ . A.3 Other results Here we state and prove auxiliary results that are used to prove the main theorems. Lemma 1 (Value factorisation). If Condition D and Condition R hold, then φ(h t ) = φ(h ′ t ) =⇒ V π (h t ) = V π (h ′ t ), ∀π ∈ Π φ ,t≥ 0.(42) Proof. By Definition 1, V π (h t ) is determined by the joint distribution of J (t) (H) when H ∼ P π (·| h t ). It therefore suffices to show that, as h t varies over histories with a fixed value of φ(h t ), both the conditional law P π (·| h t ) (Step 1) and the function H 7→ J (t) (H) (Step 2) are unchanged. Step 1. By Condition D and π ∈ Π φ , the conditional trajectory distribution factors as P π (h| h t ) = Y τ≥t ̄π a τ | φ(h τ ) K s τ +1 | φ D (h τ ), a τ .(43) Finding the Frame Workshop at RLC 20262026 Due to the unifilarity of φ = (φ D ,φ R ), every φ(h τ ) with τ ≥ t is determined by φ(h t ) and the intermediate pairs (a t ,s t+1 ),..., (a τ−1 ,s τ ). Hence, Eq. (43) depends on h t only through φ(h t ). Step 2. By Condition R, J (t) (h) = M τ≥t ℓ φ R (h τ ), a τ , s τ +1 . By the unifilarity of φ R , each φ R (h τ ) for τ ≥ t is determined by φ R (h t ) and the same intermediate pairs. Hence the local returns ℓ(φ R (h τ ),a τ ,s τ +1 ) depend on h t only through φ R (h t ). Finally, the functional V π (h t ) is determined by the distribution of J (t) (h) under H ∼ P π (· | h t ). Steps 1 and 2 show that both the law and the random variable depend on h t only through φ(h t ), so the distribution of J (t) (H) does, and hence so does V π (h t ). Lemma 2 (Factorisation of the optimal value). Assume Condition D and Condition R. Define V ∗ (h t ) := sup π∈Π V π (h t ). Then for any two histories with φ(h t ) = φ(h ′ t ), V ∗ (h t ) = V ∗ (h ′ t ), so V ∗ factors through φ as V ∗ (h t ) = ̄ V ∗ φ(h t ) for a unique ̄ V ∗ :Z → R. Proof. Take h t ,h ′ t ∈H with φ(h t ) = φ(h ′ t ) = z. For a given suffix u = (a t ,s t+1 ,a t+1 ,s t+2 ,... ) and each τ ≥ t, write h u τ := (h t ,a t ,s t+1 ,...,a τ−1 ,s τ ),(h ′ ) u τ := (h ′ t ,a t ,s t+1 ,...,a τ−1 ,s τ ) for the two histories obtained by appending the same suffix to h t and to h ′ t . Step 1. We prove by induction that φ h u τ = φ (h ′ ) u τ for all τ ≥ t.(44) At τ = t (taking the empty suffix) the claim reduces to the hypothesis φ(h t ) = φ(h ′ t ). Now, suppose Eq. (44) holds at τ . By the unifilarity of φ = (φ D ,φ R ) guaranteed by Condition D and Condition R, there is a map ν :Z ×A×S →Z with φ(h τ +1 ) = ν φ(h τ ),a τ ,s τ +1 for every h τ +1 . Applying this to both extended histories and using the inductive hypothesis, one finds that φ h u τ +1 = ν φ(h u τ ),a τ ,s τ +1 = ν φ((h ′ ) u τ ),a τ ,s τ +1 = φ (h ′ ) u τ +1 . This implies that φ D and φ R agree along the two suffix-extended histories at every τ ≥ t. Step 2. The value V π (h t ) depends on π only through its action distributions on histories extending h t . Define a relabelling Θ of these restrictions by mimicking along matched suffixes: for every finite suffix u, Θ(π) (h ′ ) u τ := π h u τ , τ ≥ t. We show V π (h t ) = V Θ(π) (h ′ t ) by checking that the trajectory law and the tail return agree. By Condition D, the conditional trajectory law factorises along the suffix through φ D : P π u| h t = Y τ≥t π a τ | h u τ K s τ +1 | φ D (h u τ ),a τ . By Step 1, φ D (h u τ ) = φ D ((h ′ ) u τ ) for every τ , and by construction Θ(π) (h ′ ) u τ = π h u τ , hence each factor is unchanged, giving P Θ(π) (·| h ′ t ) = P π (·| h t ). Step 3. By Condition R, the tail return decomposes as J (t) h t ,u = M τ≥t ℓ φ R (h u τ ),a τ ,s τ +1 , Generalised Bellman recurrence and three dualities in sequential decision-making and likewise from h ′ t . By Step 1, φ R (h u τ ) = φ R ((h ′ ) u τ ) for every τ , so the two⊕-sums have identical terms: J (t) h t ,u = J (t) h ′ t ,u for every suffix u. Since the law of the suffix and the tail-return functional both coincide, the random variable J (t) has the same distribution under (π,h t ) and under (Θ(π),h ′ t ). By Definition 1, V π (h t ) = V Θ(π) (h ′ t ). Step 4: equality of suprema. The map Θ is a bijection between the policy restrictions on histories extending h t and those extending h ′ t , its inverse mimicking in the opposite direction. Hence the sets of achievable valuesV π (h t ) : π ∈ Π andV π ′ (h ′ t ) : π ′ ∈ Π coincide, and therefore so do their suprema: V ∗ (h t ) = sup π∈Π V π (h t ) = sup π ′ ∈Π V π ′ (h ′ t ) = V ∗ (h ′ t ). As V ∗ takes a common value on every fibre h t : φ(h t ) = z, it factors through φ, defining a unique ̄ V ∗ : Z → R with V ∗ (h t ) = ̄ V ∗ (φ(h t )). Lemma 3 (History-level Bellman recursion). Assume Condition D, Condition R, and Condition A hold. For every π ∈ Π and every h t ∈H, V π (h t ) = F A∼π(·|h t ),S∼K(·|φ D (h t ),A) h ℓ φ R (h t ),A,S ⊙ V π (h t ,A,S) i .(45) Proof. The argument is identical to Steps 1–3 of the proof of Theorem 1 but with ̄π(a| z) replaced by π(a | h t ). Those steps only use Conditions D, R, and A. None of them require π ∈ Π φ (that restriction is invoked only afterwards, through Lemma 1, to re-express V π as a function of z = φ(h t )). B On the role of Condition D Here we show that Condition D is formally trivial if arbitrary history-valued statistics are allowed. However, this does not make its role vacuous as part of Bellman’s construction. It serves four purposes. 1. It captures the compression problem, i.e., whether a proposed state z t contains all past informa- tion relevant to accounting for future uncertainty. 2. It separates dynamics-sufficiency (via φ D ) from return-sufficiency (via φ R ). Even if the full history satisfies both, a useful Bellman equation requires a common compressed state. 3. It clarifies where non-stationarity is handled: time must either be irrelevant or be included in the statistic. 4. It allows us to generalise the framework from probability theory to other uncertainty formalisms (see App. B.2). Thus, the core of Condition D is not that some φ D exists, but is that the chosen uncertainty represen- tation admits a local, recursively updateable, and hopefully compressed statistic that is compatible with the return and aggregation structures used by the Bellman recursion. B.1 Clarifications in probabilistic settings Condition D is stated as the existence of a unifilar statistic φ D :H→Z D satisfying P π (h t ) = ρ(s 0 ) t−1 Y τ =0 π(a τ h τ ) K s τ +1 φ D (h τ ),a τ .(46) Here, we consider some examples. History-dependent dynamics. For an arbitrary process, the trivial statistic φ D (h t ) = h t withZ D = H satisfies Condition D, but gives no compression and no tractable Bellman equation in general. Finding the Frame Workshop at RLC 20262026 The recursion is then merely a recursion on the tree of histories, since h t+1 = (h t ,a t ,s t+1 ). In other words, every stochastic process is Markov when the state is taken to be its complete past. Markovian dynamics. If Condition D is satisfied with φ D (h t ) = s t , then the scenario corresponds to a Markov decision process. If the transition dynamics are time-inhomogeneous, then φ D (h t ) = s t is generally not sufficient, but φ D (h t ) = (t,s t ) is. Thus, a time-inhomogeneous model can still satisfy Condition D after augmenting the state by adding a clock, soZ D =S× N. Partially-observable Markov decision processes (POMDPs). In a POMDP, the state s t is not directly observed. Instead, observations are obtained from a probabilistic mapping S → ∆(Ω), where Ω is a set of possible observations for the agent. However, these observations are not dynamically- sufficient in general. POMDPs with known dynamics can be converted into a belief-space MDP, where each state is a probability distribution representing the agent’s belief about the hidden state s ∈ S . The belief state b t ∈ ∆(S), updated by Bayes’ rule from the previous belief, the action, and the observation, is the canonical dynamics-sufficient statistic. In this case, Condition D is the statement that the belief process is Markovian, even though the observation process may not be. B.2 Beyond probability The same distinction between trivial full-history representations and a useful compressed represen- tation can be generalised to more general notions of uncertainty. LetM(Y ) denote the space of uncertainty objects over a space Y . An uncertainty kernel is a map L : X →M(Y ). WhenM = ∆, these are ordinary Markov kernels. More generally,M may be a powerset functor for nondeterministic systems, a semiring-valued distribution functor for weighted systems, or a credal set for imprecise probabilities (Nardo, 2024). In such a setting, Condition D should be understood as requiring local unrolling of future uncertainty through the statistic φ D . That is, for each policy π, there is a one-step uncertainty kernel Λ π :Z D →M(A×S) such that the uncertainty object over future suffixes from h t is obtained by first drawing, selecting, or weighting one local action-state pair according to Λ π (φ D (h t )), and then continuing from the updated history (h t ,a t ,s t+1 ). Abstractly, if the uncertainty calculus has a sequential composition operation, written here as bind, this takes the schematic form P t π (·| h t ) = Λ π (φ D (h t )) bind (a t ,s t+1 )7→ P t+1 π (·| h t ,a t ,s t+1 ) , with the unifilar update φ(h t+1 ) = ν φ(h t ),a t ,s t+1 . For ordinary probability, this is the usual one-step decomposition of the conditional trajectory law. For other uncertainty calculi, it says that future uncertainty can still be generated recursively from a state-dependent local uncertainty object. Nondeterministic uncertainty. LetM(X) =P(X) be the powerset of possible outcomes. An uncertainty kernel is then a set-valued map L : X →P(Y ). If ̄ S(z,a)⊆S is the set of possible next states from statistic z and action a, then Condition D says that the possible future suffixes from h t are generated by choosing s t+1 ∈ ̄ S(φ D (h t ),a t ) and then continuing from the updated statistic. The full history again gives a trivial representation, and the useful question is whether the set of possible futures depends only on a smaller state. Generalised Bellman recurrence and three dualities in sequential decision-making Imprecise and robust uncertainty. For robust or imprecise models,M(X) may be a set of prob- ability measures on X . A one-step model may specify an ambiguity set ̄ K(z,a)⊆ ∆(S) rather than a single transition kernel. A dynamics statistic is sufficient when this ambiguity set, and the way it is updated through time, depends on the past only through φ D (h t ). However, Bellman recursions for such models generally require more than this: the ambiguity sets must compose in a dynamically consistent way. In robust MDPs this is known as the rectangularity requirement (Wiese- mann et al., 2013). In the present framework, that additional requirement belongs to the analogue of Condition A: the aggregation F must respect the chosen sequential composition of uncertainty objects. C Active inference and the free-energy principle This appendix develops the connection between Eq. (23) and the active inference framework (Friston et al., 2015; Da Costa et al., 2020; Sajid et al., 2021). Active inference casts the symmetry between beliefs and preferences as constitutive of agency: preferences are represented as prior preferences encoded in a generative model, and the agent’s task is to minimise variational and expected free energy across perception and action, respectively. The framework has been developed extensively as a descriptive model of biological agency (Friston, 2010), with sustained engagement and critique (Biehl et al., 2021; Sajid et al., 2021; Millidge, 2024). Additionally, there exist interesting relationships between active inference and control-as-inference. Millidge et al. (2020) have shown that the two frameworks differ primarily in how reward enters the generative model — i.e., priors over outcomes in active inference, auxiliary optimality variables in control-as-inference, and that the underlying inferential structures are related by the process of using inference procedures to compute policies. In the following, we explain how the distinction between active inference vs control-as-inference introduced by Millidge et al. (2020) corresponds, in our framework, to a choice of how the object duality is implemented. 6 C.1 Object duality in active inference For a finite outcome or trajectory space and β > 0, define ̃ P β (h) := exp[βJ(h)] Z β , Z β := X h exp[βJ(h)].(47) Then, for any predictive trajectory law Q π , −E Q π log ̃ P β (H) =−β E Q π [J(H)] + logZ β .(48) Eq. (48) is the formal expression of the active-inference identification of preferences with prior probabilities: the agent treats high-return trajectories as a priori more likely under a generative model, whose observable marginal is the Gibbs measure q. This is the ‘biased generative model’ construction discussed by Friston et al. (2015) and analysed critically by Millidge et al. (2021). Thus, prior preferences encode the original return up to a positive scale and a policy-independent constant. This is the part of active inference that is captured by object duality. Moreover, maximising the expected return is equivalent to minimising the cross-entropy H(Q π ; ̃ P β ) =−E H∼Q π log ̃ P β (H) ,(49) since Z does not depend on π. 6 We view our framework as neutral between the active-inference and standard-RL philosophies. That said, the broader programmatic claims of active inference (that the free energy principle is a normative theory of biological agency, that all of perception and action reduces to variational inference) lie outside the scope of our framework, being claims about the empirical scope of the object duality, not about its formal content. Finding the Frame Workshop at RLC 20262026 C.2 Variational free energy for perceptual inference Let us now expand the setting to consider a partially observable structure, which distinguishes the latent state s ∈ S from observations o ∈ O, whereO is a set of possible observations. We assume that the agent does not necessarily have access to the realisation of the latent states directly through- out, but only sees the observations which are sampled from a likelihood function P (o| s). In active inference, the agent’s actions only affect the state directly through a Markovian transition kernel P (s | a), and only affects the agent’s observations indirectly through the effect of their actions on the hidden state. This is a consequence of the Markov blanket assumption, which defines active and sensory states (a and o respectively) as a statistical boundary between the agent and its environment. The agent is also assumed to possess internal states, which allow it to perform perceptual and active inference. Perceptual inference is modelled as variational inference, where the agent maintains an approximate posterior Q(s | o) that aims to approximate the true Bayesian posterior P (s | o). The variational free energy (VFE) is given by F (o;Q) := E Q(s|o) [logQ(S | o)− logP (o,S)].(50) Factorising P (o,s) into the product of a likelihood and marginal state distribution gives the standard complexity–accuracy decomposition F (o;Q) = D KL (Q(S | o)∥P (S)) | z complexity − E Q(S|o) logP (o| S) |z accuracy .(51) Alternatively, using P (o,s) = P (s| o)P (o) gives F (o;Q) =− logP (o) |z surprisal +D KL (Q(S | o)∥P (S | o)) | z Divergence .(52) Variational free energy therefore upper-bounds the surprise− logP (o), with equality when the ap- proximate posterior is exact. Finally, the VFE can be rearranged as the sum of an energy and entropy term, which can be related to the Helmholtz free energy: F (o;Q) =− E Q(S|o) logP (o,S) | z Energy −H[Q(S|o)] | z Entropy .(53) C.3 Expected free energy for policy selection Bringing preferences back into the picture, consider a preference model ̃ P β (O) over observations. The expected free energy (EFE) of a given open-loop policy is defined as (Parr et al., 2022) G(π) =− E Q π (O) log ̃ P β (O) | z pragmatic value − E Q π (O) D KL Q π (S | O)∥Q π (S) |z epistemic value .(54) The pragmatic value, as we have seen in Eq. (48), corresponds to the expected return, whereas the epistemic term favours policies expected to produce observations that are informative about the hidden state. C.4 Sophisticated inference and Bellman recursion. Standard finite-horizon active-inference schemes typically evaluate open-loop sequences of actions. Because future actions are then not conditioned on future outcomes or beliefs, this scheme need not satisfy Bellman’s principle beyond a one-step horizon as shown in (Da Costa et al., 2023). Sophisti- cated inference instead evaluates expected free energy recursively over counterfactual future beliefs, implementing a tree search over belief states using an open-loop policy (Friston et al., 2021). In the terminology of this paper, a recursively updated belief supplies a dynamics-sufficient statistic, Generalised Bellman recurrence and three dualities in sequential decision-making while recursive evaluation of stagewise expected free energy supplies the time-consistent continua- tion structure needed for Bellman recursion. For finite-horizon MDPs with known transitions and state preferences ̃ P β (s) ∝ expβR(s), Da Costa et al. (2023) show that, in the limit β →∞, the standard expected free energy is Bellman optimal at horizon one but not in general at longer horizons, whereas their recursive sophisticated scheme yields a Bellman-optimal state–action policy on any finite horizon. Their extension to the POMDP setting shows that under the same assumptions and in the limit as β →∞, EFE-minimising open-loop policies are also reward-maximising policies. D Derivations of results in assessment duality This appendix summarises existing results that prove that each of the three families of non-standard Bellman equations recovered in Sec. 3.2 — robust (Eq. (27)), risk-sensitive (Eq. (28)), and soft (Eq. (29)) — can be re-expressed as a standard Bellman recursion with regularised returns. The proofs share a common structure, which is set in App. D.1 and then applied in Apps. D.2–D.4. D.1 A unified perspective: regularised returns and the Gibbs identity The technical background of the proofs is the Donsker–Varadhan / Gibbs variational identity (Ortega & Braun, 2013; Geist et al., 2019; Brekelmans et al., 2022). For any reference probability measure μ∈ ∆(X ), any measurable f :X → R with E μ [exp(βf )] <∞, and any β > 0, then the following holds: 1 β log E X∼μ [exp(βf (X))] =sup ν∈∆(X ) E X∼ν f (X) − 1 β D KL (ν∥μ) ,(55) with the supremum attained at the ‘Gibbs tilt’ dν ⋆ dμ (x) = exp βf (x) E μ exp βf (X) .(56) For β < 0 the identity holds with the supremum replaced by an infimum and the same attaining tilt, and the KL coefficient−1/β = 1/|β| remains non-negative. In the limit β → −∞ the KL term in the variational form is dominated by the expectation and the right-hand side of Eq. (55) converges to a hard infimum of f over the support of μ. This is the bridge between the risk-sensitive and robust formulations. All three proofs that follow share the same template: 1. Identify the non-linear aggregation appearing in the target Bellman equation. 2. Apply Eq. (55) (or its β → −∞ limit) to rewrite that aggregation as a sup/inf of an ordinary expectation under an auxiliary measure Q, minus a divergence. 3. Absorb the divergence into the local return, producing a conjugate ℓ that depends on Q. 4. Verify that the resulting recursion is the standard Bellman recursion with F = E,⊕ = ⊙ = +, and conjugate return ℓ. The auxiliary measure Q plays a different role in each case — a policy in the soft case, a transition kernel in the risk-sensitive case, and a transition kernel constrained to an ambiguity set in the robust case — but the structural derivation is identical. In the language of the (β a ,β s ) terms as used in Eq. (26), the various formulations result from applying Eq. (55) at either the state site (risk-sensitive: β a = +∞, β s finite), the action site (soft: β a finite, β s = 0), or in the limit at both sites (robust: β a = +∞, β s =−∞). Details of the calculations and corresponding references are provided below. Finding the Frame Workshop at RLC 20262026 D.2 Proof of Eq. (29) — the soft Bellman equation The link between the soft Bellman equation and reward regularisation has been investigated by Husain et al. (2021) and Brekelmans et al. (2022) (see also (Ziebart, 2010; Haarnoja et al., 2017; Levine, 2018; Geist et al., 2019; Nachum et al., 2017; Schulman et al., 2017)). Proof. Let π 0 ∈ Π φ be a reference policy with π 0 (a| s) > 0 wherever any candidate policy assigns positive probability. The soft Bellman equation (Eq. (29)) can be re-stated as V ∗ soft (s) = 1 β a log X a∈A π 0 (a| s) exp β a Q ∗ soft (s,a) ,(57) where we are using the soft Q-function Q ∗ soft (s,a) := r(s,a) + γ E S ′ ∼K(·|s,a) V ∗ soft (S ′ ) .(58) Applying the Gibbs identity (Eq. (55)) at each s with μ = π 0 (·| s), f = Q ∗ (s,·), and β = β a > 0, gives the following identity: 1 β a log E A∼π 0 (·|s) h exp(β a Q ∗ soft (s,A)) i = sup μ∈∆(A) E A∼μ Q ∗ soft (s,A) − 1 β a D KL μ∥π 0 (·| s) . (59) Combining Eq. (57) with Eq. (59), and expanding D KL (μ∥π 0 ) = E A∼μ log(μ(A)/π 0 (A|s)) , leads to V ∗ (s) = sup μ∈∆(A) E A∼μ r(s,A) + γ E S ′ ∼K(·|s,A) V ∗ (S ′ ) − 1 β a log μ(A) π 0 (A|s) = sup μ∈∆(A) E A∼μ,S ′ ∼K(·|s,A) h ℓ soft (s,A,S ′ ;μ) + γ V ∗ (S ′ ) i ,(60) where the soft local return is ℓ soft (s,a;μ) = r(s,a) − 1 β a log μ(a| s) π 0 (a| s) .(61) Eq. (60) is a standard Bellman optimality equation with aggregation E, composition +, transition kernel K(·|s,a), and local return ℓ soft . From Eq. (56), the maximising policy can be seen to be the Gibbs tilt μ ∗ (a| s) = π 0 (a| s) exp(β a Q ∗ soft (s,a)) P a ′ ∈A π 0 (a ′ | s) exp(β a Q ∗ soft (s,a ′ )) .(62) An equivalent derivation, used elsewhere in the literature (Haarnoja et al., 2017; Levine, 2018), starts from the trajectory-level conjugate return (assuming γ = 1) J π conj (h) := J(h) − 1 β a log dP π dP 0 (h),(63) so that E h∼P π [J π conj (h)] = E h∼P π [J(h)]− β −1 a D KL (P π ∥P 0 ). Under Condition D with a shared transition kernel, the dynamics terms cancel in the density ratio, and for π ∈ Π φ one has log dP π dP 0 (h) = X τ≥0 log π a τ | φ(h τ ) π 0 a τ | φ(h τ ) .(64) Thus, J π conj satisfies Condition R with the local return as given in Eq. (61); substituting this into T ∗ (Eq. (15)) and applying Eq. (55) recovers Eq. (57). The forward and reverse routes therefore close the loop between the soft Bellman equation and its conjugate-return representation. Generalised Bellman recurrence and three dualities in sequential decision-making D.3 Proof of Eq. (28) — the risk-sensitive Bellman equation The link between the risk-sensitive Bellman equation and reward regularisation has been investigated by Petersen et al. (2000); Dupuis et al. (2000); Hansen & Sargent (2011); Osogami (2012); Bäuerle & Rieder (2014). Proof. The risk-sensitive Bellman equation (Eq. (28)) reads V ∗ (s) = max a∈A r(s,a) + γ β s log E S ′ ∼K(·|s,a) h exp β s V ∗ (S ′ ) i .(65) We treat the two regimes of β s separately. The two cases share the same local return; only the sup/inf alternation differs. Risk-seeking case (β s > 0). Applying Eq. (55) at each (s,a) with μ = p(·|s,a), f = V ∗ , and β = β s leads to 1 β s log E S ′ ∼K(·|s,a) h exp β s V ∗ (S ′ ) i =sup q∈∆(S) E S ′ ∼q h V ∗ (S ′ ) i − 1 β s D KL q K(·| s,a) . (66) Substituting this expression into the right-hand side of Eq. (65) leads to V ∗ (s) = max a∈A sup q∈∆(S) r(s,a) + γ E S ′ ∼q V ∗ (S ′ ) − γ β s D KL q∥p(·| s,a) = max a∈A sup q∈∆(S) E S ′ ∼q h ℓ risk (s,a,S ′ ;q) + γ V ∗ (S ′ ) i ,(67) where the risk-sensitive conjugate local return is ℓ risk (s,a,s ′ ;q) = r(s,a) + γ β s log K(s ′ | s,a) q(s ′ ) .(68) Using Eq. (56) one can find that the distribution that extremises Eq. (67), q ∗ , is the Gibbs tilt given by q ∗ (s ′ | s,a) = p(s ′ |s,a) exp(β s V ∗ (s ′ )) E s ′ ∼K(·|s,a) [exp(β s V ∗ (s ′ ))] ,(69) which concentrates on high-value next states. Risk-averse case (β s < 0). The same calculation goes through with the supremum replaced by an infimum. For β s < 0, the Gibbs identity reads 1 β s log E S ′ ∼K(·|s,a) exp(β s V ∗ (S ′ )) = inf q∈∆(S) E S ′ ∼q [V ∗ (S ′ )]− 1 β s D KL q∥K(·| s,a) , (70) where−1/β s = 1/|β s | > 0, so that the KL term still enters with a non-negative coefficient. Substi- tuting this into Eq. (65) yields V ∗ (s) = max a∈A inf q∈∆(S) E S ′ ∼q h ℓ risk (s,a,S ′ ;q) + γ V ∗ (S ′ ) i ,(71) with the same local return as Eq. (68). There, the sign of β s is preserved inside ℓ risk , so that the coefficient−γ/β s = γ/|β s | in front of log(q/p) is positive, and the KL penalty γ|β s | −1 D KL (q∥p) is non-negative. The distribution that extremises Eq. (71), q ∗ , is again a Gibbs tilt, but now it is concentrated on low-value next states — the agent maximises against an adversarial nature whose deviations from p are KL-penalised. Finding the Frame Workshop at RLC 20262026 D.4 Proof of Eq. (27) — the robust Bellman equation The link between the robust Bellman equation and reward regularisation has been investigated by Husain et al. (2021); Derman et al. (2021). Proof. The robust Bellman equation (Eq. (27)) reads V ∗ (s) = max a∈A r(s,a) + γ min q∈Q(s,a) E S ′ ∼q h V ∗ (S ′ ) i , Q(s,a)⊆ ∆(S),(72) where Q(s,a) is the (rectangular) ambiguity set at (s,a). Let us introduce the convex-analytic indicator I Q(s,a) (q) = ( 0if q ∈Q(s,a), +∞ if q /∈Q(s,a). (73) For any function V :S → R, min q∈Q(s,a) E S ′ ∼q h V (S ′ ) i =inf q∈∆(S) n E S ′ ∼q h V (S ′ ) i +I Q(s,a) (q) o ,(74) because any q /∈ Q(s,a) contributes +∞ to the right-hand side and is excluded from the infimum. Substituting Eq. (74) into Eq. (72) yields V ∗ (s) = max a∈A inf q∈∆(S) n r(s,a) + γ E S ′ ∼q h V ∗ (S ′ ) i + γI Q(s,a) (q) o = max a∈A inf q∈∆(S) E S ′ ∼q h ℓ robust (s,a,S ′ ;q) + γ V ∗ (S ′ ) i ,(75) where the robust conjugate local return is ℓ rob (s,a,s ′ ;q) = r(s,a) + I Q(s,a) (q).(76) Above, the γ multiplying the indicator has been absorbed into I Q (since γ · (+∞) = +∞ and γ· 0 = 0). The distribution extremising Eq. (75) at (s,a), q ∗ , is any element ofQ(s,a) that attains min q∈Q E q [V ∗ ]. Note that the robust Bellman equation is the singular limit β s →−∞ of the risk-sensitive Bellman equation under a KL-ball ambiguity set. Also, note that some of these different regularisations can be combined; for instance, one can add KL-like terms over policies and transitions to obtain soft risk-sensitive solutions (Derman et al., 2021). E Distortion aggregators as deformed expectations This appendix proves that aggregators that can be written as Eq. (35) can be expressed as an ordinary expectation of the same return J under deformed dynamics Q, and gives a general formula for Q in terms of the distortion function g. For this purpose, let us focus on the case of real returns J(H) ∈ R where H ∼ P π . The return cumulative distribution function, survival function, and lower quantile function are given by F J (x) = P π (J≤ x), S J (x) = P π (J > x) = 1−F J (x), F −1 J (u) = infx : F J (x)≥ u. (77) A distortion is an increasing function g : [0, 1]→ [0, 1] with g(0) = 0 and g(1) = 1. The associated aggregator is the Choquet integral of J against the capacity A7→ g(P π (A)), ̃ F g [J] := Z J d(g◦ P π ) = Z ∞ 0 g S J (x) dx− Z 0 −∞ 1− g S J (x) dx.(78) Generalised Bellman recurrence and three dualities in sequential decision-making We will use the bounded-support form of Eq. (78): if a ≤ J ≤ b almost surely, then the translation equivariance of the Choquet integral 7 together with the tail-integral formula 8 for the non-negative variable J− a gives ̃ F g [J] = a + Z b a g S J (x) dx.(79) E.1 The smooth case: an explicit density Proposition 1 (Deformed-measure representation). Assume g ∈ C 1 ([0, 1]) (so g ′ ≥ 0) and that J is bounded with continuous distribution function F J . Define a measure Q onT by dQ dP π = g ′ S J (J) , S J (J) : ω 7→ S J J(ω) = 1− F J J(ω) .(80) Then Q is a probability measure onT , Q≪ P π , and ̃ F g [J] = E Q [J] = E P π Jg ′ (S J (J)) .(81) Proof. Let us first prove that Q is a probability measure. Because F J is continuous, the probability integral transform gives U := F J (J)∼ Unif[0, 1], hence S J (J) = 1−U ∼ Unif[0, 1]. The density g ′ (S J (J)) is non-negative (g increasing) and E P π g ′ (S J (J)) = Z 1 0 g ′ (v) dv = g(1)− g(0) = 1.(82) Thus, Q defined by Eq. (80) is a probability measure absolutely continuous with respect to P π . As a second step, let us show that ̃ F g [J] = Z 1 0 F −1 J (p)g ′ (1− p) dp.(83) Let a≤ J≤ b. Starting from Eq. (79) and substituting p = F J (x) results in Z b a g S J (x) dx = Z b a g 1− F J (x) dx = Z 1 0 g(1− p) dF −1 J (p).(84) Integrating by parts, with d[g(1− p)] =−g ′ (1− p) dp, Z 1 0 g(1− p) dF −1 J (p) = h g(1− p)F −1 J (p) i 1 0 + Z 1 0 F −1 J (p)g ′ (1− p) dp.(85) The boundary term is g(0)F −1 J (1)− g(1)F −1 J (0) = 0· b− 1· a =−a. Substituting into (79), ̃ F g [J] = a + − a + Z 1 0 F −1 J (p)g ′ (1− p) dp = Z 1 0 F −1 J (p)g ′ (1− p) dp,(86) proving Eq. (83). To conclude the proof, couple J to its own quantile function through U = F J (J) ∼ Unif[0, 1], so that J = F −1 J (U ) almost surely and S J (J) = 1− U . Then E P π Jg ′ (S J (J)) = E F −1 J (U )g ′ (1− U ) = Z 1 0 F −1 J (u)g ′ (1− u) du.(87) By Eq. (83), this equals ̃ F g [J], proving Eq. (81). 7 That is, the fact that F g [c + Z] = c + F g [Z], which holds because g◦ P π is normalised. 8 The tail-integral formula says E[Y ] = R ∞ 0 P(Y > x) dx = R ∞ 0 S Y (x) dx with S Y (x) = P(Y > x). Finding the Frame Workshop at RLC 20262026 Eq. (80) reweights each outcome by g ′ evaluated at that outcome’s own upper-tail rank S J (J) ∈ [0, 1]. Outcomes in the lower tail of J and outcomes in the upper tail are up- or down-weighted according to the local slope of g there. The representation provided in Eq. (80) exists for every distortion function g. When g is not differ- entiable (e.g. the kinked g of CVAR) or J has atoms, then Eq. (80) must be replaced by a Lebesgue– Stieltjes construction. For technical details, please see Föllmer & Schied (2025); Dhaene et al. (2006). E.2 Example: conditional value at risk Take the worst-tail CVaR at level α ∈ (0, 1), i.e. the average of the lowest α-fraction of returns. Its distortion and (a.e.) derivative are g(s) = 1 α s− (1− α) + , g ′ (s) = 1 α 1s > 1− α,(88) so that g(1− α) = 0, g(1) = 1. The density in Eq. (80) becomes dQ dP π = 1 α 1 S J (J) > 1− α = 1 α 1 F J (J) < α = 1 α 1 J≤ q α (J) ,(89) the reweighting onto the worst α-tail used in the main text. The kink of g at s = 1− α is harmless when F J is continuous, since S J (J )∼ Unif[0, 1] places no mass at the single point 1− α. F Related work The framework developed in this paper synthesises and extends a substantial body of prior work across reinforcement learning, optimal control, statistics, theoretical neuroscience, and category theory. Sec. 3 discusses various formal antecedents, including active inference, control-as-inference, the soft Bellman equation, risk-sensitive, robust, and entropy-regularised families. In this section, we situate the framework within other research programmes from which these results are drawn, addressing those aspects of the literature that are conceptual or programmatic rather than formal, and indicating in each case where our contribution relates to existing work. F.1 Connections to related reinforcement learning frameworks The ideas developed in this paper are complementary to several existing research directions that modify, extend, or generalise Bellman recursion. For instance, classic work on general value functions (Sutton et al., 2011) extends value functions beyond cumulative reward to arbitrary predictive questions specified by cumulants and termination conditions. From the perspective of the present work, this corresponds to changing the object prop- agated through Bellman recursion while preserving its recursive structure. Reward machines (Icarte et al., 2018) propose to augment the environment by including automaton states that encode task progress, thereby transforming non-Markovian reward specifications into Markovian ones. Reward machines can be seen as an example of restoring Condition R through state augmentation. Bich et al. (2015) study which aggregation operators admit recursive dynamic-programming formu- lations, while Tang et al. (2025) develop a general framework for recursively aggregating rewards beyond the additive discounted sum. These works investigate which reward aggregators admit recur- sive dynamic programming formulations; by contrast, aggregation duality assumes such a recursive formulation exists and studies equivalence transformations between different Bellman recursions. Oh et al. (2020) propose to automatically discover Bellman-style update rules through automatic search over Bellman update rules, providing an interesting complement to the view taken in this work. Their work treats the Bellman backup itself as an object to be optimised over a space of candidate operators. Finally, Rahme & Adams (2019) reformulates reinforcement learning in terms Generalised Bellman recurrence and three dualities in sequential decision-making of partition functions and free energies rather than conventional value functions. By doing this, this work is closely related to object duality. F.2 Bayes-adaptive RL, empowerment, and information-theoretic objectives A related literature treats epistemic (belief-related) and pragmatic (reward-related) value symmet- rically by absorbing exploration into the reward structure. Bayes-adaptive MDPs (BAMDPs), in- troduced by Duff (2002) and developed by Ghavamzadeh et al. (2015); Zintgraf et al. (2020), lift the planning problem to belief-space so that the agent’s posterior over MDP parameters becomes part of the state. Information-directed sampling (Russo & Van Roy, 2014; 2018) trades off regret and information gain in a unified objective. Empowerment, introduced by Klyubin et al. (2005) and developed by Salge et al. (2014), defines an information-theoretic intrinsic reward as the channel capacity from actions to future states. Related lines of work include curiosity-driven exploration (Pathak et al., 2017) and intrinsic motivation (Oudeyer & Kaplan, 2007). These approaches operationalise specific instances of the belief–reward symmetry without develop- ing it as a structural principle. Empowerment is a particularly clean example: the reward is defined as a functional of the belief over future states, which is a direct instance of the object-swap duality applied to a particular class of objectives. BAMDPs are instances of state augmentation restoring Condition D on a posterior-augmented state. Our framework provides the structural language in which these constructions can be understood as choices of (J,F) pairs satisfying Condition R and Condition A with appropriate state augmentation where necessary. We do not derive any new ex- ploration algorithm, and view this literature as a source of concrete instantiations of our framework rather than a competitor to it. F.3 Categorical approaches A line of work treats Markov decision processes from the perspective of universal algebra, coalgebra, and category theory. Feys et al. (2018) formalise an MDP as a coalgebra m = ⟨u,t⟩ : S → R× (∆S) A , combining reward and transition functions, and develop a coinductive theory of long- term value. Their two main contributions are a new proof principle they call contraction coinduction, based on Banach’s fixpoint theorem, and a characterisation of the discounted value function as the unique morphism into a final coalgebra via a generalised notion of corecursive algebra. A more recent and rapidly developing research programme is the categorical cybernetics approach to RL (Hedges & Rodríguez Sakamoto, 2023; 2025). The starting observation is that value iteration can be represented as precomposition with a specific optic — a categorical structure originating in lens theory (Clarke et al., 2024; Riley, 2018). These works extend this to parametrised optics that apply to action-value functions and depend on samples, treating major RL algorithms (value iteration, policy iteration, Q-learning, deep Q-learning, actor-critic) as instances of a categorical construction in which the Bellman update is the backward pass of an optic representing the agent– environment interaction. The categorical literature provides what is, in principle, the natural mathematical home for the framework developed in this paper. The conditions D, R, and A that we formulate are the kinds of structures that Markov categories (Fritz, 2020) can naturally accommodate. The recursive de- composition of value is related to the corecursive-algebra construction of Feys et al. (2018). Our framework should be understood as a non-categorical presentation of structures that admit a cate- gorical formulation. Our contribution relative to this literature is twofold. First, the existing cate- gorical literature treats one or another of the three dualities at a time: for example, Feys et al. (2018) develop the coalgebraic structure of the dynamics–reward pair without addressing the aggregation choice. Second, our framework is formulated in elementary terms, which makes it accessible to the broader optimal-control and RL communities while remaining compatible with the categorical formulations. We view the explicit categorical development of the triadic synthesis as a promising direction for future work — in particular, an articulation of the universal property satisfied by the Bellman state φ when all three conditions hold simultaneously.