Paper deep dive
Active Inference as a Convex Markov Decision Process
Nikola Milosevic, Nicolás Hinrichs, Nico Scherf
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/23/2026, 3:11:52 AM
Summary
The paper formulates Active Inference (AIF) as a convex Markov Decision Process (MDP) by minimizing Expected Free Energy (EFE). It demonstrates that EFE minimization for closed-loop policies is equivalent to optimizing a convex functional of state-action marginals, combining linear pragmatic terms (reward maximization) with a nonlinear epistemic term (entropy maximization). The authors derive a mirror descent algorithm to solve this convex MDP and discuss the implications for performative reinforcement learning where world-model learning and policy optimization are coupled.
Entities (8)
Relation Signals (6)
Expected Free Energy → isminimizedby → Active Inference
confidence 97% · Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE)
Active Inference → isformulatedas → Convex Markov Decision Process
confidence 96% · We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP).
Mirror Descent → isusedfor → EFE Minimization
confidence 94% · derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals
World-Model Learning → iscoupledwith → Policy Optimization
confidence 92% · coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning
Epistemic Value → introduces → Nonlinear Component
confidence 91% · the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning.
Pragmatic Terms → areequivalentto → Reward Maximization
confidence 90% · the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.
Tags
Links
- Source: https://arxiv.org/abs/2607.20152v1
- Canonical: https://arxiv.org/abs/2607.20152v1
Trouble viewing inline? Open PDF directly →
Full Text
78,693 characters extracted from source content.
Expand or collapse full text
Active Inference as a Convex Markov Decision Process Nikola Milosevic Max Planck Institute for Human Cognitive and Brain Sciences Leipzig, Germany nmilosevic@cbs.mpg.de &Nicolás Hinrichs Max Planck Institute for Human Cognitive and Brain Sciences Leipzig, Germany nhinrichs@cbs.mpg.de &Nico Scherf Max Planck Institute for Human Cognitive and Brain Sciences Leipzig, Germany nscherf@cbs.mpg.de Corresponding author: nmilosevic@cbs.mpg.de. ORCID 0000-0003-1904-8867. Abstract Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees. Keywords active inference ⋅· expected free energy ⋅· convex MDP ⋅· mirror descent ⋅· reinforcement learning 1 Introduction Figure 1: The episodic active inference setting considered in this work: A) the environment admits a true partially observable decision process, while the agent learns a latent fully observable process using variational inference. B) In contrast to standard variational inference, the two decision processes are coupled: the variational distribution ν is used both in variational model learning and in action selection through the behavior policy β=π∘νβ=π ν. The agent acts in the environment using the behavior policy β=π∘νβ=π ν for multiple episodes, then uses the data to fit its model by Variational Free Energy (VFE) minimization to obtain m=(p,ν)m=(p,ν), and uses that model to optimize its policy in imagination by minimizing the Expected Free Energy (EFE). Model-based policy optimization (MBPO) lets an agent learn a world model from environment interaction and then optimize a reactive policy on imagined rollouts inside that model. In reinforcement learning (RL), the properties of MBPO are well understood: algorithms converge under suitable assumptions, and policy-improvement guarantees can be stated for the expected reward [12]. Active Inference (AIF) [5, 6] can also be read as MBPO. The agent learns a generative model of the environment, then selects actions by minimising an expected free energy (EFE) that scores imagined future outcomes against a preference distribution. Implementations use similar architectural components as recent world model-based RL methods [8, 10, 18]: a recurrent world model, an actor, and a critic, trained on imagined environment interactions. The optimization properties of EFE minimization, however, are far less understood than those of reward-maximizing MBPO. It is not known whether the EFE minimum exists, whether it is unique, or what kind of optimization problem the combined VFE–EFE minimization constitutes. This blocks the transfer of efficient algorithms, and their convergence and policy-improvement guarantees, from the RL literature to the simulation of AIF agents. AIF is formulated with inference, not policy optimization, in mind. To bring it within reach of the RL toolbox we make two modifications to the standard protocol (cf. [5]). First, we take an episodic view: rather than updating on every environment step, VFE and EFE minimization each run over one or more full episodes while the other’s target is held fixed, see Fig.˜1. This is only a mild departure since standard AIF already freezes the model and variational posterior during planning when it imagines future timesteps. We simply extend that separation to the data-collection phase. Second, in place of the mean-field posterior over the future we use a causal one, so that imagination is Markov and can be realised by ancestral sampling, meaning the agent can sample its own imagined rollouts efficiently and in a physically meaningful manner. With these two modifications in place, our central claim is that EFE minimization under a fixed model is a convex MDP [29, 30], the recent generalisation of RL in which the objective can be an arbitrary convex functional of the state–action marginal at future timesteps rather than a linear one (as with expected reward). The per-step EFE is exactly of this form. Its extrinsic-value and ambiguity terms are linear in the occupancy, while the epistemic value, the negative state-marginal entropy, supplies a single convex term such that the whole of convex-MDP theory applies. At the exact-posterior limit this epistemic term coincides with the mutual information between states and observations, recovering the familiar information-gain reading of active inference. We return to this in Section˜3. In particular, convex MDPs admit extended dynamic-programming methods that remain scalable and sample-efficient. Finally, we show that allowing the agent to refit the model from data of the real environment significantly complicates the optimization problem. Active inference sits at the underexplored intersection of convex MDPs and performative reinforcement learning [22, 16, 24], where the reward and the environment dynamics respond to the agent’s deployed policy. We take the observation-preference reading of the EFE as primary: it carries the mutual-information interpretation of epistemic value, and because the same recognition density scores both imagined inference and online inference, it makes the planning objective self-consistent with the agent’s perception. Contributions. (i) Structure. We present the episodic policy optimization setting for AIF, where the EFE, without modification, becomes a convex functional of the imagined step-t state marginal, splitting into linear terms (a latent-MDP reward) and a single convex nonlinearity (the epistemic value). This gives EFE minimization the structure of a convex MDP. (i) Algorithm and rate. We solve EFE minimization with a mirror-descent-inspired algorithm that yields a policy-dependent reward (a fixed reward, and a variable curiosity bonus). The resulting subproblem is a combination of soft RL [9] and maximum entropy exploration [11], solved in closed form by softmax dynamic programming. Our MD scheme (MD-AIF) converges at rate O(1/K)O(1/K) by relative smoothness. (i) Closing the loop. Refitting the world model to policy-induced data makes the scheme performative [22]. We show that a performatively stable policy–model pair exists under mild assumptions, and sketch a path towards a performative AIF algorithm. 2 Episodic Active Inference We consider a finite-horizon reward-free POMDP ℳ=(,,,P,E,σ)M=(S,A,O,P,E,σ), where st∈,at∈s_t ,a_t , and ot∈o_t are states, actions, and observations at time t∈[T]=0,…,Tt∈[T]=\0,…,T\, with ,,S,A,O finite. P and E are transition and emission kernels with full support and σ is a start state distribution. For the sake of clarity, we consider the memoryless finite-horizon setting in the main text. However, our results extend to the history-dependent and infinite horizon settings, see §3.2. A memoryless behavior policy β:→Δ()β:O→ (A) induces the environment trajectory law pβ(τ)=σ(s0)E(o0|s0)∏t=0T−1β(at|ot)P(st+1|st,at)E(ot+1|st+1).p^β(τ)=σ(s_0)E(o_0|s_0) _t=0^T-1\,β(a_t|o_t)P(s_t+1|s_t,a_t)E(o_t+1|s_t+1). (1) An active inference agent consists of a variational world model m=(pθ,νϕ)m=(p_θ, _φ) with model parameters θ and variational parameters ϕφ. We work primarily in the tabular parametrization setting and drop parameter indexes following [2]. Further, the agent deploys its behavior policy in the real environment by chaining latent-state inference and action selection. Throughout, we consider composed behavior policies β=π∘νβ=π ν, where memoryless policies are of the form β(at|ot)=∑stπ(at|st)ν(st|ot),β(a_t|o_t)= _s_tπ(a_t|s_t)\,ν(s_t|o_t), i.e. the agent first infers the latent state through the recognition density ν and then acts through the latent-state-conditioned policy π used in EFE optimization. We consider an episodic model-based policy-optimization setting in which the agent alternates between 1. Model learning: explore the environment using β, collect trajectory data D over several episodes, and fit the world model m=(p,ν)m=(p,ν) by minimising variational free energy (VFE) on the data, 2. Policy optimization: improve the behavior policy by minimising expected free energy (EFE) on imagined trajectory data with respect to the latent policy π, using the latest world model. One full pass through the active inference loop is π↦deploy(π)↦VFEm((π))↦EFEπ(m((π)))=:AIF(π),π \ deploy\ D(π) \ VFE\ m(D(π)) \ EFE\ π (m(D(π)) )=:AIF(π), (2) and a fixed point satisfies π∗=AIF(π∗)π^*=AIF(π^*). The VFE and EFE steps can either be run to completion or interleaved episodically. Crucially, the latent policy space Π=∏t(Δ) = _t( _A)^S does not depend on the model and the natural question is under which conditions repeated application of AIFAIF admits such a fixed point and whether efficient algorithms exist that converge to it. 2.1 VFE: World Model Training The model m=(p,ν)m=(p,ν) that the EFE phase plans against is the output of a preceding perception/learning phase: minimization of VFE on data gathered in the real environment. We record the VFE in the form that matches the EFE below, with the same recognition density, so that the full loop (2) is well posed. Generative model and recognition. With actions treated as given, the agent’s generative model over a trajectory τ=(s0,o0,a0,…,aT−1,sT,oT)τ=(s_0,o_0,a_0,…,a_T-1,s_T,o_T) factorises as p(o0:T,s0:T|a0:T−1)=p(s0)p(o0|s0)∏t=0T−1p(ot+1|st+1)p(st+1|st,at),p(o_0:T,s_0:T|a_0:T-1)=p(s_0)p(o_0|s_0) _t=0^T-1p(o_t+1|s_t+1)p(s_t+1|s_t,a_t), (3) and states are inferred with the history-conditioned recognition density ν(s0:T|o0:T,a0:T−1)=∏t=0Tν(st|ot)ν(s_0:T|o_0:T,a_0:T-1)= _t=0^Tν(s_t|o_t). Again, under direct parametrization, we drop the parameter indexes and consider the memoryless case. The history-dependent case is discussed in Section˜3.2. VFE. Let (π)D(π) be the distribution of histories hT=(o0:T,a0:T−1)h_T=(o_0:T,a_0:T-1) obtained by running β=π∘νβ=π ν in the real environment. The variational free energy of m on data D is ℱ(m;)=h∼[−logp(o0:T|a0:T−1)]⏟surprise+h∼DKL(ν(s0:T|h)∥p(s0:T|h))⏟recognition gap≥ 0,F(m;D)= E_h [- p(o_0:T|a_0:T-1) ]_surprise+ E_h \,D_KL (ν(s_0:T|h)\,\|\,p(s_0:T|h) )_recognition gap\,≥\,0, (4) where p(s0:T|o,a)p(s_0:T|o,a) is the exact Bayesian posterior under (3). The bound is tight precisely when ν equals that posterior. This non-negative recognition gap is the same quantity that controls EFE accuracy. VFE-optimality of ν is what makes the EFE phase score against an accurate posterior. The model-learning phase targets m∗(π)∈argminmℱ(m;(π)),m^*(π)∈ _m\ F(m;D(π)), (5) where the optimum m∗m^* generally depends on the policy π used to generate the trajectory data via β=π∘νβ=π ν. 2.2 EFE: Planning in a Latent MDP Given a model m=(p,ν)m=(p,ν), a latent policy π induces the purely imagined trajectory law qπ(τ;m)=q0(s0)p(o0|s0)∏t=0T−1πt(at|st)p(st+1|st,at)p(ot+1|st+1),q^π(τ;m)=q_0(s_0)\,p(o_0|s_0) _t=0^T-1 _t(a_t|s_t)\,p(s_t+1|s_t,a_t)\,p(o_t+1|s_t+1), (6) with predictive state marginals ρtπ(s)=ℙqπ(st=s) _t^π(s)=P_q^π(s_t=s) of qπq^π at time t satisfying the recursion ρt+1π(s′)=∑s,ap(s′|s,a)πt(a|s)μtπ(s,a),μtπ(s,a)≔ρtπ(s)πt(a|s) _t+1^π(s )= _s,ap(s |s,a)\, _t(a|s)\, _t^π(s,a), _t^π(s,a) _t^π(s) _t(a|s) (7) where ρ0π=q0 _0^π=q_0 is given. For example, it can be an additional factor q0(s0)=ν(s0)q_0(s_0)=ν(s_0) in the variational posterior or derived from the recognition density q0(s0)=ν(s0|o0)q_0(s_0)=ν(s_0|o_0) where o0∼o_0 is sampled from a dataset (cf. Dreamer [10]). Sampling is ancestral: the agent acts causally in imagination. Then we define the AIF planning problem as follows. Given an active inference agent with model m, a (biased) observation preference p~ p, and a latent trajectory distribution qπ(⋅;m)q^π(·;m), define the finite-horizon EFE by (π;m) (π;m) =∑t=0TGt(π;m)=∑t=0Tqπ[gt(st,ot)], = _t=0^TG_t(π;m)= _t=0^TE_q^π\! [\,g_t(s_t,o_t)\, ], (8) gt(st,ot) g_t(s_t,o_t) =logρtπ(st)⏟predictive marginal−logν(st|ot)−logp(ot|st)−logp~(ot)⏟=:−logp~(st,ot)(biased model), = _t^π(s_t)_predictive marginal - ν(s_t|o_t)- p(o_t|s_t)- p(o_t)_=:- p(s_t,o_t)\ (biased model), (9) where ν is the agent’s recognition density. The planning (policy optimization) problem is π∗(m)∈argminπ(π;m).π^*(m)∈ _πG(π;m). (10) This particular definition of the EFE is one of many variants used throughout the literature [5, 21, 19]. It is the one that appears in policy optimization [18, 17], designed for consistency with the agent’s behavior policy (β=π∘νβ=π ν) and tractability in simulation. The epistemic value as mutual information. The intrinsic value is the negative state-marginal entropy −ℋ(ρt)-H( _t), which under Eq.˜8 appears in place of the mutual information because the recognition density ν is held fixed. If ν is replaced by the exact model posterior p(s|o)p(s|o) (the idealised case, attained at VFE convergence (4)), this term becomes the negative mutual information −It(S;O)-I_t(S;O) between states and observations at time t. This is the classical epistemic/information-gain value of active inference [21, 19]. The two readings coincide exactly at EFE convergence, so the convex-MDP structure preserves the information-theoretic interpretation of the EFE. Only the nonlinearity is named differently, depending on whether the recognition density is variational or exact. Convex-MDP structure. We show in §3 that minπ(π;m) _πG(π;m) is a convex MDP for fixed m, solvable by repeated dynamic programming with policy-dependent rewards. These rewards are precisely the EFE gradients with respect to the state-marginal of trajectory law (6). Note that this structure is independent of the exact EFE variant, see Appendix A, of history dependence, see §3.2, and of parameter uncertainty, since the model and variational parameters are fixed during planning. 3 EFE Minimization is a Convex MDP Our main structural result is that EFE minimization is a convex MDP under a fixed given model m=(p,ν)m=(p,ν). This can be read off immediately from the objective, once the EFE is brought into its state-action marginal form. Lemma 1. We write μπ=(μtπ)t=0Tμ^π=( _t^π)_t=0^T. The EFE Eq.˜8 can be written as (π)=Γ(μπ)≔⟨ℓ,μπ⟩+Φ(μπ),G(π)= (μ^π) \; ,μ^π + (μ^π), (11) where ℓ≔ℓt(s,a)=−o∼p(⋅|s)logp~(s,o) _t(s,a)=-E_o p(·|s) p(s,o), Φ(μπ)=−∑tℋ(ρtπ) (μ^π)=- _tH(ρ^π_t) is the neg. marginal Shannon entropy, and ⟨⋅,⋅⟩ ·,· is the Euclidean inner product on ℝ×[T]R^S×A×[T]. The inner product is linear in μπμ^π whenever the preference distribution p~ p is independent of μπμ^π and Φ is (not strictly) convex, since −ℋ-H is strictly convex, but −ℋ(∑aμπ(⋅,a))-H( _aμ^π(·,a)) is only convex in μπμ^π. 3.1 Planning as a Convex MDP While G can be written as a functional of some μt∈Δ(×) _t∈ (S×A), the marginals can not move freely on the entire simplex, since they are fully determined by π through Eq.˜6. Like the state-marginals, the state-action marginals μtπ _t^π that arise from some policy π must satisfy flow constraints for all t∈[T−1]t∈[T-1] and s′∈\ s : ∑aμ0π(s,a) _a _0^π(s,a) =q0(s),∑a′μt+1π(s′,a′)=∑s,ap(s′|s,a)μtπ(s,a). =q_0(s), _a _t+1^π(s ,a )= _s,ap(s |s,a)\, _t^π(s,a). (12) These constraints are a result of our choice of planning distribution, which turns imagination into a Markov process. Definition 1 (Occupancy polytope [23, 20]). Let =(μt)t=0T∈(Δ×)T+1|(12) holds,K=\( _t)_t=0^T∈( _S×A)^T+1| eq:flow holds\, (13) which is a compact convex polytope. Every μ∈μ corresponds to a unique policy πt(a|s)=μt(s,a)/∑a′μt(s,a′) _t(a|s)= _t(s,a)/ _a _t(s,a ) wherever ρt(s)>0 _t(s)>0. Due to the one-to-one correspondence of π and μπμ^π (specifically, π↦μπ μ^π is a bijection onto the interior of K), the problems minπ(π) _πG(π) and minμ∈Γ(μ) _μ (μ) are equivalent. Proposition 1 (EFE minimization is a convex MDP). Finite-horizon EFE minimization under a fixed model is a finite-horizon convex MDP, meaning minπ(π)≡minμ∈Γ(μ), _πG(π)≡ _μ (μ), (14) where μπ=(μtπ)tμ^π=(μ^π_t)_t is the collection of all step-t state-action marginals under qπq^π and Eq.˜14 is a convex program over policy-induced state-action marginals, i.e. a convex MDP [29, 20]. Proof. By Lemma˜3, Γ is convex. Further, K is a convex polytope [23, 20], thus Eq.˜14 is a convex program in state-action occupancies, exactly the convex MDP (concave-utility) problem of [20]. ∎ Optimization problems of this kind are referred to as convex [29], general-utility [30, 27], or concave-utility MDPs [20] in the RL literature. Specifically, EFE minimization has the structure of maximum-entropy exploration [11] combined with standard reward maximization, here in the episodic/finite-horizon setup studied by [20]. Convex MDPs are more general than standard MDPs. While standard MDPs also have a convex program representation, the objective there is linear, and the nonlinear case does not admit global value functions. However, specialized dynamic programming methods exist, which we will exploit in the algorithmic development of §4. 3.2 History-Dependent and Infinite-Horizon Extensions So far we have discussed the finite-horizon memoryless setting. However, this was mostly for clarity, and our results readily apply to history-dependent and infinite-horizon formulations. To restore the convex MDP structure on the history process, one must augment the state space with the space of histories. Note that this state is analogous to the imagined state of the Dreamer [10, 18] architecture. Table 1: Infinite-horizon extensions of the EFE objective. The two axes correspond to whether entropy is computed after temporal aggregation of occupancies or before it, and whether the temporal aggregation is discounted or stationary. Discounted Stationary time → state dγπ=(1−γ)∑t=0∞γtμtπγagg(π)=⟨ℓ,dγπ⟩−ℋ(dπ) aligned d_γ^π&=(1-γ) _t=0^∞γ^t _t^π\\[2.0pt] G_γ^agg(π)&= ,d_γ^π -H(d^π) aligned d∞π=limT→∞1T∑t=0T−1μtπ∞agg(π)=⟨ℓ,d∞π⟩−ℋ(d∞π) aligned d_∞^π&= _T→∞ 1T _t=0^T-1 _t^π\\[2.0pt] G_∞^agg(π)&= ,d_∞^π -H(d_∞^π) aligned state → time γstep(π)=(1−γ)∑t=0∞γtGt(π) _γ^step(π)=(1-γ) _t=0^∞γ^tG_t(π) ∞step=limT→∞1T∑t=0T−1Gt _∞^step= _T→∞ 1T _t=0^T-1G_t Lemma 2 (History augmentation). The history-dependent version of Eq.˜8 given by Eq.˜45 induces a convex MDP on the augmented state x=(h,s)x=(h,s). Generalizing to infinite horizons is also possible, however, it comes with modeling choices. Due to the nonlinearity of the objective, one must choose how to combine state and time averages, see Table˜1. The key distinction is that the entropy operator does not commute with time aggregation, but we must aggregate time in the infinite-horizon formulation to avoid infinite-dimensional decision variables. By concavity of ℋH and Jensen’s inequality, ℋ(dπ)≥(1−γ)limT→∞∑t=0Tγtℋ(μtπ),dπ(s,a)=(1−γ)limT→∞∑t=0Tγtμtπ(s,a),H(d^π)≥(1-γ) _T→∞ _t=0^Tγ^tH( _t^π), d^π(s,a)=(1-γ) _T→∞ _t=0^Tγ^t _t^π(s,a), (15) with equality only when all μtπ _t^π are identical. Details on existence and algorithmic consequences are given in Appendix D.2, see also [27]. 4 Solving EFE Minimization by Soft RL The convex nonlinearity of the EFE makes the per-step reward depend on the policy’s state-marginal, so a single value function cannot capture the objective globally. Convex MDP methods resolve this by linearising the objective at each iterate. Policy optimization algorithms based on mirror descent [4, 14, 20] are particularly well-suited for EFE minimization: every step replaces the EFE by its first-order surrogate around the current state-marginal, turning the convex MDP into a sequence of ordinary soft-MDP problems whose reward is recomputed between iterations. The reason why this works is that MD lets us choose a Bregman divergence DΨD_ through the convex generator Ψ in its update μk+1∈argminμ∈⟨∇Γ(μk),μ⟩+η−1DΨ(μ∥μk),μ^k+1∈ _μ \ ∇ (μ^k),μ +η^-1D_ (μ\,\|\,μ^k) \, (16) where η is a stepsize. We choose DΨ(μ∥μk)=∑t,sρt(s)DKL(πtμ(⋅|s)∥πtμk(⋅|s)),D_ (μ\|μ^k)= _t,s _t(s)D_KL(π^μ_t(·|s)\|π^μ^k_t(·|s)), (17) where Ψ is the negative conditional entropy, see Appendix C and [20] for details. With this choice, Eq.˜16 has a closed-form solution that is exactly the soft-Bellman backup used in Algorithm˜1. Algorithm˜1 is the active-inference instance of MD-CURL [20, Alg. 2], which is a convex MDP solver using mirror descent [4]. Here it is specialized by the EFE objective, whose nonlinearity is the state-marginal negative entropy Φ . The linearized per-iteration reward rk=−∇Γ(μk)r^k=-∇ (μ^k) carries the state-marginal term −logρtk(s)−1- _t^k(s)-1 and the static preference reward −ℓt- _t. Algorithm 1 MD-AIF 1:world model m=(p,ν)m=(p,ν); preference p~t p_t; horizon T; step size η; init. belief q0q_0; current policy π¯ π. 2:MD-AIF(πk;m,p~t,T,η,q0)MD-AIF(π^k;m, p_t,T,η,q_0) computes the update Eq.˜16. 3:μ0←q0 _0← q_0; 4:for t=0,…,T−1t=0,…,T-1 do 5: μt+1(s′)←∑s,ap(s′|s,a)π¯(a|s)μt(s) _t+1(s )← _s,ap(s |s,a)\, π(a|s)\, _t(s) 6:⊳ linearized reward: negative EFE gradient r=−∇Γ(μ)r=-∇ (μ) 7:for all t,s,at,s,a do 8: rt(s,a)←ologp~(s,o)−logρt(s)−1r_t(s,a) _o p(s,o)\;-\, _t(s)-1 9:⊳ Backward pass: soft (free-energy) value iteration 10:VT+1(⋅)←0V_T+1(·)← 0 11:for t=T,…,0t=T,…,0 do 12: Qt(s,a)←rt(s,a)+∑s′p(s′|s,a)Vt+1(s′)Q_t(s,a)← r_t(s,a)+ _s p(s |s,a)\,V_t+1(s ) 13: Vt(s)←η−1log∑aπ¯(a|s)exp(ηQt(s,a))V_t(s)←η^-1 _a π(a|s) \! (η\,Q_t(s,a) ) 14:⊳ Mirror step: multiplicative (softmax) policy update 15:for all t,st,s do 16: πt(a|s)←π¯(a|s)exp(ηQt(s,a))∑a′π¯(a′|s)exp(ηQt(s,a′)) _t(a|s)← π(a|s) \! (η\,Q_t(s,a) ) _a π(a |s) \! (η\,Q_t(s,a ) ) 17:return πt _t for all t Natural Policy Gradient of the EFE. With the conditional-negative-entropy generator Ψ , MD-AIF approximates natural gradient descent on the EFE over the policy manifold [3]. The proximal term of (16) is, to second order, the squared Riemannian norm DΨ(μ∥μk)=12∥πμ−πμk∥k2+O(∥δ∥3),k=⨁t,sρtπ(s)diag(πtk(⋅|s))−1D_ (μ\,\|\,μ^k)= 12\, \|π^μ-π^μ^k \|_F^k^2+O(\|δ\|^3), ^k= _t,s _t^π(s)\,diag( _t^k(·|s))^-1 (18) and the step πtk+1(⋅|s)=πtk(⋅|s)+ηkFtk(s)−1Qtk(s,⋅)+O(ηk2) _t^k+1(·|s)= _t^k(·|s)+ _k\,F_t^k(s)^-1\,Q_t^k(s,·)+O( _k^2) is precisely the first-order expansion of the multiplicative update of Algorithm˜1. MD-AIF is therefore an efficient implementation of Kakade’s natural policy gradient (NPG; [13]) on the EFE, see also [25, 3] for the general duality between MD and NPG. Proposition 2 (Convergence of MD-AIF). Run Algorithm 1 for K steps with a fixed model m and step size η=1/Lη=1/L from a full-support μ0∈μ^0 . Then Γ is L-smooth relative to Ψ on the flow polytope with constant L=12T(T+1)L= 12T(T+1), and min0≤k≤K(πk;m)−(π⋆;m)≤LDΨ(μ⋆∥μ0)K+1=O(1/K), _0≤ k≤ KG(π^k;m)-G(π ;m)\ ≤\ L\,D_ (μ \|μ^0)K+1\ =\ O(1/K), where DΨD_ is the Bregman divergence generated by Ψ(μ)=∑t,s,aμt(s,a)logπtμ(a|s) (μ)= _t,s,a _t(s,a) π^μ_t(a|s). sketch. The proof relies on showing that the algorithm solves the mirror descent update exactly for the chosen Bregman divergence DΨ(μ||μ′)D_ (μ||μ ) and that Γ is relatively smooth w.r.t. Ψ . One then invokes the relative smoothness bound [15] for the rate. ∎ 5 Performative Active Inference 5.1 Closing the Loop The fixed-model results above assume the planner sees a model that does not change as the policy changes. Closing the active inference loop (2) breaks that assumption: after deploying π the agent refits its world model to the data that π itself induced, so the reward ℓπ _π and the augmented dynamics PπP_π that the next planning phase optimizes against are functions of the deployed policy. This is exactly the structure of performative reinforcement learning [22, 16, 24], in which the trajectory distribution reacts to the deployed decision rule. A single round of the closed loop maps a deployed policy to its model-conditioned mirror step. Writing mπ=m⋆((π))m_π=m (D(π)) for the VFE-refit model and QπQ^π for the soft action-value produced by the backward recursion of Proposition˜4 under mπm_π, the performative mirror operator :Π→ΠP: → (Definition˜4) is the softmax update (π)t(a|x)=πt(a|x)exp(ηQtπ(x,a))∑bπt(b|x)exp(ηQtπ(x,b)),P(π)_t(a|x)\;=\; _t(a|x)\, \! (η\,Q^π_t(x,a) ) _b _t(b|x)\, \! (η\,Q^π_t(x,b) ), (19) and a performatively stable policy is a fixed point π⋆=(π⋆)π =P(π ): a policy that is already optimal for the model its own deployment produces. Existence follows from a standard fixed-point argument once the loop is continuous and the policy simplex compact. Proposition 3 (Existence, informal; see Proposition˜5). Under a support floor on the model class and continuity of the policy-to-model map, the performative mirror operator P has a fixed point π⋆=(π⋆)π =P(π ). Existence does not imply i) that the retraining iterates reach such a point rapidly, and i) that a given fixed point has the desired properties, e.g. the model converges to the true environment dynamics. We instead hypothesize a simplified regime in which a guarantee could hold and demonstrate the active-inference feature that obstructs it. 5.2 Toward a Convergence Guarantee A clean case for full AIF convergence is i) well-specified tabular parametrization i) with exact-recognition, and i) state coverage under all deployed policies. If all three hold, the deployed policy only changes how frequently each transition is seen, never which are seen or what the model converges to. The refit target is then policy-independent and the coupled iteration is a one-way cascade of two contractions, the policy converging to the model-optimal solution. However, the coverage assumption is unlikely to hold under diverse behavior policies in complex environments, where AIF is arguably most promising. The epistemic drive in the EFE actively pushes mass onto under-visited states and so helps coverage naturally, but it does not certify a uniform minimum of state coverage. When coverage fails, the refit target becomes policy-dependent. Moreover, the parameter-based novelty terms in the EFE may induce complex π-dependence. Ultimately, the cascade can close into an unstable feedback loop. This is the regime of performative RL [22, 16]. We therefore leave a convergence analysis to future work, where natural routes are reduction to online learning via dataset aggregation [26] or the mixed delayed repeated retraining of [24]. The experiments of Section˜6 instead probe the closed loop with the simplifying assumptions: interleaving MD-AIF planning with fully-observed model refitting and measuring how fast the learned kernel approaches the truth. 6 Experiments We test the two structural predictions of the convex-MDP view on deterministic gridworlds: that MD-AIF (Proposition˜4) converges at the O(1/K)O(1/K) rate of Proposition˜2 with the policy-dependent reward driving broad state coverage, and that closing the model-learning loop accelerates identification of the environment. Environment, hyperparameter, and baseline details are in App. F. Figure 2: MD-AIF on gridworld environments. (A, top two rows) 5×55× 5 deterministic gridworld. Left: EFE convergence (log–log scale) for MD-AIF, RL (no novelty), and EFE gradient descent, together with the theoretical O(1/k)O(1/k) rate (dashed). MD-AIF converges faster and to a lower value, driven by regularized dynamic programming. Right: Per-step imagined occupancy ρt(s) _t(s) at convergence (k=100k=100) for MD-AIF (top) and RL (bottom) at five timesteps t∈0,3,6,9,12t∈\0,3,6,9,12\. MD-AIF spreads probability mass broadly across the grid before concentrating toward the goal, whereas RL greedily channels mass along the direct path. Both rows share the same color scale. (B) 10×1010× 10 deterministic gridworld with uniform preference (pure epistemic drive). Left: Mean total-variation error TV¯(p^,p⋆) TV( p,p ) of the model fit as a function of environment steps under the interleaved model–policy loop. MD-AIF’s information-gain drive produces broader state coverage, accelerating model learning relative to RL and EFE gradient descent. AIF agents [5] plan myopically over short horizons (T=3T\!=\!3 exact; T=5T\!=\!5 via Monte Carlo with Nmc=100N_ mc=100 samples). Right: Per-state model error TV(p^,p⋆)=a∼πTTV(p^(⋅|s,a),p⋆(⋅|s,a))TV( p,p )=E_a _T\,TV( p(·|s,a),p (·|s,a)) after 2500 environment steps. Convergence and occupancy. On a 5×55× 5 grid with a Manhattan-distance preference, we compare MD-AIF with entropy-regularised RL and Euclidean gradient descent on the EFE. MD-AIF attains a lower EFE and tracks the theoretical O(1/K)O(1/K) line (Fig.˜2A, left), while its imagined occupancy spreads across the grid before concentrating on the goal, in contrast to the direct channel taken by RL (Fig.˜2A, right). Model learning. With a uniform preference, the EFE reduces to a pure epistemic drive. Interleaving policy optimization with model refitting, the performative loop of MD-AIF yields broader coverage and faster reduction of the model error TV¯(p^,p⋆) TV( p,p ) than RL, EFE gradient descent, or short-horizon EFE planning [5], see Fig.˜2B. 7 Conclusion For closed-loop policies under a fixed model, expected free energy minimization is a convex MDP: the pragmatic, ambiguity, and recognition terms form a linear latent-MDP reward, and the epistemic value is the only nonlinearity, a convex negative state-marginal entropy. This places EFE in the convex MDP class and makes its algorithms available, notably MD, which agrees with the natural policy gradient on the EFE up to second order and converges at its standard O(1/K)O(1/K) relative-smoothness rate. Refitting the model and the recognition density to policy-induced data closes the loop and makes it performative. Here we prove existence of a stable policy-model pair (Proposition˜5) and leave convergence to future work. Appendix Appendix A The expected free energy variants Definition 2 (EFE variants). The general integrand (8) specialises as follows. • Information gain (observation preference). Setting p~t(st,ot)=p(st|ot)p~(ot) p_t(s_t,o_t)=p(s_t|o_t) p(o_t) in (8) yields gtIG(st,ot)≔logρtπ(st)−logp(st|ot)−logp~t(ot).g^IG_t(s_t,o_t) _t^π(s_t)- p(s_t|o_t)- p_t(o_t). (20) This reflects an idealized planning scenario, where the agent can compute the Bayesian state posterior under its own model. • Approximate information gain. Setting p~t(st,ot)=ν(st|ot)p~(ot) p_t(s_t,o_t)=ν(s_t|o_t) p(o_t) in (8) yields gtAIG(st,ot)≔logρtπ(st)−logν(st|ot)−logp~t(ot),g^AIG_t(s_t,o_t) _t^π(s_t)- ν(s_t|o_t)- p_t(o_t), (21) which implements the same information gain term as above, but assuming variational inference is performed in the future, too. • Risk–ambiguity (state preferences). Placing preferences on states, p~(s) p(s), and omitting recognition gives (cf. [5]) gtRA(st,ot)≔logρtπ(st)−logp(ot|st)−logp~(st).g^RA_t(s_t,o_t) _t^π(s_t)- p(o_t|s_t)- p(s_t). (22) This is equivalent to setting p~t(st,ot)=p(ot|st)p~t(ot) p_t(s_t,o_t)=p(o_t|s_t) p_t(o_t) in (8). • Action Complexity. Optionally, add the action-complexity term to obtain gtAC(st,at,ot)=gt(st,ot)+logπt(at|st)π¯t(at|st).g^AC_t(s_t,a_t,o_t)=g_t(s_t,o_t)+ _t(a_t|s_t) π_t(a_t|s_t). Action complexity is used in Control as Inference formulations of RL [9] and in deep learning based AIF implementations [18] and it can be seen as a replacement for the policy prior in myopic AIF implementations [5]. Appendix B EFE minimization as a convex MDP We develop the occupancy form of the general EFE; the variants follow by substituting their linear cost. Using Eq.˜6 as the trajectory distribution, the agent plans by acting in imagination and confines optimization to the dynamically feasible occupancies (those satisfying the Chapman–Kolmogorov flow constraints below). This is also the structure which allows us to write EFE minimization as a convex MDP. Lemma 3. We write μπ=(μtπ)t=0Tμ^π=( _t^π)_t=0^T. The EFE Eq.˜8 can be written as (π;m)=∑t=0T∑s∈∑a∈μtπ(s,a)[logρtπ(s)−∑o∈p(o|s)logp~(s,o)],G(π;m)= _t=0^T _s _a _t^π(s,a) [ _t^π(s)- _o p(o|s) p(s,o) ], (23) or compactly (π)=⟨ℓ,μπ⟩+Φ(μπ),G(π)=\; ,μ^π + (μ^π), (24) where ℓ≔ℓt(s,a)=−o∼p(⋅|s)logp~(s,o) _t(s,a)=-E_o p(·|s) p(s,o), Φ(μπ)=−∑tℋ(ρtπ) (μ^π)=- _tH(ρ^π_t) is the neg. marginal Shannon entropy, and ⟨⋅,⋅⟩ ·,· is the Euclidean inner product on ℝ×[T]R^S×A×[T]. Proof. Starting from (8), the one-step expected free energy at time t depends only on the marginal of the trajectory at t: Gt(π;m)=∑s∈∑o∈ρtπ(s)p(o|s)[logρtπ(s)−logp~(s,o)].G_t(π;m)= _s _o _t^π(s)p(o|s) [ _t^π(s)- p(s,o) ]. (25) Summing over timesteps on both sides and expanding ρtπ(s)=∑aμtπ(s,a) _t^π(s)= _a _t^π(s,a) yields the first result. Collecting terms and marginalizing the first μtπ _t^π yields the second. ∎ Lemma 4. G is convex as a functional of μπμ^π. Further, for each t, the functional GtG_t is convex in μt _t, and ρt _t. Proof. By Eq.˜24, (π) (π) =⟨ℓ,μπ⟩+Φ(μπ)=∑t=0T[⟨ℓt,μπ⟩−ℋ(ρtπ)], =\; ,μ^π + (μ^π)=\; _t=0^T [ _t,μ^π -H( _t^π) ], (26) where the second inner product is on ℝ×R^S×A. Further, −ℋ-H is convex as a function of ρt _t. Since marginalization is linear and a composition of a linear and a convex function is convex, −ℋ-H is convex as a function of μt _t, but not strictly convex. The step-t inner products are linear (hence, convex) in μt _t whenever −logp~- p is independent of μt _t, which holds for all variants (Definition˜2). Since sums of convex functions are convex, each GtG_t is convex in μt _t. G and Φ(μπ) (μ^π) are also convex in μπμ^π. ∎ In the state-action marginals, all variants in Appendix A share the same structure: a linear cost plus a convex negative-entropy term. They differ in the linear cost ℓ and in whether the nonlinearity is the joint negative entropy R (with action complexity) or the state-marginal negative entropy Φ (without action compexity). All optimization machinery below depends only on this distinction. Definition 3 (Occupancy Entropies). Throughout we write Γ(⋅)≔⟨ℓ,⋅⟩+Φ(⋅), (·) ,· + (·), so that the general EFE is (π;m)=Γ(μπ)=⟨ℓ,μπ⟩+Φ(μπ).G(π;m)= (μ^π)= ,μ^π + (μ^π). Write the occupancy entropies Φ(μ) (μ) =∑t,s,aμt(s,a)log∑a′μt(s,a′) = _t,s,a _t(s,a) _a _t(s,a ) (state negative entropy), (state negative entropy), (27) Ψ(μ) (μ) =∑t,s,aμt(s,a)logμt(s,a)∑a′μt(s,a′) = _t,s,a _t(s,a) _t(s,a) _a _t(s,a ) (conditional negative entropy), (conditional negative entropy), (28) R(μ) R(μ) =∑t,s,aμt(s,a)logμt(s,a) = _t,s,a _t(s,a) _t(s,a) (state-action negative entropy). (state-action negative entropy). (29) Note that R=Φ+ΨR= + . Appendix C Details on Mirror Descent Active Inference We solve the inner problem minπ(π) _πG(π), by mirror descent (Bregman proximal gradient) with the conditional-negative-entropy generator Ψ : μk+1∈argminμ∈⟨∇Γ(μk),μ⟩+ηk−1DΨ(μ∥μk),μ^k+1∈ _μ \ ∇ (μ^k),μ + _k^-1D_ (μ\,\|\,μ^k) \, (30) where DΨ(μ∥μk)=∑t,sρtμ(s)DKL(πtμ(⋅|s)∥πtμk(⋅|s))D_ (μ\|μ^k)= _t,sρ^μ_t(s)D_KL(π^μ_t(·|s)\|π^μ^k_t(·|s)) (31) is the Bregman divergence generated by Ψ . We use Ψ throughout: it yields the closed-form softmax/DP update below. Proposition 4 (MD-AIF as softmax dynamic programming). The mirror step (16) with generator Ψ is solved in closed form by the multiplicative (softmax) policy update πtk+1(a|s)∝πtk(a|s)exp(ηkQtk(s,a)), _t^k+1(a|s)\; \; _t^k(a|s)\, \! ( _k\,Q_t^k(s,a) ), where the free-energy action value QtkQ_t^k is computed by the backward soft recursion QTk Q_T^k =rTk, =r_T^k, (32) Qtk(s,a) Q_t^k(s,a) =rtk(s,a)+∑s′p(s′|s,a)Vt+1k(s′), =r_t^k(s,a)+ _s p(s |s,a)V_t+1^k(s ), (33) Vtk(s) V_t^k(s) =∑aπtk(a|s)[Qtk(s,a)−ηk−1logπtk(a|s)πtk−1(a|s)], = _a _t^k(a|s) [Q_t^k(s,a)- _k^-1 _t^k(a|s) _t^k-1(a|s) ], (34) with the linearized reward rtk=−∇Γ(μk)tr_t^k=-∇ (μ^k)_t. Proof. The surrogate reward is the negative gradient of the smooth part Γ=⟨ℓ,μ⟩+Φ = ,μ + at the current iterate. Φ depends on μ only through the state marginals ρt(s)=∑aμt(s,a) _t(s)= _a _t(s,a) and differentiating the step-t marginal directly gives rtk(s,a)=−ℓt(s,a)−logρtk(s)−1r_t^k(s,a)=- _t(s,a)- _t^k(s)-1. With Ψ(μ)=∑t,s,aμt(s,a)logπtμ(a|s) (μ)= _t,s,a _t(s,a) π^μ_t(a|s) the Bregman divergence factorises as DΨ(μ∥μk)=∑t,sρtμ(s)DKL(πtμ(⋅|s)∥πtμk(⋅|s)).D_ (μ\|μ^k)= _t,sρ^μ_t(s)D_KL(π^μ_t(·|s)\|π^μ^k_t(·|s)). Substituting μt(s,a)=ρt(s)πt(a|s) _t(s,a)= _t(s) _t(a|s), the subproblem (16) becomes, up to π-independent terms, minπ∑t,sρtπ(s)∑aπt(a|s)[−rtk(s,a)+ηk−1logπt(a|s)πtk(a|s)] _π\ _t,s _t^π(s) _a _t(a|s) [-r_t^k(s,a)+ _k^-1 _t(a|s) _t^k(a|s) ] subject to (12). Introducing multipliers for the flow constraints identifies them with the value function VtkV_t^k and the action value QtkQ_t^k above, see [20] for details. For each (t,s)(t,s) the inner minimization over the simplex is an entropy-regularised linear program whose stationarity condition (one multiplier for normalisation) gives the Gibbs solution πtk+1(a|s)∝πtk(a|s)exp(ηkQtk(s,a)) _t^k+1(a|s) _t^k(a|s) ( _kQ_t^k(s,a)). Since QtkQ_t^k depends only on Vt+1kV_t+1^k, a single backward pass t=T,…,0t=T,…,0 solves the system. This is mirror descent modified policy iteration [14, 28, 7] on a finite horizon problem [20]. ∎ See 2 Proof. The softmax update solves the mirror descent update on Γ with generator Ψ exactly and keeps each iterate interior, so the relative smoothness bound [15, Thm. 3.1] with η=1/Lη=1/L holds along the trajectory as long as ⟨∇2Γ⟩≤L⟨∇2Ψ⟩ ∇^2 ≤ L ∇^2 , which holds as a direct consequence of Theorem 1 below. ∎ Lemma 5 (Markov contraction). For a Markov kernel K:→K:Y , ν>0ν>0 on Y, signed v on Y: ‖Kv‖Kν≤‖v‖ν\ \|Kv\|_Kν≤\|v\|_ν. Proof. (Kv)(z)2=(∑yK(z|y)v(y))2≤(Kν)(z)∑yK(z|y)v(y)2ν(y)(Kv)(z)^2= ( _yK(z|y)v(y) )^2≤(Kν)(z) _yK(z|y) v(y)^2ν(y) by Cauchy–Schwarz; divide by (Kν)(z)(Kν)(z), sum over z, use ∑zK(z|y)=1 _zK(z|y)=1. ∎ Theorem 1 (Relative smoothness). On the polytope tangent space TK, the EFE Γ is L-smooth relative to Ψ on K with L=12T(T+1)L= 12T(T+1). Proof. Relative smoothness for twice differentiable Γ and Ψ is the condition ∇2Γ(μ)⪯12T(T+1)∇2Ψ(μ)\ ∇^2 (μ) 12T(T+1)\,∇^2 (μ) on TK, see [15]. We read the two Fisher metrics off the identity R=Φ+ΨR= + of Definition˜3 at the level of Bregman divergences. The Bregman divergence of the joint negentropy R is the sum of per-step relative entropies, and is generally linear in the generating function ∑tDKL(μt∥μt′)⏟DR=∑tDKL(ρt∥ρt′)⏟DΦ+∑t,sρt(s)DKL(πt(⋅∣s)∥πt′(⋅∣s))⏟DΨ. _tD_KL( _t\| _t )_D_R= _tD_KL( _t\| _t )_D_ + _t,s _t(s)\,D_KL\! ( _t(· s)\| _t (· s) )_D_ . The KL-divergence obeys DKL(x,x+δx)=12‖δx‖x2+o(‖δx‖2)D_KL(x,x+δ x)= 12\|δ x\|_x^2+o(\|δ x\|^2), where ‖v‖x2=∑iv(i)2x(i)\|v\|^2_x= _i v(i)^2x(i) is the fisher length at v. Expanding each term to second order yields the Pythagorean split of the per-step Fisher metric and identifies all three quadratic forms at once: ∑t‖δμt‖μt2⏟∇2R[δμ]=∑t‖δρt‖ρt2⏟∇2Γ[δμ]=∇2Φ[δμ]+∑t,sρt(s)∥δπt(⋅∣s)∥πt2⏟∇2Ψ[δμ]. _t\|δ _t\|^2_ _t_∇^2R[δμ]= _t\|δ _t\|^2_ _t_∇^2 [δμ]=∇^2 [δμ]+ _t,s _t(s)\,\|δ _t(· s)\|^2_ _t_∇^2 [δμ]. (35) The only nonlinear part of Γ is Φ=−∑tℋ(ρt) =- _tH( _t), and Φ depends on μ only through the linear map ρ(⋅)=∑aμ(⋅,a)ρ(·)= _aμ(·,a), so its Hessian carries no conditional part, ∇2Γ[δμ]=∑t‖δρt‖ρt2.∇^2 [δμ]= _t\|δ _t\|^2_ _t. Second, the DΨD_ block of (35) gives ∇2Ψ[δμ]=∑t∑sρt(s)∥δπt(⋅∣s)∥πt2.∇^2 [δμ]= _t _s _t(s)\,\|δ _t(· s)\|^2_ _t. Thus ∇2Γ∇^2 and ∇2Ψ∇^2 are the marginal and conditional blocks of the same per-step Fisher metric. Relative smoothness is the statement that L∇2ΨL∇^2 upper bounds ∇2Γ∇^2 for some finite L. The marginal variation δρtδ _t is fully determined by the conditional components ρt′δπt′t′<t\ _t δ _t \_t <t through the flow map Eq.˜12. Writing wt′:=ρt′⊗δπt′w_t := _t \! \!δ _t for the conditional component of δμt′δ _t (so ∑s,awt′=∑sρt′∑aδπt′=0 _s,aw_t = _s _t _aδ _t =0), and linearising ρt+1=Pπtρt _t+1=P_ _t _t with δρ0=0δ _0=0 and Pπ=π∘P_π=π P gives: δρt=∑t′<tKt′→twt′,Kt′→t:=Pπt−1⋯Pπt′+1⏟δμt′→δμtP⏟μt′→ρt′+1.δ _t= _t <tK_t → t\,w_t , K_t → t:= P_ _t-1·s P_ _t +1_δ _t →δ _t\, P P_π_ _t → _t +1. As a composition of row-stochastic kernels, Kt′→tK_t → t is itself a Markov kernel. Applying Lemma˜5, ‖Kt′→twt′‖ρt≤‖wt′‖μt′=∑sρt′(s)∥δπt′(⋅|s)∥πt′2.\|K_t → t\,w_t \|_ _t\;≤\;\|w_t \|_ _t = _s _t (s)\,\|δ _t (·|s)\|^2_ _t . The triangle and Cauchy–Schwarz inequalities over the t sources give ∥δρt∥ρt2≤(∑t′<t∥Kt′→twt′∥ρt)2≤t∑t′<t∑sρt′(s)∥δπt′(⋅|s)∥πt′2\|δ _t\|^2_ _t≤ ( _t <t\|K_t → tw_t \|_ _t )^2≤ t _t <t _s _t (s)\,\|δ _t (·|s)\|^2_ _t and summing over t with ∑t=0Tt=12T(T+1) _t=0^Tt= 12T(T+1) gives ∇2Γ[δμ]=∑t‖δρt‖ρt2 ∇^2 [δμ]= _t\|δ _t\|^2_ _t ≤12T(T+1)∑t′∑sρt′(s)∥δπt′(⋅|s)∥πt′2 ≤ 12T(T+1) _t _s _t (s)\,\|δ _t (·|s)\|^2_ _t =12T(T+1)∇2Ψ[δμ], = 12T(T+1)\,∇^2 [δμ], i.e. Γ is L-smooth relative to Ψ on the flow-polytope tangent with L=12T(T+1)L= 12T(T+1). ∎ Appendix D History dependent and infinite horizon formulations The episodic memoryless development of Appendices˜B, C and E transfers to history-dependent policies and posterior densities, as well as to the discounted and average-reward settings with appropriate extensions of the standard EFE definition. D.1 History-dependence Consider the finite-horizon reward-free POMDP ℳ=(,,,P,E,σ)M=(S,A,O,P,E,σ) as before. Further, let the history at time t be ht=(o0:t,a0:t−1)h_t=(o_0:t,a_0:t-1), updated by ht+1=ht⋅(at,ot+1)h_t+1=h_t·(a_t,o_t+1) with h0=(o0)h_0=(o_0), and let ℋ=⋃t<T(×)t×H= _t<T(A×O)^t×O be the space of histories. The behavior policy β:ℋ×→Δ()β:H×T→ (A) in the history-dependent case is of the form β(at|ht)=∑stπ(at|ht,st)ν(st|ht)β(a_t|h_t)= _s_tπ(a_t|h_t,s_t)ν(s_t|h_t) (36) and induces the environment trajectory law pβ(τ)=σ(s0)E(o0|s0)∏t=0T−1β(at|ht)P(st+1|st,at)E(ot+1|st+1).p^β(τ)=σ(s_0)E(o_0|s_0) _t=0^T-1\,β(a_t|h_t)P(s_t+1|s_t,a_t)E(o_t+1|s_t+1). (37) Given a model m=(p,ν)m=(p,ν) with history-dependent variational density ν(st|ht)ν(s_t|h_t), a latent policy π(at|ht,st)π(a_t|h_t,s_t) induces the purely imagined trajectory law qπ(τ;m)=q0(s0)p(o0|s0)∏t=0T−1πt(at|ht,st)p(st+1|st,at)p(ot+1|st+1),q^π(τ;m)=q_0(s_0)\,p(o_0|s_0) _t=0^T-1 _t(a_t|h_t,s_t)\,p(s_t+1|s_t,a_t)\,p(o_t+1|s_t+1), (38) with predictive history-state marginals ρtπ(s,h)=ℙπ(st=s,ht=h)ρ^π_t(s,h)=P_π(s_t=s,h_t=h). Lemma 6 (Augmented-state recursion). For history-dependent variational density ν(st|ht)ν(s_t|h_t), define the augmented state space =ℋ×X=H×S and the augmented kernel p(x′|x,a)=p((s′,h′)|(s,h),a)=[h′=h⋅(a,o′)]p(o′|s′)p(s′|s,a).p(x |x,a)=p ((s ,h )|(s,h),a )=1[h =h·(a,o )]\,p(o |s )\,p(s |s,a). (39) Then the imagined process (38) satisfies ρ0π=q0,ρt+1π(x′)=∑x,ap(x′|x,a)πt(a|x)ρtπ(x), _0^π=q_0, _t+1^π(x )= _x,ap(x |x,a)\, _t(a|x)\, _t^π(x), (40) where ρtπ((s,h))=ℙ(st=s,ht=h)ρ^π_t((s,h))=P(s_t=s,h_t=h) are the history-state marginals of the trajectory distribution q at time t. Proof. The imagined law (38) is a product of per-step kernels, hence the law of a Markov process, and ht=(o0:t,a0:t−1)h_t=(o_0:t,a_0:t-1) is a deterministic function of the prefix, so xt=(ht,st)x_t=(h_t,s_t) is determined by it. Condition on the prefix (s0:t,o0:t,a0:t−1)(s_0:t,o_0:t,a_0:t-1) together with ata_t. The only factors carrying the next variables are p(st+1|st,at)p(ot+1|st+1)p(s_t+1|s_t,a_t)\,p(o_t+1|s_t+1), so (st+1,ot+1)(s_t+1,o_t+1) has conditional law p(s′|st,at)p(o′|s′)p(s |s_t,a_t)\,p(o |s ), depending on the past only through (st,at)(s_t,a_t); and ht+1=ht⋅(at,ot+1)h_t+1=h_t·(a_t,o_t+1) is a deterministic function of (ht,at,ot+1)(h_t,a_t,o_t+1). Since the append is injective, for fixed (h,a,h′)(h,a,h ) at most one o′o satisfies h′=h⋅(a,o′)h =h·(a,o ), so marginalising ot+1o_t+1 gives p(x′|x,a) p(x |x,a) =Pr[xt+1=(h′,s′)|xt=(h,s),at=a] = \! [x_t+1=(h ,s )|x_t=(h,s),\,a_t=a ] (41) =∑o′[h′=h⋅(a,o′)]p(o′|s′)p(s′|s,a), = _o 1[h =h·(a,o )]\,p(o |s )\,p(s |s,a), (42) which depends on the past only through (xt,at)(x_t,a_t): (xt)t(x_t)_t is a controlled Markov chain with the augmented kernel. The action satisfies at∼πt(⋅|ht,st)=πt(⋅|xt)a_t _t(·|h_t,s_t)= _t(·|x_t), so the law of total probability gives ρt+1π(x′) _t+1^π(x ) =∑x,aPr[xt+1=x′|xt=x,at=a]πt(a|x)ρtπ(x) = _x,a \! [x_t+1=x |x_t=x,a_t=a ]\, _t(a|x)\, _t^π(x) (43) =∑x,ap(x′|x,a)πt(a|x)ρtπ(x), = _x,ap(x |x,a)\, _t(a|x)\, _t^π(x), (44) which is (40), where ρtπ(x) _t^π(x) is written as a measure on X. The base case fixes ρ0π _0^π to the law of the initial augmented state x0=(h0,s0)x_0=(h_0,s_0) prescribed by (38), written q0q_0 as a measure on X. ∎ In the history-dependent case, we would define the EFE as ℰ(π;m)=∑t=0T∑x∈∑a∈πt(a|x)ρtπ(x)[logρtπ(x)−∑o∈p(o|s)logp~(x)],E(π;m)= _t=0^T _x _a _t(a|x)\, _t^π(x) [ _t^π(x)- _o p(o|s) p(x) ], (45) where we overload notation and write logp~(x)=logp~(s,o) p(x)= p(s,o) as a density on X. See 2 Proof. Lemma˜6 applies directly to ℰE, so the optimization problem minπℰ(π;m) _πE(π;m) (46) is a convex MDP, but on the augmented state-space X. ∎ Remark 1 (Augmented state in practice). The history component of x∈x makes |||X| grow with t. In a function-approximation implementation, h is typically summarised by a recurrent state, e.g. write x=f(h,s)x=f(h,s). Replace s→xs→ x and the tabular sweeps of Algorithm˜1 remain unchanged. The memoryless variant (Definition˜2) drops h entirely and recovers a standard S-state convex-MDP solver. D.2 Infinite-horizon formulations Extending the EFE to the infinite horizon introduces a choice absent in the finite-horizon case: how the future is aggregated inside the nonlinear entropy term (cf. Table˜1). Throughout we work in the stationary regime where ℳM, ℓ , and π are time-homogeneous on the augmented state x=f(h,s)x=f(h,s) of Remark˜1, now over t∈ℕt . Discounted occupancy. Fix γ∈(0,1)γ∈(0,1) and a stationary π. The normalized discounted occupancy dπ(x,a)=(1−γ)∑t=0∞γtμtπ(x,a),∑x,adπ(x,a)=1,d^π(x,a)=(1-γ) _t=0^∞γ^t\, _t^π(x,a), _x,ad^π(x,a)=1, (47) collapses the time-indexed flow constraints (12), summed against the weights (1−γ)γt(1-γ)γ^t, into the single Bellman-flow constraint ∑adπ(x′,a)=(1−γ)q0(x′)+γ∑x,ap(x′|x,a)dπ(x,a),x′∈. _ad^π(x ,a)=(1-γ)\,q_0(x )+γ _x,ap(x |x,a)\,d^π(x,a), x . (48) The discounted occupancy polytope γ=d∈Δ(×):(48)K_γ=\d∈ (X×A): eq:disc-flow\ is again compact convex, and on full-support policies π↦dπ d^π is a bijection with inverse π(a|x)=d(x,a)/d(x)π(a|x)=d(x,a)/d(x) (Definition˜1). Two aggregation conventions. Since entropy is not additive across the discount weights, the finite-horizon objective =⟨ℓ,μ⟩−∑tℋ(ρt)G= ,μ - _tH( _t) has two inequivalent stationary analogues. (a) Aggregate, then score. Apply the entropy to the single aggregate occupancy, γagg(π)=⟨ℓ,dπ⟩−ℋ(dπ(x))=⟨ℓ,dπ⟩+Φ(dπ),G_γ^agg(π)= ,d^π -H (d^π(x) )= ,d^π + (d^π), (49) the maximum-entropy-exploration objective of [11]. This is a linear functional minus the entropy of a single d∈γd _γ, hence convex, and is the cleanest extension: all of Appendices˜B and C applies with μt _t replaced by d and the finite recursion of Proposition˜4 by the stationary soft-Bellman fixed point Qk(x,a) Q^k(x,a) =rk(x,a)+γ∑x′p(x′|x,a)Vk(x′), =r^k(x,a)+γ _x p(x |x,a)\,V^k(x ), (50) Vk(x) V^k(x) =∑aπk(a|x)[Qk(x,a)−ηk−1logπk(a|x)πk−1(a|x)], = _aπ^k(a|x) [Q^k(x,a)- _k^-1 π^k(a|x)π^k-1(a|x) ], (51) with rk=−∇(⟨ℓ,d⟩+Φ)(dk)r^k=-∇( ,d + )(d^k), solved to a contraction tolerance inside each mirror step. The natural-gradient reading and O(1/K)O(1/K) rate carry over with the discounted relative-smoothness constant [14]. (b) Score, then aggregate. Apply the entropy per step, γstep(π)=∑t=0∞γt[⟨ℓ,μtπ⟩−ℋ(μtπ(x,a))],G_γ^step(π)= _t=0^∞γ^t [ , _t^π -H ( _t^π(x,a) ) ], (52) the term-by-term analogue of the finite-horizon EFE. By strict concavity of ℋH on the mixture dπ=(1−γ)∑tγtμtπd^π=(1-γ) _tγ^t _t^π, ℋ(dπ)≥(1−γ)∑tγtℋ(μtπ),H (d^π )\;≥\;(1-γ) _tγ^t\,H ( _t^π ), (53) with equality iff all μtπ _t^π coincide; hence (a)≠(b), the aggregate convention crediting occupancy spread across time as if spread within a step. Objective (52) is convex in (μt)t( _t)_t but not a function of d alone, so it must be optimized over per-step occupancies (truncated at ∼(1−γ)−1 (1-γ)^-1), and the backward recursion no longer collapses to a single fixed point. Average-reward limit. As γ→1γ→ 1, convention (a) yields the stationary occupancy d∞π(x,a)=limT→∞1T∑t=0T−1μtπ(x,a),d^π_∞(x,a)= _T→∞ 1T _t=0^T-1 _t^π(x,a), (54) which requires more than Assumption 2: under a unichain assumption [23] on pπ(x′|x)=∑ap(x′|x,a)π(a|x)p_π(x |x)= _ap(x |x,a)π(a|x), every stationary π induces a unique d∞πd_∞^π independent of q0q_0 and the limit exists. The per-step averages ℓ¯(π)=limT1T∑tμtπ[ℓ] (π)= _T 1T _tE_ _t^π[ ] and ℋ¯(π)=limT1T∑tℋ(μtπ) H(π)= _T 1T _tH( _t^π) again satisfy ℋ¯(π)≥ℋ(d∞π) H(π) (d_∞^π), so aggregation and EFE computation do not commute. The convex-MDP structure and the mirror/natural-gradient solver survive in average-reward form for ℋ(d∞π)H(d_∞^π), e.g. [1]; the full AIF loop we leave to future work. Appendix E Performative active inference In the design of MD-AIF, we made the strong assumption, that the learned model stays fixed during policy optimization. However, the interesting question is whether the optimization problem remains well-posed if the agent is allowed to update its model during policy optimization, which would restore the AIF problem in full spirit. Consider, for example, the Algorithm 2. From the optimized policy’s perspective, now both the reward and the dynamics are functions of the policy, or equivalently, the occupancy. This puts AIF squarely into the framework of performative RL [16, 24]. Algorithm 2 Performative MD-AIF with variable world model (sketch). 1:initial world model m0=(p0,ν0)m^0=(p^0,ν^0); horizon T; initial policy π0π^0; iterations K. 2:initialise πt0(⋅|x)←π¯t(⋅|x)π^0_t(·|x)← π_t(·|x) (or uniform) for all t,xt,x 3:for k=0,1,…,K−1k=0,1,…,K-1 do 4: mk+1←MIN-VFE(mk;πk)m^k+1 -VFE(m^k;π^k) 5: p~k+1,q0k+1←Extract(mk+1) p^k+1,q^k+1_0 (m^k+1) 6: πk+1←MD-AIF(πk;mk+1,p~k+1,T,η,q0k+1)π^k+1 -AIF(π^k;m^k+1, p^k+1,T,η,q^k+1_0) 7:return πKπ^K To see this, consider the one pass of the full loop (2). Deploying π (behavior β=π∘νβ=π ν) in the real environment induces the history distribution (π)D(π); the perception step returns the VFE-optimal model m⋆((π))∈argminm∈ℳℱ(m;(π))=:(pπ,νπ),m (D(π))∈ _m F (m;D(π) )\;=:\; (p_π, _π ), and this refit model is what the planner sees: it supplies a linear pseudo-cost ℓπ _π and the augmented transition kernel PπP_π of Lemma˜6, ℓπ:=ℓ(m⋆((π))),Pπ:=p(m⋆((π))). _π:= (m (D(π)) ), P_π:=p (m (D(π)) ). Thus the decision-dependence is the composition π↦(π)↦VFEm⋆((π))↦(ℓπ,Pπ),π\ \ D\ \ D(π)\ \ VFE\ \ m (D(π))\ \ ( _π,P_π ), i.e. it runs through the agent’s own model-learning step, not through a primitive environment response: the reward and dynamics react to π only because the model is refit to π’s data. Passing to occupancies through the normalisation πμ(a|x)=μ(x,a)/∑bμ(x,b)π^μ(a|x)=μ(x,a)/ _bμ(x,b) (cf. [16], eq. (2)), set ℓμ:=ℓπμ _μ:= _π^μ, Pμ:=PπμP_μ:=P_π^μ, and the EFE pseudo-reward rμ:=−ℓμr_μ:=- _μ. Assumption 1 (Smoothness of performative map; cf. [16]). We assume the model-learning loop is (εr,εp)( _r, _p)-sensitive: for all occupancies μ,μ′μ,μ , ‖ℓμ−ℓμ′‖≤εr‖μ−μ′‖,‖Pμ−Pμ′‖≤εp‖μ−μ′‖.\| _μ- _μ \|≤ _r\,\|μ-μ \|, \|P_μ-P_μ \|≤ _p\,\|μ-μ \|. This is [16], Assumption 1, transported onto the refit map μ↦m⋆((πμ))μ m (D(π^μ)); the proof below uses only continuity of μ↦(ℓμ,Pμ)μ ( _μ,P_μ). Definition 4 (Performative mirror operator). Let Π=∏t=0T∏x∈Δ() = _t=0^T _x (A) be the (episodic) policy simplex. Deploying π∈Ππ∈ induces the model mπ=m⋆((π))m_π=m (D(π)), the occupancy μπ∈(mπ)μ^π (m_π), and the linearized reward rtπ=−(∇μΓ(μ;mπ)|μ=μπ)t.r^π_t=- ( _μ (μ;m_π) |_μ=μ^π )_t. Let QπQ^π be the soft action-value produced from rπr^π by the backward recursion of Proposition˜4 under mπm_π. The performative mirror operator :Π→ΠP: → is the single step (π)t(a|x)=πt(a|x)exp(ηQtπ(x,a))∑bπt(b|x)exp(ηQtπ(x,b)).P(π)_t(a|x)\;=\; _t(a|x)\, \! (η\,Q^π_t(x,a) ) _b _t(b|x)\, \! (η\,Q^π_t(x,b) ). (55) A fixed point π⋆=(π⋆)π =P(π ) is a performatively stable point. Assumption 2 (Support floor). For the fixed model m=(p,ν)m=(p,ν) in the model class, there exists an ϵ0>0\, _0>0 with p(s′|s,a)≥ϵ0p(s |s,a)≥ _0, p(o|s)>0p(o|s)>0, q0(s)>0q_0(s)>0; ℳM (tabular, floored at ϵ0 _0) is compact convex. Proposition 5 (Existence). Under Assumption 2 and the sensitivity assumption 1, P has a fixed point. Proof. Π is a nonempty compact convex polytope. The map P of (55) is single-valued and maps into Π , since each (π)t(⋅|s)P(π)_t(·|s) is an explicit normalised positive vector in Δ() (A). Further, it is continuous, since the softmax is smooth and π↦Qπ Q^π is a finite composition of continuous maps in mπm_π, which depends continuously on π by Assumption 1. Brouwer’s theorem gives π⋆=(π⋆)π =P(π ). ∎ Remark 2 (Fixed points are inner-optimal). At a fixed point π⋆=(π⋆)π =P(π ) the multiplicative update (55) is inactive, so Qtπ⋆(x,a)Q^π _t(x,a) is constant across suppπt⋆(⋅|x)suppπ _t(·|x). With Qπ⋆=−∇Γπ⋆(μπ⋆)Q^π =-∇ _π (μ^π ) this is the variational inequality ⟨∇Γπ⋆(μπ⋆),μ−μπ⋆⟩≥ 0∀μ∈(mπ⋆), ∇ _π (μ^π ),\,μ-μ^π \ ≥\ 0 ∀\,μ (m_π ), (56) the first-order condition for minμΓπ⋆(μ) _μ _π (μ). Since Γπ⋆ _π is convex (Proposition˜1), (56) is sufficient for global optimality. With action-complexity (or an additional Ψ -regulariser), the joint-entropy barrier of Proposition˜1 keeps the optimum interior and existence (Proposition˜5) yields a performatively stable point. Without action complexity, only the support-stationary point of (56), as (55) cannot populate an unplayed action. Appendix F Experimental details F.1 Convergence experiment A deterministic 5×55× 5 grid with four actions (left, right, up, down); wall collisions result in staying in place. The agent starts at the top-left corner (state 0) and the goal is at the bottom-right corner (state 2424). The preference distribution is logp~(s)∝−αd(s,sgoal),α=0.5, p(s) -α\,d(s,s_ goal), α=0.5, (57) where d is the Manhattan distance, normalised to a probability simplex. Methods. Three optimizers are compared on the EFE objective with planning horizon H=12H=12 and K=100K=100 iterations each. • MD-AIF (Algorithm˜1). Mirror-descent with the conditional-entropy generator Ψ , constant step size η=0.05η=0.05, matching the assumption of Proposition˜2. The linearized reward at iteration k is rtk(s,a)=logp~(s)−logρtk(s)r_t^k(s,a)= p(s)- _t^k(s), combining the preference term and the novelty term −logρtk(s)- _t^k(s) that rewards imagined states currently under low occupancy. Policy initialised uniformly. • RL (greedy, no novelty). Identical to MD-AIF but with the novelty term removed, i.e. rtk(s,a)=logp~(s)r_t^k(s,a)= p(s). This reduces to soft value iteration on a static reward, equivalent to entropy-regularised policy gradient toward p~ p. • EFE gradient descent. Euclidean gradient descent on Γ directly in the softmax logit space θt∈ℝS×A _t ^S× A, using exact reverse-mode autodiff through the occupancy recursion ρt+1(s′)=∑s,ap^(s′|s,a)πt(a|s)ρt(s) _t+1(s )= _s,a p(s |s,a) _t(a|s) _t(s). Constant step size η=0.05η=0.05. Convergence plot. The y-axis shows Γ(ρk)−Γ⋆ (ρ^k)- , where Γ⋆ is the minimum across all three methods and all iterates. The O(1/k)O(1/k) reference line is fitted to MD-AIF’s initial gap: C/kC/k with C=Γ(ρ0)−Γ⋆C= (ρ^0)- . Both axes are logarithmic. Occupancy snapshots. For each method at convergence (k=Kk=K), the per-step imagined state occupancy ρt(s) _t(s) is shown at five evenly spaced timesteps t∈0,⌊H/4⌋,⌊H/2⌋,⌊3H/4⌋,H=0,3,6,9,12t∈\0, H/4 , H/2 , 3H/4 ,H\=\0,3,6,9,12\. Both MD-AIF and RL use the same color scale so occupancy concentrations are directly comparable across rows. F.2 Model-learning experiment A deterministic 10×1010× 10 grid (S=100S=100 states, A=4A=4 actions). Start: top-left (state 0); goal: bottom-right (state 9999). The log-preference is uniform (logp~(s)=const p(s)=const) so the EFE reduces to pure state-entropy maximization; the only drive is the novelty term −logρtk(s)- _t^k(s). Algorithms. Each of R=20R=20 rounds proceeds as follows: 1. Run K=120K=120 policy-optimization steps on the current estimated transition model p p (one mirror/gradient step per inner iteration). 2. Deploy the resulting policy: sample E=5E=5 episodes of length H=25H=25 from the true environment by ancestral sampling (at∼πt(⋅|st)a_t _t(·|s_t), st+1∼p⋆(⋅|st,at)s_t+1 p (·|s_t,a_t)). 3. Refit p p by Dirichlet-MAP with pseudocount α0=10−3 _0=10^-3: p^(s′|s,a)=(N(s,a,s′)+α0)/∑s′(N(s,a,s′)+α0). p(s |s,a)=(N(s,a,s )+ _0)/ _s (N(s,a,s )+ _0). Both the policy optimization and the model update use a constant step size η=0.05η=0.05 throughout (no decay). Four planning algorithms are compared. All share the same deployment and refitting procedure. • MD-AIF. Mirror-descent EFE minimization (Algorithm˜1), η=0.05η=0.05. • RL. MD-AIF without entropy. • EFE gradient descent. Euclidean gradient on logits, η=0.05η=0.05. • AIF T=3T=3 (exact, [5]). For each state s, enumerate all 43=644^3=64 action sequences of length Tdc=3T_ dc=3, evaluate G(π|s)=∑t′>t∑s′bt′(s′)logbt′(s′)G(π|s)= _t >t _s b_t (s ) b_t (s ) for belief state b, form π∝∑seq:a0=aexp(−G(seq|s))π _seq:a_0=a (-G(seq|s)), and marginalise. • AIF T=5T=5 (MC). Same as above but with Tdc=5T_ dc=5 and Nmc=100N_ mc=100 randomly sampled sequences per policy computation (Monte Carlo). Evaluation. After each deployment round the model error is measured as the mean total-variation distance between the estimated and true kernels: TV¯(p^,p⋆)=1SA∑s,a12∥p^(⋅|s,a)−p⋆(⋅|s,a)∥1. TV( p,p )= 1SA _s,a 12\| p(·|s,a)-p (·|s,a)\|_1. (58) The x-axis reports cumulative environment steps. The grey dashed line marks the error of the Dirichlet-MAP estimate with no data (prior only): TV¯≈(S−1)/S=(100−1)/100=0.99 TV≈(S-1)/S=(100-1)/100=0.99. References [1] J. Adamczyk, V. Makarenko, S. Tiomkin, and R. V. Kulkarni (2025) Average-reward soft actor-critic. Reinforcement Learning Journal 6, p. 412–430. Cited by: §D.2. [2] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan (2021) On the theory of policy gradient methods: optimality, approximation, and distribution shift. Journal of Machine Learning Research 22 (98), p. 1–76. Cited by: §2. [3] S. Amari (1998) Natural gradient works efficiently in learning. Neural computation 10 (2), p. 251–276. Cited by: §4, §4. [4] A. Beck and M. Teboulle (2003) Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters 31 (3), p. 167–175. Cited by: §4, §4. [5] L. Da Costa, T. Parr, N. Sajid, S. Veselic, V. Neacsu, and K. Friston (2020) Active inference on discrete state-spaces: a synthesis. Journal of Mathematical Psychology 99, p. 102447. Cited by: 3rd item, 4th item, 4th item, §1, §1, §2.2, Figure 2, §6. [6] K. Friston, F. Rigoli, D. Ognibene, C. Mathys, T. Fitzgerald, and G. Pezzulo (2015) Active inference and epistemic value. Cognitive Neuroscience 6 (4), p. 187–214. Cited by: §1. [7] M. Geist, B. Scherrer, and O. Pietquin (2019-09–15 Jun) A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2160–2169. External Links: Link Cited by: Appendix C. [8] D. Ha and J. Schmidhuber (2018) Recurrent world models facilitate policy evolution. Advances in neural information processing systems 31. Cited by: §1. [9] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: 4th item, item (i). [10] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2020) Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.2, §3.2. [11] E. Hazan, S. M. Kakade, K. Singh, and A. Van Soest (2019) Provably efficient maximum entropy exploration. In Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR, Vol. 97, p. 2681–2691. Cited by: §D.2, item (i), §3.1. [12] M. Janner, J. Fu, M. Zhang, and S. Levine (2019) When to trust your model: model-based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32. Cited by: §1. [13] S. M. Kakade (2001) A natural policy gradient. Advances in neural information processing systems 14. Cited by: §4. [14] G. Lan (2023) Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming 198 (1), p. 1059–1106. Cited by: Appendix C, §D.2, §4. [15] H. Lu, R. M. Freund, and Y. Nesterov (2018) Relatively smooth convex optimization by first-order methods, and applications. SIAM Journal on Optimization 28 (1), p. 333–354. Cited by: Appendix C, Appendix C, §4. [16] D. Mandal, S. Triantafyllou, and G. Radanovic (2023) Performative reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR, Vol. 202, p. 23642–23680. Cited by: Appendix E, Appendix E, §1, §5.1, §5.2, Assumption 1, Assumption 1. [17] P. Mazzaglia, T. Verbelen, O. Catal, and B. Dhoedt (2022) The free energy principle for perception and action: a deep learning perspective. Entropy 24 (2), p. 301. Cited by: §2.2. [18] P. Mazzaglia, T. Verbelen, and B. Dhoedt (2021) Contrastive active inference. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: 4th item, §1, §2.2, §3.2. [19] B. Millidge, A. Tschantz, and C. L. Buckley (2021) Whence the expected free energy?. Neural Computation 33 (2), p. 447–482. External Links: Document Cited by: §2.2, §2.2. [20] B. M. Moreno, M. Brégère, P. Gaillard, and N. Oudjane (2024) Efficient model-based concave utility reinforcement learning through greedy mirror descent. In International Conference on Artificial Intelligence and Statistics, p. 2206–2214. Cited by: Appendix C, §3.1, §3.1, §4, §4, §4, Definition 1, Proposition 1. [21] T. Parr, L. Da Costa, and K. Friston (2019-12) Markov blankets, information geometry and stochastic thermodynamics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 378 (2164), p. 20190159. External Links: ISSN 1364-503X, Document, Link, https://royalsocietypublishing.org/rsta/article-pdf/doi/10.1098/rsta.2019.0159/1316769/rsta.2019.0159.pdf Cited by: §2.2, §2.2. [22] J. C. Perdomo, T. Zrnic, C. Mendler-Dünner, and M. Hardt (2020) Performative prediction. In Proceedings of the 37th International Conference on Machine Learning (ICML), PMLR, Vol. 119, p. 7599–7609. Cited by: item (i), §1, §5.1, §5.2. [23] M. L. Puterman (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley. Cited by: §D.2, §3.1, Definition 1. [24] B. Rank, S. Triantafyllou, D. Mandal, and G. Radanovic (2024) Performative reinforcement learning in gradually shifting environments. In Proceedings of the 40th Conference on Uncertainty in Artificial Intelligence (UAI), PMLR, Vol. 244, p. 3041–3075. Cited by: Appendix E, §1, §5.1, §5.2. [25] G. Raskutti and S. Mukherjee (2015) The information geometry of mirror descent. IEEE Transactions on Information Theory 61 (3), p. 1451–1457. Cited by: §4. [26] S. Ross, G. Gordon, and D. Bagnell (2011-11–13 Apr) A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, G. Gordon, D. Dunson, and M. Dudík (Eds.), Proceedings of Machine Learning Research, Vol. 15, Fort Lauderdale, FL, USA, p. 627–635. External Links: Link Cited by: §5.2. [27] P. P. Santos, A. Sardinha, and F. S. Melo (2025) The number of trials matters in infinite-horizon general-utility markov decision processes. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §3.1, §3.2. [28] L. Xiao (2022) On the convergence rates of policy gradient methods. Journal of Machine Learning Research 23 (282), p. 1–36. Cited by: Appendix C. [29] T. Zahavy, B. O’Donoghue, G. Desjardins, and S. Singh (2021) Reward is enough for convex mdps. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34. Cited by: §1, §3.1, Proposition 1. [30] J. Zhang, A. Koppel, A. S. Bedi, C. Szepesvári, and M. Wang (2020) Variational policy gradient method for reinforcement learning with general utilities. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. Cited by: §1, §3.1.