Paper deep dive
An Irreducible Quantum Advantage in Aligning World Models with Reality
Josep Lumbreras, Hailan Ma, Jayne Thompson, Mile Gu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/21/2026, 3:37:02 AM
Summary
This paper demonstrates an irreducible quantum advantage in aligning world models with reality. It proves that for certain complex, non-Markovian true worlds, any finite classical world model suffers from a non-vanishing alignment gap, characterized by value deviancy, loss of decision resolution, and mean decision loss. In contrast, a quantum world model using a single qutrit can reproduce the true world's statistics exactly, ensuring perfect alignment of optimal agent policies.
Entities (9)
Relation Signals (6)
Quantum World Model → usesresource → Qutrit
confidence 98% · admits a quantum world model using a single qutrit that reproduces it exactly
Quantum World Model → providesadvantageover → Classical World Model
confidence 97% · We show that this is false for classical world models... In contrast, each such true world admits a quantum world model... that reproduces it exactly
Quantum World Model → achieves → Perfect Alignment
confidence 96% · ensuring that the optimal policies of the real and virtual worlds remain perfectly aligned.
Classical World Model → suffersfrom → Value Deviancy
confidence 95% · We show that this is false for classical world models... Its expected-reward estimates also retain a nonvanishing average error.
Classical World Model → suffersfrom → Loss of Decision Resolution
confidence 95% · every finite classical model fails along the same possible trajectory: it either loses the ability to distinguish actions...
Classical World Model → suffersfrom → Mean Decision Loss
confidence 95% · repeatedly assigns the highest expected reward to suboptimal actions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World models provide digital simulacra of the true world, allowing agents to be trained and tested before costly real-world deployment. At each time step, they receive an action and generate an observation and reward matching the statistics of the true world. In complex environments where present outcomes depend on events far in the past, this requires memory. One might expect that, by increasing memory, we can always build a model accurately enough to align the optimal agent policies of the real and virtual worlds. We show that this is false for classical world models, even when the true world itself is classical. We construct true worlds for which every finite classical model fails along the same possible trajectory: it either loses the ability to distinguish actions when the true world clearly prefers one, or repeatedly assigns the highest expected reward to suboptimal actions. Its expected-reward estimates also retain a nonvanishing average error. In contrast, each such true world admits a quantum world model using a single qutrit that reproduces it exactly: its reward estimates and preferred actions always match those of the true world, ensuring that the optimal policies of the real and virtual worlds remain perfectly aligned.
Tags
Links
- Source: https://arxiv.org/abs/2608.19779v1
- Canonical: https://arxiv.org/abs/2608.19779v1
Trouble viewing inline? Open PDF directly →
Full Text
187,989 characters extracted from source content.
Expand or collapse full text
An Irreducible Quantum Advantage in Aligning World Models with Reality Josep Lumbreras Email: josep.lz@ntu.edu.sg Affiliation: Centre for Quantum Technologies, Nanyang Technological University, Singapore Affiliation: Nanyang Quantum Hub, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore Hailan Ma Email: hailanma0413@gmail.com Affiliation: Centre for Quantum Technologies, Nanyang Technological University, Singapore Affiliation: Nanyang Quantum Hub, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore Jayne Thompson Email: thompson.jayne2@gmail.com Affiliation: College of Computing and Data Science, Nanyang Technological University, Singapore Affiliation: Centre for Quantum Technologies, Nanyang Technological University, Singapore Affiliation: Nanyang Quantum Hub, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore Mile Gu Email: mgu@quantumcomplexity.org Affiliation: Centre for Quantum Technologies, Nanyang Technological University, Singapore Affiliation: Nanyang Quantum Hub, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore Abstract World models provide digital simulacra of the true world, allowing agents to be trained and tested before costly real-world deployment. At each time step, they receive an action and generate an observation and reward matching the statistics of the true world. In complex environments where present outcomes depend on events far in the past, this requires memory. One might expect that, by increasing memory, we can always build a model accurately enough to align the optimal agent policies of the real and virtual worlds. We show that this is false for classical world models, even when the true world itself is classical. We construct true worlds for which every finite classical model fails along the same possible trajectory: it either loses the ability to distinguish actions when the true world clearly prefers one, or repeatedly assigns the highest expected reward to suboptimal actions. Its expected-reward estimates also retain a nonvanishing average error. In contrast, each such true world admits a quantum world model using a single qutrit that reproduces it exactly: its reward estimates and preferred actions always match those of the true world, ensuring that the optimal policies of the real and virtual worlds remain perfectly aligned. From a robot navigating crowded supermarkets to an autonomous vehicle operating on busy streets, a central ambition of reinforcement learning is to train agents to navigate ever more complex environments 58; 48; 55; 28. Such training, however, depends on data that, in physical environments, may be scarce, slow to collect, or economically costly. Meanwhile, a successful agent must be able to act in all potential scenarios, many of which are difficult to test because doing so could endanger people or equipment, or incur significant economic loss 10; 45. World models address this bottleneck by providing surrogate environments in which agents can instead be trained, stress-tested, and benchmarked 59; 24; 26. At their operational core, they accept the same actions as the environment where the agent will ultimately be deployed - the true world - and generate corresponding observations and rewards over successive time steps. But even setting aside how such models are developed, how do physical resources fundamentally constrain what worlds they can represent? Memory is a key constraint. Complex true worlds can be highly non-Markovian: previous actions, outcomes, and rewards can change their response to the same action in the distant future. Reproducing this contextual behavior requires memory (see Fig. 1). Whenever the model receives an action and generates an outcome, it updates this memory, enabling the consequences of a future action to depend on what came before 52; 35; 41. In complex environments, relevant context may extend over long horizons, and memory requirements can grow in tandem. Consequently, memory is a central bottleneck for building world models that remain accurate over extended timescales 53; 11; 39. Figure 1: Classical and quantum world models. The true world represents reality (a). At each time step, it receives an action ata_t and returns an outcome yt+1y_t+1 that consists of an observation and reward. A world model reproduces this input–output behavior through recurrent internal dynamics, updating its memory from one simulated step to the next. In a classical world model (b), a classical register with a finite number of bits carries the memory, and it evolves under classical stochastic updates. In a quantum world model (c), the memory is encoded in a finite number of qubits and evolves through quantum operations. This memory burden has operational consequences. A world model implicitly assigns an action-value to each candidate action, representing its expected cumulative reward if taken. If memory limitations distort a world model, the resulting deviancy can erase the separation between a clearly preferable action and its competitors, or reverse their ordering altogether 22; 18; 5. Even when such failures occur only for specific action-outcome sequences, the effects could be disastrous. An autonomous vehicle may seldom encounter a pedestrian in its path, yet correctly ranking the merits of braking and accelerating is crucial 50; 42. Training or benchmarking agents on a world model that reverses this ranking can induce misalignment that adversaries could exploit to cause highly undesirable outcomes. Could quantum world models, where context is enabled by repeated interactions with a quantum memory, offer unique advantages in removing this misalignment? We answer in the affirmative. We introduce a family of true worlds where any finite-memory classical model exhibits an alignment gap that cannot be removed. Along certain fixed future action-outcome trajectories, the expected future rewards assigned to potential actions retain a non-vanishing average error ε bounded away from 00. Any classical world model either loses the ability to resolve the true world’s preferred action or nominates suboptimal actions as optimal actions at least half of the time - incurring an average reward loss of at least ε . This alignment gap of ε cannot be reduced by increasing the size of the classical memory, as long as it remains finite. We then introduce a class of quantum models with a single qutrit (see Fig. 1 (c)) that can avoid these limitations. Every action it judges optimal is also optimal in the true world in every reachable future trajectory. Our work thus demonstrates an irreducible quantum advantage in aligning world models with certain true worlds. I Framework Modeling True Worlds. The heart of our problem is to build concise models of true worlds—ones that reproduce their statistical black-box behavior from the perspective of any agent interacting with them. We must therefore first define this behavior operationally. We regard a true world (see Fig. 1 (a)) as an environment with which an agent interacts over discrete time steps t=0,1,…t=0,1,…. At time t, the world receives an action At∈A_t and returns an outcome Yt+1=(Ot+1,Rt+1)∈⊆×ℝY_t+1=(O_t+1,R_t+1) ×R, consisting of an observation Ot+1∈O_t+1 and a reward Rt+1∈ℝR_t+1 . Larger rewards represent more desirable outcomes, while costs are represented by negative rewards. Here, we take the action and outcome alphabets A and Y to be finite. Let ht=(at−1,yt)h_t=(a_t-1,y_t) denote the action-outcome pair generated from the tht^th interaction. Each instance of a true world at time t then has a history h←t=h1h2…ht h_t=h_1h_2… h_t. The operational behavior of a true world can then be entirely encapsulated by Pr⋆(y∣h←,a) _ (y h,a), the probability that it emits outcome y when receiving action a. A faithful world model is a machine that reproduces this conditional action-outcome behavior. A standard finite-state classical realization is a controlled hidden Markov model 52; 35; 41 - a framework that underpins applications ranging from speech recognition to biological sequence analysis 4; 52; 16; 37. These machines contain a memory M with a finite set =1,⋯,NS=\1,·s,N\ of N distinct physical states. When receiving an action a, their dynamics can then be completely described by transition elements (Dy(a))ji (D_y^(a) )_ji :=Pr(Yt+1=y,St+1=j|St=i,At=a) := (Y_t+1=y,S_t+1=j|S_t=i,A_t=a ) (1) representing the probability that a machine in memory state St=iS_t=i transitions to j while emitting the outcome y after receiving action a. Given an initial probability distribution zkin=Pr(S0=k)z^in_k= (S_0 = k) over memory states and action sequence a0a1…a_0a_1…, the dynamics (Dy(a))ji (D_y^(a) )_ji then completely determines the model’s probability of emitting outcome Pr(y∣h←,a) _ M(y h,a) on seeing action a at time t for any history h← h (see Appendix A.4 for details). A world model is then faithful if it is operationally indistinguishable from the true world Pr⋆Pr_ , such that Pr(y∣h←,a)=Pr⋆(y∣h←,a) _ M(y h,a)= _ (y h,a) for all h← h and a. Each world model is thus specified by the tuple =(,,,zin,) M= (A,Y,S,z^in,D ), where =Dy(a)D=\D_y^(a)\ is the collection of N×N× N transition matrices describing transition dynamics on the model memory for each action-outcome pair (a,y)(a,y). Physically, such a model operates as a sequence of stochastic interactions on its memory M (see Fig. 1 (b)). Thus, M enables a world model to exhibit complex non-Markovian behavior: without it, a world model’s outcome behavior cannot depend on history. Thus, the number N of distinct configurations available to the memory system M provides a measure of the complexity required to model a true world. 11 1 Specifically, this measure is sometimes referred to as its generative complexity for stochastic models 47. Note that it is distinct from the statistical complexity used to measure the minimal memory needed to store an agent’s belief state of the environment 57. This is because environmental HMMs need not be unifilar, whereas belief state updates are 54. Operationally, N is often referred to as the physical memory dimension of M, defined as the largest number of states that can be perfectly distinguished in a single use 23; 20; 8; 21; 29. Rewards and Value Functions. In practice, a perfectly faithful model can have immense generative complexity. Any practical world model is often an approximation - identifying a model where Pr(y∣h←,a)Pr(y h,a) is a sufficiently good approximation of Pr⋆(y∣h←,a)Pr_ (y h,a). But what constitutes being sufficiently good? To answer this, we first need to review the main impetus for world models: true-world replacements for training and benchmarking agents to obtain policies that yield higher rewards. Specifically, world models are designed to house agents. An agent’s behavior is dictated by their policy: a probability distribution π(a|h←t)π(a| h_t) governing what actions a the agent would take on seeing history h←t h_t. When acting on a world model M with action-outcome response PrM(y|h←,a)Pr_M(y| h,a), each policy induces a sequence of rewards governed by random variables Rt|π,R_t|_π, M governing the reward at each time-step t. Let 0≤γ<10≤γ<1 denote the discount factor, which captures the preference for immediate over future rewards: when γ=0γ=0, only the next reward is considered, whereas as γ→1γ→ 1, rewards are weighted increasingly equally over time. The resulting discounted cumulative reward, or simply the return, is represented by the random variable Gπ,:=∑k=0∞γkRk+1|π,,G_π, M:= _k=0^∞γ^kR_k+1|_π, M, (2) which has played a dominant role in benchmarking the efficacy of a policy in reinforcement learning 58; 51. We can then introduce πoptπ^opt_ M as the theoretical optimal policy in a given world model M, one that achieves the maximum expected return ⟨Gπ,⟩ G_π, M . Here and throughout, policies are understood as history-dependent decision rules: the optimal policy specifies which action to take after every possible history encountered during interaction with the world. For true worlds with input-output response Pr⋆(y|h←,a)Pr_ (y| h,a), we obtain return Gπ,⋆G_π, and true optimal policy π⋆optπ^opt_ . Indeed, since the true world is operationally indistinguishable from a perfectly faithful model =⋆ M= , our subsequent exposition holds both for world models and true worlds. Given a particular history h← h, the reward potential of taking different actions can then be captured by the action-value function. Consider the policy πopt|aπ^opt_ M|_a that involves taking immediate action at=a_t=a at history h← h, and thereafter making decisions according to the optimal policy πoptπ^opt_ M. The action-value function Q,aopt(h←)=⟨∑k=0∞γkRt+k+1|πMopt|a,,h←⟩Q_ M,a^opt( h)= _k=0^∞γ^kR_t+k+1|_π^opt_M|_a, M, h (3) then captures the maximum expected reward we can potentially extract after taking a possibly non-optimal action a according to world model M. Taking the supremum of this quantity over all a then gives us the value function Vopt(h←)=⟨∑k=0∞γkRt+k+1|πopt,,h←⟩V_ M^opt( h)= _k=0^∞γ^kR_t+k+1|_π^opt_ M, M, h (4) that represents the reward potential of that particular history h← h under world model M. If given for the true world ⋆ , these value functions would immediately allow an agent to identify the best action at each time to maximize expected reward. Benchmarking World Models. These quantities immediately provide operationally meaningful methods to assess candidate models. Consider a candidate model M of a true world ⋆ ; an obvious measure is to look at the differences in their action-value and value functions eQ(h←) e_Q M( h) =maxa∈|Q⋆,aopt(h←)−Q,aopt(h←)|, = _a |Q_ ,a^opt( h)-Q_ M,a^opt( h) |, eV(h←) e_V M( h) =|V⋆opt(h←)−Vopt(h←)| = |V_ ^opt( h)-V_ M^opt( h) | (5) Given a particular history h← h, the first captures the maximum disagreement between M and ⋆ on the reward potential of various possible actions. The second captures their disagreement on the reward potential of h← h itself. For such disagreements to be significant, they should not be isolated to single points in time. To formalize this, let the reachable action-outcome trajectory ℱ=h←1,h←2,… F= h_1, h_2,… be an infinite sequence of action-outcome histories that have strictly positive conditional probability of occurrence in the true world. That is, (i) h←t=h1h2⋯ht h_t=h_1h_2·s h_t for some fixed sequence of action–outcome pairs ht=(at−1,yt)h_t=(a_t-1,y_t), and (i) Pr⋆(yt∣h←t−1,at−1)>0 _ (y_t h_t-1,a_t-1)>0 for every t. The quantities e¯Q|ℱ e_Q M|_ F :=lim infT→∞1T∑t=1TeQ(h←t), := _T→∞ 1T _t=1^Te_Q M( h_t), (6) e¯V|ℱ e_V M|_ F :=lim infT→∞1T∑t=1TeV(h←t), := _T→∞ 1T _t=1^Te_V M( h_t), (7) then capture the deviation of assigned action-values and values, averaged across time, for different trajectories that an agent in the true world can experience. Thus, deploying an M where they strongly deviate to benchmark various policies could lead us to very different - and likely erroneous - conclusions about its efficacy. This motivates us to define the following measure of model deviancy: Definition 1 (Value Deviancy). A world model M is ϵε-deviant if there exists a reachable action-outcome trajectory ℱ F such that e¯V|ℱ≥ϵ e_V M|_ F≥ε, and e¯Q|ℱ≥ϵ e_Q M|_ F≥ε. That is, its average disagreement with the true world in the reward potential along this trajectory is at least ϵε for both values and action-values. A second approach to benchmarking candidate world models is to focus on how deviations can lead to different conclusions about optimal agent actions. Along these lines, we first introduce the decision margin, which captures how accurately we need to estimate action-rewards to decide the optimal action based on a world model M. Specifically, given a history h← h, define a=aopt|h←ta=a_ M^opt|_ h_t as the action that leads to the optimal reward according to M, and a′=a′|h←ta =a _ M|_ h_t as the second-best action - the one that attains maximum reward subject to the condition that a′≠a ≠ a. The decision margin, g(h←)=QM,aopt(h←t)−Q,a′opt(h←t)g_ M( h)=Q^opt_M,a( h_t)-Q^opt_ M,a ( h_t) (8) represents the action-value gap between taking the best action vs its closest competitor according to the world model M. When =⋆ M= , the true action-value gap g⋆(h←)g_ ( h) is considered a measure of how robust the optimal policy is in ⋆ , as a perturbation of up to g⋆(h←)/2g_ ( h)/2 cannot change conclusions on optimal action 18; 5. This then allows us to define another potential point of failure for a candidate classical model - loss of decision resolution. Definition 2 (Loss of decision resolution). Let ℱ=h1h2⋯ F=h_1h_2·s be a reachable action–outcome trajectory, with prefixes h←t=h1⋯ht h_t=h_1·s h_t. For ε>0 >0, a world model M loses decision resolution of magnitude ε along ℱ F if, for every δ>0δ>0 and every T∈ℕT , there exists t≥Tt≥ T such that g⋆(h←t)≥εg_ ( h_t)≥ , g(h←t)<δg_ M( h_t)<δ. We say simply that M loses decision resolution along ℱ F if this condition holds for some ε>0 >0. Thus, along one possible continuing interaction, there is always improvement when taking the best action over its closest competitor in the true world. However, a model M suffering loss of decision resolution may rank them as progressively closer in value, such that agents being trained via M will find it increasingly difficult to make the correct decision. A complementary question is whether the model’s preferred action is correct. Let a=aopt|h←ta_ M=a^opt_ M|_ h_t be a selected action that maximize the action-value in a world model M. At a given history h←t h_t, an agent trained to perform optimally on M, when deployed in the true world, would thus suffer a loss of ℓ(h←t) _ M( h_t) :=V⋆opt(h←t)−Q⋆,aopt(h←t), :=V_ ^opt( h_t)-Q^opt_ ,a_ M\!( h_t), (9) even if all subsequent actions are chosen optimally. It is also commonly known as the regret of the current decision and naturally vanishes when M is faithful. Of course, an agent that performs optimally on M can continue to make non-optimal decisions, which is then captured by the mean decision loss: Definition 3 (Mean decision loss). For ε>0 >0, a world model M exhibits mean decision loss of at least ε along a reachable action-outcome trajectory ℱ=h1,h2,… F=h_1,h_2,… if ℓ¯(ℱ) _ M( F) :=lim infT→∞1T∑t=1Tℓ(h←t)≥ϵ := _T→∞ 1T _t=1^T _ M( h_t)≥ε (10) This quantity represents the asymptotic average loss incurred when decisions are optimized using a distorted model. Decision resolution and decision loss capture complementary requirements: a world model should preserve a clear true-world preference, and the action it prefers should be optimal when deployed. I Results Complex True Worlds. Our first result is to establish that true worlds can be very complex. Recall that the models introduced above simply as world models, denoted by M, are classical, with dynamics governed by the stochastic transition matrices Dy(a)D_y^(a) in (1). To make this distinction explicit, we henceforth refer to them as classical world models and denote them by the tuple =(,,,zin,) C= (A,Y,S,z^in,D ), where S is the set of physical states contained in its memory. We say that a classical world is finite if this set is finite - a requirement that must be true for any world model that can be physically realized. Ideally, we would then be able to simulate every true world arbitrarily closely in terms of model deviancy, loss of decision resolution, and mean decision loss. Our first result establishes that for model deviancy, this gap cannot be closed (see Appendix B.5): Result 1. There exist true worlds and a fixed ε>0 >0 such that, for each of these worlds, every finite classical world model C is ε -deviant. This implies that there will always be action-outcome trajectories where our model C predicts action-values and values that disagree with the true world by an average of at least ϵε through the trajectory. Next, we establish that this fallibility of classical models extends to loss of decision resolution, and mean decision loss. To do so, we first introduce a definition of a treacherous true world - one that cannot be replaced by a finite-memory classical model without serious degradations when used to optimise agent actions: Definition 4 (Classically treacherous true world). For ε>0 >0, a true world is classically ε -treacherous if there exists a reachable action–outcome trajectory ℱ F such that, for any finite-memory classical model C, one of the following holds: (a) C loses decision resolution of magnitude at least ε along ℱ F; (b) C exhibits a mean decision loss of at least ε and recommends a suboptimal action at least half the time, asymptotically, along ℱ F. We emphasize that the constant ε applies to every classical memory dimension and every classical world model - and does not approach 00 in the limit of large classical memory. In a classically treacherous true world, no finite-dimensional classical world model reproduces the decision-relevant statistics well enough to support reliable action selection along ℱ F. Its action-values either fail to separate the candidate actions or lead the agent to deploy persistently suboptimal actions in the true world. Our second result establishes that classical treacherous true worlds exist: Result 2. There exist true worlds that are classically ε treacherous - any finite-memory classical model that attempts to model such a true world will either suffer loss of decision resolution of magnitude ε or recommend suboptimal decisions resulting in a mean decision loss of at least ε along certain action-outcome trajectories. For the construction of such true worlds, see Appendix B. The decision separation is proved in Appendix B.4. Thus, replacing such treacherous environments with classical world models for benchmarking agents, agent training, or policy planning may therefore leave the resulting decisions vulnerable to a sophisticated adversary. Quantum world models. Can quantum models alleviate these fundamental limitations? Recall that an N-dimensional classical memory supports N physical states, which we labeled by integers k=1,2,…Nk=1,2,… N. In contrast, a quantum d-level system can prepare such states in quantum superposition, such that each pure state of the system can be described by any normalized superposition |ϕ⟩=∑k=1dck|k⟩ φ=Σ^d_k=1c_k k in a d-dimensional Hilbert space. The basis states |k⟩ k are perfectly distinguishable and play the same role as the N distinct configurations of a classical memory. A general probabilistic mixture in which state |ϕj⟩ _j is prepared with probability pjp_j is described by a positive, unit-trace density operator ρ=∑jpj|ϕj⟩⟨ϕj|ρ= _jp_j _j _j, which generalizes a classical probability distribution over physical states. When the quantum memory receives an action a, its dynamics are completely described by outcome-labelled quantum operations ℰy(a)y∈\E_y^(a)\_y . Each ℰy(a)E_y^(a) is completely positive and trace nonincreasing, while ∑y∈ℰy(a) _y E_y^(a) is trace preserving for every a∈a . Thus, for each action a, the operations ℰy(a)y∈\E_y^(a)\_y form a quantum instrument 49; 17. To make the correspondence with the classical transition elements explicit, let Jt=iJ_t=i denote that the incoming quantum memory is prepared in state |i⟩ i, and let Jt+1=jJ_t+1=j denote the result of reading the outgoing memory in the same basis. The instrument elements then satisfy ⟨j|ℰy(a)(|i⟩⟨i|)|j⟩ jE_y^(a)( i\! i) j :=Pr(Yt+1=y,Jt+1=j∣Jt=i,At=a). := (Y_t+1=y,J_t+1=j J_t=i,A_t=a ). This is the joint probability that the model emits outcome y and its memory is read as j, conditioned on the incoming basis state i and supplied action a. It is the direct quantum counterpart of (Dy(a))ji (D_y^(a) )_ji. Unlike in the classical case, these basis-resolved probabilities do not completely specify the dynamics, since the operations ℰy(a)E_y^(a) also describe the evolution of superpositions and coherences. Given an initial memory prepared in state ρinρ^in and a sequence of supplied actions, the instrument operations completely determine the model’s conditional outcome probabilities Pr(y∣h←,a) _ Q(y h,a) for every generated history h← h. A quantum world model is faithful when it is operationally indistinguishable from the true world, such that Pr(y∣h←,a)=Pr⋆(y∣h←,a) _ Q(y h,a)= _ (y h,a) for every reachable history h← h, action a, and outcome y. Then a quantum world model is specified by the tuple =(,,ℋQ,ρin,) Q= (A,Y,H_Q,ρ^in, E ), where =ℰy(a)(a,y)∈× E=\E_y^(a)\_(a,y) ×Y is the collection of instrument operations describing the memory dynamics for every action–outcome pair (a,y)(a,y). Physically, such a model operates through a sequence of quantum-instrument interactions on its memory, in direct analogy with the stochastic interactions of a classical world model. The number d is its physical memory dimension: the largest number of memory states that can be perfectly distinguished in a single use. Quantum advantage for model-based decisions. We now combine the preceding classical limitations with an exact quantum realization, obtaining a strict separation between classical and quantum world models for the same true world. We begin with its most direct operational consequence: the action ultimately deployed by the agent. A world model generates simulated futures from which the agent evaluates its candidate actions; the action assigned the largest value is then selected for deployment in the true world. The first quantum result shows that the physical realization of the model’s memory can determine whether this procedure identifies a true-world-optimal action. Result 3. For some fixed ε>0 >0, there exists a family of classically ε -treacherous true worlds, each of which admits an exact quantum world model Q with a single qutrit of memory. This separation holds for all discount factors γ∈[0,1)γ∈[0,1), with the same ε . Quantum advantage for value estimation. The previous result concerns the action ultimately selected from the model’s action values. We now ask the more stringent question of how accurately the model reproduces the values themselves. The separation remains robust: no finite classical memory, however large, can eliminate the dimension-independent gap in the action-values and optimal value, whereas the same qutrit world model reproduces them exactly. The advantage therefore cannot be overcome merely by allocating more finite classical storage; it arises from the physical encoding of the model’s memory. Result 4. For the same family of true worlds, there is a fixed ε>0 >0 such that every finite-memory classical world model is ε -deviant, whereas the corresponding single-qutrit quantum world model Q is exact. The last two results describe complementary consequences of the same representational limitation. The first concerns the action selected using the model, while the second concerns the numerical action values used to compare the candidate actions. Increasing the size of a finite classical memory may postpone the discrepancy to longer histories, but cannot remove its asymptotic average. In contrast, the same three-dimensional quantum memory reproduces the conditional dynamics, action values, and model-based decisions exactly. The Methods Section IV gives an overview of the true-world dynamics, why every finite classical model fails to reproduce them, and how we construct the corresponding exact quantum world model. The true world is defined formally in Appendix B.1. The decision separation is proved in Appendix Subsection B.4, while the action-value and optimal-value bounds are proved together in Appendix Subsection B.5. The exact qutrit world model is constructed and verified in Appendix Section C. I Discussion Here, we showed how a world model processes information fundamentally changes its capacity to align with the true world - both in estimating reward potential and identifying the actions needed to realize it. Every finite-dimensional classical model unavoidably assigns erroneous action-values along certain action–outcome trajectories, with a mean error bounded away from zero independently of memory dimension. Along these trajectories, each model either makes vanishing distinctions between candidate actions when the true world has a clear preference or selects a suboptimal action at least half the time. It therefore cannot reliably align model-optimal decisions with reality, and increasing its memory cannot remove this misalignment. A quantum model with a single qutrit, by contrast, reproduces every action-conditioned future after every reachable history, yielding exact values and true-world-optimal actions. To the best of our knowledge, this is the first work to establish such an irreducible quantum advantage in world-model alignment, separating a fixed finite quantum system from all finite-dimensional classical counterparts. The comparison applies at the level of recurrent physical memory in general controlled stochastic input–output machines, encompassing finite hidden-state and partially observable Markov decision process-style generative models 52; 35; 9 and connecting to recurrent methods for partial observability and finite-precision architectures with finite-automaton characterizations 30; 65; 36. The broader consequence of this separation is that the physical medium used to store a world model’s internal state cannot always be treated as merely an implementation detail. Modern research has made major advances in how such states are learned and used for prediction and control, from Dyna-style model-based reinforcement learning to recurrent latent simulators 59; 25; 27; 26; 28, with recent extensions to language-based agents 63. A complementary line learns predictive representations directly in latent space 40; 1; 2; 46. Despite their different architectures and training objectives, these approaches generally assume that the learned state is carried by a conventional classical memory. Our results show that this assumption can impose a fundamental limit: there exist true worlds for which no representation supported by finite classical memory can preserve alignment, regardless of how it is parameterized or learned, whereas the single-qutrit model preserves it exactly. Our separation therefore identifies a physical resource that complements advances in neural representation learning rather than competing with them. Learning determines what internal state a world model constructs; its physical encoding can determine what that state can faithfully represent. A natural next step is to harness this alignment advantage within learned neural world models and understand what our results imply under realistic noise, finite-shot estimation, and training constraints. We also emphasize that our results concern the modeling of entirely classical true worlds: although their internal memories are quantum, our quantum world models interact with agents entirely through classical random variables representing actions, observations, and rewards. The resulting quantum advantage is therefore directly relevant to settings in which conventional world models are used. Beyond this, our alignment advantage may provide another building block towards quantum-enhanced reinforcement learning. Quantizing an agent’s internal memory and processing can yield memory and energetic advantages that grow without bound for suitable families of strategies 15; 61. Meanwhile, coherent quantum access to a world model or environment can enable different trajectories to be explored in superposition, leading to speed-ups in learning 64; 14. It would be exciting to determine how these advantages can be combined, enabling potential simultaneous speedups in learning, world-model alignment, and reduced memory and energy costs during inference. IV Methods IV.1 The resettable FRDN clock Our results use the same true world, and we now state how it is constructed. The true world is built around a resettable clock. After every reset, it independently draws a hidden lifetime L∈ℕ0L _0. If allowed to continue, the clock produces L consecutive Ticks followed by a Break, at which point it resets and draws a new lifetime. Viewed first as an uncontrolled process, each reset starts a new independent run. A run consists of L Ticks followed by a Break and therefore lasts L+1L+1 steps. The Breaks are renewal events, with independent and identically distributed inter-renewal times L+1L+1. The underlying clock is thus a discrete-time renewal process. The renewal process above describes how the clock behaves when each run is simply allowed to unfold. To obtain the true world used in our results, we retain this renewal law but introduce actions that determine whether the current run advances, is tested, or is deliberately reset. Whenever a reset occurs, the clock independently draws a fresh lifetime from the same distribution. The action set is =W,M,PA=\W,M,P\, and the observation set is =T,BO=\T,B\, where T denotes a Tick and B the end of the current run. At each step, the clock receives one action and returns an observation together with an action-dependent reward. Immediately after a reset, the clock has age t=0t=0. Wait (W) allows the current run to continue for one further step. If the clock returns a Tick (T), the run survives and its age increases from t to t+1t+1. If it returns a Break (B), the run ends and the age resets to zero. Thus, the clock’s age is the number of consecutive Wait–Tick pairs since its most recent reset. Maintain (M) represents preventive maintenance. It incurs a fixed cost, ends the current run, and resets the age to zero without testing whether the run would have survived another step. Probe (P) performs precisely this test: at age t, it has the same Tick–Break probabilities as Wait, but the clock resets after either observation. The three actions therefore offer distinct ways of interacting with the clock. Maintain accepts a certain cost in exchange for an immediate reset. Probe acts as a one-step wager on the remaining lifetime: a Tick can produce a positive reward, whereas a Break can incur a cost. Probe may therefore be preferable when another Tick is sufficiently likely, while Maintain may be preferable when a Break is more likely. Wait has a different role because a Tick leaves the current run active and exposes the clock at the next age. The complete outcome kernel and reward assignment are specified in Appendix B.1. See Fig. 2 for an illustrative summary of this true world. Let ℱtick=h1h2⋯ F_tick=h_1h_2·s denote the trajectory in which every action–outcome pair hth_t records a Wait action followed by a Tick. The history after t interactions is h←t=h1h2⋯ht h_t=h_1h_2·s h_t, with h←0=∅ h_0= , and therefore contains t consecutive Wait–Tick interactions following a reset. This is the trajectory used in all four results. Every finite history h←t h_t is reachable, and the decision at clock age t is evaluated conditional on this history. We denote such histories as h←ttick h_t^tick Given h←ttick h_t^tick, the preceding Ticks imply that the hidden lifetime satisfies L≥tL≥ t. If the next action is Wait or Probe, the conditional probability of returning another Tick is therefore S(t) S(t) :=Pr⋆(T∣h←ttick,W)=Pr⋆(T∣h←ttick,P) :=Pr_ (T h_t^tick,W)=Pr_ (T h_t^tick,P) =Pr(L≥t+1∣L≥t)=Pr(L≥t+1)Pr(L≥t). =Pr(L≥ t+1 L≥ t)= (L≥ t+1) (L≥ t). (11) Thus, S(t)S(t) is the age-dependent statistic that enters the evaluation of the available actions at h←t h_t. A world model need not store S(t)S(t) explicitly, but its memory and readout must jointly reproduce this dependence to predict the next observation and assign the actions their correct values. This construction defines a family of true worlds, indexed by the lifetime law and the rewards. We study a concrete member inspired by early examples of stochastic processes with finite-dimensional linear representations from Fox, Rubin, Dharmadhikari and Nadkarni (FRDN) 13; 19; 12. Fix λ∈(0,1/2]λ∈(0,1/2] and α∈ℝα with α/π∉ℚα/π , and set Pr(L=ℓ) (L= ) =λℓsin2(ℓα2),ℓ≥1, =λ ^2\! ( α2 ), ≥ 1, (12) and Pr(L=0)=1−∑ℓ=1∞Pr(L=ℓ) (L=0)=1- _ =1^∞ (L= ). Substituting the above lifetime law (12) into the conditional survival probability (11) gives S(t)=f(tα)S(t)=f(tα) for t≥1t≥ 1, where f is a continuous, nonconstant, and 2π2π-periodic function. Thus, the sinusoidal dependence of the lifetime distribution is inherited by the conditional probability of observing another Tick, producing an oscillatory dependence on the clock age. The closed form of f and its derivation are given in Appendix B.1. The irrational phase increment of discrete ages t in f(tα)f(tα) prevents the age dependence from synchronizing with any finite cycle. Even if the clock ages are divided into any finite collection of regularly repeating classes, the probabilities within each class continue to explore the full profile of f. Operationally, the prediction-relevant information carried by the clock therefore never reduces to a finite periodic label. The rewards translate this predictive structure into decisions. For the reward assignment used in our main results, the true optimal action continues to switch between Probe and Maintain as the clock ages: Probe is preferred when S(t)S(t) lies above a fixed threshold, and Maintain when it lies below. This switching persists for every discount factor γ∈[0,1)γ∈[0,1). It is not essential that these particular two actions compete. Other reward choices for the same renewal clock make the optimal action switch between Wait and Maintain instead, as discussed in Appendix B.4. Thus, the fact that Wait is dominated in our main construction is a convenient way of isolating a clean decision boundary, rather than a structural property of the world. Figure 2: The resettable FRDN clock world. The displayed signs illustrate a representative payoff regime: Wait offers a reward +RW+R_W if the clock advances but risks a failure cost −CW-C_W; Maintain accepts the known cost −CM-C_M of an immediate reset; and Probe also resets, with reward +RP+R_P or cost −CP-C_P determined by whether the clock would have advanced. The diagram thus contrasts risky continuation under Wait, a certain outcome under Maintain, and an outcome-dependent bet under Probe, creating a nontrivial decision problem. IV.2 Why finite classical memory fails and a qutrit succeeds The relevant contrast is how the two types of memory respond to the increasing clock age along the fixed reachable trajectory ℱtick F_tick. Every additional Wait–Tick interaction applies the same update to a classical world model’s memory. Perron–Frobenius theory implies that repeated application of this update to any finite classical memory eventually approaches a finite collection of limiting behaviours 56; 51. Thus, for some finite spacing p, the model’s conditional predictions converge separately along the interlaced sequences of ages t=np+rt=np+r, with r∈0,…,p−1r∈\0,…,p-1\. The true clock does not settle into such a finite pattern. As noted above, the conditional probability of one more Tick satisfies S(t)=f(tα)S(t)=f(tα), where f is continuous, nonconstant, and 2π2π-periodic. Since α/π∉ℚα/π , for every finite spacing p, the phase increment pαpα remains irrational relative to 2π2π. Weyl equidistribution 38 therefore implies that the phases (np+r)α(np+r)α explore the full circle within every interlaced sequence of ages. Consequently, S(np+r)S(np+r) continues to sample the same nonconstant profile of f rather than converging to one limiting probability. The action-dependent rewards transfer this persistent age dependence to the action-values and values. In our construction, Probe and Maintain both reset the clock, so their future contributions are the same and their comparison retains the oscillation of S(t)S(t). The value obtained by selecting the highest-valued action retains a corresponding age dependence. This remains true for every discount factor γ∈[0,1)γ∈[0,1). Combining these facts with Perron–Frobenius theory and Weyl equidistribution gives a positive lower bound on the asymptotic mean action-value and value errors of every finite classical world model. A common bound can be chosen independently of the memory dimension, the particular classical model, and the discount factor. Increasing the finite memory may postpone the discrepancy to later clock ages, but cannot make either asymptotic mean error vanish. This is the ε -deviancy of Definition 1, established in Result 1. The same mismatch also affects which action is assigned the highest value. The true preferred action switches between Probe and Maintain as the clock ages. Every finite classical world model must therefore either become arbitrarily indecisive at increasingly late histories where the true preference remains separated by a fixed positive amount, or recommend a truly suboptimal action at least half of the time along ℱtick F_tick and incur a positive mean decision loss. This is the ε -treachery of Definition 4, established in Result 2. In contrast, the same clock is reproduced by a concrete world model whose memory is a single qutrit. This model tracks the increasing clock age through a phase in its quantum memory. After the first Tick following Wait, the memory lies in the two-dimensional subspace spanned by |0⟩,|1⟩\|0 ,|1 \. Let Π01:=|0⟩⟨0|+|1⟩⟨1| _01:=|0 0|+|1 1| be the projector onto this subspace, and let X and Z denote the corresponding Pauli operators. The unitary phase rotation is Uα=eiαZ/2U_α=e^iα Z/2, while the complete instrument operation associated with a Tick following Wait is ℰtick(W)(ρ) _tick^(W)(ρ) :=λ(e−rXUαerXΠ01)ρ(e−rXUαerXΠ01)†. :=λ (e^-rXU_αe^rX _01 )ρ (e^-rXU_αe^rX _01 ) . (13) Here UαU_α advances the phase by α. The surrounding factors e±rXe^± rX adjust the relative amplitudes so that the phase also determines the Tick probability. Without them, the map ρ↦λUαρUα†ρ λ U_αρ U_α would give the constant Tick probability λ. Because these factors are inverses, adjacent copies cancel when the operation is repeated, allowing the phase rotations to accumulate. The parameter r is chosen so that the resulting probabilities reproduce (11); its value is derived in Appendix C.1. Since α/πα/π is irrational, the accumulated phase never closes into a finite cycle. Different clock ages are instead associated with generally nonorthogonal qutrit states along this phase orbit. Along ℱtick F_tick, this is precisely the operation applied after every Wait–Tick interaction. The qutrit memory therefore follows the same histories h←t h_t along which the finite classical models are assessed. The remaining instrument operations are constructed in Appendix C.1, and their agreement with the true world after every reachable history is verified in Appendix C.3. The finite-classical decision-loss and value-error bounds are proved in Appendices B.4 and B.5, respectively. IV.3 Numerical illustration of the separation We complement the analytical results by fitting classical world models with different memory dimensions N to the Wait–Tick dynamics of the clock. For each N, we optimize the initial probabilities of the N internal memory states and the probabilities of producing a Tick while moving between these states after a Wait action. The models are fitted over the clock ages t=0,…,1000t=0,…,1000, with each age weighted by its probability of occurrence. The complete fitting and evaluation procedure is given in Appendix D. Since the optimization is nonconvex, the curves below show the best models obtained numerically; the separation itself follows from the analytical result above. We write Pr(Tick∣h←,W)Pr_ C(Tick h,W) for the probability returned by a classical model when it is initialized after history h← h. We consider the representative clock parameters λ=0.4λ=0.4 and α=π/2α=π/ 2. Figure 3 compares the true world probability Pr⋆(Tick∣h←ttick,W)Pr_ (Tick h_t^tick,W) with the corresponding probabilities predicted by the fitted classical models. The qutrit world model reproduces the true curve exactly. Increasing N allows a classical model to follow the true probability over a larger initial range of clock ages. At later ages, its prediction loses the nonrepeating dependence on t and approaches the finitely many limiting predictions described above. For the fitted models shown here, the limiting prediction is effectively a single value. An error in the predicted Tick probability is therefore directly an error in the return induced by the world model for Wait. Figure 4 shows the uniform average of this error over h←1tick,…,h←ttick h_1^tick,…, h_t^tick. Since Wait is one of the candidate actions, this quantity lower-bounds the largest action-value error across the candidate actions. Increasing N postpones the discrepancy to longer histories, but Theorem 28 (see Appendix B.5) guarantees that no finite N eliminates its asymptotic average. Figure 3: Conditional Tick probabilities. Probability of one additional Tick after h←t h_t for λ=0.4λ=0.4 and α=π/2α=π/ 2. The gray curve is the true-world probability, which the qutrit world model reproduces exactly. The colored curves show the predictions of the fitted classical models with memory dimensions N=5N=5 (blue), N=50N=50 (orange), and N=500N=500 (green). Increasing N allows the model to reproduce the nonrepeating dependence on the clock age over a larger initial range. At later ages, the predictions approach the finite limiting behavior imposed by finite classical memory. Figure 4: Average Wait action-value error for one-step setting. The one-step rewards are set to RW=CW=CM=CP=1R_W=C_W=C_M=C_P=1 and RP=−1R_P=-1, with γ=0γ=0. Consequently, the plotted quantity for each T is 1T∑t=1T|Q⋆,Wopt(h←t)−QM,Wopt(h←t)| 1T _t=1^T |Q_ ,W^opt( h_t)-Q_M,W^opt( h_t) |. The dashed red line marks the dimension-independent lower bound on its asymptotic average derived in Appendix B.3. Blue, orange, and green correspond to N=5N=5, N=50N=50, and N=500N=500, respectively. Here the average Q-gap represents the errors of classical models, and lower curves mean better models with low error. Larger memories postpone the discrepancy to later clock ages but do not eliminate its asymptotic average. The qutrit world model has zero error for every T; its curve coincides with the horizontal axis and is omitted for clarity. Acknowledgements JL thanks Alessandro Luongo and Aditya Chidambaram for their help in revising the manuscript. This work is supported by the National Research Foundation of Singapore through the NRF Investigatorship Program (Award No. NRF-NRFI09-0010), the National Quantum Office, hosted in A*STAR, under its Centre for Quantum Technologies Funding Initiative (S24Q2d0009), the RIE 2025 AQAS projects S25Q9D001 and S25Q9D002, the Singapore Ministry of Education Tier 1 Grant RT4/23 and RG91/25 and the RIE25 Japan-Singapore Joint Call on Quantum (Project ID H25-MRO3490). HL gratefully acknowledges support from Schmidt Sciences, LLC through the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship. Use of generative AI. The authors derived the main technical results and produced main manuscript drafts. OpenAI’s ChatGPT and Codex were used to assist with correctness checks and proofreading. All content was reviewed and verified by the authors. Code Availability Statement Code and data for reproducing the results in this work are available at https://github.com/tuliplan/quantum-world-model-RL. Roadmap. Section A develops the general framework for true worlds and classical and quantum world models. In particular, Theorem 12 shows that exact world models preserve optimal values, action values and optimal decisions. Section B then constructs the true world and the fixed reachable all-Tick trajectory ℱtick F_tick used to establish all four results. The proofs proceed as follows: • World-model deviancy. Theorems 28 and 29, proved in Subsection B.5, show that every finite classical world model has mean action-value and optimal-value errors bounded below by a common positive constant. This proves Result 1. • Classically treacherous true worlds. Theorem 23 and Corollaries 24 and 25, proved in Subsection B.4, show that every finite classical world model either loses decision resolution or incurs persistent decision loss through suboptimal actions. This proves Result 2. • Quantum advantage for model-based decisions. Section C constructs a world model with a single qutrit and proves in Theorem 30 that it reproduces the FRDN true world exactly after every reachable history. Combining this exact realization with the classical treachery result proves Result 3. • Quantum advantage for value estimation. Combining the classical value-error bounds with the same exact qutrit realization, whose action-value and optimal-value errors vanish after every reachable history, proves Result 4. Section D gives the numerical procedures used for the illustrations. Notation. Throughout the appendix, ℕ=1,2,…N=\1,2,…\, ℕ0=0,1,2,…N_0=\0,1,2,…\, and [N]=1,…,N[N]=\1,…,N\. For a finite set X, let Δ() (X) denote the probability distributions on X; equivalently, ΔN−1=Δ([N]) _N-1= ([N]). Finally, we write (x)+ (x)_+ :=maxx,0, := \x,0\, E 1\E\ :=1,if E holds,0,otherwise, := cases1,&if $E$ holds,\\ 0,&otherwise, cases (14) for the positive part of x∈ℝx and the indicator of an event or condition E, respectively. Appendix A World models and value functions For the definitions below, use the following fixed notation. Let A be a finite action set, let ⊆×ℝY ×R be a finite alphabet of observation–reward outcomes. For y=(oy,ry)∈y=(o_y,r_y) , write oyo_y for its observation component and ryr_y for its reward component. We assume that rewards are bounded: |ry|≤Rmax|r_y|≤ R_ for every y∈y . For t≥0t≥ 0, define ℋt H_t :=(×)t, :=(A×Y)^t, ℋ H :=⋃t≥0ℋt. := _t≥ 0 H_t. (15) A length-t history prefix can be written as h←t h_t =h1h2⋯ht=(a0,y1,…,at−1,yt), =h_1h_2·s h_t=(a_0,y_1,…,a_t-1,y_t), (16) hk h_k :=(ak−1,yk),|h←t|=t, :=(a_k-1,y_k), | h_t|=t, (17) with h←0=∅ h_0= . When no time index is needed, we write h∈ℋh∈ H for a generic finite history. For h∈ℋsh∈ H_s and g∈ℋtg∈ H_t, write h⌢g∈ℋs+th g∈ H_s+t for their concatenation. For a one-step extension by a∈a and y∈y , we continue to use the shorter notation hayhay. A deterministic history string carries no true-world or world-model label. Its generating source is instead indicated by the probability law or by the corresponding random variable. Uppercase letters denote random variables and lowercase letters their realizations. A.1 World models We first define the true world—the environment in reinforcement-learning terminology—and the world model used as a conditional simulator. Definition 5. A true world is specified by the tuple ⋆=(,,ℋ,Pr⋆) W_ =(A,Y, H,Pr_ ), where (i) A is the finite set of actions that the true world can receive; (i) ⊆×ℝY ×R is the finite set of possible observation–reward outcomes y=(o,r)y=(o,r); (i) ℋ H is the common finite action–outcome history space defined above (15); (iv) Pr⋆:ℋ×→Δ()Pr_ : H×A→ (Y) is the conditional outcome law, so that Pr⋆(y∣h,a)Pr_ (y h,a) is the probability of outcome y after history h when the agent takes action a. Its reachable history tree ℋ⋆⊆ℋ H_ H is the smallest set containing ∅ such that hay∈ℋ⋆hay∈ H_ whenever h∈ℋ⋆h∈ H_ , a∈a , y∈y , and Pr⋆(y∣h,a)>0Pr_ (y h,a)>0. Definition 6 (World model). A world model is specified by the tuple =(,,ℳ,min,PrM,TM), M= (A,Y,M,m^in,Pr_M,T_M ), (18) where ℳM is the state space of the model’s recurrent memory, min∈ℳm^in is its initial memory state, A is the set of actions, ⊆×ℝY ×R is the set of observations-rewards, PrM:ℳ×⟶Δ() _M:M×A (Y) (19) is its conditional outcome law, and TM=TM,ya:ℳ⟶ℳ(a,y)∈× T_M= \T_M,y^a:M \_(a,y) ×Y (20) is its family of outcome-conditioned memory updates. Thus, from memory state m and supplied action a, the model generates Y∼PrM(⋅∣m,a)Y _M(· m,a) and updates its memory to TM,Ya(m)T_M,Y^a(m). The value of TM,ya(m)T_M,y^a(m) on a branch for which PrM(y∣m,a)=0Pr_M(y m,a)=0 may be chosen arbitrarily. This abstract representation isolates the recurrent input–output interface needed to define value functions. At this level, PrMPr_M and TMT_M are operational maps and need not be specified independently in a physical realization. For the classical and quantum world models considered below, they are induced, respectively, by the transition matrices Dy(a)D_y^(a) and the instrument operations ℰy(a)E_y^(a), which directly describe the physical dynamics of the model’s memory. An encoder for M is a separate initialization interface EM:ℋ⟶ℳ,EM(∅)=min. E_M: H , E_M( )=m^in. (21) For a world model query following history h, the encoder supplies the memory state EM(h)E_M(h) from which the rollout is initialized. Once initialized, the rollout evolves entirely through PrMPr_M and TMT_M. The encoder belongs to the agent and provides the interface through which the world model is queried. For a query history h, it initializes the model’s recurrent memory in the state EM(h)E_M(h), after which the model generates the rollout through its internal dynamics. The memory dimension of a world model refers exclusively to this recurrent physical memory: it is the largest number of memory states that can be perfectly distinguished without error in a single use 23; 20; 8; 21. A world model is consistent and exact with respect to a true world if PrMPr_M reproduces the outcome probability of the true world and both descriptions of the outcome-conditioned memory updates TMT_M and the encoder EME_M remain synchronized on reachable true-world branches. We formalize this notion below. Definition 7 (Predictive consistency and exactness). Let M be a world model and let EME_M be an encoder for it. The pair (,EM)( M,E_M) is predictively consistent with ⋆ W_ on ⊆ℋ G H if, for every h∈h∈ G, a∈a , and y∈y , PrM(y∣EM(h),a)=Pr⋆(y∣h,a). _M(y E_M(h),a)=Pr_ (y h,a). (22) The pair is exact on G if it is predictively consistent there and, whenever Pr⋆(y∣h,a)>0Pr_ (y h,a)>0, TM,ya(EM(h))=EM(hay). T_M,y^a\! (E_M(h) )=E_M(hay). (23) It is an exact realization of the true world if it is exact on the reachable history tree ℋ⋆ H_ . When the encoder is fixed by context, we use the shorter statement that M is exact relative to EME_M. A.2 Value functions in world models Throughout this subsection, fix an abstract recurrent world model M together with an encoder EME_M. The history-indexed rollout laws and value functions therefore depend on the pair (,EM)( M,E_M). For notational economy, we suppress the encoder dependence from their subscripts. The world model generates simulated outcomes, whereas the agent converts the resulting reward sequences into scores or value functions in order to evaluate its current policy. Now we formally define the concept of policy. Definition 8. A policy is a stochastic kernel from the complete classical history space to the action set, equivalently a map π:ℋ⟶Δ(),g⟼π(⋅∣g). π: H (A), g π(· g). (24) Thus, for every g∈ℋg∈ H, π(a∣g) π(a g) ≥0, ≥ 0, ∑a∈π(a∣g) _a π(a g) =1. =1. (25) Let Πℋ:=π:ℋ⟶Δ() _ H:= \π: H (A) \ (26) denote the class of all such policies. The same policy π∈Πℋπ∈ _ H is used in the true-world and world-model continuation laws below. During simulation, it is conditioned on the complete query history extended by the generated continuation. Fix a query history h∈ℋh∈ H and a policy π∈Πℋπ∈ _ H. A model-generated rollout begins from H0M H_0^M :=h, :=h, M0 M_0 :=EM(h). :=E_M(h). (27) At rollout depth k≥0k≥ 0, AkM A_k^M ∼π(⋅∣HkM), π(· H_k^M), (28) Yk+1M=(Ok+1M,Rk+1M) Y_k+1^M=(O_k+1^M,R_k+1^M) ∼PrM(⋅∣Mk,AkM), _M(· M_k,A_k^M), (29) Hk+1M H_k+1^M :=HkMAkMYk+1M, :=H_k^MA_k^MY_k+1^M, (30) Mk+1 M_k+1 :=TM,Yk+1MAkM(Mk). :=T_M,Y_k+1^M^A_k^M(M_k). (31) Thus, the policy conditions on the accumulated classical history HkMH_k^M, whereas the world model generates its next outcome using only its recurrent memory MkM_k. The conditional true-world benchmark begins from the same query history, H0⋆ H_0 :=h, :=h, (32) and evolves according to Ak⋆ A_k ∼π(⋅∣Hk⋆), π(· H_k ), (33) Yk+1⋆=(Ok+1⋆,Rk+1⋆) Y_k+1 =(O_k+1 ,R_k+1 ) ∼Pr⋆(⋅∣Hk⋆,Ak⋆), _ (· H_k ,A_k ), (34) Hk+1⋆ H_k+1 :=Hk⋆Ak⋆Yk+1⋆. :=H_k A_k Y_k+1 . (35) More explicitly, let ζn=(a0,y1,…,an−1,yn) _n=(a_0,y_1,…,a_n-1,y_n) (36) be a deterministic continuation of length n∈ℕ0n _0 for a history h. For 1≤k≤n1≤ k≤ n, let ζk _k :=(a0,y1,…,ak−1,yk), :=(a_0,y_1,…,a_k-1,y_k), ζ0 _0 :=∅ := (37) denote its length-k prefix. The complete history at rollout depth k is then h⌢ζkh _k. Define the model-memory state associated with this continuation recursively m0(h,ζ0) m_0(h, _0) :=EM(h), :=E_M(h), mk+1(h,ζk+1) m_k+1(h, _k+1) :=TM,yk+1ak(mk(h,ζk)), :=T_M,y_k+1^a_k (m_k(h, _k) ), (38) for 0≤k<n.0≤ k<n. The probability of ζn _n under the world model rollout is PrM,πh(ζn)=∏k=0n−1π(ak∣h⌢ζk)PrM(yk+1∣mk(h,ζk),ak). _M,π^h( _n)= _k=0^n-1π\! (a_k h _k )Pr_M\! (y_k+1 m_k(h, _k),a_k ). (39) The corresponding true-world probability continuation law after h is Pr⋆,πh(ζn)=∏k=0n−1π(ak∣h⌢ζk)Pr⋆(yk+1∣h⌢ζk,ak). _ ,π^h( _n)= _k=0^n-1π\! (a_k h _k )Pr_ \! (y_k+1 h _k,a_k ). (40) The continuation laws above determine the corresponding rollout expectations, denoted by M,πhE_M,π^h and ⋆,πhE_ ,π^h. For X∈M,⋆X∈\M, \ and a∈a , we write X,πh,aE_X,π^h,a for expectation under the corresponding rollout initialized at h, with only the first action-selection rule replaced by A0=aalmost surely. A_0=a surely. (41) All outcome-generation and memory-update rules remain unchanged, and the policy π supplies the actions from rollout depth k=1k=1 onward. In the fixed-first-action laws, π(⋅∣h)π(· h) is not used: the policy controls only the actions at rollout depths k≥1k≥ 1. Now we define the policy values for the true world and world model. Unless stated otherwise, all quantities are defined for a fixed discount factor γ∈[0,1)γ∈[0,1). Definition 9 (Policy values). Let π∈Πℋπ∈ _ H, h∈ℋh∈ H, γ∈[0,1)γ∈[0,1) and a∈a . For X∈M,⋆X∈\M, \, define VXπ(h) V_X^π(h) :=X,πh[∑k=0∞γkRk+1], :=E_X,π^h [ _k=0^∞γ^kR_k+1 ], (42) QXπ(h,a) Q_X^π(h,a) :=X,πh,a[∑k=0∞γkRk+1]. :=E_X,π^h,a [ _k=0^∞γ^kR_k+1 ]. (43) Thus, VXπ(h)V_X^π(h) follows π from the first action, whereas QXπ(h,a)Q_X^π(h,a) takes a first and follows π thereafter. For later use, define the expected one-step rewards of the model in memory state m and of the true world after history h by rM(m,a) r_M(m,a) :=∑y∈PrM(y∣m,a)ry, := _y Pr_M(y m,a)r_y, (44) r⋆(h,a) r_ (h,a) :=∑y∈Pr⋆(y∣h,a)ry. := _y Pr_ (y h,a)r_y. (45) When γ=0γ=0, the action-values reduce to these expected one-step rewards. A.2.1 Optimal values and Bellman equations The material in this subsection is standard in reinforcement learning and dynamic programming. We briefly recall the definitions and results needed below, including Bellman recursions, contraction and fixed-point characterizations, and their connection with policy optimization. For complete treatments and rigorous proofs, see 58; 51; 7; 60. The true world is Markov when the complete history is used as its state, whereas a rollout of the world model is Markov in its recurrent memory state. Accordingly, for bounded functions u:ℋ→ℝu: H and v:ℳ→ℝv:M , define (⋆u)(h):=maxa∈[ (T_ u)(h):= _a [ r⋆(h,a) r_ (h,a) +γ∑y∈Pr⋆(y∣h,a)u(hay)], +γ _y Pr_ (y h,a)u(hay) ], (46) and (Mv)(m):=maxa∈[ (T_Mv)(m):= _a [ rM(m,a) r_M(m,a) +γ∑y∈PrM(y∣m,a)v(TM,ya(m))]. +γ _y Pr_M(y m,a)v(T_M,y^a(m)) ]. (47) Since rewards are bounded and γ<1γ<1, both operators are γ-contractions in the supremum norm. They therefore have unique bounded fixed points. Denote these fixed points temporarily by U⋆U_ and uMu_M, U⋆ U_ =⋆U⋆, =T_ U_ , uM u_M =MuM. =T_Mu_M. (48) We next identify these fixed points using the policy values defined above. Let Gn:=∑k=0n−1γkRk+1 G_n:= _k=0^n-1γ^kR_k+1 (49) be the n-step return and G∞:=∑k=0∞γkRk+1G_∞:= _k=0^∞γ^kR_k+1. Backward induction gives (⋆n0)(h) (T_ ^n0)(h) =supπ∈Πℋ⋆,πh[Gn], = _π∈ _ HE_ ,π^h[G_n], (50) (Mn0)(EM(h)) (T_M^n0)(E_M(h)) =supπ∈ΠℋM,πh[Gn], = _π∈ _ HE_M,π^h[G_n], (51) where 00 denotes the zero function. Indeed, conditioning on the first action and outcome produces the Bellman recursions: the first action is optimized, and after each outcome the later actions may be chosen separately on the resulting history branch. Conversely, these choices define a single history policy because each generated outcome is included in the history observed by the policy. If |Rk+1|≤Rmax|R_k+1|≤ R_ for every k≥0k≥ 0, |G∞−Gn|≤Rmaxγn1−γ |G_∞-G_n|≤ R_ γ^n1-γ (52) uniformly over policies and initial conditions. Note that since γ∈[0,1)γ∈[0,1) we have G∞G_∞ bounded. Hence the finite-horizon suprema converge to the corresponding infinite-horizon suprema. Since value iteration also converges to the unique fixed points, U⋆(h) U_ (h) =supπ∈ΠℋV⋆π(h), = _π∈ _ HV_ ^π(h), (53) uM(EM(h)) u_M(E_M(h)) =supπ∈ΠℋVMπ(h). = _π∈ _ HV_M^π(h). (54) The same argument with the first action fixed gives the corresponding action-value identities. We can therefore introduce the optimal quantities without ambiguity. Definition 10 (Optimal true-world value functions). For h∈ℋh∈ H and a∈a , define V⋆opt(h) V_ ^opt(h) :=supπ∈ΠℋV⋆π(h), := _π∈ _ HV_ ^π(h), (55) Q⋆opt(h,a) Q_ ^opt(h,a) :=supπ∈ΠℋQ⋆π(h,a). := _π∈ _ HQ_ ^π(h,a). (56) The fixed-point identification established above gives V⋆opt(h)=U⋆(h). V_ ^opt(h)=U_ (h). (57) Consequently, the true-world optimal action-value satisfies Q⋆opt(h,a)=r⋆(h,a)+γ∑y∈Pr⋆(y∣h,a)V⋆opt(hay), Q_ ^opt(h,a)=r_ (h,a)+γ _y Pr_ (y h,a)V_ ^opt(hay), (58) and V⋆opt(h)=maxa∈Q⋆opt(h,a). V_ ^opt(h)= _a Q_ ^opt(h,a). (59) Now we define the optimal values in the world model. Definition 11 (Optimal world-model value functions). For h∈ℋh∈ H and a∈a , define VMopt(h) V_M^opt(h) :=supπ∈ΠℋVMπ(h), := _π∈ _ HV_M^π(h), (60) QMopt(h,a) Q_M^opt(h,a) :=supπ∈ΠℋQMπ(h,a). := _π∈ _ HQ_M^π(h,a). (61) The world-model Bellman operator acts on memory space. We therefore write its fixed point and associated action-value function as vMopt(m) v_M^opt(m) :=uM(m), :=u_M(m), (62) qMopt(m,a) q_M^opt(m,a) :=rM(m,a)+γ∑y∈PrM(y∣m,a)vMopt(TM,ya(m)). :=r_M(m,a)+γ _y Pr_M(y m,a)v_M^opt (T_M,y^a(m) ). (63) They satisfy vMopt(m)=maxa∈qMopt(m,a). v_M^opt(m)= _a q_M^opt(m,a). (64) The history-indexed value functions are related to these memory-state functions through the encoder VMopt(h) V_M^opt(h) =vMopt(EM(h)), =v_M^opt(E_M(h)), (65) QMopt(h,a) Q_M^opt(h,a) =qMopt(EM(h),a). =q_M^opt(E_M(h),a). (66) Hence VMopt(h)=maxa∈QMopt(h,a). V_M^opt(h)= _a Q_M^opt(h,a). (67) All suprema above are pointwise in the displayed history h and range over Πℋ _ H. For a value Vopt(h)V^opt(h), the policy selects the first and all subsequent actions. For an action-value Qopt(h,a)Q^opt(h,a), the displayed first action a is fixed and the policy selects the actions from rollout depth k=1k=1 onward. Thus Bellman optimality defines a function of the current state. For the true world, the state is the complete history. For the model, the optimal history-indexed quantities depend on the supplied history through the memory state EM(h)E_M(h). A.3 Exact models preserve value functions The next theorem shows that exactness preserves the complete trajectory law under every history policy, and therefore preserves both policy-evaluation and optimal value functions. Theorem 12. Let M be a world model and let EME_M be an encoder such that (,EM)( M,E_M) is an exact realization of ⋆ W_ in the sense of Definition 7. Then, for every π∈Πℋπ∈ _ H, every reachable query history h∈ℋ⋆h∈ H_ , every n≥0n≥ 0, and every continuation ζn _n, PrM,πh(ζn)=Pr⋆,πh(ζn). _M,π^h( _n)= _ ,π^h( _n). (68) The analogous equality holds when the first action is forced to a∈a in both rollouts. Consequently, VMπ(h) V_M^π(h) =V⋆π(h), =V_ ^π(h), QMπ(h,a) Q_M^π(h,a) =Q⋆π(h,a). =Q_ ^π(h,a). (69) Moreover, VMopt(h) V_M^opt(h) =vMopt(EM(h))=V⋆opt(h), =v_M^opt(E_M(h))=V_ ^opt(h), (70) QMopt(h,a) Q_M^opt(h,a) =qMopt(EM(h),a)=Q⋆opt(h,a). =q_M^opt(E_M(h),a)=Q_ ^opt(h,a). (71) Consequently, the sets of optimal actions also agree argmaxa∈QMopt(h,a)=argmaxa∈Q⋆opt(h,a). _a Q_M^opt(h,a)= _a Q_ ^opt(h,a). (72) Proof. Fix h∈ℋ⋆h∈ H_ and π∈Πℋπ∈ _ H. For a continuation prefix ζk _k, set gk g_k :=h⌢ζk, :=h _k, μk _k :=mk(h,ζk). :=m_k(h, _k). (73) We prove simultaneously that the model and true-world probabilities of every prefix agree and that every prefix having positive common probability satisfies gk g_k ∈ℋ⋆, ∈ H_ , μk _k =EM(gk). =E_M(g_k). (74) At k=0k=0, both laws assign probability one to the empty continuation, g0=h∈ℋ⋆g_0=h∈ H_ , and μ0=EM(h) _0=E_M(h). Suppose the claims hold for ζk _k. If its common probability is zero, every extension of that prefix has probability zero under both laws. Otherwise, exactness of the pair (,EM)( M,E_M) gives PrM(y∣μk,a)=PrM(y∣EM(gk),a)=Pr⋆(y∣gk,a) _M(y _k,a)=Pr_M(y E_M(g_k),a)=Pr_ (y g_k,a) (75) for every a∈a and y∈y . Together with the induction hypothesis, multiplying by the common prefix probability and the common policy factor gives PrM,πh(ζk+1) _M,π^h( _k+1) =PrM,πh(ζk)π(a∣gk)PrM(y∣μk,a) = _M,π^h( _k)π(a g_k)Pr_M(y _k,a) =Pr⋆,πh(ζk)π(a∣gk)Pr⋆(y∣gk,a) = _ ,π^h( _k)π(a g_k)Pr_ (y g_k,a) =Pr⋆,πh(ζk+1), = _ ,π^h( _k+1), (76) If the extended prefix has positive probability, then Pr⋆(y∣gk,a)>0Pr_ (y g_k,a)>0. Hence gkay∈ℋ⋆g_kay∈ H_ , and the memory-update condition in (23) gives TM,ya(μk)=TM,ya(EM(gk))=EM(gkay). T_M,y^a( _k)=T_M,y^a(E_M(g_k))=E_M(g_kay). (77) This completes the induction and proves (68). The same induction applies when the first action is fixed to a∈a . At rollout depth zero, both the model and true-world processes receive the same supplied action a; from depth one onward, both use the same policy π. Therefore VMπ(h) V_M^π(h) =V⋆π(h), =V_ ^π(h), QMπ(h,a) Q_M^π(h,a) =Q⋆π(h,a). =Q_ ^π(h,a). (78) Taking suprema over Πℋ _ H gives the optimal value and action-value equalities. The memory-state identities follow from (65) and (66). ∎ A.4 Finite-dimensional classical world models Here we formally define classical world models. We state some notation that we will use for classical world models. For real vectors u,v∈ℝdu,v ^d, write ⟨u,v⟩ u,v for their inner product, and write ∈ℝd1 ^d for the all-ones vector. The identity matrix is d×d∈ℝd×dI_d× d ^d× d. A non-negative matrix D∈ℝ≥0d×dD ^d× d_≥ 0 is column-substochastic if ⟨,Du⟩≤⟨,u⟩ 1,Du ≤ 1,u for every non-negative vector u∈ℝ≥0du ^d_≥ 0. We call D column-stochastic if the inequality is an equality for every u∈ℝ≥0du ^d_≥ 0. When the column convention is clear, we simply say substochastic and stochastic. Now we formally define our notion of classical world model. A classical world model of memory dimension N uses a stochastic physical memory with N perfectly distinguishable configurations, labeled by i∈[N]i∈[N]. A general state of this memory is represented by a probability vector z∈Δ([N])z∈ ([N]). In POMDP terminology, z is the distribution over the N internal configurations; physically, its entries are the preparation probabilities of one N-state memory system. We use N for this physical memory dimension, independently of the continuum of possible state distributions. The dynamics of this memory take the standard form of a finite-state POMDP, equivalently the non-negative hidden-state subclass of a controlled observable-operator model 33; 62. This representation assigns a non-negative matrix to each action–outcome branch. It is particularly convenient here because both branch probabilities and posterior distributions over states are obtained from products of the same matrices, in direct parallel with the quantum-instrument representation that we use for quantum world models 62; 32; 17. Related operator representations have also been useful in recent statistical analyses of POMDPs 34; 43. Definition 13. An N-dimensional classical world model is specified by N C_N =(,,[N],zin,), = (A,Y,[N],z^in,D ), :=Dy(a)(a,y)∈×, := \D_y^(a) \_(a,y) ×Y, (79) where [N]=1,…,N[N]=\1,…,N\ labels the perfectly distinguishable physical memory configurations, zin∈Δ([N])z^in∈ ([N]) is the initial memory state, A is the set of actions, ⊆×ℝY ×R is the set of observations-rewards, and each Dy(a)∈ℝ≥0N×ND_y^(a) _≥ 0^N× N is column-substochastic, and the family satisfies (∑y∈Dy(a))=for every a∈. 1 T ( _y D_y^(a) )=1 T every a . (80) The entry (Dy(a))ji(D_y^(a))_ji is the joint probability that the model generates y and moves from memory configuration i to configuration j when supplied with action a. The corresponding abstract memory-state space is C=Δ([N])Z_C= ([N]). The branch matrices induce the outcome law PrC(y∣z,a) _C(y z,a) :=⟨,Dy(a)z⟩ := 1,D_y^(a)z (81) and, whenever this probability is nonzero, the conditional memory update TC,ya(z) T_C,y^a(z) :=Dy(a)z⟨,Dy(a)z⟩. := D_y^(a)z 1,D_y^(a)z . (82) After fixing zrefz_ref, N C_N induces the abstract recurrent tuple (,,C,zin,PrC,TC)(A,Y,Z_C,z^in,Pr_C,T_C) required by Definition 6. The maps PrCPr_C and TCT_C are derived from D and the fixed zero-probability convention. For history-indexed queries, write z0:=zinz_0:=z^in and, for h=(a0,y1,…,at−1,yt)h=(a_0,y_1,…,a_t-1,y_t), define Dh D_h :=Dyt(at−1)⋯Dy1(a0), :=D_y_t^(a_t-1)·s D_y_1^(a_0), D∅ D_ :=N×N. :=I_N× N. (83) The encoder associated with the model is EC(h)=Dhz0⟨,Dhz0⟩ E_C(h)= D_hz_0 1,D_hz_0 (84) whenever the denominator is nonzero, and EC(h)=zrefE_C(h)=z_ref otherwise. This encoder is derived from the model’s branch dynamics and is the standard forward-filtering preparation obtained by conditioning the initial state zinz^in through the same branch matrices Dy(a)D_y^(a) that govern the subsequent simulation 52; 35; 62. Moreover, for strictly positive probability histories, its form is unique in order to fulfill relation (23) associated to the dynamics of the matrices D. The encoder remains a separate interface and is not a component of N C_N, but it is not an additional free parameter in the classical results. Throughout the classical results below, all history-indexed quantities use this canonical encoder. A.5 Finite-dimensional quantum world models A quantum world model of memory dimension d uses a d-level physical memory with Hilbert space ℋQ≃ℂdH_Q ^d. Write (ℋQ) L(H_Q) for the linear operators on ℋQH_Q and define (ℋQ):=ρ∈(ℋQ):ρ⪰0,Trρ=1. (H_Q):= \ρ∈ L(H_Q):ρ 0,\ Trρ=1 \. (85) Its states are density operators ρ∈(ℋQ)ρ (H_Q). A set of quantum states can be perfectly distinguished in a single use only when their supports are mutually orthogonal, so such a set contains at most d states. For each candidate action, a quantum world model replaces non-negative branch matrices by quantum instrument elements, forming a quantum instrument for each action 49; 17. The same definitions have also been used to describe quantum POMDPs 3; 44. Definition 14. A d-dimensional quantum world model is specified by d Q_d =(,,ℋQ,ρin,), = (A,Y,H_Q,ρ^in, E ), E :=ℰy(a)(a,y)∈×, := \E_y^(a) \_(a,y) ×Y, (86) where ℋQ≃ℂdH_Q ^d is the Hilbert space of the memory, ρin∈(ℋQ)ρ^in (H_Q) is its initial state, and each ℰy(a):(ℋQ)⟶(ℋQ) _y^(a): L(H_Q) L(H_Q) (87) is completely positive and trace nonincreasing. For every a∈a , ∑y∈ℰy(a) _y E_y^(a) is trace preserving, so ℰy(a)y∈\E_y^(a)\_y forms a quantum instrument. The corresponding abstract memory-state space is Q=(ℋQ)Z_Q=S(H_Q). The instrument operations induce the outcome law PrQ(y∣ρ,a) _Q(y ρ,a) :=Tr[ℰy(a)(ρ)] :=Tr\! [E_y^(a)(ρ) ] (88) and, whenever this probability is nonzero, the conditional memory update TQ,ya(ρ) T_Q,y^a(ρ) :=ℰy(a)(ρ)Tr[ℰy(a)(ρ)]. := E_y^(a)(ρ)Tr[E_y^(a)(ρ)]. (89) Choose a reference state ρref∈Q _ref _Q and set TQ,ya(ρ)=ρrefT_Q,y^a(ρ)= _ref on zero-probability branches. Thus, PrQPr_Q and TQT_Q provide the abstract recurrent representation required by Definition 6; they are induced by E. For history-indexed queries, write ρ0:=ρin _0:=ρ^in and, for h=(a0,y1,…,at−1,yt)h=(a_0,y_1,…,a_t-1,y_t), define ℰh _h :=ℰyt(at−1)∘⋯∘ℰy1(a0), :=E_y_t^(a_t-1) ·s _y_1^(a_0), ℰ∅ _ :=id. :=id. (90) The encoder associated with the instrument dynamics is EQ(h)=ℰh(ρ0)Tr[ℰh(ρ0)] E_Q(h)= E_h( _0)Tr[E_h( _0)] (91) whenever the denominator is nonzero, and EQ(h)=ρrefE_Q(h)= _ref otherwise. The expression (91) identifies the memory state associated with a history. For every history h satisfying Tr[ℰh(ρ0)]>0Tr[E_h( _0)]>0, it obeys the update relation (23) induced by E. On zero-probability histories, ρref _ref is an arbitrary fixed convention. The classical and quantum specifications above induce operational recurrent representations of the form in Definition 6. In each case, the outcome law and conditional memory update are derived from the outcome-labelled physical dynamics rather than supplied as independent model data. Their respective memory dimensions are N and d, independently of the number of statistical states in Δ([N]) ([N]) or density operators in (ℋQ)S(H_Q). Appendix B The FRDN true world and finite-dimensional classical gap This section constructs the true world used to prove our main results. We begin with the Fox–Rubin–Dharmadhikari–Nadkarni (FRDN) renewal process 13; 19; 12; 62. The original process generates a stochastic sequence and has neither actions nor rewards. Its output probabilities admit a finite-dimensional linear realization: they can be computed through products of fixed finite-dimensional matrices. Nevertheless, the same probabilities cannot be generated by any finite-state hidden Markov model. We use them below to define the action-conditioned dynamics of a controlled world and assign rewards to its possible outcomes. B.1 The FRDN true world We first specify the renewal law that determines the outcome probabilities of the true world between two resets. We will equip our agent with the actions Wait, Probe and Maintain. Suppose that, after a reset, the agent repeatedly chooses Wait. The true world draws an auxiliary random lifetime L∈ℕ0L _0. Conditional on L=ℓL= , it produces Tick (an observation) on the first ℓ Wait actions and Break (another observation) on the next one, after which the process resets. At each reset, a fresh independent copy of L is drawn. Let pℓ:=Pr(L=ℓ)p_ :=Pr(L= ) be the probability that a run contains exactly ℓ Ticks. For the FRDN process, fix λ∈(0,1/2]λ∈(0,1/2] and α∈ℝα such that α/π∉ℚα/π , and define pℓ p_ :=λℓsin2(ℓα2),ℓ≥1, :=λ ^2\! ( α2 ), ≥ 1, (92) p0 p_0 :=1−∑ℓ=1∞pℓ. :=1- _ =1^∞p_ . (93) This is a probability law because ∑ℓ≥1pℓ≤λ/(1−λ)≤1 _ ≥ 1p_ ≤λ/(1-λ)≤ 1. After t consecutive Ticks, the observed history implies that L≥tL≥ t. We therefore define the survival probability Φ(t) (t) :=Pr(L≥t)=∑ℓ=t∞pℓ,t≥1, :=Pr(L≥ t)= _ =t^∞p_ , t≥ 1, Φ(0) (0) :=1, :=1, (94) and the conditional probability of one additional Tick, S(t) S(t) :=Pr(L≥t+1∣L≥t)=Φ(t+1)Φ(t), :=Pr(L≥ t+1 L≥ t)= (t+1) (t), (95) for t≥0t≥ 0. The irrationality assumption implies pℓ>0p_ >0 for every ℓ≥1 ≥ 1. Consequently, Φ(t)>0 (t)>0 for every t, so the conditional probability S(t)S(t) in (95) is well defined. These quantities are probabilities: pℓp_ is a probability mass, Φ(t) (t) is a survival probability, and S(t)S(t) is a conditional probability. They determine the true-world kernel below: after t consecutive Wait–Tick outcomes since the last reset, the next Wait produces Tick with probability S(t)S(t) and Break with probability 1−S(t)1-S(t). For t≥1t≥ 1, summing the geometric series gives Φ(t) (t) =λt[A−Bcos(tα+φ)], =λ^t [A-B (tα+ ) ], (96) A A :=12(1−λ),B:=12|1−λeiα|, := 12(1-λ),\,\,B:= 12|1-λ e^iα|, (97) where (1−λeiα)−1=|1−λeiα|−1eiφ(1-λ e^iα)^-1=|1-λ e^iα|^-1e^i . Since A>B>0A>B>0, S(t) S(t) =f(tα),t≥1, =f(tα), t≥ 1, (98) f(x) f(x) :=λA−Bcos(x+α+φ)A−Bcos(x+φ). :=λ A-B (x+α+ )A-B (x+ ). (99) The function f is continuous, 2π2π-periodic, and nonconstant. Moreover, f(x)∈[0,1]f(x)∈[0,1]: the irrational orbit tαmod2π\tα 2π\ is dense since α/π∉ℚα/π , f(tα)=S(t)∈[0,1]f(tα)=S(t)∈[0,1], and f is continuous. For t≥1t≥ 1, S(t)=f(tα)S(t)=f(tα). The value of S(0)S(0) is defined separately and plays no role in the asymptotic lower bound. We now use this probability to define the true controlled world. For an outcome label yoay_o^a, the superscript records the action a, while the subscript records the observation o. The action index is mnemonic and is not an additional component of the observation–reward pair. For special reward choices, labels associated with different actions may denote the same element of the outcome alphabet; this is unambiguous because each branch is indexed by both its action and its outcome. Definition 15 (FRDN true world). In the sense of Definition 5, the FRDN true controlled world is ⋆FRDN W_ ^FRDN :=(FRDN,FRDN,ℋ,Pr⋆). := (A_FRDN,Y_FRDN, H,Pr_ ). (100) Its action set is FRDN _FRDN :=W,M,P, :=\W,M,P\, (101) where W, M, and P denote Wait, Maintain, and Probe, respectively. Let :=T,BO:=\T,B\ be the observation set, where T denotes Tick and B denotes Break. Fix parameters CW,CM,CP>0C_W,C_M,C_P>0 and RW,RP∈ℝR_W,R_P , and define the possible observation–reward outcomes by yTW y_T^W :=(T,RW), :=(T,R_W), yBW y_B^W :=(B,−CW), :=(B,-C_W), yBM y_B^M :=(B,−CM), :=(B,-C_M), (102) yTP y_T^P :=(T,RP), :=(T,R_P), yBP y_B^P :=(B,−CP). :=(B,-C_P). (103) The outcome set is FRDN _FRDN :=yTW,yBW,yBM,yTP,yBP. :=\y_T^W,y_B^W,y_B^M,y_T^P,y_B^P\. (104) For h∈ℋh∈ H, let ℓ(h) (h) be the length of the terminal sequence of consecutive Wait–Tick action–outcome pairs in h. Equivalently, ℓ(∅) ( ) :=0, :=0, (105) ℓ(hay) (hay) :=ℓ(h)+1,a=Wandy=yTW,0,otherwise. := cases (h)+1,&a=W\ and\ y=y_T^W,\\ 0,&otherwise. cases (106) With S(t)S(t) denoting the conditional Tick probability defined in (95), the true outcome law is Pr⋆(y∣h,W) _ (y h,W) :=S(ℓ(h)),y=yTW,1−S(ℓ(h)),y=yBW,0,otherwise, := casesS( (h)),&y=y_T^W,\\ 1-S( (h)),&y=y_B^W,\\ 0,&otherwise, cases (107) Pr⋆(y∣h,M) _ (y h,M) :=1,y=yBM,0,otherwise, := cases1,&y=y_B^M,\\ 0,&otherwise, cases (108) Pr⋆(y∣h,P) _ (y h,P) :=S(ℓ(h)),y=yTP,1−S(ℓ(h)),y=yBP,0,otherwise. := casesS( (h)),&y=y_T^P,\\ 1-S( (h)),&y=y_B^P,\\ 0,&otherwise. cases (109) Here and throughout the FRDN construction, the history space is instantiated using the FRDN alphabets: ℋ:=⋃t≥0(FRDN×FRDN)t. H:= _t≥ 0 (A_FRDN×Y_FRDN )^t. (110) The lifetime L is an auxiliary construction used to specify the renewal law, while ℓ(h) (h) is a deterministic function of the observed history; neither introduces an additional state variable into the true-world tuple. The Wait–Tick branch increments ℓ(h) (h), whereas Wait–Break, Maintain, and either Probe outcome reset it to zero. Using the one-step reward definition (45), the expected immediate rewards are r⋆(h,W) r_ (h,W) =(RW+CW)S(ℓ(h))−CW, =(R_W+C_W)S( (h))-C_W, (111) r⋆(h,M) r_ (h,M) =−CM, =-C_M, (112) r⋆(h,P) r_ (h,P) =(RP+CP)S(ℓ(h))−CP. =(R_P+C_P)S( (h))-C_P. (113) For the controlled FRDN world, the reachable history tree ℋ⋆ H_ is generated by the kernels above. Equivalently, a finite string is reachable if and only if every appended outcome has positive conditional probability given the preceding history and queried action. B.2 Convergence of finite-dimensional classical memories via Perron–Frobenius The goal of this subsection is to isolate the finite-dimensional classical memory constraint imposed by Perron–Frobenius theory on non-negative matrices. Along the all-Tick trajectory, a finite-dimensional classical world model updates its distributions over states by repeatedly applying the same non-negative Tick branch and renormalizing. Perron–Frobenius theory implies that such N-dimensional non-negative matrix dynamics cannot track an irrational rotation forever: after passing to finitely many arithmetic subsequences, its normalized distributions over states converge. More precisely, for some period p, each residue class r∈0,…,p−1r∈\0,…,p-1\ collects the times t=r+kpt=r+kp for k∈ℕ0k _0, and the classical memory along each such subsequence has a limiting state. This will imply that the classical Tick predictions become asymptotically constant on each residue class. Let N C_N be an arbitrary N-dimensional classical world model in the sense of Definition 13, with action set FRDNA_FRDN and outcome alphabet FRDNY_FRDN. Let DT D_T :=DyTW(W) :=D_y_T^W^(W) (114) be its Wait–Tick branch matrix, and let z0∈ΔN−1z_0∈ _N-1 be its initial distribution over states. Define the all-Tick action–outcome trajectory by ℱtick F_tick :=h1h2⋯, :=h_1h_2·s, ht h_t :=(W,yTW),t≥1, :=(W,y_T^W), t≥ 1, h←0 h_0 :=∅, := , h←t h_t :=h1h2⋯ht,t≥1. :=h_1h_2·s h_t, t≥ 1. (115) Thus, h←t+1 h_t+1 =h←tWyTW, = h_tWy_T^W, ℓ(h←t) ( h_t) =t. =t. (116) Moreover, Pr⋆(yTW∣h←t,W) _ (y_T^W h_t,W) =S(t)=Φ(t+1)Φ(t)>0, =S(t)= (t+1) (t)>0, (117) so every prefix h←t h_t is reachable. Each h←t h_t can serve as the common root of either a true-world continuation or a world-model rollout, with H0⋆=H0M=h←t. H_0 =H_0^M= h_t. (118) A general history h satisfying ℓ(h)=t (h)=t need not equal h←t h_t; only its terminal Wait–Tick suffix has length t. Along these prefixes, define the model survival probability and conditional Tick prediction by ΦC(t) _C(t) :=⟨,DTtz0⟩, := 1,D_T^tz_0 , (119) SC(t) S_C(t) :=PrC(yTW∣EC(h←t),W). :=Pr_C\! (y_T^W E_C( h_t),W ). (120) Here, ΦC(t) _C(t) is the probability that the model assigns to the length-t all-Tick prefix, whereas SC(t)S_C(t) is its conditional probability of one further Tick. Whenever ΦC(t)>0 _C(t)>0, EC(h←t) E_C( h_t) =DTtz0ΦC(t), = D_T^tz_0 _C(t), SC(t) S_C(t) =ΦC(t+1)ΦC(t). = _C(t+1) _C(t). (121) If ΦC(t0)=0 _C(t_0)=0 for some t0t_0, then ΦC(t)=0 _C(t)=0 for every t≥t0t≥ t_0. By the fixed-reference convention following (84), the encoded memory state and its Tick prediction are then eventually constant, so the claims below are immediate. We therefore treat the case ΦC(t)>0 _C(t)>0 for every t. Our main technical tool is Perron–Frobenius theory, which characterizes the asymptotic behavior of powers of DTD_T and hence of the conditional probability SC(t)S_C(t). We use the following standard facts that can be found in 56; 6. Perron–Frobenius theory for non-negative matrices. We use the column-vector convention of Definition 13. Dji>0D_ji>0 means that one step can carry mass from state i to state j. Equivalently, the directed graph of a non-negative matrix D∈ℝ≥0N×ND _≥ 0^N× N has an edge i→ji→ j whenever Dji>0D_ji>0; then (Dt)ji>0(D^t)_ji>0 precisely when there is a positive-weight path of length t from i to j. A set of states is strongly connected if each state can reach every other, and D is irreducible when its graph is strongly connected. The strongly connected components can be ordered so that a simultaneous permutation of rows and columns puts D in Frobenius normal form, ΠDΠ=(B10⋯0∗B2⋱0∗⋯∗Bm). D T= pmatrixB_1&0&·s&0\\ *&B_2& & \\ & & &0\\ *&·s&*&B_m pmatrix. (122) Each diagonal block BjB_j is irreducible or a 1×11× 1 zero block, while the off-diagonal blocks describe paths between distinct components. We call BjB_j reachable from a non-negative vector z if some mass initially in the support of z can enter that component; equivalently, there exist τ≥0τ≥ 0 and a state i in BjB_j such that (Dτz)i>0(D^τz)_i>0. Blocks that are not reachable from z never contribute to DtzD^tz. For an irreducible block B, let ρB:=max|μ|:μ∈spec(B) _B:= \|μ|:μ (B)\ be its spectral radius. Its period is the greatest common divisor of the lengths of all closed paths from any fixed state back to itself, hB:=gcdt≥1:(Bt)ii>0; h_B:= \t≥ 1:(B^t)_i>0\; (123) the value is independent of i. The Perron–Frobenius theorem gives positive left and right eigenvectors at ρB _B, and states that the eigenvalues on the spectral circle |μ|=ρB|μ|= _B are exactly ρBe2πik/hB,k=0,1,…,hB−1. _Be^2π ik/h_B, k=0,1,…,h_B-1. (124) These are the peripheral eigenvalues. Since hB≤dimB≤Nh_B≤ B≤ N, the integer LN:=lcm(1,2,…,N) L_N:=lcm(1,2,…,N) (125) is divisible by every possible block period. Hence every peripheral eigenvalue μ of B obeys μLN=ρBLN. μ^L_N= _B^L_N. (126) Passing to a fixed residue class modulo LNL_N therefore removes all Perron–Frobenius phases. We now prove the main technical result used to establish the value-function gap. Lemma 16. Let DT∈ℝ≥0N×ND_T _≥ 0^N× N be column-substochastic and LN:=lcm(1,2,…,N)L_N:=lcm(1,2,…,N). Let z0∈ΔN−1z_0∈ _N-1, and define ΦC(t):=⟨,DTtz0⟩. _C(t):= 1,D_T^tz_0 . (127) Assume that ΦC(t)>0 _C(t)>0 for all sufficiently large t. Then, for every q∈ℝ≥0Nq _≥ 0^N with q≤q 1 and every r∈0,1,…,LN−1r∈\0,1,…,L_N-1\, there exists cq,r∈[0,1]c_q,r∈[0,1] such that limn→∞⟨q,DTnLN+rz0⟩⟨,DTnLN+rz0⟩=cq,r. _n→∞ q,D_T^nL_N+rz_0 1,D_T^nL_N+rz_0 =c_q,r. (128) In particular, taking q=DTq=D_T T1 gives constants cr∈[0,1]c_r∈[0,1] such that SC(nLN+r)=ΦC(nLN+r+1)ΦC(nLN+r)⟶cr. S_C(nL_N+r)= _C(nL_N+r+1) _C(nL_N+r) c_r. (129) Proof. Fix r and abbreviate D:=DTD:=D_T and L:=LNL:=L_N. Column substochasticity gives ΦC(t+1)=⟨D,Dtz0⟩≤⟨,Dtz0⟩=ΦC(t). _C(t+1)= D T1,D^tz_0 ≤ 1,D^tz_0 = _C(t). (130) Thus, if ΦC(t) _C(t) is positive for all sufficiently large t, it is in fact positive for every t. We may therefore define the normalized column vector zr:=Drz0⟨,Drz0⟩∈ΔN−1. z_r:= D^rz_0 1,D^rz_0 ∈ _N-1. (131) Since DnL+r=DnLDrD^nL+r=D^nLD^r, ⟨q,DnL+rz0⟩⟨,DnL+rz0⟩=⟨q,DnLzr⟩⟨,DnLzr⟩. q,D^nL+rz_0 1,D^nL+rz_0 = q,D^nLz_r 1,D^nLz_r . (132) It is therefore enough to study the subsequence nLnL from an arbitrary initial distribution z; below we write z=zrz=z_r. Put D in Frobenius normal form (122) and delete all blocks that are not reachable from z. They never receive mass from z, so this does not change DtzD^tz or either scalar in (132). We henceforth restrict D to the reachable blocks. Let ρ:=maxjρBj ρ:= _j _B_j (133) be the largest spectral radius among its diagonal blocks. The hypothesis ΦC(t)>0 _C(t)>0 implies ρ>0ρ>0. The main contribution of DtD^t in the inner product ⟨,Dtz⟩ 1,D^tz has exponential rate ρ. Choose a reachable block B with spectral radius ρB=ρ _B=ρ. By reachability, there exist p≥0p≥ 0 and a state a in B such that (Dpz)a>0(D^pz)_a>0. Let eae_a denote the basis vector of that state within the block, and let u>0u>0 be a left Perron vector of B, written as Bu=ρuB Tu=ρ u, and let B1_B denote the all-ones vector on that block. For some η>0η>0, B≥ηu1_B≥η u componentwise. Keeping only paths that enter B at a and subsequently remain in B gives, for t≥pt≥ p, a(t):=⟨,Dtz⟩ a(t):= 1,D^tz ≥(Dpz)a⟨B,Bt−pea⟩ ≥(D^pz)_a\, 1_B,B^t-pe_a ≥(Dpz)aη⟨u,Bt−pea⟩ ≥(D^pz)_aη\, u,B^t-pe_a (134) =(Dpz)aη⟨u,ea⟩ρt−p. =(D^pz)_aη u,e_a ρ^t-p. (135) Thus, a(t)≥Cpρt,Cp:=(Dpz)aη⟨u,ea⟩ρ−p>0. a(t)≥ C_pρ^t, C_p:=(D^pz)_aη u,e_a ρ^-p>0. (136) We also use the standard consequence of Jordan normal form that, for any finite matrix D and vectors u,vu,v, the scalar sequence ⟨u,Dtv⟩ u,D^tv is a finite sum of polynomial–exponential terms pμ(t)μtp_μ(t)μ^t, where μ ranges over eigenvalues of D (31, Sec. 3.1). In particular, for all sufficiently large t, a(t)=⟨,Dtz⟩=∑μ∈spec(D)∖0pμ(t)μt, a(t)= 1,D^tz = _μ (D) \0\p_μ(t)μ^t, (137) where each pμp_μ is a polynomial of t. Every eigenvalue of D lies in a diagonal Frobenius block, and hence has modulus at most ρ. Moreover, if |μ|=ρ|μ|=ρ, then μ belongs to a block of spectral radius ρ and is therefore peripheral for that block. Perron-Frobenius (126) then gives μnL=ρnLμ^nL=ρ^nL. The lower bound (135) ensures that the terms with |μ|=ρ|μ|=ρ do not all cancel after passing to t=nLt=nL. Indeed, after substituting t=nLt=nL in (137), all terms with |μ|<ρ|μ|<ρ are exponentially smaller, while every term with |μ|=ρ|μ|=ρ satisfies μnL=ρnLμ^nL=ρ^nL by (126). Hence the terms with |μ|=ρ|μ|=ρ combine into ρnLp(n)ρ^nLp(n) for a real polynomial p. If p were the zero polynomial, then a(nL)=o(ρnL)a(nL)=o(ρ^nL), contradicting (136). Therefore p is not zero. Let m be its degree and A its leading coefficient. Then a(nL)=ρnLnm(A+o(1)). a(nL)=ρ^nLn^m (A+o(1) ). (138) Since a(nL)>0a(nL)>0 for all sufficiently large n, necessarily A>0A>0. For the numerator in (132), set bq(t):=⟨q,Dtz⟩. b_q(t):= q,D^tz . (139) It has a Jordan expansion of the same form, and the componentwise inequality 0≤q≤0≤ q 1 gives 0≤bq(t)≤a(t). 0≤ b_q(t)≤ a(t). (140) After substituting t=nLt=nL, the terms with |μ|=ρ|μ|=ρ in the numerator combine into ρnLp~(n)ρ^nL p(n) for another real polynomial p~ p, while all terms with |μ|<ρ|μ|<ρ are exponentially smaller. The inequality (140) implies that p~ p has degree at most m: if it had larger degree, then bq(nL)b_q(nL) would eventually either be negative or larger than a(nL)a(nL). Therefore bq(nL)=ρnLnm(Bq+o(1)), b_q(nL)=ρ^nLn^m (B_q+o(1) ), (141) where Bq=0B_q=0 when the numerator has strictly smaller order. Dividing (140) by ρnLnmρ^nLn^m and taking the limit gives 0≤Bq≤A0≤ B_q≤ A. Therefore ⟨q,DnLz⟩⟨,DnLz⟩⟶BqA∈[0,1]. q,D^nLz 1,D^nLz B_qA∈[0,1]. (142) Together with (132), this proves (128). Finally, column substochasticity implies 0≤D≤0≤ D T1 1, and ⟨D,Dtz0⟩=⟨,Dt+1z0⟩=ΦC(t+1). D T1,D^tz_0 = 1,D^t+1z_0 = _C(t+1). (143) The choice q=Dq=D T1 therefore yields (129). ∎ Corollary 17. Under the hypotheses of Lemma 16, for every residue r there exists z∞,r∈ΔN−1z_∞,r∈ _N-1 such that EC(h←nLN+r)=DTnLN+rz0⟨,DTnLN+rz0⟩⟶z∞,r. E_C( h_nL_N+r)= D_T^nL_N+rz_0 1,D_T^nL_N+rz_0 z_∞,r. (144) Consequently, every continuous function of the classical distribution over states converges to a constant on each residue class. Proof. For the jjth standard basis vector eje_j, [EC(h←nLN+r)]j=⟨ej,DTnLN+rz0⟩⟨,DTnLN+rz0⟩. [E_C( h_nL_N+r) ]_j= e_j,D_T^nL_N+rz_0 1,D_T^nL_N+rz_0 . (145) Since 0≤ej≤0≤ e_j 1, the lemma gives convergence of every coordinate. The limiting coordinates are non-negative and sum to one, so they define z∞,r∈ΔN−1z_∞,r∈ _N-1. ∎ B.3 Gap for the conditional probabilities We now turn the convergence of finite classical memories established in Lemma 16 and Corollary 17 into an operational separation for prediction, action-values, and values along the Tick histories. We start with a standard tool that converts the problem of approximating the sequence f(tα)f(tα) along arithmetic subsequences into the simpler problem of approximating the function f by constants in phase average. Recall that f is given in (99) and S(t)=f(tα)S(t)=f(tα) for t≥1t≥ 1. We use Weyl equidistribution to compare the sequence f(tα)f(tα) with the phase average of f. Since LNα/(2π)∉ℚL_Nα/(2π) , Weyl equidistribution (38, Chap. 1) gives, for every residue r∈0,…,LN−1r∈\0,…,L_N-1\ and every Riemann-integrable 2π2π-periodic function g, limK→∞1K∑n=0K−1g((nLN+r)α) _K→∞ 1K _n=0^K-1g\! ((nL_N+r)α ) =12π∫02πg(x)x. = 12π _0^2πg(x)\,dx. (146) In particular, (146) applies both to the continuous functions used in the value bounds and to the indicator functions used in (193), since the latter have only finitely many discontinuities. We shall repeatedly use the following stability consequence of Weyl equidistribution. Lemma 18. Fix r∈0,…,LN−1r∈\0,…,L_N-1\. Let F:ℝ→ℝF:R be continuous and 2π2π-periodic, and let cn→c_n→ c in ℝR. Then limK→∞1K∑n=0K−1|F((nLN+r)α)−cn| _K→∞ 1K _n=0^K-1 |F ((nL_N+r)α )-c_n | (147) =12π∫02π|F(x)−c|x. = 12π _0^2π|F(x)-c|\,dx. (148) Proof. The reverse triangle inequality gives |1K∑n=0K−1|F((nLN+r)α)−cn| | 1K _n=0^K-1 |F ((nL_N+r)α )-c_n | . −1K∑n=0K−1|F((nLN+r)α)−c|| 40.00006pt .- 1K _n=0^K-1 |F ((nL_N+r)α )-c | | ≤1K∑n=0K−1|cn−c|. ≤ 1K _n=0^K-1|c_n-c|. (149) Since cn→c_n→ c, convergence implies limK→∞1K∑n=0K−1|cn−c|=0. _K→∞ 1K _n=0^K-1|c_n-c|=0. (150) On the other hand, Weyl equidistribution (146) applied to the continuous 2π2π-periodic function x↦|F(x)−c|x |F(x)-c| gives limK→∞1K∑n=0K−1|F((nLN+r)α)−c|=12π∫02π|F(x)−c|x. _K→∞ 1K _n=0^K-1 |F ((nL_N+r)α )-c |= 12π _0^2π|F(x)-c|\,dx. (151) Combining the above proves (148). ∎ The following constant will be the one appearing in our lower bounds, κFRDN:=minc∈[0,1]12π∫02π|f(x)−c|x. _FRDN:= _c∈[0,1] 12π _0^2π|f(x)-c|\,dx. (152) Because f is continuous and nonconstant, κFRDN>0 _FRDN>0. The following proposition combines this constant approximation gap with the residue-class convergence established above. Proposition 19 (Finite-dimensional classical prediction gap). For every finite-dimensional classical world model over (FRDN,FRDN)(A_FRDN,Y_FRDN), in the sense of Definition 13, lim infT→∞1T∑t=1T|S(t)−SC(t)|≥κFRDN. _T→∞ 1T _t=1^T|S(t)-S_C(t)|≥ _FRDN. (153) Proof. First suppose that ΦC(t0)=0 _C(t_0)=0 for some t0t_0. Since DTD_T is column-substochastic, ΦC(t+1)≤ΦC(t) _C(t+1)≤ _C(t), so ΦC(t)=0 _C(t)=0 for every t≥t0t≥ t_0. By the fixed-reference convention following (84), we then have EC(h←t)=zrefE_C( h_t)=z_ ref for all t≥t0t≥ t_0. Therefore SC(t)=PrC(yTW∣zref,W)=:cref∈[0,1] S_C(t)=Pr_C(y_T^W z_ ref,W)=:c_ ref∈[0,1] (154) eventually. Since α/(2π)∉ℚα/(2π) , Weyl equidistribution (146) gives limT→∞1T∑t=1T|S(t)−SC(t)| _T→∞ 1T _t=1^T|S(t)-S_C(t)| =12π∫02π|f(x)−cref|x = 12π _0^2π|f(x)-c_ ref|\,dx ≥κFRDN. ≥ _FRDN. (155) Thus, the claim holds in this case. Hence, from now on, assume ΦC(t)>0 _C(t)>0 for all t. Fix a residue class r∈0,…,LN−1r∈\0,…,L_N-1\. By Lemma 16, SC(nLN+r)⟶cr. S_C(nL_N+r) c_r. (156) Applying Lemma 18 with F(x) F(x) :=f(x), :=f(x), cn c_n :=SC(nLN+r), :=S_C(nL_N+r), c c :=cr, :=c_r, (157) gives limK→∞1K∑n=0K−1|S(nLN+r)−SC(nLN+r)| _K→∞ 1K _n=0^K-1|S(nL_N+r)-S_C(nL_N+r)| =12π∫02π|f(x)−cr|x≥κFRDN. = 12π _0^2π|f(x)-c_r|\,dx≥ _FRDN. (158) For T=KLNT=KL_N, decomposition into residue classes gives 1KLN∑t=0KLN−1|S(t)−SC(t)| 1KL_N _t=0^KL_N-1|S(t)-S_C(t)| =1LN∑r=0LN−11K∑n=0K−1|S(nLN+r)−SC(nLN+r)|. = 1L_N _r=0^L_N-1 1K _n=0^K-1|S(nL_N+r)-S_C(nL_N+r)|. (159) Taking K→∞K→∞ and using (158) gives lim infK→∞1KLN∑t=0KLN−1|S(t)−SC(t)|≥κFRDN. _K→∞ 1KL_N _t=0^KL_N-1|S(t)-S_C(t)|≥ _FRDN. (160) Since the summands lie in [0,1][0,1], shifting the indices from 0,…,KLN−10,…,KL_N-1 to 1,…,KLN1,…,KL_N is asymptotically irrelevant. For arbitrary T, let K=⌊T/LN⌋K= T/L_N . Then since all terms are nonnegative, 1T∑t=1T|S(t)−SC(t)| 1T _t=1^T|S(t)-S_C(t)| ≥KLNT(1KLN∑t=1KLN|S(t)−SC(t)|). ≥ KL_NT ( 1KL_N _t=1^KL_N|S(t)-S_C(t)| ). (161) Since KLN/T→1KL_N/T→ 1, taking the lower limit proves (153). ∎ B.4 Model-selected actions and decision loss We now study the action selected by an agent that acts greedily with respect to a finite-dimensional classical world model on the all-Tick query histories. To make the subsection self-contained, we repeat the decision quantities introduced in the main text. For a query history h and world model M, fix a model-greedy action a^M(h)∈argmaxa∈FRDNQMopt(h,a). a_M(h)∈ *arg\,max_a _FRDNQ_M^opt(h,a). (162) Choose likewise a true-world optimal action a⋆(h)∈argmaxa∈FRDNQ⋆opt(h,a). a_ (h)∈ *arg\,max_a _FRDNQ_ ^opt(h,a). (163) The corresponding model and true-world decision margins are gM(h) g_M(h) :=QMopt(h,a^M(h))−maxa≠a^M(h)QMopt(h,a), :=Q_M^opt\! (h, a_M(h) )- _a≠ a_M(h)Q_M^opt(h,a), g⋆(h) g_ (h) :=Q⋆opt(h,a⋆(h))−maxa≠a⋆(h)Q⋆opt(h,a). :=Q_ ^opt\! (h,a_ (h) )- _a≠ a_ (h)Q_ ^opt(h,a). (164) Each margin is the difference between the largest and second-largest action-values, and therefore vanishes when the two largest values are tied. The true-world loss incurred by deploying the model-greedy action is ℓM(h) _M(h) :=V⋆opt(h)−Q⋆opt(h,a^M(h)). :=V_ ^opt(h)-Q_ ^opt\! (h, a_M(h) ). (165) This is the return sacrificed by the current model-selected action when all subsequent actions are chosen optimally in the true world. Along a reachable action–outcome trajectory ℱ=h1h2⋯ F=h_1h_2·s, with prefixes h←t=h1⋯ht h_t=h_1·s h_t, define ℓ¯M(ℱ) _M( F) :=lim infT→∞1T∑t=1TℓM(h←t). := _T→∞ 1T _t=1^T _M( h_t). (166) Definition 20 (Loss of decision resolution). A world model M loses decision resolution along a reachable trajectory ℱ=h1h2⋯ F=h_1h_2·s if there exists ε>0 >0 such that, for every δ>0δ>0 and every T∈ℕT , there is a t≥Tt≥ T satisfying g⋆(h←t) g_ ( h_t) ≥ε, ≥ , gM(h←t) g_M( h_t) <δ. <δ. (167) For a finite-dimensional classical world model along ℱtick F_tick, write a^C(t) a_C(t) :=a^C(h←t), := a_C( h_t), ℓC(t) _C(t) :=ℓC(h←t), := _C( h_t), gC(t) g_C(t) :=gC(h←t), :=g_C( h_t), g⋆(t) g_ (t) :=g⋆(h←t). :=g_ ( h_t). (168) At each prefix h←t h_t, all three candidate actions are evaluated, independently of the Wait action that extends the all-Tick trajectory. Fix 0<η<1−λ0<η<1-λ and specialize the reward parameters to RW R_W :=−(1+η), :=-(1+η), CW C_W :=1+η, :=1+η, CM C_M :=η, :=η, RP R_P :=1−λ−η, :=1-λ-η, CP C_P :=λ+η. :=λ+η. (169) Thus, Wait gives reward −(1+η)-(1+η) after either outcome, Maintain gives the deterministic reward −η-η, and Probe gives reward 1−λ−η1-λ-η after Tick and −(λ+η)-(λ+η) after Break. Maintain and both Probe outcomes reset the clock to age zero. Consequently, Maintain and Probe have the same discounted continuation term; their comparison depends only on their expected immediate rewards. The next lemma computes the resulting true-world optimal actions and decision margins. Lemma 21. For the rewards in (169), every γ∈[0,1)γ∈[0,1) and t≥0t≥ 0 satisfy Q⋆opt(h←t,P)−Q⋆opt(h←t,M)=S(t)−λ, Q_ ^opt( h_t,P)-Q_ ^opt( h_t,M)=S(t)-λ, Q⋆opt(h←t,W)−Q⋆opt(h←t,M)<−λ. Q_ ^opt( h_t,W)-Q_ ^opt( h_t,M)<-λ. (170) Consequently, the true optimal-action set is ⋆(t) _ (t) :=argmaxa∈FRDNQ⋆opt(h←t,a) := *arg\,max_a _FRDNQ_ ^opt( h_t,a) =P,S(t)>λ,P,M,S(t)=λ,M,S(t)<λ, = cases\P\,&S(t)>λ,\\ \P,M\,&S(t)=λ,\\ \M\,&S(t)<λ, cases (171) and g⋆(t)=|S(t)−λ|. g_ (t)=|S(t)-λ|. (172) Proof. For t≥0t≥ 0, define Ht H_t :=(S(t)−λ)+. := (S(t)-λ )_+. (173) We construct a candidate Bellman fixed point whose value depends on a history only through its clock age. At age zero, we define u0 u_0 :=−η+H01−γ, := -η+H_01-γ, (174) and for t≥1t≥ 1, define ut u_t :=−η+γu0+Ht. :=-η+γ u_0+H_t. (175) By the defining equation for u0u_0, the same formula also holds at t=0t=0. Given the above quantities, we now define our candidate function that will be the fixed-point equation of the Bellman value function equation (47) with the optimal Bellman operator ⋆T_ defined in (46). For a history h, let U(h) U(h) :=uℓ(h). :=u_ (h). (176) Since 0≤Ht≤1−λ0≤ H_t≤ 1-λ, the function U is bounded. To verify that U is the optimal value, it is useful to separate the action-specific terms entering the Bellman operator. For a bounded function F and an action a, define (ℬ⋆,aF)(h) (B_ ,aF )(h) :=∑y∈FRDNPr⋆(y∣h,a)[ry+γF(hay)]. := _y _FRDNPr_ (y h,a) [r_y+γ F(hay) ]. (177) Then (⋆F)(h)=maxa∈FRDN(ℬ⋆,aF)(h). (T_ F)(h)= _a _FRDN (B_ ,aF )(h). (178) Later we will identify ℬ⋆,aUB_ ,aU with Q⋆optQ_ ^opt, only after we prove that U=V⋆optU=V_ ^opt. Fix a history h with ℓ(h)=t (h)=t. Maintain gives the deterministic reward −η-η and resets the clock to age zero. Its continuation value under U is therefore u0u_0, so (ℬ⋆,MU)(h) (B_ ,MU )(h) =−η+γu0. =-η+γ u_0. (179) For Probe, Tick occurs with probability S(t)S(t) and gives reward 1−λ−η1-λ-η, whereas Break occurs with probability 1−S(t)1-S(t) and gives reward −(λ+η)-(λ+η). Both outcomes reset the clock to age zero. Hence (ℬ⋆,PU)(h) (B_ ,PU )(h) =S(t)[1−λ−η+γu0] =S(t) [1-λ-η+γ u_0 ] +(1−S(t))[−λ−η+γu0] + (1-S(t) ) [-λ-η+γ u_0 ] =−η+γu0+S(t)−λ. =-η+γ u_0+S(t)-λ. (180) Subtracting the Maintain (179) gives (ℬ⋆,PU)(h)−(ℬ⋆,MU)(h) (B_ ,PU )(h)- (B_ ,MU )(h) =S(t)−λ. =S(t)-λ. (181) Therefore, the larger of the Maintain and Probe is max(ℬ⋆,MU)(h),(ℬ⋆,PU)(h) \ (B_ ,MU )(h), (B_ ,PU )(h) \ =−η+γu0+(S(t)−λ)+ =-η+γ u_0+ (S(t)-λ )_+ =−η+γu0+Ht =-η+γ u_0+H_t =ut. =u_t. (182) It remains to show that Wait never exceeds this value. Wait gives reward −(1+η)-(1+η) after either outcome. A Tick, occurring with probability S(t)S(t), increases the clock age to t+1t+1, whereas a Break resets it to zero. Thus, (ℬ⋆,WU)(h) (B_ ,WU )(h) =−(1+η)+γ[S(t)ut+1+(1−S(t))u0]. =-(1+η)+γ [S(t)u_t+1+ (1-S(t) )u_0 ]. (183) Subtracting the Maintain and using ut+1−u0=Ht+1−H0u_t+1-u_0=H_t+1-H_0 gives (ℬ⋆,WU)(h)−(ℬ⋆,MU)(h)=−1+γS(t)(ut+1−u0) (B_ ,WU )(h)- (B_ ,MU )(h)=-1+γ S(t)(u_t+1-u_0) =−1+γS(t)(Ht+1−H0). =-1+γ S(t)(H_t+1-H_0). (184) Since 0≤S(t)≤10≤ S(t)≤ 1, H0≥0H_0≥ 0, and Ht+1≤1−λH_t+1≤ 1-λ, (ℬ⋆,WU)(h)−(ℬ⋆,MU)(h) (B_ ,WU )(h)- (B_ ,MU )(h) ≤−1+γ(1−λ) ≤-1+γ(1-λ) <−1+(1−λ) <-1+(1-λ) =−λ. =-λ. (185) Thus, Wait is strictly below Maintain. Combining this with (182), we obtain (⋆U)(h) (T_ U)(h) =maxa∈FRDN(ℬ⋆,aU)(h) = _a _FRDN (B_ ,aU )(h) =ut=U(h). =u_t=U(h). (186) Hence U is a bounded fixed point of the true-world Bellman optimality operator. By uniqueness of the bounded fixed point, U(h)=V⋆opt(h)for every h∈ℋ. U(h)=V_ ^opt(h) every h∈ H. (187) We may now identify the action-specific operators ℬ⋆,aB_ ,a with the optimal action-values through Q⋆opt(h,a) Q_ ^opt(h,a) =(ℬ⋆,aU)(h), = (B_ ,aU )(h), (188) for which we use the expression in (56). Equations (181) and (185) therefore give Q⋆opt(h←t,P)−Q⋆opt(h←t,M) Q_ ^opt( h_t,P)-Q_ ^opt( h_t,M) =S(t)−λ, =S(t)-λ, Q⋆opt(h←t,W)−Q⋆opt(h←t,M) Q_ ^opt( h_t,W)-Q_ ^opt( h_t,M) <−λ. <-λ. (189) Moreover, because S(t)≥0S(t)≥ 0, Q⋆opt(h←t,W)−Q⋆opt(h←t,M) Q_ ^opt( h_t,W)-Q_ ^opt( h_t,M) <−λ≤S(t)−λ. <-λ≤ S(t)-λ. (190) Thus using (B.4) we determine that Wait is also strictly below Probe, and the two largest action-values are always those of Probe and Maintain. Consequently using (B.4), Probe is uniquely optimal when S(t)>λS(t)>λ, Maintain is uniquely optimal when S(t)<λS(t)<λ, and they are tied when S(t)=λS(t)=λ. This proves (171). Since the top two action-values are those of Probe and Maintain, their separation is g⋆(t) g_ (t) =|Q⋆opt(h←t,P)−Q⋆opt(h←t,M)| = |Q_ ^opt( h_t,P)-Q_ ^opt( h_t,M) | =|S(t)−λ|, =|S(t)-λ|, (191) which proves (172). ∎ We next determine how often the true optimal action switches. From (99), f(x)−λ f(x)-λ =2λBsin(α/2)sin(x+φ+α/2)A−Bcos(x+φ). = 2λ B (α/2) (x+ +α/2)A-B (x+ ). (192) The denominator is strictly positive. Moreover, sin(α/2)≠0 (α/2)≠ 0 because α/πα/π is irrational. Thus, f(x)−λf(x)-λ is a nonzero multiple of a shifted sine divided by a positive function. The sets on which it is positive and negative each occupy one half of a period. Since S(t)=f(tα)S(t)=f(tα) for t≥1t≥ 1, Weyl equidistribution (146) gives, for every r∈0,…,LN−1r∈\0,…,L_N-1\, limK→∞1K∑n=0K−1S(nLN+r)>λ _K→∞ 1K _n=0^K-1 1\! \S(nL_N+r)>λ \ =12, = 12, limK→∞1K∑n=0K−1S(nLN+r)<λ _K→∞ 1K _n=0^K-1 1\! \S(nL_N+r)<λ \ =12. = 12. (193) The possible term with nLN+r=0nL_N+r=0 does not affect either limit. Irrationality also implies that S(t)=λS(t)=λ for at most one integer t≥1t≥ 1, so ties do not affect the asymptotic frequencies. A small result we will need for our proof is the continuity of the action-value and value functions with respect to the distributions of classical states. Lemma 22 (Continuity of finite-classical value functions). Fix a finite-dimensional classical world model and γ∈[0,1)γ∈[0,1). Then vCoptv_C^opt is continuous on ΔN−1 _N-1, and qCopt(⋅,a)q_C^opt(·,a) is continuous for every a∈a . Proof. Let v be a bounded continuous function on ΔN−1 _N-1. For each action a and outcome y, consider the weighted branch term Fa,yv(z):=PrC(y∣z,a)v(TC,ya(z)). F_a,y^v(z):=Pr_C(y z,a)\,v\! (T_C,y^a(z) ). (194) It is continuous wherever PrC(y∣z,a)>0Pr_C(y z,a)>0. If zn→z_n→ z and PrC(y∣z,a)=0Pr_C(y z,a)=0, then |Fa,yv(zn)|≤‖v‖∞PrC(y∣zn,a)⟶0. |F_a,y^v(z_n) |≤\|v\|_∞Pr_C(y z_n,a) 0. (195) Hence every weighted branch term extends continuously through zero-probability points. The expected one-step reward rC(⋅,a)r_C(·,a) is linear and therefore continuous. It follows that the Bellman optimality operator maps continuous functions to continuous functions. Starting from the zero function, its iterates are continuous and converge uniformly to the unique fixed point vCoptv_C^opt. Thus vCoptv_C^opt is continuous. Equation (63) then implies that qCopt(⋅,a)q_C^opt(·,a) is continuous for every action a. ∎ We now combine continuity with the residue-class convergence of finite classical memories. On each residue class, the model’s action-value vector converges. Its limiting vector either has a tie at the top, in which case the predicted decision margin vanishes, or has a unique maximizer, in which case the model eventually selects one fixed action. The latter cannot track the true switching between Probe and Maintain. Theorem 23 (Limits of finite classical decisions). For every discount factor γ∈[0,1)γ∈[0,1), every N∈ℕN , every N-dimensional classical world model over (FRDN,FRDN)(A_FRDN,Y_FRDN), and every model-greedy selection in (162), at least one of the following alternatives holds: • The decision margin of the classical model becomes arbitrarily small, lim inft→∞gC(t)=0. _t→∞g_C(t)=0. (196) • The model-selected action is truly suboptimal on at least half of the prefixes of ℱtick F_tick in the long run, lim infT→∞1T∑t=1Ta^C(t)∉⋆(t)≥12. _T→∞ 1T _t=1^T 1 \ a_C(t) _ (t) \≥ 12. (197) Moreover, the first alternative holds if and only if there is a residue r∈0,…,LN−1r∈\0,…,L_N-1\ for which gC(nLN+r)⟶0. g_C(nL_N+r) 0. (198) Proof. If the model eventually assigns zero probability to the all-Tick branch, the fixed-reference convention makes EC(h←t)E_C( h_t) eventually equal to zrefz_ref. Otherwise, Corollary 17 applies. Hence, in either case, for every residue r∈0,…,LN−1r∈\0,…,L_N-1\ there is a distribution over states z∞,r∈ΔN−1z_∞,r∈ _N-1 such that EC(h←nLN+r)⟶z∞,r. E_C( h_nL_N+r) z_∞,r. (199) Using the bridge relation (66) and Lemma 22, the corresponding action-value vectors satisfy n,r _n,r :=(QCopt(h←nLN+r,a))a∈FRDN := (Q_C^opt( h_nL_N+r,a) )_a _FRDN ⟶(qCopt(z∞,r,a))a∈FRDN=:r. (q_C^opt(z_∞,r,a) )_a _FRDN=:q_r. (200) Let gr∞g_r^∞ be the difference between the largest and second-largest components of rq_r. The top two gap is a continuous function of a finite vector, so gC(nLN+r)⟶gr∞. g_C(nL_N+r) g_r^∞. (201) Since the number of residue classes is finite, lim inft→∞gC(t)=min0≤r<LNgr∞. _t→∞g_C(t)= _0≤ r<L_Ng_r^∞. (202) Equations (201) and (202) prove the final equivalence in the theorem. In particular, if gr∞=0g_r^∞=0 for some r, the first alternative holds. Suppose instead that gr∞>0g_r^∞>0 for every residue. Then rq_r has a unique maximizing action, denoted by ara_r, which is the model-greedy action (162). Thus, convergence implies a^C(nLN+r)=ar a_C(nL_N+r)=a_r (203) for all sufficiently large n. If ar=Pa_r=P, Lemma 21 shows that the selected action is suboptimal whenever S(nLN+r)<λS(nL_N+r)<λ. If ar=Ma_r=M, it is suboptimal whenever S(nLN+r)>λS(nL_N+r)>λ. If ar=Wa_r=W, it is suboptimal for every n. Equation (193) therefore shows that the selected action is suboptimal on at least half of the histories in every residue class. Averaging over the residue classes first for T=KLNT=KL_N, and then observing that an incomplete final block is negligible, proves (197). ∎ The first alternative becomes an operational failure only if the predicted margin vanishes away from true-world ties. The following corollary shows that this is precisely what happens. Corollary 24 (Loss of decision resolution). Under the hypotheses of Theorem 23, suppose that lim inft→∞gC(t)=0 _t→∞g_C(t)=0. Then the classical world model loses decision resolution along ℱtick F_tick in the sense of Definition 20. More precisely, there is a constant εres>0 _res>0 and an increasing sequence tj→∞t_j→∞ such that g⋆(tj) g_ (t_j) ≥εres, ≥ _res, gC(tj) g_C(t_j) ⟶0. 0. (204) The constant εres _res depends only on the true-world function f and the threshold λ, and is therefore independent of γ, the classical memory dimension, and the chosen classical model. Proof. By the final assertion of Theorem 23, choose a residue r∈0,…,LN−1r∈\0,…,L_N-1\ such that gC(nLN+r)⟶0. g_C(nL_N+r) 0. (205) It remains to choose a subsequence in this residue class on which the true decision margin remains positive. Equation (192) shows that f−λf-λ is not identically zero. Define εres _res :=12maxx∈[0,2π]|f(x)−λ|. := 12 _x∈[0,2π]|f(x)-λ|. (206) Then εres>0 _res>0. By continuity, there exists a nonempty open interval Ires⊂[0,2π)I_res⊂[0,2π) such that |f(x)−λ| |f(x)-λ| ≥εresfor every x∈Ires. ≥ _res every x∈ I_res. (207) Define the 2π2π-periodic indicator χIres(x) _I_res(x) :=xmod2π∈Ires. := 1 \x 2π∈ I_res \. (208) This function is Riemann integrable, since it has discontinuities only at the endpoints of IresI_res. Because LNα/(2π)∉ℚL_Nα/(2π) , Weyl equidistribution (146) gives limK→∞1K∑n=0K−1χIres((nLN+r)α)=|Ires|2π>0. _K→∞ 1K _n=0^K-1 _I_res ((nL_N+r)α )= |I_res|2π>0. (209) Hence the phases (nLN+r)αmod2π(nL_N+r)α 2π enter IresI_res infinitely often. Choose an increasing sequence of such visits njn_j and set tj:=njLN+rt_j:=n_jL_N+r, discarding a possible initial term with tj=0t_j=0. Then |S(tj)−λ| |S(t_j)-λ| =|f(tjα)−λ|≥εres. = |f(t_jα)-λ |≥ _res. (210) Using (172), we obtain g⋆(tj)≥εresg_ (t_j)≥ _res. At the same time, gC(tj)→0g_C(t_j)→ 0 by the choice of the residue class r. This is precisely the condition in (167). ∎ If the predicted margin stays positive, each residue class instead has an eventually fixed selected action. We now lower-bound the mean decision loss of these actions on ℱtick F_tick. Corollary 25 (Decision loss). Under the hypotheses of Theorem 23, suppose that lim inft→∞gC(t) _t→∞g_C(t) >0. >0. (211) Then the mean decision loss defined in (166) satisfies ℓ¯C(ℱtick) _C\! ( F_tick ) ≥εdec>0, ≥ _dec>0, (212) where εdec _dec :=min12π∫02π(λ−f(x))+dx, := \ 12π _0^2π (λ-f(x) )_+\,dx, 12π∫02π(f(x)−λ)+dx>0. 42.00003pt 12π _0^2π (f(x)-λ )_+\,dx \>0. (213) The constant εdec _dec depends only on the true world and is independent of both the classical memory dimension and the chosen model. Proof. By (202), the assumption (211) implies gr∞>0g_r^∞>0 for every residue class. The proof of Theorem 23 therefore gives, for each r∈0,…,LN−1r∈\0,…,L_N-1\, an action ara_r such that a^C(nLN+r)=ar a_C(nL_N+r)=a_r (214) for all sufficiently large n. Fix a residue r and write t=nLN+rt=nL_N+r. Lemma 21 shows that Wait is strictly below both Probe and Maintain. Hence V⋆opt(h←t)=maxQ⋆opt(h←t,P),Q⋆opt(h←t,M). V_ ^opt( h_t)= \Q_ ^opt( h_t,P),Q_ ^opt( h_t,M) \. (215) If ar=Pa_r=P, then, for all sufficiently large n, ℓC(t) _C(t) =V⋆opt(h←t)−Q⋆opt(h←t,P) =V_ ^opt( h_t)-Q_ ^opt( h_t,P) =(Q⋆opt(h←t,M)−Q⋆opt(h←t,P))+ = (Q_ ^opt( h_t,M)-Q_ ^opt( h_t,P) )_+ =(λ−S(t))+. = (λ-S(t) )_+. (216) If ar=Ma_r=M, then ℓC(t) _C(t) =V⋆opt(h←t)−Q⋆opt(h←t,M) =V_ ^opt( h_t)-Q_ ^opt( h_t,M) =(Q⋆opt(h←t,P)−Q⋆opt(h←t,M))+ = (Q_ ^opt( h_t,P)-Q_ ^opt( h_t,M) )_+ =(S(t)−λ)+. = (S(t)-λ )_+. (217) Finally, if ar=Wa_r=W, then ℓC(t) _C(t) ≥Q⋆opt(h←t,M)−Q⋆opt(h←t,W)>λ. ≥ Q_ ^opt( h_t,M)-Q_ ^opt( h_t,W)>λ. (218) For the Probe case, Weyl equidistribution (146) gives limK→∞1K∑n=0K−1(λ−S(nLN+r))+ _K→∞ 1K _n=0^K-1 (λ-S(nL_N+r) )_+ =12π∫02π(λ−f(x))+x. = 12π _0^2π (λ-f(x) )_+\,dx. (219) For the Maintain case, it gives limK→∞1K∑n=0K−1(S(nLN+r)−λ)+ _K→∞ 1K _n=0^K-1 (S(nL_N+r)-λ )_+ =12π∫02π(f(x)−λ)+x. = 12π _0^2π (f(x)-λ )_+\,dx. (220) Both phase averages are strictly positive by (192). By the definition (213), each is at least εdec _dec. Moreover, 0≤(λ−f(x))+≤λ, 0≤ (λ-f(x) )_+≤λ, (221) and therefore εdec≤12π∫02π(λ−f(x))+x≤λ. _dec≤ 12π _0^2π (λ-f(x) )_+\,dx≤λ. (222) Thus, the uniform Wait loss in (218) is also at least εdec _dec. Consequently, every residue class has asymptotic mean decision loss at least εdec _dec. Averaging over the finitely many residue classes gives ℓ¯C(ℱtick)≥εdec. _C\! ( F_tick )≥ _dec. (223) ∎ With εtr:=minεres,εdec>0 _tr:= \ _res, _dec\>0, Theorem 23 and Corollaries 24 and 25 show that, along the fixed reachable trajectory ℱtick F_tick, every finite-dimensional classical world model either loses decision resolution at level εtr _tr or selects a true-world-suboptimal action with lower asymptotic frequency at least 1/21/2 and has mean decision loss at least εtr _tr. Both the trajectory and the constant are independent of the classical memory dimension and the chosen model. B.4.1 About other optimal actions The choice of competing actions in the preceding results is not a property of the dynamics, but a consequence of the reward assignment. That choice makes Probe and Maintain differ by S(t)−λS(t)-λ for every γ∈[0,1)γ∈[0,1), which gives a particularly direct arbitrary-discount proof. An analogous one-step construction can instead make Wait and Maintain the competing actions. At γ=0γ=0, choose RW R_W =1−λ−η, =1-λ-η, CW C_W =λ+η, =λ+η, CM C_M =η, =η, RP R_P =−CP, =-C_P, (224) where 0<η<1−λ0<η<1-λ and CP>ηC_P>η. The corresponding true action-values on the all-Tick prefixes are Q⋆opt(h←t,W) Q_ ^opt( h_t,W) =S(t)−λ−η, =S(t)-λ-η, Q⋆opt(h←t,M) Q_ ^opt( h_t,M) =−η, =-η, Q⋆opt(h←t,P) Q_ ^opt( h_t,P) =−CP. =-C_P. (225) Probe is then strictly suboptimal, whereas Wait is optimal when S(t)>λS(t)>λ and Maintain is optimal when S(t)<λS(t)<λ. By (192) and (193), each case has asymptotic frequency 1/21/2 on every residue class. The same residue-class convergence argument therefore gives the same decision dichotomy: every finite-dimensional classical world model either loses decision resolution or, if its predicted margin remains positive, selects a true-world suboptimal action on at least half of the queried histories and incurs a strictly positive mean decision loss. Thus, the use of Probe and Maintain in the arbitrary-discount construction is a technical convenience and should not be interpreted as implying that advancing the clock with Wait can never be optimal. B.5 Estimation errors for value functions The preceding subsection concerned the ordering of the action-values. We now study their numerical accuracy along the all-Tick trajectory. For a world model M and history h, define eQM(h) e_Q^M(h) :=maxa∈FRDN|Q⋆opt(h,a)−QMopt(h,a)|, := _a _FRDN |Q_ ^opt(h,a)-Q_M^opt(h,a) |, (226) eVM(h) e_V^M(h) :=|V⋆opt(h)−VMopt(h)|. := |V_ ^opt(h)-V_M^opt(h) |. (227) Along a reachable trajectory ℱ=h1h2⋯ F=h_1h_2·s, with prefixes h←t=h1⋯ht h_t=h_1·s h_t, define e¯QM(ℱ) e_Q^M( F) :=lim infT→∞1T∑t=1TeQM(h←t), := _T→∞ 1T _t=1^Te_Q^M( h_t), (228) e¯VM(ℱ) e_V^M( F) :=lim infT→∞1T∑t=1TeVM(h←t). := _T→∞ 1T _t=1^Te_V^M( h_t). (229) We apply these quantities below to finite-dimensional classical world models along ℱtick F_tick. Lemma 21 already determines the relative true-world action-values: Probe differs from Maintain by S(t)−λS(t)-λ, while Wait is strictly below Maintain. To obtain the absolute value functions needed below, it therefore remains only to compute the common baseline supplied by Maintain. Because Maintain gives the immediate reward −η-η and resets the clock, this requires a single Bellman step. For convenience, define the continuous 2π2π-periodic function H(x) H(x) :=(f(x)−λ)+, := (f(x)-λ )_+, (230) and the age-independent reset baseline bγ b_γ :=−η+γV⋆opt(h←0). :=-η+γ V_ ^opt( h_0). (231) Lemma 26 (True FRDN value functions). For the rewards in (169), every γ∈[0,1)γ∈[0,1), and every t≥0t≥ 0, the optimal value and Probe action-value at the all-Tick prefix h←t h_t satisfy V⋆opt(h←t) V_ ^opt( h_t) =−η+γV⋆opt(h←0)+(S(t)−λ)+, =-η+γ V_ ^opt( h_0)+ (S(t)-λ )_+, (232) Q⋆opt(h←t,P) Q_ ^opt( h_t,P) =−η+γV⋆opt(h←0)+S(t)−λ. =-η+γ V_ ^opt( h_0)+S(t)-λ. (233) In particular, for t≥1t≥ 1, V⋆opt(h←t) V_ ^opt( h_t) =−η+γV⋆opt(h←0)+H(tα), =-η+γ V_ ^opt( h_0)+H(tα), (234) Q⋆opt(h←t,P) Q_ ^opt( h_t,P) =−η+γV⋆opt(h←0)+f(tα)−λ. =-η+γ V_ ^opt( h_0)+f(tα)-λ. (235) Moreover, the reset value is V⋆opt(h←0) V_ ^opt( h_0) =−η+(S(0)−λ)+1−γ. = -η+ (S(0)-λ )_+1-γ. (236) Proof. Fix t≥0t≥ 0. Maintain gives the deterministic immediate reward −η-η and resets the clock to age zero. Hence, by the Bellman action-value relation (58), Q⋆opt(h←t,M) Q_ ^opt( h_t,M) =−η+γV⋆opt(h←0)=bγ. =-η+γ V_ ^opt( h_0)=b_γ. (237) Thus, bγb_γ is the age-independent action-value of Maintain. The first relation in (170) gives Q⋆opt(h←t,P) Q_ ^opt( h_t,P) =Q⋆opt(h←t,M)+S(t)−λ =Q_ ^opt( h_t,M)+S(t)-λ =bγ+S(t)−λ =b_γ+S(t)-λ =−η+γV⋆opt(h←0)+S(t)−λ. =-η+γ V_ ^opt( h_0)+S(t)-λ. (238) This proves (233). The second relation in (170) shows that Wait is strictly below Maintain. Therefore, the value–action-value relation (59) reduces to V⋆opt(h←t) V_ ^opt( h_t) =maxQ⋆opt(h←t,M),Q⋆opt(h←t,P) = \Q_ ^opt( h_t,M),Q_ ^opt( h_t,P) \ =maxbγ,bγ+S(t)−λ = \b_γ,b_γ+S(t)-λ \ =bγ+(S(t)−λ)+ =b_γ+ (S(t)-λ )_+ =−η+γV⋆opt(h←0)+(S(t)−λ)+. =-η+γ V_ ^opt( h_0)+ (S(t)-λ )_+. (239) This proves (232). Taking t=0t=0 in this identity gives V⋆opt(h←0) V_ ^opt( h_0) =−η+γV⋆opt(h←0)+(S(0)−λ)+. =-η+γ V_ ^opt( h_0)+ (S(0)-λ )_+. (240) Solving this scalar fixed-point equation yields V⋆opt(h←0) V_ ^opt( h_0) =−η+(S(0)−λ)+1−γ, = -η+ (S(0)-λ )_+1-γ, (241) which is (236). Finally, for t≥1t≥ 1, (98) gives S(t)=f(tα)S(t)=f(tα), and hence (S(t)−λ)+ (S(t)-λ )_+ =(f(tα)−λ)+=H(tα). = (f(tα)-λ )_+=H(tα). (242) Substitution into (232) and (233) gives (234) and (235), respectively. ∎ The baseline bγb_γ in (231) is the age-independent action-value of Maintain. The Probe action-value and optimal value can therefore be written as Q⋆opt(h←t,P) Q_ ^opt( h_t,P) =bγ+S(t)−λ, =b_γ+S(t)-λ, V⋆opt(h←t) V_ ^opt( h_t) =bγ+(S(t)−λ)+. =b_γ+ (S(t)-λ )_+. (243) Thus, Probe adds the signed, age-dependent advantage S(t)−λS(t)-λ, while optimization between Probe and Maintain replaces this signed profile by its positive part. Consequently, for t≥1t≥ 1, the discount factor changes only the additive baseline bγb_γ; the nonconstant phase profiles f−λf-λ and H are independent of γ. Lemma 27 (Residue limits of finite-dimensional classical value functions). Fix γ∈[0,1)γ∈[0,1) and an N-dimensional classical world model over (FRDN,FRDN)(A_FRDN,Y_FRDN). For every r∈0,…,LN−1r∈\0,…,L_N-1\, there exist constants qP,rq_P,r and vrv_r such that QCopt(h←nLN+r,P) Q_C^opt( h_nL_N+r,P) ⟶qP,r, q_P,r, VCopt(h←nLN+r) V_C^opt( h_nL_N+r) ⟶vr. v_r. (244) Proof. If ΦC(t0)=0 _C(t_0)=0 for some t0t_0, the fixed-reference convention makes EC(h←t)E_C( h_t) equal to zrefz_ref for every t≥t0t≥ t_0. Otherwise, Corollary 17 gives, for each residue r, a distribution over states z∞,rz_∞,r such that EC(h←nLN+r)→z∞,rE_C( h_nL_N+r)→ z_∞,r. Hence such a limit z∞,rz_∞,r exists in either case. The bridge relations (65) and (66), together with Lemma 22, give (244). ∎ We first use the Probe action to lower-bound the action-value error. Theorem 28 (Action-value error for finite classical models). Fix the rewards in (169). For every γ∈[0,1)γ∈[0,1), every N∈ℕN , and every N-dimensional classical world model over (FRDN,FRDN)(A_FRDN,Y_FRDN), e¯QC(ℱtick) e_Q^C\! ( F_tick ) ≥κFRDN>0. ≥ _FRDN>0. (245) Proof. Fix the classical model and a residue r∈0,…,LN−1r∈\0,…,L_N-1\. By Lemma 27, cn,r c_n,r :=λ+QCopt(h←nLN+r,P)−bγ :=λ+Q_C^opt( h_nL_N+r,P)-b_γ ⟶cr:=λ+qP,r−bγ, c_r:=λ+q_P,r-b_γ, (246) where bγ=−η+γV⋆opt(h←0)b_γ=-η+γ V_ ^opt( h_0). For nLN+r≥1nL_N+r≥ 1, Lemma 26 gives |Q⋆opt(h←nLN+r,P)−QCopt(h←nLN+r,P)| |Q_ ^opt( h_nL_N+r,P)-Q_C^opt( h_nL_N+r,P) | =|f((nLN+r)α)−cn,r|. = |f ((nL_N+r)α )-c_n,r |. (247) The possible term with nLN+r=0nL_N+r=0 does not affect the average. Applying Lemma 18 to f gives limK→∞1K∑n=0K−1|Q⋆opt(h←nLN+r,P)−QCopt(h←nLN+r,P)| _K→∞ 1K _n=0^K-1 |Q_ ^opt( h_nL_N+r,P)-Q_C^opt( h_nL_N+r,P) | =12π∫02π|f(x)−cr|x≥κFRDN. = 12π _0^2π|f(x)-c_r|\,dx≥ _FRDN. (248) For the final inequality, if cr∉[0,1]c_r∉[0,1], project it onto [0,1][0,1]. Since f(x)∈[0,1]f(x)∈[0,1], this projection cannot increase the integrand, and the claim follows from (152). By (226), the error eQCe_Q^C dominates the Probe action-value error. Decomposing an average of length KLNKL_N into the LNL_N residue classes and using (248) therefore gives lim infK→∞1KLN∑t=0KLN−1eQC(h←t) _K→∞ 1KL_N _t=0^KL_N-1e_Q^C( h_t) ≥κFRDN. ≥ _FRDN. (249) For fixed γ, all value functions are bounded, so shifting from the indices 0,…,KLN−10,…,KL_N-1 to 1,…,KLN1,…,KL_N is asymptotically irrelevant. For arbitrary T, let K=⌊T/LN⌋K= T/L_N . Nonnegativity gives 1T∑t=1TeQC(h←t) 1T _t=1^Te_Q^C( h_t) ≥KLNT(1KLN∑t=1KLNeQC(h←t)), ≥ KL_NT ( 1KL_N _t=1^KL_Ne_Q^C( h_t) ), (250) and KLN/T→1KL_N/T→ 1. This proves (245). ∎ The previous theorem uses a single action. For the optimal value, the relevant quantity that gives the lower bound is κval _val :=minc∈[0,1−λ]12π∫02π|H(x)−c|x. := _c∈[0,1-λ] 12π _0^2π|H(x)-c|\,dx. (251) Equation (192) shows that H vanishes on one nonempty open set and is positive on another. Hence H is continuous and nonconstant, so κval>0 _val>0. Theorem 29 (Optimal-value error for finite classical models). Fix the rewards in (169). For every γ∈[0,1)γ∈[0,1), every N∈ℕN , and every N-dimensional classical world model over (FRDN,FRDN)(A_FRDN,Y_FRDN), e¯VC(ℱtick) e_V^C\! ( F_tick ) ≥κval>0. ≥ _val>0. (252) Proof. Fix the classical model and a residue r∈0,…,LN−1r∈\0,…,L_N-1\. By Lemma 27, dn,r d_n,r :=VCopt(h←nLN+r)−bγ⟶dr:=vr−bγ, :=V_C^opt( h_nL_N+r)-b_γ d_r:=v_r-b_γ, (253) where bγ=−η+γV⋆opt(h←0)b_γ=-η+γ V_ ^opt( h_0). For nLN+r≥1nL_N+r≥ 1, Lemma 26 gives |V⋆opt(h←nLN+r)−VCopt(h←nLN+r)| |V_ ^opt( h_nL_N+r)-V_C^opt( h_nL_N+r) | =|H((nLN+r)α)−dn,r|. = |H ((nL_N+r)α )-d_n,r |. (254) Lemma 18 applied to H therefore gives limK→∞1K∑n=0K−1|V⋆opt(h←nLN+r)−VCopt(h←nLN+r)| _K→∞ 1K _n=0^K-1 |V_ ^opt( h_nL_N+r)-V_C^opt( h_nL_N+r) | =12π∫02π|H(x)−dr|x≥κval. = 12π _0^2π|H(x)-d_r|\,dx≥ _val. (255) As before, the possible index nLN+r=0nL_N+r=0 is irrelevant. If dr∉[0,1−λ]d_r∉[0,1-λ], projection onto this interval cannot increase the distance from H(x)∈[0,1−λ]H(x)∈[0,1-λ], which proves the final inequality. Decomposing into residue classes as in the proof of Theorem 28, and then accounting for the incomplete final block, proves (252). ∎ Taking εest:=minκFRDN,κval>0 _est:= \ _FRDN, _val\>0, Theorems 28 and 29 show that both mean errors are at least εest _est along the same fixed trajectory ℱtick F_tick, uniformly over all finite classical memory dimensions and all γ∈[0,1)γ∈[0,1). Appendix C Exact qutrit realization This appendix constructs a quantum world model FRDN Q_FRDN with a three-dimensional memory, together with a separate encoder EQE_Q, such that the pair (FRDN,EQ)( Q_FRDN,E_Q) exactly realizes the FRDN true world of Definition 15. The construction adapts the qutrit realization of the FRDN renewal process in 17 to the quantum-instrument world models of Definition 14. We first specify the instrument associated with each action, then identify the memory states used to initialize history-dependent queries, and finally verify the probability and memory-update conditions in (22) and (23). C.1 Qutrit instruments for Wait, Maintain, and Probe Let ℋQ≃ℂ3H_Q ^3, with computational basis |0⟩,|1⟩,|2⟩\|0 ,|1 ,|2 \, and let Π01:=|0⟩⟨0|+|1⟩⟨1|. _01:=|0 0|+|1 1|. (256) We write X and Z for the Pauli matrices on Π01ℋQ _01H_Q, and write 3I_3 for the identity operator on ℋQH_Q. Since α/π∉ℚα/π , one has 0<1−λ|1−λeiα|<1. 0< 1-λ|1-λ e^iα|<1. (257) Because tanh(2r) (2r) increases continuously from 00 to 11 for r>0r>0, there is a unique r>0r>0 satisfying tanh(2r)=1−λ|1−λeiα|. (2r)= 1-λ|1-λ e^iα|. (258) We start by defining the operators that will form the Wait branch. Define the Tick operator AT A_T :=Kr,αΠ01, :=K_r,α _01, Kr,α K_r,α :=λe−rXeiαZ/2erX. := λ\,e^-rXe^iα Z/2e^rX. (259) Thus AT|2⟩=0A_T|2 =0. For the choice (258), a direct calculation gives det(Kr,α†Kr,α)=λ2,Tr(Kr,α†Kr,α)=1+λ2. (K_r,α K_r,α)=λ^2, (K_r,α K_r,α)=1+λ^2. (260) The eigenvalues of Kr,α†Kr,αK_r,α K_r,α are therefore 11 and λ2λ^2. Hence AT†AT≤3A_T A_T _3, so 3−AT†ATI_3-A_T A_T is positive. The two effects AT†ATA_T A_T and 3−AT†ATI_3-A_T A_T will represent Tick and Break, respectively. We now specify the reset state. Choose θα _α by tanθα=e2rtan[12arctan(λsinα1−λcosα)], _α=e^2r \! [ 12 \! ( λ α1-λ α ) ], (261) and define |ξ⟩:=eiθα2|0⟩+e−iθα2|1⟩. |ξ := e^i _α 2|0 + e^-i _α 2|1 . (262) Finally set ω ω :=λ(1+λ)sin2(α/2)(1−λ)(1+λ2−2λcosα), := λ(1+λ) ^2(α/2)(1-λ)(1+λ^2-2λ α), (263) ρ0 _0 :=ω|ξ⟩⟨ξ|+(1−ω)|2⟩⟨2|. :=ω|ξ ξ|+(1-ω)|2 2|. (264) For 0<λ≤1/20<λ≤ 1/2, one has 0<ω≤10<ω≤ 1. For each action, the following are the only nonzero branches of the corresponding instrument; all other branches indexed by y∈FRDNy _FRDN are the zero map. For Wait, ℰyTW(W)(ρ) _y_T^W^(W)(ρ) :=ATρAT†, :=A_Tρ A_T , ℰyBW(W)(ρ) _y_B^W^(W)(ρ) :=Tr[(3−AT†AT)ρ]ρ0. :=Tr\! [(I_3-A_T A_T)ρ ] _0. (265) For Maintain, ℰyBM(M)(ρ) _y_B^M^(M)(ρ) :=Tr(ρ)ρ0. :=Tr(ρ) _0. (266) For Probe, ℰyTP(P)(ρ) _y_T^P^(P)(ρ) :=Tr[AT†ATρ]ρ0, :=Tr\! [A_T A_Tρ ] _0, (267) ℰyBP(P)(ρ) _y_B^P^(P)(ρ) :=Tr[(3−AT†AT)ρ]ρ0. :=Tr\! [(I_3-A_T A_T)ρ ] _0. (268) These maps define a valid quantum instrument for each action. The Wait–Tick branch is a Kraus map, while the remaining nonzero branches are measure-and-prepare maps associated with positive effects. Moreover, Tr(AT†ATρ)+Tr[(3−AT†AT)ρ]=Trρ. (A_T A_Tρ)+Tr\! [(I_3-A_T A_T)ρ ]=Trρ. (269) Thus the Wait and Probe instruments are trace preserving when their branches are summed. The Maintain map is trace preserving because Trρ0=1Tr _0=1. C.2 Clock memories and query initialization To derive the normalized clock memories, it is useful first to retain the probability of the all-Tick branch in the trace of a subnormalized operator. Define ρ~t:=(ℰyTW(W))t(ρ0),t≥0. ρ_t:= (E_y_T^W^(W) )^t( _0), t≥ 0. (270) For t≥1t≥ 1, the first Tick branch removes the |2⟩|2 component, hence ρ~t ρ_t =ωKr,αt|ξ⟩⟨ξ|(Kr,α†)t, =ω\,K_r,α^t|ξ ξ|(K_r,α )^t, (271) Kr,αt K_r,α^t =λt/2e−rXeitαZ/2erX. =λ^t/2e^-rXe^itα Z/2e^rX. (272) Using (258)–(264), one obtains Trρ~t ρ_t =λt(12(1−λ)−14eitα1−λeiα−14e−itα1−λe−iα) =λ^t ( 12(1-λ)- 14 e^itα1-λ e^iα- 14 e^-itα1-λ e^-iα ) =∑ℓ=t∞λℓsin2(ℓα2)=Φ(t). = _ =t^∞λ ^2\! ( α2 )= (t). (273) The case t=0t=0 gives Trρ0=1=Φ(0)Tr _0=1= (0). Define the normalized clock memories ρt:=ρ~tΦ(t),t≥0. _t:= ρ_t (t), t≥ 0. (274) For t≥1t≥ 1, these states are pure ρt=|ηt⟩⟨ηt|,|ηt⟩=e−rXeitαZ/2erX|ξ⟩‖e−rXeitαZ/2erX|ξ⟩‖. _t=| _t _t|, | _t = e^-rXe^itα Z/2e^rX|ξ \|e^-rXe^itα Z/2e^rX|ξ \|. (275) A conditional rollout after t≥1t≥ 1 consecutive Wait–Tick outcomes may therefore be initialized by preparing |ηt⟩| _t in the qubit subspace, while a reset initializes the mixed state ρ0 _0. Moreover, for every t≥1t≥ 1, there exists a qutrit unitary UtU_t such that Ut|0⟩ U_t|0 =|ηt⟩, =| _t , ρt _t =Ut|0⟩⟨0|Ut†. =U_t|0 0|U_t . (276) Thus the clock state required after t consecutive Wait–Tick outcomes can be prepared directly. The mixed reset state ρ0 _0 is initialized by its fixed preparation procedure. Let EQE_Q be the canonical encoder associated with the instrument products as in (91), choosing ρref=ρ0 _ref= _0 on zero-probability histories. Since every nonzero instrument operation other than Wait–Tick resets the memory, the encoder satisfies EQ(h) E_Q(h) =ρℓ(h),h∈ℋ⋆. = _ (h), h∈ H_ . (277) In particular, EQ(h←t)=ρtE_Q( h_t)= _t. C.3 Exactness of the qutrit world model The reset state (264) and the instrument operations (265)–(268) define the three-dimensional quantum world model FRDN Q_FRDN :=(FRDN,FRDN,ℋQ,ρ0,FRDN), := (A_FRDN,Y_FRDN,H_Q, _0, E_FRDN ), FRDN E_FRDN :=ℰy(a)(a,y)∈FRDN×FRDN, := \E_y^(a) \_(a,y) _FRDN×Y_FRDN, (278) where all unlisted instrument operations are the zero map. The encoder EQE_Q in (277) is the separate initialization interface used for history-indexed queries. We use PrQPr_Q and TQT_Q below as shorthand for the outcome law and conditional updates induced by FRDN E_FRDN. Using ρ~t+1=ℰyTW(W)(ρ~t) ρ_t+1=E_y_T^W^(W)( ρ_t) and Trρ~t=Φ(t)Tr ρ_t= (t), the nonzero branch identities are, for every t≥0t≥ 0, PrQ(yTW∣ρt,W) _Q(y_T^W _t,W) =Trρ~t+1Φ(t)=Φ(t+1)Φ(t)=S(t), = Tr ρ_t+1 (t)= (t+1) (t)=S(t), TQ,yTWW(ρt) T_Q,y_T^W^W( _t) =ρt+1, = _t+1, PrQ(yBW∣ρt,W) _Q(y_B^W _t,W) =1−S(t), =1-S(t), TQ,yBWW(ρt) T_Q,y_B^W^W( _t) =ρ0, = _0, PrQ(yBM∣ρt,M) _Q(y_B^M _t,M) =1, =1, TQ,yBMM(ρt) T_Q,y_B^M^M( _t) =ρ0, = _0, PrQ(yTP∣ρt,P) _Q(y_T^P _t,P) =S(t), =S(t), TQ,yTPP(ρt) T_Q,y_T^P^P( _t) =ρ0, = _0, PrQ(yBP∣ρt,P) _Q(y_B^P _t,P) =1−S(t), =1-S(t), TQ,yBPP(ρt) T_Q,y_B^P^P( _t) =ρ0. = _0. (279) We write VQπ(h)V_Q^π(h) and Qπ(h,a)Q_Q^π(h,a) for the history-indexed policy values of FRDN Q_FRDN defined in Definition 9, and VQopt(h)V_Q^opt(h) and QQopt(h,a)Q_Q^opt(h,a) for its history-indexed optimal values defined in Definition 11. The optimal quantities have the equivalent memory-state representations given by (65) and (66). Theorem 30 (Exact qutrit realization). The tuple FRDN Q_FRDN defined above is a quantum world model of memory dimension 33 in the sense of Definition 14. Together with the encoder EQE_Q in (277), it forms an exact model–encoder pair for the FRDN true world ⋆FRDN W_ ^FRDN of Definition 15, in the sense of Definition 7. Consequently, for every discount factor γ∈[0,1)γ∈[0,1), policy π∈Πℋπ∈ _ H, reachable history h∈ℋ⋆h∈ H_ , and action a∈FRDNa _FRDN, the policy values defined in Definition 9 satisfy VQπ(h) V_Q^π(h) =V⋆π(h), =V_ ^π(h), Qπ(h,a) Q_Q^π(h,a) =Q⋆π(h,a). =Q_ ^π(h,a). (280) The optimal quantities defined in Definitions 10 and 11 likewise satisfy VQopt(h) V_Q^opt(h) =V⋆opt(h), =V_ ^opt(h), QQopt(h,a) Q_Q^opt(h,a) =Q⋆opt(h,a). =Q_ ^opt(h,a). (281) Hence, by (226) and (227), eQQ(h) e_Q^Q(h) =eVQ(h)=0 =e_V^Q(h)=0 (282) after every reachable history. For the rewards (169), every γ∈[0,1)γ∈[0,1), and every reachable history h∈ℋ⋆h∈ H_ , each model-greedy action a^Q(h) a_Q(h) satisfying (162) is true-world optimal a^Q(h)∈argmaxa∈FRDNQ⋆opt(h,a). a_Q(h)∈ *arg\,max_a _FRDNQ_ ^opt(h,a). (283) The corresponding decision margin and decision loss satisfy gQ(h) g_Q(h) =g⋆(h), =g_ (h), ℓQ(h) _Q(h) =0. =0. (284) Consequently, for every reachable action–outcome trajectory ℱ F, e¯Q(ℱ) e_Q^Q( F) =e¯VQ(ℱ)=ℓ¯Q(ℱ)=0. = e_V^Q( F)= _Q( F)=0. (285) Proof. Fix a reachable history h∈ℋ⋆h∈ H_ and set t=ℓ(h)t= (h). By the encoder construction (277), EQ(h) E_Q(h) =ρt. = _t. (286) Using the outcome and update rules (88) and (89), the branch identities (279), and the zero maps for all unlisted outcomes, the qutrit model reproduces the true-world kernel (109). Therefore, for every a∈FRDNa _FRDN and y∈FRDNy _FRDN, PrQ(y∣EQ(h),a) _Q\! (y E_Q(h),a ) =Pr⋆(y∣h,a). =Pr_ (y h,a). (287) This is precisely the probability condition (22) in Definition 7. The same branch identities reproduce the clock recursion (106): Wait–Tick maps ρt _t to ρt+1 _t+1, whereas Wait–Break, Maintain, and either Probe outcome prepare ρ0 _0. Hence, whenever Pr⋆(y∣h,a)>0Pr_ (y h,a)>0, TQ,ya(EQ(h)) T_Q,y^a\! (E_Q(h) ) =EQ(hay). =E_Q(hay). (288) This is the memory-update condition (23). The pair (FRDN,EQ)( Q_FRDN,E_Q) is therefore exact on the complete reachable history tree according to Definition 7. The policy-value equalities now follow from Theorem 12. Taking the suprema over the same class of history policies gives the corresponding equalities for the optimal values. Equality of the quantum and true-world optimal action-values implies that every action satisfying the model-greedy rule (162) is true-world optimal. Substitution into (164) and (165) gives gQ(h) g_Q(h) =g⋆(h), =g_ (h), ℓQ(h) _Q(h) =0. =0. (289) Since these identities hold after every reachable history, the corresponding trajectory means vanish along every reachable action–outcome trajectory. ∎ Appendix D Numerical procedures We numerically optimize the finite-dimensional classical world models introduced above. Specifically, we specialize the classical world model definition in Definition 13 to the Wait–Tick branch and focus on DT:=DyTW(W)D_T:=D_y_T^W^(W), where W denotes the Wait action and yTWy_T^W denotes the Tick outcome under Wait. The initial hidden distribution over states is z0∈ΔN−1z_0∈ _N-1, and DT∈ℝ≥0N×ND_T _≥ 0^N× N is column-substochastic: ⊤DT≤⊤.1 D_T 1 . For the consecutive Wait–Tick history h←t h_t, the classical survival probability is ΦC(t):=⟨,DTtz0⟩. _C(t):= 1,D_T^tz_0 . (290) Whenever ΦC(t)>0 _C(t)>0, its conditional probability of one further Tick is SC(t) S_C(t) =PrC(yTW∣EC(h←t),W)=ΦC(t+1)ΦC(t). =Pr_C\! (y_T^W E_C( h_t),W )= _C(t+1) _C(t). (291) This approximates the true conditional probability S(t) S(t) =Pr⋆(yTW∣h←t,W)=Φ(t+1)Φ(t). =Pr_ \! (y_T^W h_t,W )= (t+1) (t). (292) We optimize DTD_T and z0z_0 by minimizing the weighted Bernoulli cross-entropy ℒ=−∑t=0Tfitwt[S(t)logSC(t)+(1−S(t))log(1−SC(t))]. =- _t=0^T_fitw_t [S(t) S_C(t)+ (1-S(t) ) (1-S_C(t) ) ]. (293) For each classical memory dimension N, we fix the fitting horizon to Tfit=1000T_fit=1000 and fit the model on the complete age grid t=0,…,Tfitt=0,…,T_fit. Thus, each optimization step uses 10011001 analytically evaluated conditional Tick probabilities. We use the natural-frequency weighting wt=Φ(t)∑s=0TfitΦ(s),w_t= (t) _s=0^T_fit (s), which weights each age according to its occurrence probability under the uncontrolled renewal process. We optimize the initial distribution over states z0z_0 and the Tick-transition matrix DTD_T using Adam with learning rate 10−310^-3 for 30003000 optimization steps. After each Adam update, we project z0z_0 onto ΔN−1 _N-1 and each augmented transition column onto ΔN _N. This projected-gradient procedure preserves non-negativity, normalization of the initial distribution over states, and column-substochasticity of DTD_T, while allowing exact zero entries. For each N, we perform 1010 independent random restarts and retain the feasible iterate with the lowest training loss across all restarts. The resulting parameters provide a best-found classical approximation of memory dimension N to the FRDN conditional Tick sequence. After fitting, the selected model is evaluated without further optimization or model selection. The evaluation horizon is specified separately for each figure to display the relevant short- or long-horizon behavior. References Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15619–15629. Cited by: §I. Bardes et al. (2024) A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471. Cited by: §I. Barry et al. (2014) J. Barry, D. T. Barry, and S. Aaronson Quantum partially observable markov decision processes. Phys. Rev. A 90, p. 032311. External Links: Document, Link Cited by: §A.5. Baum and Petrie (1966) L. E. Baum and T. Petrie Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics 37 (6), p. 1554 – 1563. External Links: Document, Link Cited by: §I. Bellemare et al. (2016) M. G. Bellemare, G. Ostrovski, A. Guez, P. S. Thomas, and R. Munos Increasing the action gap: new operators for reinforcement learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, p. 1476–1483. Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Berman and Plemmons (1994) A. Berman and R. J. Plemmons Nonnegative matrices in the mathematical sciences. Classics in Applied Mathematics, Vol. 9, SIAM, Philadelphia. Cited by: §B.2. Bertsekas (2012) D. Bertsekas Dynamic programming and optimal control. Vol. 4, Athena scientific. Cited by: §A.2.1. Brunner et al. (2014) N. Brunner, M. Kaplan, A. Leverrier, and P. Skrzypczyk Dimension of physical systems, information processing, and thermodynamics. New Journal of Physics 16 (12), p. 123050. External Links: Document Cited by: §A.1, §I. Cassandra et al. (1994) A. R. Cassandra, L. P. Kaelbling, and M. L. Littman Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence, AAAI’94, p. 1023–1028. Cited by: §I. Chua et al. (2018) K. Chua, R. Calandra, R. McAllister, and S. Levine Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, p. 4759–4770. Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Deng et al. (2023) F. Deng, J. Park, and S. Ahn Facing off world model backbones: RNNs, transformers, and S4. In Advances in Neural Information Processing Systems, Vol. 36, p. 72904–72930. External Links: Document, Link Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Dharmadhikari and Nadkarni (1970) S. W. Dharmadhikari and M. G. Nadkarni Some regular and non-regular functions of finite Markov chains. The Annals of Mathematical Statistics 41 (1), p. 207–213. Cited by: Appendix B, §IV.1. Dharmadhikari (1963) S. W. Dharmadhikari Sufficient conditions for a stationary process to be a function of a finite Markov chain. The Annals of Mathematical Statistics 34 (3), p. 1033–1041. Cited by: Appendix B, §IV.1. Dunjko et al. (2016) V. Dunjko, J. M. Taylor, and H. J. Briegel Quantum-enhanced machine learning. Phys. Rev. Lett. 117, p. 130501. External Links: Document, Link Cited by: §I. Elliott et al. (2022) T. J. Elliott, M. Gu, A. J. P. Garner, and J. Thompson Quantum adaptive agents with efficient long-term memories. Phys. Rev. X 12, p. 011007. External Links: Document, Link Cited by: §I. Ephraim and Merhav (2002) Y. Ephraim and N. Merhav Hidden markov processes. IEEE Transactions on Information Theory 48 (6). External Links: Document Cited by: §I. Fanizza et al. (2024) M. Fanizza, J. Lumbreras, and A. Winter Quantum theory in finite dimension cannot explain every general process with finite memory. Communications in Mathematical Physics 405 (2), p. 50. External Links: Document Cited by: §A.4, §A.5, Appendix C, §I. Farahmand (2011) A. Farahmand Action-gap phenomenon in reinforcement learning. In Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (Eds.), Vol. 24, p. . External Links: Link Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Fox and Rubin (1968) M. Fox and H. Rubin Functions of processes with Markovian states. The Annals of Mathematical Statistics 39 (3), p. 938–946. Cited by: Appendix B, §IV.1. Gallego et al. (2010) R. Gallego, N. Brunner, C. Hadley, and A. Acín Device-independent tests of classical and quantum dimensions. Physical Review Letters 105 (23). External Links: ISSN 1079-7114, Link, Document Cited by: §A.1, §I. Ghafari et al. (2019) F. Ghafari, N. Tischler, J. Thompson, M. Gu, L. K. Shalm, V. B. Verma, S. W. Nam, R. B. Patel, H. M. Wiseman, and G. J. Pryde Dimensional quantum memory advantage in the simulation of stochastic processes. Phys. Rev. X 9, p. 041013. External Links: Document, Link Cited by: §A.1, §I. Grimm et al. (2020) C. Grimm, A. Barreto, S. Singh, and D. Silver The value equivalence principle for model-based reinforcement learning. Advances in neural information processing systems 33, p. 5541–5552. Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Gu et al. (2012) M. Gu, K. Wiesner, E. Rieper, and V. Vedral Quantum mechanics can reduce the complexity of classical models. Nature communications 3 (1), p. 762. External Links: Document Cited by: §A.1, §I. Ha and Schmidhuber (2018a) D. Ha and J. Schmidhuber World models. arXiv preprint arXiv:1803.10122. Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Ha and Schmidhuber (2018b) D. Ha and J. Schmidhuber World models. Zenodo. External Links: Document, Link Cited by: §I. Hafner et al. (2020) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations, Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2555–2565. External Links: Link Cited by: §I. Hafner et al. (2025) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse control tasks through world models. Nature 640 (8059), p. 647–653. Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Hardy (2001) L. Hardy Quantum theory from five reasonable axioms. External Links: quant-ph/0101012, Link Cited by: §I. Hausknecht and Stone (2015) M. Hausknecht and P. Stone Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposium Series, Cited by: §I. Horn and Johnson (1985) R. A. Horn and C. R. Johnson Matrix analysis. Cambridge University Press. Cited by: §B.2. Hsu et al. (2008) D. Hsu, S. M. Kakade, and T. Zhang A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences 78, p. 1460–1480. External Links: Document, ISSN 00220000, Link Cited by: §A.4. Jaeger (2000) H. Jaeger Observable operator models for discrete stochastic time series. Neural Comput. 12 (6), p. 1371–1398. External Links: ISSN 0899-7667, Link, Document Cited by: §A.4. Jin et al. (2020) C. Jin, S. M. Kakade, A. Krishnamurthy, and Q. Liu Sample-efficient reinforcement learning of undercomplete pomdps. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. External Links: ISBN 9781713829546 Cited by: §A.4. Kaelbling et al. (1998) L. P. Kaelbling, M. L. Littman, and A. R. Cassandra Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1), p. 99–134. External Links: ISSN 0004-3702, Document, Link Cited by: §A.4, §I, §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Korsky and Berwick (2019) S. A. Korsky and R. C. Berwick On the computational power of rnns. CoRR abs/1906.06349. External Links: Link Cited by: §I. Krogh et al. (1994) A. Krogh, M. Brown, I. S. Mian, K. Sjölander, and D. Haussler Hidden markov models in computational biology: applications to protein modeling. Journal of Molecular Biology 235 (5), p. 1501–1531. External Links: Document, Link Cited by: §I. Kuipers and Niederreiter (2012) L. Kuipers and H. Niederreiter Uniform distribution of sequences. Courier Corporation. Cited by: §B.3, §IV.2. Laird and Clark (2025) E. J. Laird and C. Clark On memory: a comparison of memory mechanisms in world models. External Links: 2512.06983, Document, Link Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. LeCun et al. (2022) Y. LeCun et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62 (1), p. 1–62. Cited by: §I. Littman and Sutton (2001) M. Littman and R. S. Sutton Predictive representations of state. Advances in neural information processing systems 14. Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Liu and Feng (2024) H. X. Liu and S. Feng Curse of rarity for autonomous vehicles. nature communications 15 (1), p. 4808. External Links: Document Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Liu et al. (2022) Q. Liu, A. Chung, C. Szepesvari, and C. Jin When is partially observable reinforcement learning not scary?. In Proceedings of Thirty Fifth Conference on Learning Theory, P. Loh and M. Raginsky (Eds.), Proceedings of Machine Learning Research, Vol. 178, p. 5175–5220. External Links: Link Cited by: §A.4. Lumbreras et al. (2026) J. Lumbreras, R. C. Huang, Y. Hu, M. Fanizza, and M. Gu Reinforcement learning for quantum processes with memory. arXiv preprint arXiv:2603.25138. Cited by: §A.5. M. Moerland et al. (2023) T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker Model-based reinforcement learning: a survey. Foundations and Trends in Machine Learning 16 (1), p. 1–118. External Links: Document, Link Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Maes et al. (2026) L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §I. Marzen and Crutchfield (2017) S. E. Marzen and J. P. Crutchfield Nearly maximally predictive features and their dimensions. Physical Review E 95 (5), p. 051301. Cited by: footnote 1. Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518 (7540), p. 529–533. External Links: Document Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Monràs and Winter (2016) A. Monràs and A. Winter Quantum learning of classical stochastic processes: The completely positive realization problem. Journal of Mathematical Physics 57 (1), p. 015219. Cited by: §A.5, §I. O’Kelly et al. (2018) M. O’Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi Scalable end-to-end autonomous vehicle testing via rare-event simulation. Advances in neural information processing systems 31. Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Puterman (2014) M. L. Puterman Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons. Cited by: §A.2.1, §I, §IV.2. Rabiner (1989) L.R. Rabiner A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE 77 (2), p. 257–286. External Links: Document Cited by: §A.4, §I, §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Robine et al. (2023) J. Robine, M. Höftmann, T. Uelwer, and S. Harmeling Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Ruebeck et al. (2018) J. B. Ruebeck, R. G. James, J. R. Mahoney, and J. P. Crutchfield Prediction and generation of binary markov processes: can a finite-state fox catch a markov mouse?. Chaos: An Interdisciplinary Journal of Nonlinear Science 28 (1), p. 013109. External Links: ISSN 1054-1500, Document, Link, https://pubs.aip.org/aip/cha/article-pdf/doi/10.1063/1.5003041/14614248/013109_1_online.pdf Cited by: footnote 1. Schrittwieser et al. (2020) J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature 588 (7839), p. 604–609. Cited by: An Irreducible Quantum Advantage in Aligning World Models with Reality. Seneta (2006) E. Seneta Non-negative matrices and markov chains. 2 edition, Springer Series in Statistics, Springer, New York. Cited by: §B.2, §IV.2. Shalizi and Crutchfield (2001) C. R. Shalizi and J. P. Crutchfield Computational mechanics: pattern and prediction, structure and simplicity. Journal of statistical physics 104 (3), p. 817–879. Cited by: footnote 1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. Second edition, The MIT Press. External Links: Link Cited by: §A.2.1, §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Sutton (1991) R. S. Sutton Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin 2 (4), p. 160–163. Cited by: §I, An Irreducible Quantum Advantage in Aligning World Models with Reality. Szepesvári (2010) C. Szepesvári Algorithms for reinforcement learning. Morgan & Claypool Publishers. Cited by: §A.2.1. Thompson et al. (2025) J. Thompson, P. M. Riechers, A. J. P. Garner, T. J. Elliott, and M. Gu Energetic advantages for quantum agents in online execution of complex strategies. Phys. Rev. Lett. 135, p. 160402. External Links: Document, Link Cited by: §I. Vidyasagar (2011) M. Vidyasagar The complete realization problem for hidden markov models: a survey and some new results. Mathematics of Control, Signals, and Systems 23, p. 1–65. External Links: Document Cited by: §A.4, §A.4, Appendix B. Yu et al. (2026) X. Yu, B. Peng, R. Xu, Y. Shen, P. He, S. Nath, N. Singh, J. Gao, and Z. Yu Reinforcement world model learning for llm-based agents. arXiv preprint arXiv:2602.05842. Cited by: §I. Zeng et al. (2023) P. Zeng, Y. He, F. R. Yu, and V. C. Leung Quantum reinforcement learning with quantum world model. In GLOBECOM 2023-2023 IEEE Global Communications Conference, p. 01–06. Cited by: §I. Zhang et al. (2019) A. Zhang, Z. C. Lipton, L. Pineda, K. Azizzadenesheli, A. Anandkumar, L. Itti, J. Pineau, and T. Furlanello Learning causal state representations of partially observable environments. arXiv preprint arXiv:1906.10437. Cited by: §I.