Paper deep dive
Does Learning Recurrent Networks Matter in Partially Observable RL?
Zekang Wang, Zhe He, Gaojie Jin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/12/2026, 7:02:24 AM
Summary
This paper investigates whether learning the recurrent network module is necessary for solving Partially Observable Markov Decision Processes (POMDPs) in Reinforcement Learning. The authors propose a parameter-efficient architecture that uses a frozen recurrent network (GRU or LRU) combined with a trainable nonlinear MLP readout, inspired by Reservoir Computing and Generalized Reservoir Computing theories. Experiments demonstrate that frozen recurrent networks can support long-term memory recall and achieve competitive performance across diverse POMDPs while reducing trainable parameters by over 50% compared to fully trainable counterparts.
Entities (9)
Relation Signals (8)
Frozen Recurrent Network → reduces → trainable parameters
confidence 95% · using less than half the number of trainable parameters required by their fully trainable counterparts.
Frozen Recurrent Network → solves → POMDP
confidence 95% · frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs
PQN → uses → Frozen Recurrent Network
confidence 95% · Overview of our PQN agent with frozen recurrence... We encode vector and visual observations differently.
PQN → uses → MLP Readout
confidence 95% · The right panel shows the architecture of our simple MLP in detail... Q_theta(h_t, .)
Frozen Recurrent Network → isbasedon → Reservoir Computing
confidence 90% · Motivated by the theoretical foundations of Reservoir Computing (RC)... we study a simple architecture that combines a fixed recurrent network
Frozen Recurrent Network → istheoreticallysupportedby → Generalized Reservoir Computing
confidence 90% · Generalized reservoir computing (GRC)... offers a broader perspective... This suggests that ESP is unnecessary for using frozen recurrent network as memory
LRU → satisfies → Echo State Property
confidence 85% · Thus, the frozen LRU cell is a stable linear reservoir satisfying the ESP.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.13197v1
- Canonical: https://arxiv.org/abs/2610.13197v1
Trouble viewing inline? Open PDF directly →
Full Text
77,999 characters extracted from source content.
Expand or collapse full text
Does Learning Recurrent Networks Matter in Partially Observable RL? Conference: ; ; Zekang Wang Affiliation: University of Macau, Macau SAR, China email: yc67543@um.edu.mo , Zhe He Affiliation: University of Macau, Macau SAR, China email: yc67202@um.edu.mo and Gaojie Jin Note: Corresponding author. Affiliation: University of Macau, Macau SAR, China email: gaojiejin@um.edu.mo © none Abstract. Reinforcement learning (RL) under partial observability typically relies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving partially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple architecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and surprisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP. Keywords: Reinforcement Learning, POMDP, Recurrent Neural Network, Reservoir Computing Observation oto_t Trainable CNN EψE_ψ (Optional) Frozen Recurrent Network ht=fω0(ht−1,ut)h_t=f_ _0(h_t-1,u_t) MLP Readout Qθ(ht,⋅)Q_θ(h_t,·) Environment Imagesu_th_ta_tVectorsObservation Feedback PQN Learning ℒTDL_TD PredictionsRewardsUpdate θ ψ (Images)Trainable ParametersFixed ParametersParameter-free ElementsData FlowParameter UpdateMLP Readouthth_t Linear Layer LayerNorm LeakyReLU ×L× L Linear Layer Output Qθ(ht,⋅)Q_θ(h_t,·) Figure 1. Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig. 7. Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig.~ fig:frozen-recurrent-layer. 1. Introduction Reinforcement learning (RL) is commonly formulated as a Markov Decision Process (MDP), in which the current state contains all information needed for optimal decision making (Sutton and Barto, 1998). In many real-world settings, however, agents cannot directly observe the underlying state and instead receive partial or noisy observations that do not satisfy the Markov property. Such problems are naturally modeled as partially observable Markov decision processes (POMDPs). By incorporating memory, an agent can summarize its interaction history into a latent state representation that captures information missing from the current observation, thereby approximating a Markovian representation and enabling standard RL methods to be applied more effectively in partially observable settings (Kaelbling et al., 1998). Recurrent neural networks (RNNs) (Hochreiter and Schmidhuber, 1997; Elman, 1990; Cho et al., 2014), Transformers (Vaswani et al., 2017; Katharopoulos et al., 2020; Parisotto et al., 2020; Banino et al., 2022; Ni et al., 2023; Pramanik et al., 2024), and state-space models (SSMs) (Gu et al., 2022a; Gu and Dao, 2024; Orvieto et al., 2023; Lu et al., 2023; Lu et al., 2024; Morad et al., 2024) have all been used as memory modules in model-free RL to summarize trajectory information and address POMDPs (Ni et al., 2022). These memory models are usually trained jointly with the policy or value function. However, this raises the question that is learning the memory module actually necessary? Several observations suggest that it may not always be. Tallec et al. (2018) report that using an untrained Mixture-Density Recurrent Neural Network (MDRNN) from Ha and Schmidhuber (2018) leads to only a 1.16% performance increase in a model-based RL setting. More recently, Gu et al. (2022b) show that freezing the recurrent and input matrices of S4 and S4D can preserve performance on certain tasks. These findings are closely related to Reservoir Computing (RC) (Jaeger, 2001; Maass et al., 2002; Lukoševičius and Jaeger, 2009), where a randomly initialized recurrent network serves as a fixed temporal feature extractor and only the output mapping is learned. Despite these promising observations, it remains unclear whether a frozen recurrent network can serve as effective memory for general POMDPs. We therefore ask: can an agent solve POMDPs using a randomly initialized recurrent network as memory while learning only the policy or value function, and what design choices make this possible? As shown in Tab. 1, freezing the recurrent memory can substantially reduce the number of trainable parameters, motivating a simpler and more parameter-efficient approach to partially observable RL. Table 1. Parameter counts for two types of recurrent agents we used. All hidden layers have the same width. Reduction is relative to the fully trainable agent.Parameter counts for two type of recurrent agents we used. All hidden layers have same width. Reduction is relative to the fully trainable agent. Model Total Trainable Frozen Reduction PQN GRU 1,119,746 1,119,746 0 — Frozen GRU 1,119,746 331,778 787,968 70.37% LRU 858,114 858,114 0 — Frozen LRU 858,114 331,778 526,336 61.34% PPO GRU 1,385,221 1,385,221 0 — Frozen GRU 1,385,221 597,253 787,968 56.88% LRU 1,123,589 1,123,589 0 — Frozen LRU 1,123,589 597,253 526,336 46.84% To answer this question, we need to revisit Reservoir Computing (RC) mentioned before, because it is the most mature paradigm to design and leverage frozen recurrent networks. RC typically consists of two components, a reservoir and a readout. Generally, reservoir is a randomly initialized RNN and its weights remain frozen during training. The Echo State Network (ESN) (Jaeger, 2001) is a widely used architecture of RC. A key gaurantee in ESN is the Echo State Property (ESP) which indicates that the hidden state should be independent of its initial state and only retain information about the input history. However, recurrent networks used in RL are usually different from the architecture of ESN, and the ESP is not a necessary condition for recurrent networks design. Generalized reservoir computing (GRC) (Kubota et al., 2024) offers a broader perspective of ESP, which verify that a suitable readout can restore input history even the hidden state retain dependence from initialization. This suggests that ESP is unnecessary for using frozen recurrent network as memory in recurrent agent. Through extensive experiments, we find that, similar to (Tallec et al., 2018), frozen recurrent networks can match the performance of trainable counterparts across a wide range of partially observable tasks, including long-horizon delayed-recall problems, discrete and continuous control, and pixel-based POMDPs. Performance degradation appears only in a small number of task-specific cases, where freezing the recurrent network leads to modest losses. Here, we summarize the main contributions in this paper as follows: (1) We investigate whether frozen recurrent networks can serve as effective memory modules for solving POMDPs. (2) We propose a simple and effective agent architecture that uses a frozen recurrent network as memory, reducing the number of trainable parameters in RL by more than half. (3) Across a diverse set of POMDPs, we provide empirical evidence that learning the recurrent memory module is not always necessary for strong performance in partially observable RL. (4) We further show that agents with frozen recurrent memory can successfully learn visual encoders for high-dimensional partially observable control tasks. 2. Related Work Recurrent Memory in Partially Observable RL Recurrent agents learn history-dependent representations through LSTM-based architectures and online recurrent updates (Kapturowski et al., 2019; Elelimy et al., 2024). GTrXL, CoBERL, HCAM, DTQN, AGaLiTe, and RATE use attention or recurrent memory to process past information (Parisotto et al., 2020; Banino et al., 2022; Lampinen et al., 2021; Esslinger et al., 2022; Pramanik et al., 2024; Cherepanov et al., 2026). S5, FFM, and SHM provide structured recurrence and memory updates (Lu et al., 2023; Morad et al., 2023b; Le et al., 2025). MERLIN and MRA combine recurrent networks with external memory to store past experiences and retrieve information for decision making (Wayne et al., 2018; Fortunato et al., 2019). These approaches still involve learning many parameters within their recurrent memory. We explore an alternative approach by adapting existing trainable recurrent architectures into frozen recurrent memories for solving POMDPs. Reservoir Computing and RL Reservoir computing separates a fixed recurrent backbone from a trainable readout (Jaeger, 2001; Maass et al., 2002). In POMDPs, ESN-based methods exploit this separation to construct history-dependent state representations for decision making (Szita et al., 2006; Yamashita and Hamagami, 2022). With trainable actor–critic readouts, ESNs can achieve competitive performance with learned recurrent memories on several memory-intensive tasks (McKee, 2024). Related work has also explored fixed exponential memory traces for value prediction and control (Eberhard et al., 2025). However, these studies primarily focus on reservoir-specific architectures rather than frozen versions of standard recurrent networks. We take a simpler approach: directly freeze the recurrent unit of conventional architectures and train only the downstream policy or value function. Despite this minimal modification, the resulting agents achieve strong performance across a diverse range of POMDPs. RL with Frozen Recurrent Networks Conventional frozen recurrent networks can also supply memory. Agents with randomly initialized frozen LSTMs can learn visual interval-timing tasks (Deverett et al., 2019). In robotic control, gated feature extraction combines an untrained GRU’s summary of preceding observations with the current observation for TD3 (Böhm et al., 2023). However, agents with frozen RNNs and GRUs failed on accumulating-towers, range-count, and exact-count tasks under the reported A2C setup (Mochizuki-Freeman et al., 2023). However, these studies evaluate a limited range of POMDP environments. They also do not examine the theoretical basis of this phenomenon. We draw on GRC to guide the design of agents with frozen recurrent networks. Our simple architecture enables these agents to solve complex POMDPs. 3. Background 3.1. Reinforcement Learning (RL) A Markov Decision Process (MDP) (Sutton and Barto, 1998) is a tuple ℳ=⟨,,P,R,γ⟩M= ,A,P,R,γ , with state space S, action space A, state transition function P(st+1|st,at)P(s_t+1\\ |s_t,a_t), reward function R(rt|st,at)R(r_t|s_t,a_t) and discount factor γ. In RL, at each timestep t, the agent observes a state st∈s_t and selects an action ata_t with respect to its policy at∼π(⋅|st)a_t π(·|s_t), which leads the agent to a new state st+1∼P(⋅|st,at)s_t+1 P(·|s_t,a_t) while emitting a reward rt∼R(⋅|st,at)r_t R(·|s_t,a_t). The Markov property states that the next state st+1s_t+1 depends only on the current state sts_t and action ata_t, as follows: P(st+1|st,at)=P(st+1|st,at,st−1,at−1,…,s0,a0).P(s_t+1|s_t,a_t)=P(s_t+1|s_t,a_t,s_t-1,a_t-1,...,s_0,a_0). The agent’s objective is to find an optimal policy π⋆π that maximizes the expected discounted sum of returns (1) J(π,ℳ)=π,ℳ[∑t=0∞γtrt].J(π,M)=E_π,M [ _t=0^∞γ^tr_t ]. 3.2. Partially Observable RL and Recurrent Memory A Partially Observable Markov Decision Process (POMDP) extends an MDP and is defined by the tuple =⟨,,P,R,Ω,O,γ⟩P= ,A,P,R, ,O,γ , where S, A, P, R, and γ follow the definitions above, Ω is the observation space, and O(ot|st)O(o_t|s_t) is the observation function. Unlike in an MDP, the agent cannot directly observe the latent state sts_t and instead receives an observation ot∼O(⋅|st)o_t O(·|s_t). In that case, a single observation oto_t can not satisfy the Markov property, a trajectory t=(o0,a0,o1,a1,o2,a2,…,at−1,ot)x_t=(o_0,a_0,o_1,a_1,o_2,a_2,...,a_t-1,o_t) is used instead (Sutton and Barto, 1998). The objective for an agent with memory model f in a POMDP is therefore: (2) J(f,π,)=f,π,[∑t=0∞γtrt].J(f,π,P)=E_f,π,P [ _t=0^∞γ^tr_t ]. A recurrent memory model implements f through a compact hidden state th_t, updated from the previous state and current input (Hausknecht and Stone, 2015; Ni et al., 2022): (3) t=fω(t−1,ot),t=Rθ(t).h_t=f_ω(h_t-1,o_t), _t=R_θ(h_t). Here, oto_t is the current observation, and the readout RθR_θ maps the hidden state th_t to action values or policy parameters. A Gated Recurrent Unit (GRU) uses input- and state-dependent gates to control memory updates and retention (Cho et al., 2014), (4) ~t h_t =tanh(ot+(t⊙t−1)+), = \! (Wo_t+U(r_t _t-1)+b ), t _t =(1−t)⊙t−1+t⊙~t. =(1-z_t) _t-1+z_t h_t. Here, W and U are the input and recurrent weight matrices, b is a bias vector, and ~t h_t is the candidate hidden state. The reset gate tr_t controls past-state information in the candidate, while the update gate tz_t controls how much of the candidate replaces the previous hidden state. Both gates apply a sigmoid to affine functions of oto_t and t−1h_t-1; ⊙ denotes elementwise multiplication. A Linear Recurrent Unit (LRU) instead accumulates inputs in a complex-valued state through a diagonal linear recurrence (Orvieto et al., 2023), (5) t=t−1+ot,=diag(λ1,…,λN).h_t= _t-1+Bo_t, =diag( _1,…, _N). The magnitudes of λi _i determine decay and their phases determine rotation, shaping memory retention. Real-valued output projections and nonlinear mixing may surround this linear cell. 3.3. Reservoir Computing and the Echo State Property Reservoir Computing (RC) uses a dynamical system with randomly initialized recurrent networks to construct temporal features from an input sequence (Jaeger, 2001; Verstraeten et al., 2007). Its hidden state evolves with the inputs, and a learned readout maps this state to predictions or control tasks. In a classical Echo State Network (ESN), the state update is (6) t= _t= (1−α)t−1 (1-α)h_t-1 +αtanh(inot+rect−1+), +α \! (W_ino_t+W_rech_t-1+b ), where the leak rate α∈(0,1]α∈(0,1] controls the mixture of the previous state and nonlinear update. The input weights inW_in, recurrent weights recW_rec, and bias b remain fixed (Jaeger and Haas, 2004). A linear readout takes the form t=outt+outy_t=W_outh_t+b_out, although the readout can also be nonlinear. The Echo State Property (ESP) describes the asymptotic independence of the hidden state from its initialization under a common input sequence (Jaeger, 2001; Yildiz et al., 2012). In the asymptotic forgetting formulation, trajectories th_t and t′h _t driven by any admissible common input sequence converge from any pair of admissible initial states: (7) limt→∞‖t−t′‖=0. _t→∞\|h_t-h _t\|=0. For the leaky ESN, applying the 11-Lipschitz bound for tanh to two such trajectories yields (8) ‖t−t′‖2≤κ‖t−1−t−1′‖2,κ=(1−α)+α‖rec‖2.\|h_t-h _t\|_2≤κ\|h_t-1-h _t-1\|_2, κ=(1-α)+α\|W_rec\|_2. Thus κ<1κ<1 is sufficient for contraction in the induced Euclidean operator norm. A recurrent spectral radius below one alone does not supply this norm bound for a nonlinear reservoir. Forgetting an initial state is different from forgetting an input. Eq. 7 holds the driving sequence fixed; control requires distinguishing histories with different driving sequences. The distinction is explicit in a linear system with fixed ,A,B: (9) t=∑k=0t−kok.h_t= _k=0^tA^t-kBo_k. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds, and 10 evaluation per seed. Figure 2. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds and 10 evaluation episodesper seed.Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds, and 10 evaluation per seed. 3.4. Generalized Reservoir Computing Generalized Reservoir Computing (GRC) shifts the ESP requirement from the reservoir state to the output (Kubota et al., 2024). It seeks a time-invariant (TI) transformation that extracts a response determined by input history, even when the hidden state retains initialization dependence. For a readout t=Rθ(t)y_t=R_θ(h_t), independence of the output from initialization requires (10) limt→∞‖Rθ(t)−Rθ(t′)‖=0, _t→∞\|R_θ(h_t)-R_θ(h _t)\|=0, where th_t and t′h _t receive identical inputs from different initial states. The resulting response has the form t∗=Φ(ot,ot−1,…)y_t^*= (o_t,o_t-1,…) after transients. TI means that the same mapping Φ applies at every time; the output can still vary with the input history. A nonlinear readout can extract such a response by removing initialization-dependent components. For example, consider an oscillatory state (Xt,Yt)=(rtcosφt,rtsinφt)(X_t,Y_t)=(r_t _t,r_t _t) whose radius depends on input history while its phase retains initialization dependence. The transformation Xt2+Yt2=rt2X_t^2+Y_t^2=r_t^2 removes the phase. GRC demonstrates related transformations using MLP readouts to recover past inputs from chaotic and physical reservoirs without ESP (Kubota et al., 2024). Frozen LRU as Linear Reservoir Consider the LRU in Eq. 5, with t∈ℂNh_t ^N and fixed matrices and B. Its stable diagonal parameterization (Orvieto et al., 2023) takes the form (11) λj=exp(−exp(νj)+iφj),ρ=maxj|λj|<1, _j= \! (- ( _j)+i _j ), ρ= _j| _j|<1, where νj _j and φj _j are finite real parameters and i2=−1i^2=-1. Each eigenvalue has magnitude exp(−exp(νj))<1 (- ( _j))<1. For two trajectories initialized at arbitrary finite states 0h_0 and 0′h _0 and driven by the same inputs, subtracting their updates cancels otBo_t at every step. Writing Δt=t−t′ _t=h_t-h _t gives (12) Δt _t =Δt−1=tΔ0, = _t-1= ^t _0, ‖Δt‖2 \| _t\|_2 ≤‖t‖2‖Δ0‖2=ρt‖Δ0‖2. ≤\| ^t\|_2\| _0\|_2=ρ^t\| _0\|_2. The equality ‖t‖2=ρt\| ^t\|_2=ρ^t follows from diagonality. Since ρ<1ρ<1, the initial-state difference vanishes as t→∞t→∞, establishing the ESP. For bounded inputs ‖ot‖2≤M\|o_t\|_2≤ M extending into the past, the corresponding history response is (13) t∗=∑k=0∞kot−k,‖t∗‖2≤M‖21−ρ.h_t^*= _k=0^∞ ^kBo_t-k, \|h_t^*\|_2≤ M\|B\|_21-ρ. Thus, the frozen LRU cell is a stable linear reservoir satisfying the ESP. It is a conventional reservoir. Frozen GRU as General Reservoir For a GRU, ESP depends on its parameters. Sufficient conditions for incremental stability ensure initial-state forgetting (Bonassi et al., 2021), but freezing the weights alone does not enforce these conditions. A two-dimensional construction illustrates how a frozen GRU can lack state ESP while admitting a readout with output ESP. Let t=(vt,wt)⊤∈[−1,1]2h_t=(v_t,w_t) ∈[-1,1]^2 and ot∈[−1,1]o_t∈[-1,1]. Set all gate weights and biases to zero, so t=t=(1/2,1/2)⊤z_t=r_t=(1/2,1/2) . In Eq. 4, choosing =(1,0)⊤W=(1,0) , =diag(0,4)U=diag(0,4), and =0b=0 yields (14) vt v_t =12vt−1+12tanh(ot), = 12v_t-1+ 12 (o_t), wt w_t =12wt−1+12tanh(2wt−1). = 12w_t-1+ 12 (2w_t-1). The second coordinate has fixed points 00 and w∗∈(0,1)w_*∈(0,1), since tanh(2w)−w (2w)-w is positive just above zero and negative at one. Two trajectories with identical v0v_0 and inputs but with w0=0w_0=0 and w0′=w∗w _0=w_* remain separated by w∗w_*, violating state ESP. The first coordinate instead forgets its initialization. For updates starting at t=1t=1, unrolling gives vt=2−tv0+∑k=0t−12−(k+1)tanh(ot−k)v_t=2^-tv_0+ _k=0^t-12^-(k+1) (o_t-k). Extend the input sequence to t≤0t≤ 0 with any bounded prehistory and write o≤t=(ot,ot−1,…)o_≤ t=(o_t,o_t-1,…). Then (15) Φ(o≤t) (o_≤ t) =∑k=0∞2−(k+1)tanh(ot−k), = _k=0^∞2^-(k+1) (o_t-k), |vt−Φ(o≤t)| |v_t- (o_≤ t)| ≤21−t→0. ≤ 2^1-t→ 0. The bound follows from |v0|≤1|v_0|≤ 1 and the geometric tail of the series. The linear readout yt=vty_t=v_t therefore approaches a nonconstant function of input history. For a nonlinear readout yt=ℓ(vt)y_t= (v_t), where ℓ(v)=max(v,0.01v) (v)= (v,0.01v), the 11-Lipschitz property of ℓ gives |yt−ℓ(Φ(o≤t))|≤|vt−Φ(o≤t)|⟶0. |y_t- ( (o_≤ t)) |≤ |v_t- (o_≤ t) | 0. Since ℓ is strictly increasing, the function of input history remains nonconstant. Both readouts discard the initialization-dependent coordinate, illustrating the GRC principle (Kubota et al., 2024). POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed. Figure 3. POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed.POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed. Algorithm 1 Constructing and Using Frozen Recurrent Agents 1: Agent and its trainable parameter set Θlearn _learn 2: Fω0,ϕF_ _0,φ is the recurrent memory network: ω0 _0 contains all recurrent unit parameters at initialization; ϕφ denotes the trainable mixer parameters. 3: Initialize Fω0,ϕF_ _0,φ and RθR_θ 4: Use FrozenStep for each recurrent unit 5: Θlearn←ϕ,θ _learn←\φ,θ\ 6: E←IdentityE 7: if observations are images then 8: Initialize CNN EψE_ψ; set E←EψE← E_ψ 9: Θlearn←Θlearn∪ψ _learn← _learn∪\ψ\ 10: end if 11: Use E(ot)E(o_t) as the input to Fω0,ϕF_ _0,φ 12: Keep ω0 _0 fixed; update only Θlearn _learn with the RL objective 13: return agent (E,Fω0,ϕ,Rθ)(E,F_ _0,φ,R_θ) and Θlearn _learn 14: 15: function FrozenStep(c,ωc,0,t−1,t,etc, _c,0,h_t-1,u_t,e_t) 16: c: a single recurrent unit (e.g., one GRU or LRU unit). 17: ωc,0 _c,0: unit c’s initialized parameter block in ω0 _0. 18: t: time step; t−1h_t-1: the cell’s previous hidden state. 19: tu_t: input features supplied to c at step t. 20: et∈0,1e_t∈\0,1\: episode start flag. 21: if et=1e_t=1 then 22: t−1←InitialState(c)h_t-1← InitialState(c) ⊳ unit’s reset state 23: end if 24: SG←jax.lax.stop_gradientSG← jax.lax.stop\_gradient 25: ω¯c←jax.tree.map(SG,ωc,0) ω_c← jax.tree.map(SG, _c,0) 26: (t,tout)←cω¯c(t−1,t)(h_t,x_t^out)← c_ ω_c(h_t-1,u_t) 27: return t,touth_t,x_t^out ⊳ Updated state, output features 28: end function 4. Experiments Our experiments aim to answer the following research questions: • RQ1: Can frozen recurrent networks retain input over long horizons in RL? (Sec. 4.2) • RQ2: Can frozen recurrent networks solve vector observation discrete POMDPs? (Sec. 4.3) • RQ3: Can frozen recurrent networks solve vector observation continuous POMDPs? (Sec. 4.4) • RQ4: Can frozen recurrent networks solve high-dimensional image observed POMDPs? (Sec. 4.5) • RQ5: Which learning matter in frozen recurrent networks? (sec. 4.6) 4.1. Experimental Setup Baselines. Morad et al. (2023a) evaluated serveral memory archiectures and found that GRU is the most performant. Recently, Wang et al. (2026) and Lu et al. (2024) found that LRU is the most powerful and efficient state-space memory architecture for RL. We therefore pick both of them as our baselines. For the frozen variants, we simply apply Algo. 1 to the recurrent agent. Here, we use JAX-style pesudo code as an example. Agent Architecture. We show our recurrent PQN agent architecture in Fig. 1. We use a simple MLP as readout, and each MLP consists of L blocks with a Linear Layer, a Layer Normalization (Ba et al., 2016), and LeakyReLU activation. We only stop the gradient of parameters in the recurrent unit, instead of cutting off gradient computation of recurrent models, so our agent still can learn a visual encoder to compress high-dimensional image observations. We update all the trainable parameters with the PQN learning target. Fig. 7 shows our detailed architecture of frozen recurrent networks. Each frozen recurrent unit is followed by a trainable feature mixer. For PPO, we only replace the Q-value readout MLP with same separate actor and critic MLP heads. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity-only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training Figure 4. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training.MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity-only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training MemoryChain. MemoryChain is a classical one-bit delayed-recall task from bsuite (Osband et al., 2020). We use the implementation from Gymnax (Lange, 2022) for faster rollout. We train eight delay configurations with Proximal Policy Optimization (PPO) (Schulman et al., 2017), using 1010M env steps and 5 seeds at each delay. Each seed receives ten evaluation episodes. POPGym. Partially Observable Process Gym (POPGym) is a well-known benchmark for evaluating memory in partially observable RL (Morad et al., 2023a). We use POPJym (Lu et al., 2023), a JAX implementation of POPGym. Our evaluation includes six tasks: Minesweeper, NoisyStatelessCartPole, RepeatFirst, RepeatPrevious, and StatelessCartPole. Each task has three difficulty levels, giving 18 settings. We train each agent using PQN (Gallici et al., 2025) with a budget of 2020 million environment steps with 10 seeds per run. POMDP MuJoCo Playground. We also hope to apply our framework to robot control. We therefore evaluate it on the DeepMind Control Suite tasks reimplemented in MuJoCo Playground (Zakka et al., 2025). We use four environments: BallInCup, CheetahRun, ReacherHard, and WalkerWalk. We introduce partial observability by modifying the original observations. The Positional Only setting retains position information and hides velocity information. The Velocity Only setting retains velocity information and hides position information. Following the original paper, we use a budget of 6060 million environment steps per run. We train with PPO (Schulman et al., 2017) using a shared Actor-Critic trunk. Each configuration uses 10 seeds. POPGym Arcade. We extend the evaluation to pixelated POMDPs with POPGym Arcade (Wang et al., 2026). This benchmark provides both fully observable and partially observable Atari-style environments implemented in JAX. Our experiments cover the partially observable variants of BattleShip, CountRecall, MineSweeper, and Navigator. Agents receive 128×128×3128× 128× 3 RGB images as the observation from environments. To reduce computational cost and improve training efficiency, we train the models with PQN (Gallici et al., 2025). The training budget is 2020 million environment steps per seed, with five training seeds per configuration. Evaluation. We evaluate every agent after training. For each seed, we run 10 episodes and compute the mean return and variance to make the final evaluation return plots in Fig. 2, 3, 4 and 5. 4.2. RQ1: Long-Horzion Recall We use MemoryChain to compare the recall performance of Frozen and trainable recurrent networks. In this environment, the agent receives a single-bit cue at the first time step of the episode, and then receives a sequence of zero observations for a number of delay steps. The agent receives a positive reward of +1+1 if it outputs the correct initial cue at the last time step, and −1-1 otherwise. We sweep the delay 2k2^k for k∈2,…,11k∈\2,...,11\ and compare MLP, GRU, LRU, frozen GRU and frozen LRU. Here, we introduce MLP as a memoryless baseline. We implement recurrent PPO with JAX (Bradbury et al., 2018) and Equinox (Kidger and Garcia, 2021) to accelerate the training process. In Fig. 2, we plot the success rate of evaluation results. LRU achieves 100% success rate at delays up to 512 length, as expected for a state-of-the-art memory model in POMDPs. More surprisingly, frozen LRU gets 100% success rate at delays up to 256 length close to LRU. Similarly, the recall ability of frozen GRU also matches that of GRU, both achieve 100% success rate at delay of 128 length. Although the performance of frozen LRU and GRU does not exceed that of their trainable counterparts, these results still show the evidence that frozen recurrent networks have the ability to retain a cue over long horizons, indicating that simply freezing the parameters of the recurrent unit does not hurt too much of the memory retention of recurrent networks with RL objective. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes. Figure 5. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes. 4.3. RQ2: Vector Observation Discrete POMDPs POPGym introduces partial observability to some simple RL tasks, making them more challenging and long-term memory demanding. For example, in MineSweeper, the agent can only observe the number of mines nearby instead of the actual grid, which requires the agent to remember which grids have been revealed before. We use 18 environments with approximately 1000 runs on POPGym to test the performance of Frozen GRU and LRU compared with their trainable counterparts as well as MLP. We plot seven representative selected learning curves in Fig. 3(a) and the final evaluation returns for all environments in Fig. 3(b). We present only informative results and exclude very simple tasks that almost all recurrent networks can solve. Frozen recurrent networks achieve competitive performance while training substantially fewer parameters. As shown in Tab. 1, Frozen LRU trains only approximately 40% of the total agent parameters, yet achieves a final evaluation return of 0.464, compared with 0.458 for LRU (Fig. 3(b)). This corresponds to a relative increase of 1.23%. Frozen GRU trains only approximately 30% of the total agent parameters and achieves a final evaluation return of 88.70% for GRU. These results show that updating all recurrent parameters is unnecessary for effective control on the evaluated POMDP tasks. In Fig. 3(a), the learning curves of Frozen LRU and LRU closely overlap on most tasks, indicating similar training progress and final performance. Although Frozen GRU often achieves lower returns than GRU during training, its final returns are comparable to GRU. These results demonstrate that frozen recurrent networks can serve as effective memory models to solve discrete POMDPs and achieve competitive performance. 4.4. RQ3: Vector Observation Continuous POMDPs To further test the applicability of frozen recurrent networks in POMDPs, we also conducted experiments on partially observable continuous control tasks. We use MuJoCo Playground (Zakka et al., 2025), the latest JAX reimplementation of MuJoCo (Todorov et al., 2012). It consists of several challenging tasks in which agents need to control a multi-joint dynamical body. We introduce Position Only and Velocity Only settings widely used in prior work for evaluating RL agents with memory (Ni et al., 2022; Meng et al., 2021; Ni et al., 2023), which are denoted as P and V in Fig. 4, respectively. We implement wrappers to mask the positional or velocity information in their observation. Similarly, frozen recurrent networks also achieve competitive performance in partially observable continuous control tasks. In our PPO agents, Frozen LRU and Frozen GRU train approximately 53% and 43% of the total agent parameters, respectively (Tab. 1). Across the seven observation settings in Fig. 4(b), Frozen LRU achieves a final evaluation return of 791.8, compared with 831.9 for LRU, retaining 95% of LRU’s return. Frozen GRU achieves 772.9, compared with 842.8 for GRU, retaining 92% of GRU’s return. The learning curves in Fig. 4(a) show similar returns for all four agents on ReacherHard. On WalkerWalk, Frozen GRU has higher mean training returns than GRU near the end of training under both Position Only and Velocity Only settings. The learning curves also show lower returns for Frozen LRU than LRU on WalkerWalk and for Frozen GRU than GRU on CheetahRun at the end of training. These results demonstrate that frozen recurrent networks can provide sufficient and efficient memory to solve such continuous POMDPs. 4.5. RQ4: Partially Observable Visual Control In real-world scenarios, we usually regard most of the tasks as partially observable visual control tasks, in which agents need to make optimal decisions with image observations. With respect to Fig. 1 and Sec. 4.1, we can use gradients through a frozen recurrent network to learn a CNN as observation encoder to solve visual control POMDPs. We evaluate our agents on POPGym Arcade (Wang et al., 2026), a JAX implemented extension of POPGym that provides various partially observable visual control tasks. It includes very challenging and high-dimensional visual POMDPs. For example, in the partially observable versions of BattleShip and MineSweeper, each observation reveals information only about the current grid cell. The agent must therefore extract features from these high-dimensional images, retain information about previously observed cells in memory, and retrieve it to guide decisions. Notably, frozen recurrent networks also achieve competitive performance with learned visual representations. Across the four tasks in Fig. 5(b), Frozen LRU achieves a final evaluation return of 0.694, compared with 0.624 for LRU, representing a relative increase of 11.2% with lower variance. Frozen GRU achieves 0.643, retaining 95.2% of GRU’s final evaluation return of 0.676. In Fig. 5(a), the learning curves of all four agents approach similar returns on MineSweeper and Navigator. Frozen LRU reaches higher mean training returns than LRU on BattleShip, while the training curves on CountRecall fluctuate more but remain comparable at the end. These results demonstrate that frozen recurrent networks can work with learned CNN encoders to solve challenging and high-dimensional visual control POMDPs. Moreover, Frozen LRU can even outperform its trainable counterpart, making it a promising baseline for partially observable RL. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed. Figure 6. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed.Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed. 4.6. RQ5: Learning the Mixers Matter Modern recurrent architectures often use nonlinear feature mixer after each recurrent unit (Gu et al., 2021; Gu et al., 2022a; Orvieto et al., 2023; Smith et al., 2023) as well as Feed Forward Network (FFN) in transformer (Vaswani et al., 2017; Katharopoulos et al., 2020; Yang et al., 2024; De et al., 2024). In our experiments, we also adopt a similar design (Fig. 7) and conduct additional experiments to examine the effect of training these mixers while keeping the recurrent unit frozen. We find that learning the mixers significantly improves performance for both Frozen LRU and Frozen GRU. As shown in Fig. 6, training the mixers increases the final evaluation return from 0.392 to 0.464 for Frozen LRU and from 0.341 to 0.386 for Frozen GRU, corresponding to relative improvements of 18.5% and 13.2%, respectively. Even the simple design used here (Fig. 7) substantially improves agent performance. These results suggest that learning the mixers contributes to the agent’s ability to adapt and use the memory provided by frozen recurrent networks. Modern large models use attention and complex nonlinear transformations to mix features (Touvron et al., 2023; Yang et al., 2025; Gemma Team et al., 2025). Agents with frozen recurrent networks may also benefit from incorporating such designs into their mixers and readouts. 5. Discussion Researchers often assume that recurrent RL requires training a memory model. Our experiments in Sec.4.2, 4.3, 4.4 and 4.5 challenge this assumption. We show that agents can solve the tested POMDPs with frozen recurrent networks and nonlinear MLPs and mixers can learn to use frozen recurrent networks as memory. We believe this approach offers an alternative solution for partially observable RL and may also benefit recurrent RL applications in robotic learning. 6. Limitations and Future Work Our study has several limitations. First, we do not conduct a large-scale, exhaustive ablation study, and our evaluation covers a limited set of benchmarks. Computational constraints also restrict the range of baselines considered. Future work should therefore include broader ablations, additional benchmarks, and stronger baselines to better assess the generality of our findings. Second, our GRC analysis relies on specific assumptions and establishes its result under a relaxed ESP condition. In practice, random initialization, optimization dynamics, and finite-sample effects introduce additional uncertainty, so trained agents may not always satisfy the theoretical conditions exactly. Future work could investigate how closely these conditions hold in practical RL settings and how deviations from them affect performance. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters. Figure 7. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters.Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters. 7. Conclusion We investigated whether recurrent networks must be learned in partially observable RL. We first introduced a simple architecture that uses frozen recurrent networks as memory modules. Across diverse POMDPs, these frozen memories retain information over long horizons and achieve competitive performance with substantially fewer trainable parameters. We further showed that visual encoders can be trained effectively through frozen recurrent memories to solve high-dimensional visual control tasks. Overall, our results suggest that learning the recurrent module is not always necessary and frozen recurrent networks provide a simple, strong, and parameter-efficient baseline for recurrent RL. AI Use Statement We used OpenAI Codex with GPT-6 Astra for language polishing and LaTeX formatting. We also used it to write plotting code and code execution scripts. We did not retain complete prompt and conversation records. We take full responsibility for all the content, code, and results. References Ba et al. (2016) J. L. Ba, J. R. Kiros, and G. E. Hinton Layer normalization. External Links: Link, 1607.06450 Cited by: §4.1. Banino et al. (2022) A. Banino, A. P. Badia, J. C. Walker, T. Scholtes, J. Mitrovic, and C. Blundell CoBERL: contrastive BERT for reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Böhm et al. (2023) P. Böhm, P. Pounds, and A. C. Chapman Feature extraction for effective and efficient deep reinforcement learning on real robotic platforms. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 7126–7132. External Links: Document, Link Cited by: §2. Bonassi et al. (2021) F. Bonassi, M. Farina, and R. Scattolini On the stability properties of Gated Recurrent Units neural networks. Note: Revised manuscript, version 6 External Links: Document, Link, 2011.06806v6 Cited by: §3.4. Bradbury et al. (2018) JAX: composable transformations of Python+NumPy programs External Links: Link Cited by: §4.2. Cherepanov et al. (2026) E. Cherepanov, A. Staroverov, A. K. Kovalev, and A. I. Panov Recurrent action transformer with memory. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Cho et al. (2014) K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 1724–1734. External Links: Document, Link Cited by: §1, §3.2. De et al. (2024) S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, et al. Griffin: mixing gated linear recurrences with local attention for efficient language models. External Links: Link, 2402.19427 Cited by: §4.6. Deverett et al. (2019) B. Deverett, R. Faulkner, M. Fortunato, G. Wayne, and J. Z. Leibo Interval timing in deep reinforcement learning agents. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §2. Dong et al. (2020) J. Dong, R. Ohana, M. Rafayelyan, and F. Krzakala Reservoir computing meets recurrent kernels and structured transforms. In Advances in Neural Information Processing Systems, Vol. 33, p. 16785–16796. External Links: Link Cited by: §B.2. Eberhard et al. (2025) O. Eberhard, M. Muehlebach, and C. Vernade Partially observable reinforcement learning with memory traces. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 14934–14949. External Links: Link Cited by: §2. Elelimy et al. (2024) E. Elelimy, A. White, M. Bowling, and M. White Real-time recurrent learning using trace units in reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Document, Link Cited by: §2. Elman (1990) J. L. Elman Finding structure in time. Cognitive Science 14 (2), p. 179–211. External Links: Document Cited by: §1. Esslinger et al. (2022) K. Esslinger, R. Platt, and C. Amato Deep transformer Q-networks for partially observable reinforcement learning. External Links: 2206.01078, Document, Link Cited by: §2. Fortunato et al. (2019) M. Fortunato, M. Tan, R. Faulkner, S. Hansen, A. Puigdomènech Badia, G. Buttimore, C. Deck, J. Z. Leibo, and C. Blundell Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §2. Gallicchio et al. (2017) C. Gallicchio, A. Micheli, and L. Pedrelli Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, p. 87–99. External Links: Document, Link Cited by: Appendix B. Gallicchio et al. (2018) C. Gallicchio, A. Micheli, and L. Pedrelli Design of deep echo state networks. Neural Networks 108, p. 33–47. External Links: Document, Link Cited by: §B.1. Gallici et al. (2025) M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. N. Foerster, and M. Martin Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §4.1, §4.1. Gemma Team et al. (2025) Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. External Links: Link, 2503.19786 Cited by: §4.6. Gu and Dao (2024) A. Gu and T. Dao Mamba: linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, External Links: Link Cited by: §1. Gu et al. (2022a) A. Gu, K. Goel, and C. Ré Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, External Links: Link Cited by: §1, §4.6. Gu et al. (2022b) A. Gu, A. Gupta, K. Goel, and C. Ré On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems, Vol. 35, p. 35971–35983. External Links: Document, Link Cited by: §1. Gu et al. (2021) A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems, Vol. 34, p. 572–585. External Links: Link Cited by: §4.6. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. External Links: Link Cited by: §1. Hausknecht and Stone (2015) M. Hausknecht and P. Stone Deep recurrent q-learning for partially observable MDPs. In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, External Links: Link Cited by: §3.2. Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber Long short-term memory. Neural Computation 9 (8), p. 1735–1780. External Links: Document Cited by: §1. Jaeger and Haas (2004) H. Jaeger and H. Haas Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304 (5667), p. 78–80. External Links: Document Cited by: §3.3. Jaeger (2001) H. Jaeger The “echo state” approach to analysing and training recurrent neural networks. GMD Report Technical Report 148, German National Research Center for Information Technology. Note: Corrected version with an erratum, January 26, 2010 External Links: Link Cited by: §1, §1, §2, §3.3, §3.3. Kaelbling et al. (1998) L. P. Kaelbling, M. L. Littman, and A. R. Cassandra Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1–2), p. 99–134. External Links: Document Cited by: §1. Kapturowski et al. (2019) S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §2. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are RNNs: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 5156–5165. External Links: Link Cited by: §1, §4.6. Kidger and Garcia (2021) P. Kidger and C. Garcia Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021. External Links: Link Cited by: §4.2. Kubota et al. (2024) T. Kubota, Y. Imai, S. Tsunegi, and K. Nakajima Reservoir computing generalized. Note: Version 1 External Links: Document, Link, 2412.12104v1 Cited by: §1, §3.4, §3.4, §3.4. Lampinen et al. (2021) A. K. Lampinen, S. C. Y. Chan, A. Banino, and F. Hill Towards mental time travel: a hierarchical memory for reinforcement learning agents. In Advances in Neural Information Processing Systems, Vol. 34. External Links: Link Cited by: §2. Lange (2022) gymnax: a JAX-based reinforcement learning environment library External Links: Link Cited by: §4.1. Le et al. (2025) H. Le, D. Nguyen, K. Do, S. Gupta, and S. Venkatesh Stable hadamard memory: revitalizing memory-augmented agents for reinforcement learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Lu et al. (2024) C. Lu, R. Shi, Y. Liu, K. Hu, S. S. Du, and H. Xu Rethinking transformers in solving POMDPs. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 33089–33112. External Links: Link Cited by: §1, §4.1. Lu et al. (2023) C. Lu, Y. Schroecker, A. Gu, E. Parisotto, J. N. Foerster, S. Singh, and F. Behbahani Structured state space models for in-context reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36, p. 47016–47031. External Links: Document, Link Cited by: §1, §2, §4.1. Lukoševičius and Jaeger (2009) M. Lukoševičius and H. Jaeger Reservoir computing approaches to recurrent neural network training. Computer Science Review 3 (3), p. 127–149. External Links: Document, Link Cited by: §1. Maass et al. (2002) W. Maass, T. Natschläger, and H. Markram Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation 14 (11), p. 2531–2560. Cited by: §1, §2. McKee (2024) K. McKee Reservoir computing for fast, simplified reinforcement learning on memory tasks. External Links: Document, Link, 2412.13093 Cited by: §2. Meng et al. (2021) L. Meng, R. Gorbet, and D. Kulić Memory-based deep reinforcement learning for POMDPs. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5619–5626. External Links: Document, Link Cited by: §4.4. Mochizuki-Freeman et al. (2023) J. Mochizuki-Freeman, S. S. Maini, and Z. Tiganj Characterizing neural activity in cognitively inspired RL agents during an evidence accumulation task. In 2023 International Joint Conference on Neural Networks (IJCNN), External Links: Document, Link Cited by: §2. Morad et al. (2023a) S. Morad, R. Kortvelesy, M. Bettini, S. Liwicki, and A. Prorok POPGym: benchmarking partially observable reinforcement learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §4.1, §4.1. Morad et al. (2023b) S. Morad, R. Kortvelesy, S. Liwicki, and A. Prorok Reinforcement learning with fast and forgetful memory. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Document, Link Cited by: §2. Morad et al. (2024) S. Morad, C. Lu, R. Kortvelesy, S. Liwicki, J. Foerster, and A. Prorok Recurrent reinforcement learning with memoroids. In Advances in Neural Information Processing Systems, Vol. 37, p. 14386–14416. External Links: Document, Link Cited by: §1. Ni et al. (2022) T. Ni, B. Eysenbach, and R. Salakhutdinov Recurrent model-free RL can be a strong baseline for many POMDPs. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 162, p. 16691–16723. External Links: Link Cited by: §1, §3.2, §4.4. Ni et al. (2023) T. Ni, M. Ma, B. Eysenbach, and P. Bacon When do transformers shine in RL? decoupling memory from credit assignment. In Advances in Neural Information Processing Systems, Vol. 36, p. 50429–50452. External Links: Document, Link Cited by: §1, §4.4. Orvieto et al. (2023) A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 26670–26698. External Links: Link Cited by: §1, §3.2, §3.4, §4.6. Osband et al. (2020) I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvari, S. Singh, B. V. Roy, R. Sutton, D. Silver, and H. V. Hasselt Behaviour suite for reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Parisotto et al. (2020) E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, M. Botvinick, N. Heess, and R. Hadsell Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 7487–7498. External Links: Link Cited by: §1, §2. Pinna et al. (2026) M. Pinna, G. Lagomarsini, A. Ceni, and C. Gallicchio ParalESN: enabling parallel information processing in reservoir computing. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §B.3. Pramanik et al. (2024) S. Pramanik, E. Elelimy, M. C. Machado, and A. White AGaLiTe: approximate gated linear transformers for online reinforcement learning. Transactions on Machine Learning Research. External Links: Link, ISSN 2835-8856 Cited by: §1, §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: Link, 1707.06347 Cited by: Appendix A, §4.1, §4.1. Smith et al. (2023) J. T.H. Smith, A. Warrington, and S. Linderman Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §4.6. Sutton and Barto (1998) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. MIT Press. External Links: Link, ISBN 9780262193986 Cited by: §1, §3.1, §3.2. Szita et al. (2006) I. Szita, V. Gyenes, and A. Lőrincz Reinforcement learning with echo state networks. In Artificial Neural Networks—ICANN 2006, Lecture Notes in Computer Science, Vol. 4131, p. 830–839. External Links: Document Cited by: §2. Tallec et al. (2018) C. Tallec, L. Blier, and D. Kalainathan Reproducing “World Models”: is training the recurrent network really needed?. Note: Accessed: 2026-10-08Blog post External Links: Link Cited by: §1, §1. Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa MuJoCo: a physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. External Links: Document, Link Cited by: §4.4. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: open and efficient foundation language models. External Links: Link, 2302.13971 Cited by: §4.6. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. External Links: Link Cited by: §1, §4.6. Verstraeten et al. (2007) D. Verstraeten, B. Schrauwen, M. D’Haene, and D. Stroobandt An experimental unification of reservoir computing methods. Neural Networks 20 (3), p. 391–403. External Links: Document Cited by: §3.3. Wang et al. (2026) Z. Wang, Z. He, B. Zhang, E. Toledo, and S. Morad Investigating memory in model-free RL with POPGym arcade. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 126692–126733. External Links: Link Cited by: §4.1, §4.1, §4.5. Wayne et al. (2018) G. Wayne, C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska-Barwinska, J. Rae, P. Mirowski, J. Z. Leibo, A. Santoro, M. Gemici, M. Reynolds, T. Harley, J. Abramson, S. Mohamed, D. Rezende, D. Saxton, A. Cain, C. Hillier, D. Silver, K. Kavukcuoglu, M. Botvinick, D. Hassabis, and T. Lillicrap Unsupervised predictive memory in a goal-directed agent. External Links: 1803.10760, Document, Link Cited by: §2. Yamashita and Hamagami (2022) K. Yamashita and T. Hamagami Reinforcement learning for POMDP environments using state representation with reservoir computing. Journal of Advanced Computational Intelligence and Intelligent Informatics 26 (4), p. 562–569. External Links: Document Cited by: §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. External Links: Link, 2505.09388 Cited by: §4.6. Yang et al. (2024) S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 56501–56523. External Links: Link Cited by: §4.6. Yildiz et al. (2012) I. B. Yildiz, H. Jaeger, and S. J. Kiebel Re-visiting the echo state property. Neural Networks 35, p. 1–9. External Links: Document Cited by: §3.3. Zakka et al. (2025) K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel Demonstrating MuJoCo Playground. In Proceedings of Robotics: Science and Systems, External Links: Document, Link Cited by: §4.1, §4.4. Supplementary Material Observation oto_t Trainable CNN EψE_ψ (Optional) Frozen Recurrent Network ht=fω0(ht−1,ut)h_t=f_ _0(h_t-1,u_t) Actor / Critic MLPs πθ(at∣ht) _θ(a_t h_t) Vϕ(ht)V_φ(h_t) Environment Imagesutu_thth_tata_tVectorsObservation Feedback PPO Learning Policy + Value Loss −- Entropy Bonus PredictionsRollout / GAEUpdate θ,ϕθ, ψ (Images)Trainable ParametersFixed ParametersParameter-Free ElementsData FlowParameter UpdateEach MLP Headhth_t Linear Layer LayerNorm LeakyReLU ×K× K Linear Layer Output Actor: Logits Or Mean Critic: Scalar Value Figure A1. Overview of recurrent PPO agent with frozen recurrent networks. The frozen recurrent network supplies memory features to trainable actor and critic MLPs, which are optimized with PPO. Observations reach the frozen recurrent network directly for vectors or through an optional trainable CNN for images. Memory features feed separate actor and critic MLPs. The environment supplies observations and rollout data, and PPO updates the trainable parameters. The right panel shows the layers in each MLP head. Appendix A PPO Agent We show the recurrent PPO agent used in our MuJoCo Playground experiments in Fig. A1. It follows the PQN architecture in Fig. 1, with separate actor and critic MLPs in place of the PQN readout. The actor produces the policy, and the critic estimates the state value. We update the trainable parameters using PPO (Schulman et al., 2017). Appendix B Deep Reservoir Computing Deep Reservoir Computing (Deep RC) (Gallicchio et al., 2017) stacks reservoir layers to construct temporal features from input history and uses a trainable readout to learn from their hidden states. To further validate our conclusions, we conduct additional experiments with three Deep RC models: Deep Echo State Network (DeepESN), Structured Reservoir Computing (SRC), and Parallel Echo State Network (ParalESN). B.1. Deep Echo State Network (DeepESN) DeepESN extends a standard ESN by stacking multiple reservoir layers (Gallicchio et al., 2018). Let t(ℓ)h_t^( ) denote the state of layer ℓ at time t. The first layer receives t(1)=otu_t^(1)=o_t, while each higher layer receives t(ℓ)=t(ℓ−1)u_t^( )=h_t^( -1). Each layer computes a candidate state and applies a leaky update: (A1) ~t(ℓ) h_t^( ) =tanh(in(ℓ)t(ℓ)+rec(ℓ)t−1(ℓ)+(ℓ)), = \! (W_in^( )u_t^( )+W_rec^( )h_t-1^( )+b^( ) ), t(ℓ) _t^( ) =(1−α(ℓ))t−1(ℓ)+α(ℓ)~t(ℓ). =(1-α^( ))h_t-1^( )+α^( ) h_t^( ). Here, in(ℓ)W_in^( ) and rec(ℓ)W_rec^( ) are the fixed input and recurrent weight matrices, and (ℓ)b^( ) is a fixed bias. The leak rate α(ℓ)∈(0,1]α^( )∈(0,1] controls how much of the candidate replaces the previous state. For LrL_r reservoir layers, a typical readout uses (A2) t=Rθ([t(1);…;t(Lr)]),y_t=R_θ\! ([h_t^(1);…;h_t^(L_r)] ), where [⋅;⋅][\,·\,;\,·\,] denotes concatenation and RθR_θ is the trainable readout. This gives the readout access to temporal features at different depths. B.2. Structured Reservoir Computing (SRC) SRC replaces dense random projections with structured random transforms (Dong et al., 2020). For a layer with N state coordinates and input t∈ℝdu_t ^d, the basic update is (A3) t=1Nϕ([σrt−1;σint]).h_t= 1 Nφ\! (S[ _rh_t-1; _inu_t] ). Here, ϕφ is an elementwise nonlinear activation, and σr _r and σin _in scale the previous state and current input. The structured projection ∈ℝN×(N+d)S ^N×(N+d) is constructed from a square transform of the form (A4) m=γmm1m2m3,T_m=γ m\,H_mD_1H_mD_2H_mD_3, where mH_m is an orthogonally normalized Hadamard matrix, each jD_j is diagonal with independent random signs in −1,+1\-1,+1\, and γ sets the projection scale. The transform dimension m≥N+dm≥ N+d is a power of two, and the corresponding N×(N+d)N×(N+d) submatrix gives S. The fast Hadamard transform evaluates this projection in O(mlogm)O(m m) operations without storing a dense matrix. The nonlinear state update still processes the sequence one time step at a time. B.3. Parallel Echo State Network (ParalESN) ParalESN separates linear recurrence from nonlinear mixing (Pinna et al., 2026). For one layer with input tu_t, the complex-valued state follows (A5) t _t =¯t−1+α(int+), = h_t-1+α(W_inu_t+b), ¯ =(1−α)+α. =(1-α)I+α . Here, is a fixed complex diagonal matrix, inW_in is the input matrix, b is a bias, I is the identity matrix, and α∈(0,1]α∈(0,1] is the leak rate. The diagonal entries of the effective transition matrix ¯ control decay and rotation of the state coordinates. The mixer then produces real-valued features: (A6) t=tanh(Re(t+)),q_t= \! (Re(Ch_t+c) ), where C represents a fixed convolution across state coordinates, c is its bias, and ReRe extracts the real part. These features feed the next block and the trainable readout. The mixer acts at each time step, outside the temporal recurrence in Eq. A5. The recurrence therefore remains affine, so its updates compose associatively and can be evaluated by a parallel scan over a given input sequence. Layer Parameters Activation Input And Shared Recurrent Trunk Vector Input (PQN) Observation And Previous One-Hot Action; din=do+dad_in=d_o+d_a None Vector Input (PPO) Observation Only; din=dod_in=d_o None Image Input (PQN) CNN Features And Previous One-Hot Action; din=512+dad_in=512+d_a None Input Projection Linear, din→256d_in→ 256 None Recurrent Block GRU Input/Output Width 256→256256→ 256 Sigmoid & Tanh LRU Input/Output Width 256→256256→ 256 None Mixer Linear Layer Linear, 256→256256→ 256 None Mixer Normalization LayerNorm Over 256256 Features, Without Affine Parameters LeakyReLU Trunk Output And Readout Output Projection Linear, 256→256256→ 256 None Readout Hidden Block Linear, 256→256256→ 256; LayerNorm With Affine Parameters; Repeat K Times LeakyReLU Q Output (PQN) Linear, 256→da256→ d_a; One Q-Value Per Action None Actor Output (PPO) Linear, 256→da256→ d_a; Action Logits Or Gaussian Means None Critic Output (PPO) Linear, 256→1256→ 1; Scalar State Value None Table A1. Architecture Of The Recurrent Agents. The left panel contains seven training-return plots and a shared legend for DeepESN, SRC, and ParalESN and their trainable variants. The right panel compares final evaluation returns averaged across eighteen POPGym settings, with bootstrap confidence intervals. Filled markers show trainable models, and hollow markers show standard models. Figure A2. POPGym results for DeepESN, SRC, and ParalESN and their trainable variants. (a) Learning curves on the selected seven tasks over ten seeds. (b) Final evaluation returns across all 1818 POPGym tasks over ten seeds and ten evaluation episodes per seedThe left panel contains seven training-return plots and a shared legend for DeepESN, SRC, and ParalESN and their trainable variants. The right panel compares final evaluation returns averaged across eighteen POPGym settings, with bootstrap confidence intervals. Filled markers show trainable models, and hollow markers show standard models. Appendix C Reservoir Training on POPGym We compare DeepESN, SRC, and ParalESN with their trainable variants on the 18 POPGym tasks. Fig. A2(a) shows the learning curves, and Fig. A2(b) summarizes final evaluation returns across all tasks. Appendix D Detailed Evaluation Results We report detailed final evaluation returns for POPGym, POPGym Arcade, and MuJoCo Playground. For each seed, we evaluate the trained agent over 10 episodes and compute its mean return. Each heatmap entry shows the mean and sample standard deviation across seeds. We use 10 seeds for POPGym and MuJoCo Playground, and 5 for POPGym Arcade. D.1. POPGym The POPGym heatmaps report evaluation results for 1818 tasks and 1111 model configurations. Figs. A3 and A4 report the final evaluation returns for the recurrent baselines and reservoir models, respectively. Both heatmaps use a shared color scale centered at zero. Frozen GRU and Frozen LRU retain trainable mixers and outer projections. We exclude Autoencode, Concentration, CountRecall, and MultiArmedBandit because of implementation bugs in the vector environments. ParalESN achieves a higher mean final evaluation return than its trainable counterpart on six of the seven settings selected for the learning curves (Fig. A4). Layer Parameters Activation Input Scaling 128×128×3128× 128× 3 RGB Image None Conv2D 1 3→643→ 64 Channels; 5×55× 5 Kernel, Stride 22; Output 64×62×6264× 62× 62 LeakyReLU MaxPool2D 1 2×22× 2 Kernel, Stride 22; Output 64×31×3164× 31× 31 None Conv2D 2 64→12864→ 128 Channels; 3×33× 3 Kernel, Stride 22; Output 128×15×15128× 15× 15 LeakyReLU MaxPool2D 2 2×22× 2 Kernel, Stride 22; Output 128×7×7128× 7× 7 None Conv2D 3 128→256128→ 256 Channels; 3×33× 3 Kernel, Stride 22; Output 256×3×3256× 3× 3 LeakyReLU MaxPool2D 3 3×33× 3 Kernel, Stride 11; Output 256×1×1256× 1× 1 None Conv2D 4 256→512256→ 512 Channels; 1×11× 1 Kernel, Stride 11; Output 512×1×1512× 1× 1 LeakyReLU Flatten 512×1×1→512512× 1× 1→ 512 None Table A2. CNN Observation Encoder For POPGym Arcade. A heatmap reports final evaluation returns for MLP and variants of GRU and LRU across eighteen POPGym settings. Each cell contains a mean and standard deviation. Columns distinguish trainable models from variants with frozen recurrent cores. Figure A3. Detailed final evaluation returns for LRU, GRU, Frozen GRU , Frozen LRU and MLP on POPGym. Each entry shows the mean ± sample standard deviation over 10 seeds, with 10 evaluation episodes per seedA heatmap reports final evaluation returns for MLP and variants of GRU and LRU across eighteen POPGym settings. Each cell contains a mean and standard deviation. Columns distinguish trainable models from variants with frozen recurrent cores. Hyperparameter Value Initial Learning Rate 3×10−43× 10^-4 Final Learning Rate 10−1010^-10 Discount γ 0.990.99 GAE λ 0.900.90 Gradient Clipping 0.50.5 PPO Clip 0.20.2 Entropy Coefficient 00 Value-Loss Coefficient 0.50.5 Number Of Environments 64 Rollout Length 1024 Minibatches Per Epoch 16 Update Epochs 4 Total Timesteps 1010M Table A3. PPO Hyperparameters For MemoryChain. Hyperparameter Value Initial LR (Trainable RNNs) 10−510^-5 Initial LR (Fixed RNNs) 5×10−55× 10^-5 Initial LR (MLP) 5×10−75× 10^-7 Final Learning Rate 10−1010^-10 Discount γ 0.990.99 TD(λ) 0.950.95 Gradient Clipping 1010 Exploration ϵε 1→0.051→ 0.05 Number Of Environments 16 Rollout Length 128 Minibatches Per Epoch 16 Update Epochs 4 Total Timesteps 2020M Table A4. PQN Hyperparameters For POPGym. A heatmap compares final evaluation returns for DeepESN, SRC, and ParalESN and their trainable variants across eighteen POPGym settings. Each cell contains a mean and standard deviation. Figure A4. Detailed final evaluation returns for DeepESN, SRC, ParalESN, and their trainable variants on POPGym. Each entry shows the mean ± sample standard deviation over 10 seeds, with 10 evaluation episodes per seed.A heatmap compares final evaluation returns for DeepESN, SRC, and ParalESN and their trainable variants across eighteen POPGym settings. Each cell contains a mean and standard deviation. D.2. POPGym Arcade and MuJoCo Playground We report the final evaluation returns of GRU, LRU, and their frozen variants on four POPGym Arcade tasks and seven MuJoCo Playground settings in Fig. A5. We train the Arcade agents with PQN and the Playground agents with PPO. POPGym Arcade uses a separate implementation from vector POPGym, so we retain its CountRecall task. Hyperparameter Value Initial LR (Trainable RNNs) 5×10−55× 10^-5 Initial LR (Frozen RNNs) 3×10−43× 10^-4 Final Learning Rate 5×10−75× 10^-7 Discount γ 0.990.99 TD(λ) 0.650.65 Gradient Clipping 1010 Exploration ϵε 1→0.051→ 0.05 Number Of Environments 16 Rollout Length 128 Minibatches Per Epoch 4 Update Epochs 4 Total Timesteps 2020M Table A5. PQN Hyperparameters For POPGym Arcade. Hyperparameter Value Initial Learning Rate (Range) [1,5]×10−5[1,5]× 10^-5 Final Learning Rate 10−810^-8 Discount γ 0.9950.995 GAE λ 0.950.95 Gradient Clipping None PPO Clip 0.30.3 Entropy Coefficient 0.010.01 Value-Loss Coefficient 0.50.5 Number Of Environments 16 Rollout Length 128 Minibatches Per Epoch 4 Update Epochs 4 Total Timesteps 6060M Table A6. PPO Hyperparameters For MuJoCo Playground. The top heatmap reports final evaluation returns for four GRU and LRU configurations on four POPGym Arcade tasks. The bottom heatmap reports the same configurations on seven MuJoCo Playground settings. Each cell shows a mean and standard deviation, and the panels use separate color scales. Figure A5. Detailed final evaluation returns for GRU, LRU, and their frozen variants on POPGym Arcade (top) and MuJoCo Playground (bottom). Each entry shows the mean ± sample standard deviation over 5 and 10 seeds, respectively, with 10 evaluation episodes per seed.The top heatmap reports final evaluation returns for four GRU and LRU configurations on four POPGym Arcade tasks. The bottom heatmap reports the same configurations on seven MuJoCo Playground settings. Each cell shows a mean and standard deviation, and the panels use separate color scales. Appendix E Model and Training Parameters Tables A1 and A2 describe the recurrent agents and the CNN encoder used for POPGym Arcade. Here, dod_o is the vector observation dimension, dad_a is the number of discrete actions or the continuous action dimension, and dind_in is the input feature dimension. We use L=1L=1 recurrent block for MemoryChain and L=2L=2 for the other benchmarks. Each readout has K=1K=1 hidden block for MemoryChain, K=2K=2 for POPGym and POPGym Arcade, and K=3K=3 for MuJoCo Playground. PPO uses separate actor and critic readouts with a shared recurrent trunk. All LeakyReLU activations use a negative slope of 0.010.01. The CNN uses no padding, and its output feature-map dimensions are listed as channels × height × width. We use Adam with linear learning-rate decay. Tables A3–A6 list the main training hyperparameters. For MuJoCo Playground, we report the range of initial learning rates used across models and tasks.