Paper deep dive
Optimistic Policy Regularization
Mai Pham, Vikrant Vaze, Peter Chin
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 12:27:51 AM
Summary
Optimistic Policy Regularization (OPR) is a reinforcement learning mechanism that mitigates premature convergence and entropy collapse by anchoring policy updates to historically successful trajectories. It utilizes a dynamic 'Good-Episode' buffer, directional log-ratio reward shaping, and an auxiliary behavioral cloning objective to improve sample efficiency and performance in complex environments like Atari and the CAGE Challenge 2.
Entities (5)
Relation Signals (3)
Optimistic Policy Regularization → evaluatedon → Arcade Learning Environment
confidence 100% · We evaluate OPR instantiated on PPO across 49 Atari environments
Optimistic Policy Regularization → instantiatedon → Proximal Policy Optimization
confidence 100% · When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency
Optimistic Policy Regularization → generalizesto → CAGE Challenge 2
confidence 95% · OPR also generalizes to the CAGE Challenge 2 cyber-defense environment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.
Tags
Links
- Source: https://arxiv.org/abs/2603.06793v1
- Canonical: https://arxiv.org/abs/2603.06793v1
Trouble viewing inline? Open PDF directly →
Full Text
44,161 characters extracted from source content.
Expand or collapse full text
Cover Page Optimistic Policy Regularization Mai Pham, Vikrant Vaze, Peter Chin Keywords: Reinforcement Learning, Proximal Policy Optimization, Exploration, Policy Entropy, Behavioral Cloning. Summary Deep reinforcement learning agents frequently suffer from premature convergence: early entropy collapse drives exploitation of safe, low-reward behaviors and causes rare high-reward trajectories to be forgotten before they can be consolidated. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism that anchors policy optimization to histori- cally successful behavior. OPR maintains a dynamic buffer of high-performing episodes and uses (i) a directional log-ratio reward shaping signal that favors actions consistent with past successes and (i) an auxiliary behavioral cloning objective that directly reinforces these be- haviors. Instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment, achieving the highest score in 22 of 49 Atari games at the 10M-step budget despite baselines being reported at 50M steps. Beyond Atari, OPR generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. Contribution(s) 1. We introduce Optimistic Policy Regularization (OPR), a lightweight optimization mech- anism for policy-gradient reinforcement learning that mitigates premature convergence caused by early exploration collapse. OPR maintains a dynamic buffer of historically high- performing episodes and biases learning toward these behaviors through directional log- ratio reward shaping and an auxiliary behavioral cloning objective. Instantiated on PPO, OPR improves sample efficiency on the Arcade Learning Environment, achieving the high- est score in 22 of 49 Atari games at the 10M-step budget, and further generalizes to the CAGE Challenge 2 cyber-defense environment where it surpasses the competition-winning Cardiff agent while using the same underlying PPO architecture. Context:Results are reported for OPR instantiated on PPO; exploring additional base agents (e.g., value-based methods) is left to future work. arXiv:2603.06793v1 [cs.LG] 6 Mar 2026 Optimistic Policy Regularization Mai Pham, Vikrant Vaze, Peter Chin mai.p.pham.th@dartmouth.edu Thayer School of Engineering, Dartmouth College, USA Abstract Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regulariza- tion (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through direc- tional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sam- ple efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments de- spite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense envi- ronment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance. 1 Introduction Deep Reinforcement Learning (DRL) has achieved remarkable success across diverse domains, in- cluding board games (Silver et al., 2016), robotic control (Andrychowicz et al., 2020), and high- dimensional video game environments (Mnih et al., 2015). Among modern DRL algorithms, Prox- imal Policy Optimization (PPO) (Schulman et al., 2017) has emerged as a widely adopted standard due to its relative ease of tuning, robustness, and strong empirical performance. Despite these ad- vantages, model-free algorithms such as PPO often struggle to reach globally optimal policies in visually complex or delayed-reward environments, such as the Atari 2600 benchmark suite (Belle- mare et al., 2013), frequently converging to suboptimal local strategies. A central challenge underlying this behavior is the exploration–exploitation trade-off. During early training, exploration is primarily driven by the entropy of the policy’s action distribution. In en- vironments with sparse or delayed rewards, the agent may quickly discover a safe but low-reward behavior pattern, leading to a rapid collapse in policy entropy. Once exploration diminishes, the pol- icy becomes effectively pessimistic about alternative trajectories. Even when high-reward behaviors are occasionally discovered through stochastic exploration, standard on-policy updates may fail to reinforce them because the policy already assigns negligible probability mass to those actions. Problem Statement. This dynamic creates a common failure mode in actor–critic reinforcement learning: the agent prematurely commits to a locally stable but globally suboptimal policy and subsequently discards rare high-reward trajectories discovered during early exploration. Once this collapse occurs, the policy rarely revisits or reinforces these trajectories, limiting both final perfor- mance and sample efficiency. 1 Several approaches attempt to mitigate this failure mode. Entropy regularization (Mnih et al., 2016) encourages broader exploration but does so uniformly across the state space, often resulting in unfo- cused exploration. Self-Imitation Learning (SIL) (Oh et al., 2018) instead replays past experiences whose returns exceed the current value estimate. However, SIL relies on value-function estimates to select transitions and integrates less naturally with strictly on-policy optimization. To address these limitations, we introduce Optimistic Policy Regularization (OPR), a lightweight framework that anchors policy updates to historically successful trajectories discovered during train- ing. OPR maintains a curated buffer of high-performing episodes and introduces two complementary mechanisms during optimization: a directional log-ratio reward shaping signal that biases policy up- dates toward previously successful action distributions, and an auxiliary behavioral cloning objective that reinforces these behaviors directly. Unlike uniform entropy bonuses, OPR encourages exploration in regions of the state space that have empirically produced strong outcomes. By preventing the policy from forgetting rare but valuable behaviors discovered during training, OPR allows the agent to escape early local optima and con- tinue improving beyond the plateau commonly observed in actor–critic methods. In this work, we define optimism not as optimism in the face of uncertainty, but as an optimistic an- choring to empirically successful trajectories. Instead of discarding rare high-reward behaviors due to entropy collapse or conflicting gradient signals, OPR preserves and reinforces these trajectories throughout training. The key contributions of this work are as follows: 1. We introduce Optimistic Policy Regularization (OPR), a lightweight framework that mitigates premature convergence by anchoring policy updates to historically successful trajectories. 2. We propose a trajectory-based regularization mechanism combining directional log-ratio reward shaping with an auxiliary behavioral cloning objective derived from a dynamic buffer of top- performing episodes. 3. We evaluate OPR instantiated on PPO across 49 Atari environments and show that it substantially improves performance under a 10-million step training budget, outperforming standard baselines in 22 environments. We further validate OPR in the CAGE Challenge 2 cyber-defense environ- ment, where it surpasses the competition-winning Cardiff agent while using the same underlying PPO architecture. 2 Related Work Optimistic Policy Regularization (OPR) relates to several strands of reinforcement learning research, particularly work on exploration strategies, experience reuse in on-policy learning, and policy regu- larization mechanisms. Exploration in Reinforcement Learning. Managing exploration remains a central challenge in deep reinforcement learning. A common approach in actor–critic methods is entropy regulariza- tion (Williams, 1992; Mnih et al., 2016), which encourages stochastic policies by adding an entropy bonus to the objective. While effective at delaying premature convergence, entropy regularization is fundamentally unguided, promoting global stochasticity rather than targeted exploration of promis- ing behaviors. Alternative approaches include intrinsic-motivation methods such as curiosity-driven exploration (Pathak et al., 2017) and random network distillation (Burda et al., 2019), which reward novelty in the state space. In contrast to these approaches, which emphasize discovering new states, OPR focuses on retaining and reinforcing empirically successful behaviors discovered during train- ing. Recent work has also explored stabilizing exploration through pessimistic value regularization, such as Conservative Q-Learning (CQL) (Kumar et al., 2020), which penalizes value estimates for out-of-distribution actions. While such methods improve stability through pessimism, OPR adopts the complementary perspective of encouraging optimism toward historically successful trajectories. 2 More recently, adaptive exploration mechanisms have also been proposed to improve exploration behavior in PPO-style algorithms (e.g., (Lixandru, 2024)). Experience Reuse in On-Policy Methods. Experience replay is a central component of off-policy algorithms such as DQN (Mnih et al., 2015) and SAC (Haarnoja et al., 2018). Integrating replay mechanisms into on-policy algorithms like PPO is more challenging due to the mismatch between past and current policy distributions. Methods such as ACER (Wang et al., 2017) address this issue through importance sampling corrections. Closely related to our work is Self-Imitation Learning (SIL) (Oh et al., 2018), which stores past experiences and applies a behavioral cloning objective to transitions whose observed returns exceed the value estimate. Unlike SIL, which selects individual transitions based on value-function advantages, OPR operates at the trajectory level and identifies successful behaviors using episodic return statistics. Furthermore, OPR introduces a directional reward shaping signal that directly biases policy optimization toward historically successful action distributions, rather than relying solely on imitation. Policy Regularization and Trust Regions. Constraining policy updates is a core principle behind Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) and Proximal Policy Optimiza- tion (PPO) (Schulman et al., 2017), which restrict policy updates relative to the previous policy π old to maintain stability. OPR similarly introduces a regularization mechanism that anchors the policy to historically successful behaviors. However, instead of constraining updates relative to the previous policy, OPR derives its reference from trajectories that achieved high empirical returns during train- ing. This idea shares conceptual similarities with advantage-weighted and offline RL methods such as AWAC (Nair et al., 2020), which bias policy learning toward high-return actions in a dataset. Un- like these approaches, OPR constructs its reference trajectories dynamically during online training. Recent trajectory-centric methods such as Decision Transformer (Chen et al., 2021) also emphasize learning from high-return trajectories, but formulate the problem as sequence modeling rather than on-policy policy-gradient optimization. 3 Background: Proximal Policy Optimization (PPO) Proximal Policy Optimization (Schulman et al., 2017) is a policy gradient method that optimizes a surrogate objective while penalizing large updates to the policy network. Let π θ (a|s) denote a parameterized stochastic policy, and let π old denote the policy before the current update step. The PPO objective maximizes the clipped surrogate advantage: L CLIP (θ) = ˆ E t " min r t (θ) ˆ A t , clip r t (θ), 1− ε, 1 + ε ˆ A t !# , where r t (θ) = π θ (a t |s t ) π old (a t |s t ) is the probability ratio, ˆ A t is the generalized advantage estimator (Schul- man et al., 2016), and ε is a hyperparameter bounding the policy update step. To encourage sufficient exploration, an entropy bonusH(π θ (·|s t )) is typically added to the optimization objective, the actor loss (minimized during optimization) is therefore: L Actor (θ) =−L CLIP (θ)− c ent H π θ (·|s t ) ,(1) where c ent is the entropy coefficient. However, as training progresses, the policy inevitably assigns near-zero probabilities to exploratory actions, causing the entropy to collapse towards zero and nul- lifying the exploration bonus. 4 Proposed Method: Optimistic Policy Regularization While the core components of Optimistic Policy Regularization (OPR)—the Good-Episode Mem- ory Buffer and auxiliary Behavioral Cloning—can be integrated into many reinforcement learning algorithms, we formulate and evaluate it within the Proximal Policy Optimization (PPO) framework due to its strong empirical baseline performance. To prevent premature exploration collapse, OPR 3 Environment Agent (π θ ) Actiona t States t+1 , Rewardr t On-Policy Rollout Data L Actor PPO (θ) Good-Episode Buffer (M) Filter topP-th percentile episodes Directional Log-Ratio Reward Shaping L BC OPR (θ) π good (a|s t ) Expert actions Shaped rewards r OPR t Optimizer Updateθ Optimistic Policy Regularization (OPR) Figure 1: The Optimistic Policy Regularization Architecture. Top-performing trajectories are con- ditionally stored in a FIFO Good-Episode Buffer. During optimization, these historical successes provide both a directional log-ratio reward shaping signal on the primary policy and an auxiliary Behavioral Cloning objective, shielding the agent against pessimistic exploration collapse. maintains a memory of historically successful trajectories and pulls the policy toward their action distributions when learning begins to converge suboptimally. This augments the PPO objective with two mechanisms that anchor the policy to high-performing behaviors. 4.1 The Good-Episode Memory Buffer Standard on-policy algorithms discard transition data immediately after the corresponding policy update. OPR alters this paradigm by retaining a specialized "Good-Episode Buffer" M. During training, for each completed episode E i = (s (i) t ,a (i) t ,r (i) t ,π old (a (i) t |s (i) t )) T i t=0 , we compute the total episodic return R(E i ) = P T i t=0 r (i) t . To determine which episodes are retained, we maintain a sliding window of the returns from the K most recent episodes, denoted asR K =R(E i−K+1 ),...,R(E i ). An episode E i is admitted into the bufferM if its return strictly exceeds a dynamic threshold τ : τ = Percentile(R K ,P ),(2) where P ∈ [0, 100] is a percentile hyperparameter (typically P = 75). The buffer stores up to N max transitions and uses episode-level FIFO eviction to retain recent high-performing behaviors while discarding stale trajectories. During each PPO optimization epoch, on-policy mini-batches are augmented with samples from the Good-Episode Buffer. 4.2 Directional Log-Ratio Reward Shaping To guide the policy toward historically successful behaviors, OPR introduces a directional reward shaping signal derived from the log-probability ratio between the action distributions of successful trajectories and the current policy. Let π good (a|s) denote the action probability recorded during the execution of a high-performing episode stored in the Good-Episode BufferM, and let π θ (a|s) denote the current policy. For each transition (s t ,a t ) encountered during training, we compute the directional log-ratio ∆ t = logπ good (a t |s t )− logπ θ (a t |s t ),(3) which measures how much more likely the selected action was under the historically successful policy compared to the current policy. This quantity corresponds to the integrand of the directional KL divergence D KL (π good ∥ π θ ) evaluated at the sampled action. For numerical stability, the signal is smoothly bounded using a hyperbolic tangent transformation: ̃ ∆ t = clip 2 tanh(∆ t /2),−δ,δ ,(4) 4 where δ is a shaping bound hyperparameter. The reward is then multiplicatively adjusted as r OPR t = r t 1 + α ̃ ∆ t ,(5) where α controls the strength of the optimistic shaping signal. Intuitively, this mechanism increases rewards for actions consistent with historically successful trajectories and decreases rewards for actions that diverge from them. Because the shaping signal depends only on the sampled action, it provides a lightweight alternative to distribution-level KL regularization. The modified reward r OPR t is then used directly in PPO advantage estimation and policy optimization, yielding a targeted and computationally efficient learning signal. 4.3 Auxiliary Behavioral Cloning In situations where the current policy has already collapsed—assigning near-zero probabilities to historically successful actions—the log-ratio shaping signal may become weak. To provide a direct optimization signal in these cases, OPR simultaneously applies an auxiliary Behavioral Cloning (BC) objective over states and actions stored in the Good-Episode BufferM: L BC OPR (θ) =− ˆ E (s,a)∼M [logπ θ (a|s)].(6) By treating high-performing trajectories as implicit expert demonstrations, the BC loss encourages the policy to retain non-zero probability mass over actions that previously yielded high rewards, effectively reviving exploration paths that might otherwise disappear during training. The PPO objective is then optimized using the shaped rewards r OPR t , yielding the final actor objective L Total (θ) =L Actor (θ) + λ BC L BC OPR (θ),(7) where λ BC controls the influence of the auxiliary imitation objective relative to the standard PPO policy loss. This auxiliary objective is conceptually related to advantage-weighted and behavior- regularized policy learning methods, which bias policy updates toward actions associated with high returns. In contrast to methods that rely on static datasets or explicit advantage weighting, OPR dynamically constructs its reference trajectories online and integrates the resulting signal within a standard on-policy PPO optimization process. 5 Experiments We evaluate the efficacy of Optimistic Policy Regularization (OPR) on the Arcade Learning Envi- ronment (ALE) (Bellemare et al., 2013), comparing it against standard baseline actor-critic methods. Our primary experimental focus is evaluating how effectively the agent can overcome early entropy collapse to reach higher final return peaks, while using significantly fewer samples. We specifi- cally evaluate at the 10 million environment step benchmark (equivalent to 100 epochs or 40 million frames), which is precisely the window where standard PPO typically experiences its most severe entropy collapse and flatlines in complex environments. 5.1 Experimental Setup We evaluate PPO augmented with OPR across 49 Atari 2600 games from the Arcade Learning Environment. The agent uses the standard Nature CNN architecture for visual feature extraction. The Good-Episode Buffer stores up to 100 transitions from episodes whose returns exceed the 75th percentile of recent episodic returns. Reward shaping strength is controlled by α, with the shaping signal bounded by δ = 0.01, and the auxiliary Behavioral Cloning objective is weighted by λ BC = 1.0. For benchmarking, we compare OPR against several widely used reinforcement learning baselines. PPO and A2C are evaluated using our own implementations under the same 10M-step training bud- get as OPR. For additional context, we also report results from prior work for agents trained under 5 (a) 10M Steps (49 Games)(b) 50M Steps (14 Games) Figure 2: Performance dominance across the Atari suite. The left panel compares the total number of environments where each evaluated configuration achieved the highest peak score at the 10M-step benchmark across the full 49-game suite. The right panel shows the same metric for the 14-game subset evaluated after 50M environment steps. OPR maintains a clear performance advantage across both evaluation horizons. the standard 50M-step Atari protocol, including DQN (Mnih et al., 2015), Actor-Critic with Pri- oritized Experience Replay (ACPER), and Self-Imitation Learning (SIL) (Oh et al., 2018). These comparisons provide context relative to established Atari benchmarks, while highlighting the sample efficiency of OPR under a significantly smaller interaction budget. 5.2 Main Results Table 1 reports raw scores across the 49 Atari environments. Games were evaluated either upon reaching convergence or at the 100-episode mark (equivalent to 10M environment steps), whereas benchmark baselines are reported after 50M steps. This protocol highlights the sample efficiency of OPR by comparing performance achieved with only a fraction of the interaction budget typically used in Atari benchmarks. OPR records the highest score in 22 of the 49 games, compared to A2C (3), ACPER (4), and A2C+SIL (19, with 1 tie). Notably, these results are obtained with 5× fewer environment interactions than the baseline benchmarks. OPR shows a clear advantage in environments requiring sustained exploration, suggesting improved credit assignment and policy refinement under limited data. Beyond Atari, we evaluate OPR in the CAGE Challenge 2 cyber-defense environment. Using the same PPO architecture, OPR achieves a final reward of−4.2, surpassing the competition-winning Cardiff agent (−6.2). This result suggests that optimistic trajectory anchoring generalizes to more complex adversarial decision-making tasks. 5.3 Detailed Game Analysis To better understand the mechanisms driving OPR’s performance gains and its sample-efficient learning dynamics, we analyze its behavior across several representative categories of Atari envi- ronments, including sparse-reward exploration tasks, high-complexity control environments, and long-horizon strategic games. Hard Exploration (Montezuma’s Revenge, Venture): Sparse-reward environments remain among the most challenging benchmarks in Atari reinforcement learning due to the difficulty of discover- ing reward states through random exploration. In Montezuma’s Revenge, OPR achieves a score of 2500, substantially exceeding the strongest baseline result of 1100 achieved by SIL, while most other methods fail to obtain any meaningful reward. A similar pattern appears in Venture, where OPR achieves 1380 while all baseline methods remain at zero. These results indicate that OPR is able to discover reward trajectories that remain inaccessible to existing approaches within compara- ble interaction budgets. Additional improvements are observed in exploration-heavy environments 6 Table 1: Detailed Performance across 49 Atari games. OPR is evaluated at 10M steps (100 Epochs), while ACPER, SIL, and DQN baselines are evaluated at the standard 50M step (200M frame) mark. OPR outperforms or ties these high-resource baselines in 22 environments. GameDQNA2CPPO ACPERSIL OPR (Ours) Alien227.11859.21475.6390.22242.23389.0 Amidar163.6739.9757.2424.81362.02571.2 Assault537.51981.44970.6818.21812.05285.7 Asterix855.016083.34531.03533.117984.23410.0 Asteroids1007.82056.02195.11780.12259.42632.0 Atlantis58012.5 3032444.2 3010196.0 58012.5 3084781.7880040.0 BankHeist421.11333.71118.81203.21137.8506.0 BattleZone20433.310683.316053.3 15025.025075.033800.0 BeamRider2896.73931.75046.12602.42366.25701.2 Bowling46.131.240.959.331.160.8 Boxing73.299.794.6100.099.683.4 Breakout119.9501.6339.0118.5452.0111.7 Centipede2753.83857.84418.17790.17559.523521.5 ChopperCommand1090.03464.22837.21307.56710.020160.0 CrazyClimber34390.0129715.8104169.0 19918.8130185.8150110.0 DemonAttack6310.518331.411370.44777.510140.579421.0 DoubleDunk-16.1-0.5-13.5-9.821.5-0.6 Enduro276.80.0694.03113.31205.1434.0 FishingDerby-14.439.123.459.855.821.0 Freeway22.60.032.031.432.233.0 Frostbite110.0339.5314.12342.56289.89216.0 Gopher2148.09358.54546.13919.523304.21648.0 Gravitar263.3329.2414.7627.51874.21845.0 Hero12300.028008.129104.9 13299.133156.733831.0 IceHockey-4.4-4.3-2.10.0-2.43.2 Jamesbond365.1399.2511.0598.1310.817505.0 Kangaroo3340.01563.35318.05875.02888.312300.0 Krull3800.08883.97488.0 11323.210614.66791.0 KungFuMaster20500.032507.526640.0 20485.034449.26830.0 MontezumaRevenge0.05.80.00.01100.02500.0 MsPacman1016.02843.42650.01016.04025.12970.0 NameThisGame3888.011174.27241.62888.014958.28754.0 Pong18.920.820.720.920.914.2 PrivateEye100.0210.8100.0100.0661.2100.0 Qbert3800.017605.214104.2657.2104975.636652.5 Riverraid3400.013036.07963.02224.514306.17998.0 RoadRunner9562.539874.234400.08925.057071.721210.0 Robotank10.53.215.47.710.523.4 Seaquest804.51795.21680.0804.52456.51720.0 SpaceInvaders729.52466.11000.0729.52951.71945.5 StarGunner10000.057371.735000.01107.531309.23700.0 Tennis-17.0-10.3-14.4-17.0-17.3-0.6 TimePilot3952.55346.74500.03952.510811.77560.0 Tutankham270.7305.6200.0270.7340.5127.1 UpNDown9562.548131.835000.09562.553314.667114.0 Venture0.00.00.00.00.01380.0 VideoPinball21797.7391241.65000.0 21797.7461522.428558.3 WizardOfWor1550.04196.74000.01550.07088.35470.0 Zaxxon4278.8124.25000.04278.89164.26720.0 such as Gravitar and Tutankham, where OPR reaches 1845 and 127.1 respectively, remaining com- 7 petitive with the strongest baseline methods despite operating with substantially fewer environment interactions. High-Complexity Score Scaling (DemonAttack, Centipede, BattleZone): OPR also demon- strates strong performance in environments requiring sustained control and long-horizon credit as- signment once rewarding strategies are discovered. In DemonAttack, OPR achieves 79,421, dramat- ically surpassing A2C (18,331) and SIL (10,140). Similarly, in Centipede, OPR reaches 23,521, compared to 7,790 for ACPER and 7,559 for SIL. In BattleZone, OPR achieves 33,800, outper- forming all baseline methods. These environments reward consistent policy refinement and stable value propagation, suggesting that OPR not only discovers rewarding behaviors but also scales them efficiently once promising trajectories emerge. Long-Horizon Strategic Control (Jamesbond, Kangaroo, ChopperCommand): Several Atari environments require coordinated decision-making across extended action sequences. In these set- tings, OPR exhibits particularly strong advantages. In Jamesbond, OPR achieves 17,505, far ex- ceeding the strongest baseline score of 598. In Kangaroo, OPR reaches 12,300, outperforming all baselines by a substantial margin, while in ChopperCommand it achieves 20,160, significantly ex- ceeding competing methods. These results suggest that OPR effectively propagates delayed rewards and maintains stable policy improvements over long temporal horizons. Failure Cases and Dense-Reward Environments (Asterix, Atlantis, VideoPinball): While OPR achieves the highest score in a substantial number of environments, several cases remain where baseline methods perform better. For example, SIL achieves stronger results in environments such as Asterix, Qbert, and VideoPinball, while ACPER performs best in games such as Enduro and Fish- ingDerby. These environments typically feature dense reward structures or highly reactive gameplay dynamics, where large replay buffers and imitation-based updates can provide strong advantages. Nevertheless, it is important to emphasize that OPR achieves its results using only 10M environ- ment interactions, whereas all baselines are evaluated after 50M environment steps, highlighting the strong sample efficiency of the proposed approach. 5.4 Detailed Performance at 50M Steps (500 Epochs) Table 2: Detailed Performance at 50M Steps (500 Epochs) across the subset of 14 Atari games. Baseline scores correspond to standard 50M-step benchmark results reported in prior work. GameDQNA2CPPOACPERA2C+SILOPR (Ours) Alien227.11859.21475.6390.22242.24888.0 Amidar163.6739.9757.2424.81362.07155.3 Assault537.51981.44970.6818.21812.014508.2 Asterix855.016083.34531.03533.117984.235110.0 BankHeist421.11333.71118.81203.21137.8375.0 BeamRider2896.73931.75046.12602.42366.29663.6 Bowling46.131.240.959.331.158.5 Breakout119.9501.6339.0118.5452.0111.7 Centipede2753.83857.84418.17790.17559.523521.5 ChopperCommand1090.03464.22837.21307.56710.02070.0 CrazyClimber34390.0129715.8104169.019918.8130185.8150110.0 DemonAttack6310.518331.411370.44777.510140.579421.0 Pong18.920.820.720.920.914.2 Qbert3800.017605.214104.2657.2104975.636652.5 To provide a direct comparison with established Atari benchmarks, we continued training a subset of environments to the standard evaluation horizon of 50M environment interactions. Due to computational constraints, this extended training was performed on a representative subset of 14 Atari environments. The environments were not hand-selected to favor the proposed method; 8 instead, they were chosen to reflect a diverse range of gameplay dynamics, including sparse-reward exploration tasks, dense-reward arcade environments, and long-horizon control problems. Table 2 reports the resulting performance. In this experiment, OPR is applied directly to PPO in order to isolate the contribution of the proposed mechanism. This allows a controlled compari- son between PPO and PPO+OPR, while still situating the results within the context of commonly reported Atari baselines including DQN, A2C, ACPER, and A2C+SIL. Performance Under Equal Training Budgets. Across the evaluated environments, PPO aug- mented with OPR consistently matches or exceeds the performance of standard PPO and frequently outperforms other baseline methods. In particular, OPR achieves the highest score in 8 of the 14 evaluated games, including Alien, Amidar, Assault, Asterix, BeamRider, Centipede, CrazyClimber, and DemonAttack. These improvements demonstrate that the advantages observed in the earlier 10M-step evaluation are not solely due to faster early learning but persist even when all methods are given equivalent interaction budgets. Gains in Complex Environments. Several of the strongest improvements appear in environments requiring sustained control and long-horizon policy refinement. For example, in Centipede OPR reaches 23,521, significantly exceeding the best baseline score of 7,790. Similarly, in DemonAttack OPR achieves 79,421, substantially outperforming all baseline methods. These environments reward consistent policy optimization over extended trajectories, suggesting that OPR supports stable long- term policy refinement once promising strategies are discovered. Sample Efficiency. Importantly, the 50M-step evaluation should be interpreted primarily as a veri- fication of stability rather than the main performance claim of this work. The central contribution of OPR lies in its sample efficiency: the earlier results demonstrate that OPR achieves competitive or superior performance using only 10M environment interactions, whereas most benchmark meth- ods require 50M interactions. The extended training results therefore confirm that OPR retains its advantages under equal training budgets while providing substantially faster learning in low-sample regimes. 5.5 Breakthrough Learning Dynamics To better understand how OPR achieves strong performance with significantly fewer environment interactions, we analyze the learning dynamics across representative Atari environments. Figure 3 visualizes the training curves for eight environments in which OPR demonstrates particularly strong improvements. The dashed horizontal lines indicate the benchmark scores achieved by existing methods after 50M environment interactions, while the vertical dashed line marks the 10M inter- action point corresponding to our primary evaluation budget. Rapid Performance Breakthrough. Across multiple environments, OPR rapidly approaches and surpasses the final performance of baseline methods within the first 10M interactions. For example, in Amidar and Assault, OPR reaches or exceeds the strongest baseline scores well before the 10M- step mark. Similar patterns are observed in BeamRider and Centipede, where OPR quickly surpasses the PPO and A2C baselines and continues improving thereafter. These results highlight the strong sample efficiency of the proposed approach: policies trained with OPR are able to discover high- reward behaviors using substantially fewer environment interactions. Learning Beyond Early Gains. Importantly, the rapid early improvements do not lead to premature saturation. In most environments, including Asterix, CrazyClimber, and DemonAttack, the learning curves continue to improve steadily even near the 50M interaction horizon. This indicates that OPR not only accelerates the discovery of effective policies but also supports continued policy re- finement over extended training horizons. Such behavior suggests that the mechanism improves both exploration and long-horizon credit assignment rather than merely providing faster early learning. Consistency Across Diverse Environments. The selected environments span a range of gameplay characteristics, including dense-reward arcade environments (BeamRider, Assault), score-scaling 9 Figure 3: Learning dynamics of OPR on representative Atari environments where the method demonstrates strong improvements. The green curve shows the performance of PPO augmented with OPR during training, with shaded regions indicating variability across evaluations. Dashed horizontal lines denote benchmark scores achieved by baseline methods (DQN, A2C, PPO, ACPER, and A2C+SIL) after 50M environment interactions. The vertical dashed line marks the 10M-step evaluation point used for the primary comparison in this work. In many environments, OPR ap- proaches or surpasses the final baseline performance within the first 10M interactions and continues improving thereafter. Notably, several curves show no clear performance plateau even near the 50M interaction horizon, suggesting that OPR does not prematurely saturate and continues improving policies over extended training. environments (DemonAttack, CrazyClimber), and environments requiring more complex strategic behavior (Centipede, Alien). Across these diverse settings, OPR consistently shows faster perfor- mance growth compared to baseline methods, reinforcing the robustness of the proposed approach. General Applicability of OPR. It is also worth noting that OPR is not designed as a standalone reinforcement learning algorithm but rather as a general optimization mechanism that can be in- tegrated into a variety of RL agents. In this work, we apply OPR specifically to PPO in order to provide a clean and controlled comparison with widely used policy-gradient baselines. Future work may explore integrating OPR into other algorithm families, including value-based methods, where similar improvements in exploration and sample efficiency may be realized. 5.6 Cyber-Security Domain: CAGE Challenge 2 To show the general applicability of OPR beyond classic arcade games, we evaluate its performance in the Cyber Autonomous Agents (CAGE) Challenge 2, a complex cyber-defense environment. In this domain, a defender agent must protect a network against a multi-stage attacker. This environ- ment is characterized by sparse rewards and a need for high resilience against attack strategies. Figure 4 shows the training dynamics of Cardiff agent, which was the winner of the CAGE Challenge 2 competition, augmented with OPR compared against the PPO baseline and the original Cardiff 10 020000400006000080000100000 Training Episodes 400 350 300 250 200 150 100 50 0 Average Episodic Reward CAGE Challenge 2: Attacker Resilience over Time PPO Baseline Cardiff OPR (Ours) 80000850009000095000100000 Training Episodes 14 12 10 8 6 4 2 0 Average Episodic Reward Attacker Resilience (Zoomed) Cardiff OPR (Ours) Figure 4: Performance of OPR on the CAGE Challenge 2 attacker resilience environment. The right figure shows a zoomed view of the final phase, where OPR surpasses the Cardiff’s winning solution. agent. The PPO baseline fails to learn a stable strategy, remaining at substantially negative reward throughout training. In contrast, both Cardiff and OPR rapidly improve during the early training phase, approaching the near-zero reward regime that characterizes effective defensive policies. Importantly, OPR consistently achieves higher reward than the Cardiff solution during the later stages of training. As highlighted in the inset of Figure 4, OPR reaches a final average episodic re- ward of approximately−4.2, compared to roughly−6.2 for Cardiff. This represents a clear improve- ment over the competition-winning approach. Notably, this gain is achieved without environment- specific tuning, as OPR is applied directly to the same PPO architecture used in the Cardiff agent. These results demonstrate that the benefits of OPR extend beyond Atari-style benchmarks to more complex cyber-defense environments, improving both learning stability and final policy perfor- mance in adversarial settings. 6 Conclusion In this paper, we introduced Optimistic Policy Regularization (OPR), a lightweight framework for reinforcement learning that improves sample efficiency and mitigates premature convergence. OPR maintains a dynamic memory of historically successful trajectories and regularizes policy updates toward these behaviors through regularization shaping and an auxiliary behavioral cloning objec- tive. This mechanism preserves valuable exploratory behaviors while allowing policies to improve beyond early local optima. Empirically, OPR improves both sample efficiency and final performance when instantiated on PPO in the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million interac- tion budget, OPR achieves the highest score in 22 environments, outperforming baselines including A2C, SIL, PPO, and DQN. Analysis of learning dynamics shows that OPR avoids the early perfor- mance plateau commonly observed in actor–critic methods and continues improving under extended training. Beyond standard benchmarks, OPR also generalizes to a real-world cyber-defense setting in the CAGE Challenge 2 environment, where it surpasses the competition-winning Cardiff agent using the same PPO architecture. These results suggest that OPR provides a general mechanism for stabilizing exploration and improving policy optimization across reinforcement learning domains. Future work will explore extending OPR to off-policy and value-based algorithms, such as Rainbow DQN, and evaluating its effectiveness in continuous control settings. 11 References OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Pinto, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253–279, 2013. Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In In International Conference on Learning Representations, 2019. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. NeurIPS, 2021. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International confer- ence on machine learning, p. 1861–1870, 2018. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. NeurIPS, 2020. Andrei Lixandru. Proximal policy optimization with adaptive exploration, 2024. URL https: //arxiv.org/abs/2405.04664. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, p. 1928–1937, 2016. Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning, p. 3878–3887, 2018. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, p. 2778–2787, 2017. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, p. 1889–1897, 2015. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347, 2017. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. 12 Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International conference on learning representations, 2017. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine learning, volume 8, p. 229–256, 1992. 13 Supplementary Materials The following content was not necessarily subject to peer review. A Hyperparameters Tables 3 and 4 detail the full configuration and hyperparameter settings used for the Atari 2600 and CAGE Challenge 2 experiments, respectively. Table 3: Hyperparameters for Atari 2600 experiments (PPO + OPR). HyperparameterValue PPO Base Learning Rate 2.5× 10 −4 (Linear Decay) Gamma (γ)0.99 GAE Lambda (λ)0.95 Entropy Coefficient0.01 Value Function Coefficient0.25 CLIP Epsilon (ε)0.1 Max Grad Norm0.5 Batch Size256 Hidden Size512 Frames Stack4 Num Train Envs10 Num Test Envs10 Epochs100/500 Steps per Epoch100,000 OPR Specific Buffer Capacity (transitions)100 Good-Eps Percentile (P )75 Shaping ModeDirectional log-ratio Shaping Scale (α)0.5 BC Enable (λ BC )1.0 BC Epochs3 Update Interval50 Epochs Table 4: Hyperparameters for CAGE Challenge 2 experiments. HyperparameterValue Learning Rate0.002 Gamma0.99 Betas[0.9, 0.990] K Epochs6 EPS Clip0.2 Max Episodes100,000 Max Timesteps30 Update Timestep20,000 Good Episodes Buffer Maxlen1000 BC Training Interval50 Episodes 14 B Learning Dynamics in Other Games Figure 5: Breakthrough Learning Dynamics (Balanced Environments): Reward escalation for the remaining 6 environments. In these settings, OPR remains highly competitive with high-resource baselines, demonstrating stable strategy refinement even in environments where dense-reward imi- tation (SIL) or large-scale prioritized replay (ACPER) provide strong specialized advantages. 15