Paper deep dive
Boosting deep Reinforcement Learning using pretraining with Logical Options
Zihan Ye, Phil Chau, Raban Emunds, Jannis Blüml, Cedric Derstroff, Quentin Delfosse, Oleg Arenz, Kristian Kersting
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/13/2026, 12:20:11 AM
Summary
The paper introduces Hybrid Hierarchical Reinforcement Learning (H^2RL), a two-stage framework that uses differentiable symbolic logic and option-based pretraining to inject structural priors into neural reinforcement learning agents. This approach mitigates policy misalignment and reward hacking in long-horizon tasks by guiding agents toward goal-directed behavior during pretraining, while allowing for standard neural policy refinement during post-training, ultimately outperforming purely neural, symbolic, and neuro-symbolic baselines.
Entities (5)
Relation Signals (3)
H^2RL → evaluatedon → Atari Learning Environment
confidence 95% · We evaluate all methods on the Atari Learning Environment (ALE)
H^2RL → mitigates → Policy Misalignment
confidence 95% · H^2RL, a hierarchical neuro-symbolic reinforcement learning (RL) framework designed to mitigate policy misalignment in deep RL.
H^2RL → uses → Logic Manager
confidence 95% · H^2RL consists of four components: (i) a differentiable symbolic logic manager
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep reinforcement learning agents are often misaligned, as they over-exploit early reward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic architectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans' ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H^2RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment interaction. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro-symbolic baselines.
Tags
Links
- Source: https://arxiv.org/abs/2603.06565v1
- Canonical: https://arxiv.org/abs/2603.06565v1
Trouble viewing inline? Open PDF directly →
Full Text
63,498 characters extracted from source content.
Expand or collapse full text
Boosting Deep Reinforcement Learning using Pretraining with Logical Options Zihan Ye 1,2,5 , Phil Chau 1,2 , Raban Emunds 1,2,5 , Jannis Blüml 1,2,5 , Cedric Derstroff 1,2,5 , Quentin Delfosse 2 , Oleg Arenz 1,3 , Kristian Kersting 1,2,4,5 zihan.ye@tu-darmstadt.de 1 CS Department, Technical University of Darmstadt, Germany 2 AIML Group 3 IAS Group 4 DFKI 5 Hessian AI Abstract Deep reinforcement learning agents are often misaligned, as they over-exploit early re- ward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic ar- chitectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans’ ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H 2 RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment inter- action. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro- symbolic baselines. 1 Introduction In deep reinforcement learning (RL), while sparse rewards hinder exploration, dense rewards de- signed to guide agents are prone to reward hacking (Everitt et al., 2021; Skalse et al., 2022; Delfosse et al., 2025). As noted by Goodhart (1984), “When a measure becomes a target, it ceases to be a good measure.” This manifests as shortcut learning, where agents exploit spurious correlations rather than solving the intended task (Ilyas et al., 2019; Chan et al., 2020; Geirhos et al., 2020; Koch et al., 2021; Hermann et al., 2024; Delfosse et al., 2025). For instance, in multi-objective Atari games like Seaquest and Kangaroo (depicted in Fig. 1), deep RL agents often prioritize short-term and easily accessible gains. These agents focus on shooting and punching enemies until oxygen or time runs out—over essential long-horizon objectives like refilling oxygen and collecting divers (in Seaquest) or climbing ladders to reach the joey (in Kangaroo), ultimately leading to failure (Delfosse et al., 2025). To mitigate such pitfalls, many recent approaches have integrated symbolic representations to pro- vide reasoning capabilities similar to human planning (Ye et al., 2022; 2025; Delfosse et al., 2023; Grandien et al., 2024; Luo et al., 2024; Bhuyan et al., 2024; Delvecchio et al., 2025; Shindo et al., 2025; Emunds et al., 2025). However, relying on explicit symbolic reasoning during inference cre- ates significant computational overhead and latency, limiting real-time applicability (Bhuyan et al., 2024; Delvecchio et al., 2025). Moreover, due to its discrete nature, it is tricky, if not impossible, to use symbolic reasoning without extra work in a continuous action space. Alternatively, manual 1 arXiv:2603.06565v1 [cs.AI] 6 Mar 2026 reward shaping can guide exploration (Ng et al., 1999; Gupta et al., 2022), but lacks the precision of symbolic logic and requires tedious, domain-specific tuning that is difficult to generalize. Refilling OxygenReaching Joey Figure 1: Deep reinforcement learning policies are often misaligned, exemplified on neural PPO agents. Although the oxygen is running low in Seaquest (left) and the goal in Kangaroo (right) is to go up, PPO agent fails to choose the optimal ac- tions (in green). Instead, they focus on immediate rewards, e.g., keep attacking enemies. Our approach addresses these limitations by drawing on the cognitive process of scaffold- ing in human learning. Humans rarely learn through unstructured trial-and-error; they rely on explicit instruction and rules to establish fundamentals before transitioning to a phase of “free play” to achieve mastery (Bransford et al., 2000; Dreyfus & Dreyfus, 1980). For example, consider mastering a complex mo- tor skill, such as playing tennis: novices do not begin by learning from playing competi- tive matches. Rather, they start off by master- ing individual components, such as the grip, the swing, and footwork. Only after these foun- dational mechanics are internalized does the learner progress to the open-ended free play to improve their skills. This suggests that effec- tive learning requires an initial phase of struc- tured guidance to establish a behavioral prior, followed by unconstrained optimization. Specifically, based on this intuition, we introduce Hybrid Hierarchical Reinforcement Learning (H 2 RL). The core innovation of H 2 RL is the use of differentiable symbolic logic and options solely during the pretraining phase. This implicitly encodes high-level reasoning and inductive biases into the neural network’s parameters, allowing the agent to internalize long-horizon dependencies. Critically, because the symbolic engine is not required in post-training, the final agent retains the inference speed of a standard neural policy while exhibiting the structural coherence of a symbolic reasoner. Empirically, H 2 RL significantly outperforms baselines in environments with deceptive reward functions. We demonstrate that our agents avoid common misalignment pitfalls and escape reward traps that trap purely neural approaches, effectively solving the trade-off between symbolic control and neural scalability. In summary, our contributions are: (i) We introduce H 2 RL, a hierarchical neuro-symbolic reinforcement learning (RL) framework designed to mitigate policy misalignment in deep RL. Leveraging logic-informed pretrain- ing, H 2 RL embeds logic priors directly into neural policies, facilitating task-semantic aligned decision-making while eliminating the computational overhead of logic reasoning and symbol extraction at inference time. (i) We conducted ablation studies, demonstrating that logic-informed pretraining is crucial for mitigating policy misalignment in deep RL. (i) We demonstrate the versatility of our approach by confirming H 2 RL’s effectiveness as a general pretraining framework for a range of deep RL algorithms. We proceed as follows. We start off by discussing background in Sec. 2, followed by a detailed introduction of H 2 RL in Sec. 3. Before concluding, we present our experimental results and discuss related work in Secs. 4 and 5, respectively. 2 Background A key component of H 2 RL is differentiable logic reasoning; for a detailed review of first-order logic, we refer the reader to App. A. Akin to Shindo et al. (2021; 2023), by defining the initial and t-th step valuation of ground atoms as v (0) and v (t) , we make the logic manager and logic gating module differentiable in three steps: (Step 1) We encode each reasoning rule C i ∈ C as a tensor 2 Figure 2: Overview of the framework. Through logic-informed pretraining, H 2 RL embeds logic priors directly into neural policies, thereby addressing the deep policy misalignment is- sue. H 2 RL provides a two-stage training paradigm. In the first stage, the deep policy is jointly trained with the logic manager and the gating module (referred to as deep policy pretraining). In the second stage, the deep policy is further trained through direct interaction with the environment (referred to as deep policy post-training). See Sec. 3 for details. I i ∈ N G×S×L , where S is the maximum number of possible substitutions for variables, L is the maximum number of body atoms and G is the number of grounded atoms. Specifically, the tensor I i stores at position [j,k,l] the index (0 to G−1) of the grounded atom that serves as the l-th body atom when rule C i derives grounded head j using substitution k. (Step 2) To be able to learn which rules are most relevant during forward reasoning, a weight matrix W consisting of M learnable weight vectors, [w 1 ,..., w M ], is introduced. Each vector w m ∈ R C contains raw weights for the C rules. To convert these raw weights into normalized probabilities for soft rule selection, a softmax function is applied independently to each vector w m , yielding w ∗ m . (Step 3) At each step t, we compute the valuation of body atoms using the gather operation over the valuation vector v (t) , looping over the body atoms for each grounded rule. These valuations are combined using a soft logical AND (gather function) followed by a soft logical OR across substitutions: b (t) i,j,k = Y 1≤l≤L gather(v (t) , I i )[j,k,l], c (t) i,j = softor γ (b (t) i,j,1 ,...,b (t) i,j,S ).(1) Here, i indexes the rule, j the grounded head atom, and k the substitution applied to existentially quantified variables. The resulting body evaluations c (t) i,j are weighted by their assigned rule weights w ∗ m,i , and then aggregated across rules and rule sets: h (t) j,m = X 1≤i≤C w ∗ m,i · c (t) i,j , r (t) j = softor γ (h (t) j,1 ,...,h (t) j,M ), v (t+1) j = softor γ (r (t) j ,v (t) j ). (2) We provide full details of this differentiable reasoning procedure in App. E. 3 Hybrid Hierarchical Reinforcement Learning We now present the H 2 RL framework. As shown in Fig. 2, H 2 RL consists of four components: (i) a differentiable symbolic logic manager, (i) a set of pretrained option workers, (i) a neural RL policy, and (iv) an MoE gating module. H 2 RL provides a two-stage training paradigm. In the first stage of training (pretraining), the neural RL policy is jointly trained with the logic manager and the gating module. In the second stage of training (posttraining), the neural RL policy is further trained through standard interactions with the environment. We now introduce the individual modules and explain how they are jointly trained. Specifically, we use PPO as an example of a neural RL policy. 3 We consider a Markov decision process (MDP) with state spaceS , action spaceA, transition kernel p, and reward function R. In our environments, the agent observes both a low-level visual state x t ∈ R 4×84×84 (a stack of frames) and a high-level symbolic state z t , for example, an object-centric representation. Together, they form the full state s t = (x t ,z t )∈S . Logic manager and pretrained option workers. The logic manager is a differentiable logic pro- gram, parameterized by θ, that maps the symbolic state z t to a distribution over a finite set of option workersO : π logic (o t | z t ;θ) for o t ∈ O. Each option worker o ∈ O corresponds to a low-level policy π ω o (a t | x t ) that is trained separately on a subtask, such as “grab the hammer”, “use ham- mer”, or “climb up”. The workers are not restricted to neural policies, but could also be logic-based. For details on how we pretrain the options, please refer to App. G. These workers are kept fixed during agent training. Conditioned on z t , the logic manager defines a hierarchical distribution over actions by marginalizing over options: π L (a t | x t ,z t ) = P o∈O π logic (o| z t ;θ)π ω o (a t | x t ). Neural RL policy and Gating Module (MoE). In parallel to the logic manager, the neural RL policy π N (a t | x t ;φ) is jointly trained with the MoE Module on the raw visual input x t . It uses the standard actor-critic architecture for Atari-style domains: a convolutional backbone followed by a linear policy head and a linear value head. The neural controller directly parameterizes a distribution over primitive actions: a t ∼ π N (· | x t ;φ) . We introduce a gating module b ψ , as a Mixture-of- Experts (MoE) gate, to combine the output of the logic manager with the neural policy. This module b ψ outputs a distribution over logic and neural control. Depending on the configuration, b ψ can be: (i) a differentiable logic program operating on z t (logic-based gating), or (i) a convolutional network operating on x t (neural-based gating). We define: β t = (β L t ,β N t ) = b ψ (b t ), with β L t ,β N t ≥ 0 and β L t + β N t = 1, where b t = z t for logic gating and b t = x t for neural gating. In practice, b ψ produces unnormalized logits, which are passed through a softmax to obtain β t . Given the logic induced policy π L (·| x t ,z t ) in Eq. (3) and the neural policy π N (·| x t ), the final policy is a convex combination, i.e., π H (a t | x t ,z t ) = β L t π L (a t | x t ,z t ) + β N t π N (a t | x t ). Value function. The hybrid agent also maintains a value function that combines a logic critic and a neural critic. The logic critic V L (z t ;θ V ) is implemented as an MLP on symbolic features, while the neural critic V N (x t ;φ V ) shares the convolutional backbone with the PPO policy network. Consistent with the policy mixture we define V H (s t ) = β L t V L (z t ;θ V ) + β N t V N (x t ;φ V ). Training objective. We optimize the parameters of the neural PPO, the logic critic, and the gating module jointly. Let θ denote all trainable parameters of the hybrid agent (including φ, θ V , φ V , and, when trainable, the logic manager and gating parameters). For a batch of trajectories, we compute standard generalized advantage estimates A t and returns R t . The surrogate for the hybrid policy is L clip (θ) = E t h min r t (θ)A t , clip(r t (θ), 1− ε, 1 + ε)A t i , where r t (θ) = π H (a t |x t ,z t ;θ) π old H (a t |x t ,z t ) . The hybrid value function is optimized with a squared error loss: L V (θ) = E t V H (s t ;θ)− R t 2 . To encourage exploration, we add two entropy regularizers: (i) the entropy of the action distribution H π H (·| x t ,z t ) , and (i) the entropy of the gating distribution H(β t ). The full loss L(θ) =−L clip (θ) + c V L V (θ)− c ent E t H π H (·| x t ,z t ) − c gate E t H(β t ) , (3) is minimized during training. The coefficients c V , c ent , and c gate correspond to the value loss, action entropy, and gating entropy, respectively. This completes the pretraining stage. So far, we have two policies: the full H 2 RL framework policy and the neural component part of H 2 RL, the H 2 RL + policy. We then posttrain H 2 RL + via standard on-policy interaction with the environment, which results in the policy H 2 RL ++ . 4 Experimental Evaluation With H 2 RL at hand, our intention is now to investigate five research questions: (RQ1) How does H 2 RL perform compared with baselines? (RQ2) Can H 2 RL pretraining be utilized to boost other deep RL methods? (RQ3) Can H 2 RL pretraining successfully address policy misalignment? (RQ4) 4 H 2 RL + + H 2 RL + H 2 RL PPODQNNUDGEBlendRL Option-critic hDQN 0 10 2 10 3 10 4 10 5 Episodic Return Game Seaquest Kangaroo DonkeyKong Figure 3: Leveraging logic-informed pretraining, H 2 RL with its variants (bolded), outperforms baselines on challenging ALE tasks (Seaquest, Kangaroo, and DonkeyKong) with long-horizon dependencies and reward traps. Although DQN and PPO achieve high returns in Kangaroo, their learned policies remain misaligned; see Sec. 4 (RQ1 and RQ3) for details. Episodic returns are averaged over 12 environments (with 200 runs per environment). Results are presented on a log scale to normalize for the disparate reward magnitudes across games. How do the different components affect the performance of H 2 RL? (RQ5) Does H 2 RL scale to continuous action spaces? Experimental setup. We evaluate all methods on the Atari Learning Environment (ALE) (Belle- mare et al., 2013) with discrete action spaces, including Seaquest and two more challenging games Kangaroo and DonkeyKong, which present long-horizon dependencies and reward traps. Addition- ally, we assess our method and baseline approaches in the Continuous Atari Learning Environment (CALE) (Farebrother & Castro, 2024), focusing on the two more challenging games, Kangaroo and DonkeyKong, with continuous action space. Observations are preprocessed using the standard Atari pipeline: frames are converted to grayscale, resized to 84× 84, and stacked over the last 4 time steps, yielding a visual input x t . For the logic manager, the symbolic state z t is obtained through OCAtari (Delfosse et al., 2024) and used only during pretraining. Note that, in Kangaroo and Don- keyKong, the game does not advance beyond the first level; it restarts at the first level after ending. Baseline Methods. We compare H 2 RL against neural, hierarchical, and neuro-symbolic baselines, including: PPO (Schulman et al., 2017), DQN (Mnih et al., 2015), NUDGE (Delfosse et al., 2023), BlendRL (Shindo et al., 2025), Option-critic (Bacon et al., 2017), C51 (Bellemare et al., 2017), and hDQN (Kulkarni et al., 2016). All variants and baseline models are trained and evaluated under iden- tical environment settings to ensure fair comparison. Since no public open-source implementation of hDQN was available, we developed a custom version, with architecture details in App. B. Due to in- stabilities encountered during the evaluation phase, we reported the maximum performance achieved during training as a representative metric. Additionally, while we attempted to include Option-critic, existing public implementations do not natively support the DonkeyKong environment. Our efforts to adapt the framework were unsuccessful; therefore, we exclude Option-critic for DonkeyKong. All methods were trained on an RTX 2090 GPU with 12 GB. Logic manager and pretrained options. We provide the logic manager, gating rules in App. H. We pretrain the logic options using HackAtari. Details on how to train individual options are provided in App. G. Note that the logic manager rules and the individual options may not be optimal. Two-Stage Training in H 2 RL. (1) Pretraining: We jointly train the gating module and neural RL policy while the logic manager selects from a fixed set of pretrained options via argmax, yielding H 2 RL (full hybrid policy) and H 2 RL + (neural policy part). (2) Posttraining: We further train H 2 RL + via standard environment interaction to obtain H 2 RL ++ . Unless stated otherwise, H 2 RL uses PPO as the default neural policy. RQ1: H 2 RL with its variants demonstrate a significant leap in performance and outperform all compared baselines. We evaluate the performance of three variants of H 2 RL against SOTA deep RL, hierarchical RL, and neuro-symbolic RL baselines across three challenging Atari environments. The results, summarized in Fig. 3 (and in App. Tab. 13). Most notably, H 2 RL ++ achieves scores 5 Table 1: H 2 RL serves as a universal pretraining substrate for on-policy and off-policy deep RL methods. Shown are the means of the episodic return (averaged over 12 environments, 200 runs per environment) with the std of the pretrained and base policy. Although DQN and PPO achieve high returns in Kangaroo, their learned policies remain misaligned; Details see Sec. 4 (RQ3). H 2 PPO + PPOH 2 DQN + DQNH 2 C51 + C51 Seaquest1802 ±400 3247 ±881 1966 ±903 3259 ±1001 2774 ±1367 4381 ±1988 Kangaroo2754 ±2626 14592 ±491 114665 ±16932 14822 ±1810 8193 ±5928 13854 ±1021 DonkeyKong 87780 ±32786 4536 ±296 6887 ±1765 3205 ±988 183645 ±107378 3393 ±721 Table 2: H 2 RL mitigates policy misalignment via logic-informed pretraining for on-policy and off-policy deep RL methods. The table reports success rates for reaching different floors in Kanga- roo for agents pre-trained using H 2 RL, its base variant, and the logic manager (hReason). Success rates averaged over 5 seeds, with 6 runs per seed. PPODQNC51hReasonH 2 PPOH 2 DQN + H 2 C51 + Floor 20% ±0% 0% ±0% 0% ±0% 100% ±0% 100% ±0% 100% ±0% 100% ±0% Floor 30% ±0% 0% ±0% 0% ±0% 30% ±20% 60% ±10% 100% ±0% 100% ±0% Floor 40% ±0% 0% ±0% 0% ±0% 30% ±10% 50% ±10% 100% ±0% 100% ±0% that are orders of magnitude higher than the baselines, reaching 131,842 ±1,221 in Kangaroo and 216,793 ±125,655 in DonkeyKong. These environments are notorious for misleading agents with early, dense rewards. The fact that H 2 RL maintains its trajectory toward high-level objectives, where other agents plateau, suggests that our logic-informed pretraining effectively boosts the agents’ perfor- mance. Furthermore, the performance jump from the pretraining policy H 2 RL + to the post-training policy H 2 RL ++ validates our two-stage approach, showing that the symbolic scaffolding provides a foundation upon which the neural policy can eventually achieve peak performance that neither purely symbolic nor deep RL methods can reach alone. RQ2: H 2 RL can serve as a universal pretraining substrate for both on-policy and off-policy methods. For off-policy pretraining, we use the logic manager to collect a replay buffer and then train the agent on it. Tab 1 compares the average episodic returns of pretrained methods with their base variants in Seaquest, Kangaroo, and DonkeyKong. Overall, H 2 RL substantially improves episodic returns in games with long-horizon dependencies such as DonkeyKong. In contrast, for simpler games such as Seaquest, we do not observe a clear pretraining gain, which may stem from non-optimal logic- manager design or suboptimal option definitions. We note that, despite achieving high returns in Kangaroo, DQN and PPO still learn misaligned policies as shown in RQ3. RQ3: H 2 RL pretraining can mitigate policy misalignment. We report success rates for reaching different floors in Kangaroo for PPO, DQN, and C51, as well as their H 2 RL-pretrained variants. As shown in Tab. 2, rather than getting stuck in the corner of the base variants (Fig. 1), all H 2 RL- pretrained agents (across on-policy and off-policy) successfully overcome the policy misalignment issue and consistently climb upward, outperforming their base variants. RQ4: The integration of logic guidance and neural flexibility in H 2 RL is crucial for its success. We conduct an ablation study to compare H 2 PPO (the default version of H 2 RL) with the neural- only part (PPO), the pure logic manager (hReason), a hierarchical neural manager (hPPO), and an augmented PPO provided with both pixel and symbolic data (exPPO). We provide exPPO’s archi- tecture in App.C. As shown in Tab. 3, while exPPO and hPPO achieve high scores in Kangaroo (14,247 ±1,085 and 10,601 ±914 , respectively), they got trapped in the short-term reward by attacking the enemies in a corner (indicated by the success rate in Tab. 3). Instead, H 2 PPO successfully avoids reward hacking and proceeds along the intended game path without getting stuck in the corner to col- lect points by attacking enemies. This logic-guided game progression can significantly improve the return, for example, in DonkeyKong, where H 2 PPO reaches 33,657 ±14,578 , outperforming the next- 6 Table 3: The synergy between the logic and neural modules is crucial to H 2 RL’s improved performance. As an ablation study, we compare H 2 PPO (the default H 2 RL) with PPO (neural- only), hPPO (neural manager), hReason (logic manager), and exPPO (PPO with extended inputs) on Seaquest, Kangaroo, and DonkeyKong. The table reports the average episodic returns and success rates for reaching the third floor in Kangaroo. For details, see Sec. 4 (RQ3). H 2 PPOPPOhPPOhReasonexPPO Seaquest2812 ±1477 3247 ±881 1906 ±628 1281 ±963 1848 ±34 Kanga./Succ. 5351 ±4132 / 0.6 ±0.1 14592 ±491 / 010601 ±914 / 02238 ±2140 / 0.3 ±0.2 14247 ±1085 / 0 DonkeyKong33657 ±14578 4536 ±296 418 ±139 905 ±1335 4268 ±249 closest PPO by nearly an order of magnitude. In this environment, the pure logic manager (hReason) and the hierarchical neural manager (hPPO) both fail significantly, suggesting that neither symbolic nor hierarchical structure is sufficient in isolation. Furthermore, the fact that exPPO, which has ac- cess to the same symbolic information as H 2 PPO but lacks our logic-informed pretraining, fails to match H 2 PPO’s performance in Seaquest and DonkeyKong confirms that it is insufficient to simply provide symbolic state information. Instead, the results demonstrate that H 2 RL’s two-stage frame- work, which uses logic to steer learning toward goal-directed behavior, is the critical factor in over- coming long-horizon decision-making challenges that individual components alone cannot solve. H 2 RL PPOhPPOhReason 0 10 2 10 3 10 4 10 5 Episodic Return Game Kangaroo DonkeyKong Figure 4: H 2 RL effectively leverages logic reasoning in continuous action spaces and improves deep agents’ performance. We compare H 2 RL with methods applicable to continuous ac- tion space on the Kangaroo and Don- keyKong tasks in CALE, where H 2 RL consistently outperforms these base- lines. Details see Sec. 4: RQ4. RQ5: H 2 RL effectively leverages logic reasoning in con- tinuous action spaces and improves deep agents’ perfor- mance. We evaluate H 2 RL within the Continuous Atari Learning Environment (CALE) to determine whether the integration of symbolic structures benefits agents in non- discrete domains. The results, illustrated in Fig. 4 (and in Tab. 12), demonstrate that H 2 RL pretraining significantly improves upon the baseline PPO agent across diverse tasks (we compare against baselines that are able to apply to continuous-action settings, namely PPO, hPPO, and hReason). Specifically, in Kangaroo (cont.) environment, H 2 RL achieves a mean score of 84,665 ±49,767 , substan- tially outperforming both the PPO baseline (1,785 ±72 ) and the hierarchical variant, hPPO (19,854 ±18,586 ). This performance gap confirms that logic-informed pretraining provides a decisive advantage in continuous action space, where the structured guidance provided by logic is not confined to discrete settings, rather, it is equally potent for continuous action space. Discussion: The experimental results across RQ1–RQ5 provide a comprehensive validation of H 2 RL’s effectiveness to mitigate policy misalignment and as a universal pretraining framework. By outperforming neural and neuro-symbolic baselines by orders of magnitude in both discrete and continuous action spaces, H 2 RL demonstrates that symbolic scaffolding is a powerful tool for main- taining a trajectory toward high-level objectives. The ablation study further clarifies that this success is not a byproduct of simple data augmentation. Neither providing extended symbolic information to a neural agent nor using a purely hierarchical neural manager could replicate H 2 RL’s performance. This confirms that the logic-informed pretraining is the critical mechanism for overcoming long- horizon challenges. Finally, our extensions to off-policy methods and to continuous action spaces underscore the framework’s algorithmic flexibility. Collectively, these results position H 2 RL not merely as a novel RL algorithm, but as a versatile architectural paradigm that successfully bridges high-level reasoning and low-level control. 7 5 Related Work Neuro-symbolic RL. Neuro-symbolic RL seeks to combine neural function approximation with symbolic reasoning for improved generalization and interpretability.Related efforts in- clude NLRL (Jiang & Luo, 2019), Galois (Cao et al., 2022), ESPL (Guo et al., 2023), and BlendRL (Shindo et al., 2025), however, they focus on synthesizing symbolic or logical policies that map directly to raw actions. Instead of raw actions, hierarchical RL (Sutton et al., 1999; Dietterich, 2000) allows to decompose tasks into subtasks or options (Sutton et al., 1999; Bacon et al., 2017; Vezhnevets et al., 2017). NEXUS (Emunds et al., 2025) and Dylan (Ye et al., 2025) employ (neuro- )symbolic meta-policies to guide such neural options. In contrast, H 2 RL is a pretraining framework with a differentiable logic reasoner, allowing deep RL agents to inherit logic priors through pretrain- ing without the computational burden of symbolic reasoning during test-time inference. Imitation learning. Imitation learning (IL), such as GAIL (Ho & Ermon, 2016) and learning from demonstration, has a long history in machine learning (Bain & Sammut, 1995). While many prior works leverage expert demonstrations (Cheng et al., 2020; Ilhan et al., 2021), most existing methods assume access to a high-quality expert or teacher (Derstroff et al., 2024). In contrast, the Logic Manager in our framework selects among pretrained options and is explicitly non-expert (see Tab.3, hReason). Related work has also explored learning from sparse or budgeted teacher feedback (Ilhan et al., 2021) and learning whom to trust (Nunes & Oliveira, 2003). Several works combine imi- tation and RL by pretraining or regularizing policies with behavioral cloning (BC). For example, Rajeswaran et al. (2017) pretrains a policy via BC before RL, while Goecks et al. (2020) and Huang et al. (2023) incorporate an auxiliary BC loss and sample expert trajectories from a dedicated re- play buffer. In contrast, H 2 RL does not rely on recorded human demonstrations or expert policies. Instead, it injects heuristic guidance into the neural policy through logic-informed pretraining. Shortcut mitigation. To address the fragility of reward-driven agents, current mitigation strate- gies typically frame robustness as one of their goals (Pinto et al., 2017; Farebrother et al., 2018; Delfosse et al., 2025). Existing strategies are either extrinsic methods, such as domain randomiza- tion (Tobin et al., 2017) and image augmentations (Yarats et al., 2021), which increase data diversity to force invariance, or algorithmic solutions utilizing auxiliary objectives for invariant feature em- beddings (Zhang et al., 2021; Bertoin et al., 2022). We diverge from these approaches by using symbolic scaffolding (cf. Bransford et al. (2000)) to create a new training setup. Unlike classical neurosymbolic approaches that suffer from a “latency bottleneck” due to continuous symbolic rea- soning (Bhuyan et al., 2024), H 2 RL directly embeds a structural inductive bias into the neural policy while retaining the efficiency of a neural approach. Exploration. Effective exploration in deep RL typically relies on either stochastic noise or intrin- sic motivation. Foundational methods like ε-greedy (Mnih et al., 2015) and entropy maximiza- tion (Haarnoja et al., 2018) introduce random perturbations to ensure coverage, but they often result in unstructured and inefficient behavior in complex environments. To drive more directed discovery, sophisticated approaches augment the reward signal with “novelty bonuses”. This includes count- based methods (Bellemare et al., 2016), curiosity-driven prediction errors (Pathak et al., 2017), and Random Network Distillation (Burda et al., 2019). In contrast, H 2 RL approaches exploration as a structural side effect rather than an explicit objective. By employing a hierarchical structure gov- erned by a logical meta-policy, the resulting policy benefits from the structured guidance during the logic-informed pretraining phase, without requiring complex intrinsic reward signals. 6 Conclusions We proposed H 2 RL, a novel hierarchical neuro-symbolic reinforcement learning framework that leverages logic-informed pretraining to steer deep RL policies away from misalignment. Empiri- cally, we demonstrate that H 2 RL significantly enhances deep RL agents’ performance on challeng- ing, long-horizon tasks while requiring no logical reasoning at inference time. Furthermore, H 2 RL can serve as a universal pretraining substrate for both on-policy and off-policy methods, providing a 8 pathway for integrating logical reasoning into reinforcement learning in continuous action spaces. In future work, we aim to integrate H 2 RL into real-world robotic systems, where structured priors and safety-aware reasoning are critical. Additionally, we plan to scale the framework to more complex decision-making environments involving high-dimensional observations and adaptive, multi-level reasoning mechanisms. 7 Acknowledgements This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Founda- tion) under Germany ́s Excellence Strategy – EXC-3066, “The Adaptive Mind”, and EXC-3057, “Reasonable AI”. It was also funded by the German Federal Ministry of Education and Research, the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the “SenPai: XReLeaS” project. The authors gratefully acknowledge the computing time provided to them on the high-performance computer Lichtenberg at the NHR Center NHR4CES@TUDa. This is funded by the German Federal Ministry of Education and Research (BMBF) and the Hessian Ministry of Science and Research, Art and Culture (HMWK). This project has been supported by a hardware donation by NVIDIA through the Academic Grant Program. References Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2017. Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, 1995. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research (JAIR), 2013. Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2016. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2017. David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! saliency-guided q-networks for generalization in visual reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Bikram Pratim Bhuyan, Amar Ramdane-Cherif, Ravi Tomar, and T. P. Singh. Neuro-symbolic artificial intelligence: a survey. Neural Computing and Applications, 36(21), 2024. John D Bransford, Ann L Brown, Rodney R Cocking, et al. How People Learn: Brain, Mind, Expe- rience, and School: Expanded Edition (2000), volume 11. Washington, DC: National Academy Press, 2000. Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. Galois: boosting deep reinforcement learning via generalizable logic synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 9 Stephanie C. Y. Chan, Samuel Fishman, Anoop Korattikara, John F. Canny, and Sergio Guadar- rama. Measuring the reliability of reinforcement learning algorithms. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of mul- tiple oracles. In Advances in Neural Information Processing Systems (NeurIPS), 2020. Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In Proceedings of the International Conference on Machine Learning (ICML), 2017. Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and ex- plainable logical policies via neurally guided symbolic abstraction. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2023. Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Object-centric atari 2600 reinforcement learning environments. Reinforcement Learning Journal (RLJ), 2024. Quentin Delfosse, Jannis Blüml, Fabian Tatai, Théo Vincent, Bjarne Gregori, Elisabeth Dillies, Jan Peters, Constantin A. Rothkopf, and Kristian Kersting. Deep reinforcement learning agents are not even close to human intelligence. arXiv preprint arXiv:2505.21731, 2025. Giovanni Pio Delvecchio, Lorenzo Molfetta, and Gianluca Moro. Neuro-symbolic artificial intelli- gence: A task-directed survey in the black-box models era. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2025. Survey Track. Cedric Derstroff, Mattia Cerrato, Jannis Brugger, Jan Peters, and Stefan Kramer. Peer learning: Learning complex policies in groups from scratch via action recommendations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024. Thomas G. Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function De- composition. Journal of Artificial Intelligence Research (JAIR), 13:227–303, 2000. S.E. Dreyfus and Hubert Dreyfus. A five-stage model of the mental activities involved in directed skill acquisition. Distribution, 1980. Raban Emunds, Jannis Blüml, Quentin Delfosse, and Kristian Kersting. Interpretable reinforcement learning via meta-policy guidance. In The 18th European Workshop on Reinforcement Learning (EWRL), 2025. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese, 198 (27), 2021. Jesse Farebrother and Pablo Samuel Castro. Cale: Continuous arcade learning environment. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in DQN. arXiv preprint arXiv:1810.00123, 2018. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 2020. Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, John Valasek, and Nicholas R. Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020. 10 C. A. E. Goodhart. Monetary Theory and Practice: The UK Experience. Macmillan Education UK, 1984. ISBN 978-1-349-17295-5. Nils Grandien, Quentin Delfosse, and Kristian Kersting. Interpretable end-to-end neurosymbolic reinforcement learning agents. arXiv preprint arXiv:2410.14371, 2024. Jiaming Guo, Rui Zhang, Shaohui Peng, Qi Yi, Xing Hu, Ruizhi Chen, Zidong Du, Xishan Zhang, Ling Li, Qi Guo, and Yunji Chen. Efficient symbolic policy learning with differentiable symbolic expression. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), 2018. Katherine L. Hermann, Hossein Mobahi, Thomas Fel, and Michael Curtis Mozer. On the founda- tions of shortcut learning. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2024. Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and JoÃG , o GM AraÚjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research (JMLR), 2022. Tao Huang, Kai Chen, Bin Li, Yun-Hui Liu, and Qi Dou. Guided reinforcement learning with efficient exploration for task automation of surgical robot. arXiv preprint arXiv:2302.09772, 2023. Ercument Ilhan, Jeremy Gow, and Diego Perez. Student-initiated action advising via advice novelty. IEEE Transactions on Games (T-G), 2021. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019. Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In Proceedings of the Interna- tional Conference on Machine Learning (ICML), 2019. Jack Koch, Lauro Langosco, Jacob Pfau, James Le, and Lee Sharkey. Objective robustness in deep reinforcement learning. arXiv preprint arXiv:2105.14111, 2021. Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems (NeurIPS), 2016. Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro- symbolic reinforcement learning with textual explanations. In Proceedings of the International Conference on Machine Learning (ICML), 2024. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015. 11 Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning (ICML), 1999. Luís Nunes and Eugénio Oliveira. Exchanging advice and learning to trust. In Cooperative Infor- mation Agents VII, Lecture Notes in Computer Science, 2003. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the International Conference on Machine Learning (ICML), 2017. Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the International Conference on Machine Learning (ICML), 2017. Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017. Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall Press, 1995. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Hikaru Shindo, Masaaki Nishino, and Akihiro Yamamoto. Differentiable inductive logic program- ming for structured examples. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021. Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. α ilp: thinking visual scenes as differentiable logic programs. Machine Learning (MLJ), 2023. Hikaru Shindo, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Blendrl: A frame- work for merging symbolic and neural policy learning. In Proceedings of the International Con- ference on Learning Representations (ICLR), 2025. Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defin- ing and characterizing reward gaming. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Frame- work for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 112(1-2):181– 211, 1999. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal Networks for Hierarchical Reinforcement Learning. In International Conference on Machine Learning (ICML), p. 3540–3549, 2017. Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. Zihan Ye, Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Neural meta-symbolic reasoning and learning. arXiv preprint arXiv:2211.11650, 2022. 12 Zihan Ye, Oleg Arenz, and Kristian Kersting. Learning from less: Guiding deep reinforcement learn- ing with differentiable symbolic planning. RLC 2025 Workshop on Programmatic Reinforcement Learning, 2025. Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. 13 Supplementary Materials The following content was not necessarily subject to peer review. Broader Impact Statement By integrating symbolic structure with deep reinforcement learning, Hybrid Hierarchical RL (H 2 RL) seeks to improve alignment and long-horizon performance by discouraging short-term reward ex- ploitation. However, badly designed or biased logical rules may cause the agent to rigidly follow flawed paths, producing behavior that is internally consistent yet unsatisfactory and difficult for neural refinement to override. A First-Order Logic In first-order logic, a term can be a constant, a variable, or a function term constructed using a function symbol. We denote an n-ary predicate p as p/(n, [dt 1 ,..., dt n ]), where dt i represents the data type of the i-th argument. An atom is an expression of the form p(t 1 ,..., t n ), where p is an n-ary predicate symbol and t 1 ,..., t n are terms. If the atom contains no variables, it is referred to as a ground atom, or simply a fact. A literal is either an atom or the negation of an atom. We refer to an atom as a positive literal, and its negation as a negative literal. A clause is defined as a finite disjunction (∨) of literals. When a clause contains no variables, it is called a ground clause. A definite clause is a special case: a clause that contains exactly one positive literal. Formally, if A,B 1 ,...,B n are atoms, then the expression A∨¬B 1 ∨...∨¬B n constitutes a definite clause. We write definite clauses in the form of A :- B 1 ,...,B n . where A is the head of the clause, and the setB 1 ,...,B n is referred to as the body. For simplicity, we refer to definite clauses as clauses throughout this paper. The forward- chaining inference is a type of inference in first-order logic to compute logical entailment (Russell & Norvig, 1995). B Architecture of the Hierarchical DQN (hDQN) We describe here the hierarchical deep Q-network (hDQN) custom implementation. The agent follows the standard two-level hDQN structure: a meta-controller that selects a temporally-extended subgoal g, and a controller that selects primitive actions a conditioned on both the environment state and the current goal. B.1 Meta-Controller Network The meta-controller estimates the goal-values Q θ m (s,g)∈ R G ,(4) where G is the number of subgoals (varies by game: G Seaquest = 8, G Kangaroo = 6, G DonkeyKong = 10). Input/Output. • Input: stacked grayscale frames s∈ R 4×84×84 (the network expects 4 channels). • Output: goal Q-values Q θ m (s,·)∈ R G . • Normalization: the forward pass normalizes the 8-bit gray-scale input image to a [0, 1] 84×84 float array. Architecture. The meta-controller is a DQN-style CNN. The architecture is shown in Tab. 4 14 Table 4: Meta-Controller Architecture LayerConfiguration ConvolutionalConv2d(4, 32, 8, stride=4) ReLU ConvolutionalConv2d(32, 64, 4, stride=2) ReLU ConvolutionalConv2d(64, 64, 3, stride=1) ReLU Flatten Fully ConnectedLinear(64· 7· 7, 512) ReLU Fully ConnectedLinear(512, G) B.2 Controller Network The controller estimates action-values conditioned on the goal: Q θ c (s,g,a)∈ R |A| .(5) Input/Output. • Input: a goal-conditioned image tensor in R 5×84×84 (the network expects 5 channels). • Output: action Q-values Q θ c (s,g,·)∈ R |A| , where|A| = number of actions. Architecture. The controller matches the meta-controller CNN except for the first convolution input channels (cf. Tab. 5). Table 5: Controller Network Architecture LayerConfiguration ConvolutionalConv2d(5, 32, 8, stride=4) ReLU ConvolutionalConv2d(32, 64, 4, stride=2) ReLU ConvolutionalConv2d(64, 64, 3, stride=1) ReLU Flatten Fully ConnectedLinear(64· 7· 7, 512) ReLU Fully ConnectedLinear(512,|A|) C Visual Encoder Architecture of exPPO. In this section, we introduce the visual encoder architecture of exPPO. The Actor, Critic architecture stays the same with PPO (See Tabs. 6 and 7). N is an environment-dependent constant that fixes the flattened size. In our implementation, N Kangaroo = 49, yielding a linear layer with input size of 12,544. N Seaquest = 43, and N DonkeyKong = 18. 15 Table 6: Visual Encoder Architecture (Pixel Stream) LayerConfiguration ConvolutionalConv2d(4, 32, 8, stride=4) ReLU ConvolutionalConv2d(32, 64, 4, stride=2) ReLU ConvolutionalConv2d(64, 64, 3, stride=1) ReLU Flatten Fully ConnectedLinear(64· 7· 7, 512) ReLU Table 7: Object Encoder Architecture (Symbolic Stream) LayerConfiguration Fully ConnectedLinear(4, 128) ReLU Fully ConnectedLinear(128, 64) ReLU Flatten Fully ConnectedLinear(64· 4· N , 32) ReLU C.1 Fusion Trunk (Shared Representation) The model concatenates the two modality embeddings and maps them to a shared 32-dimensional hidden state: Table 8: Fusion Trunk Architecture (Pixel + Symbolic Fusion) LayerConfiguration ConcatenationConcat(y pixel ∈R 512 , y symb. ∈R 32 ) Fully ConnectedLinear(544, 32) ReLU D Training Parameters The options and models were all trained using the same hyperparameters listed in Tab. 9 because of time- and hardware constraints. The models were all trained on an Nvidia RTX 2090 with 12 GB. In our experiments, we use the CleanRL library (Huang et al., 2022) for the PPO, C51, and DQN implementations. All neural networks that use the pixel state have the same layer configuration described in Tab 10. Furthermore, a simple linear layer is added to the critic or the actor, depending on the algorithm. For the object-centric critic, the layer configuration is described in Tab 11. E Making Symbolic Reasoning Differentiable We now describe each step in detail for making the symbolic planner differentiable. 16 Table 9: Hyperparameters used when training the models. ParameterValue Gamma0.99 Learning rate0.00025 Blending entropy coefficient0.01 Entropy coefficient0.01 Clipping coefficient0.1 Maximum gradient norm0.5 Number of environments128 Number of steps128 Total timesteps40,000,000 Table 10: Pixel State Backbone Architecture LayerConfiguration ConvolutionalConv2d(4, 32, 8, stride=4) ReLU ConvolutionalConv2d(32, 64, 8, stride=2) ReLU ConvolutionalConv2d(64, 64, 3, stride=1) ReLU Flatten Fully ConnectedLinear(64· 7· 7, 512) ReLU (Step 1) Encoding Logic Programs as Tensors. To enable differentiable forward reasoning, each meta-rule is converted to a tensor representation. Each meta-rule C i ∈ C is encoded as a tensor I i ∈ N G×S×L , where S denotes the maximum number of substitutions for existentially quantified variables in the rule set, and L is the maximum number of atoms in the body of any rule. For instance, I i [j,k,l] stores the index of the l-th subgoal in the body of rule C i used to derive the j-th fact under the k-th substitution. (Step 2) Weighting and Selecting Meta-Rules. We construct the reasoning function by assigning weights that determine how multiple meta-rules are combined. (i) We fix the size of the target meta- program to be M , meaning the final program will consist of M meta-rules selected from a total of C candidates inC. (i) To enable soft selection, we define a weight matrix W = [w 1 ,..., w M ], where each w i ∈ R C assigns a real-valued weight. (i) We then apply a softmax to each w i to obtain a probability distribution over the C candidates, allowing the model to softly combine multiple meta- rules. (Step 3) Perform Differentiable Inference. Starting from a single application of the weighted meta-rules, we iteratively propagate inferred facts across T reasoning steps. We compute the valuation of body atoms for every grounded instance of a meta-rule C i ∈ C. This is achieved by first gathering the current truth values from the valuation vector v (t) using an index tensor Ii, and then applying a multiplicative aggregation across subgoals: b (t) i,j,k = L Y l=1 gather(v (t) , Ii)[j,k,l],(6) 17 Table 11: Object-Centric Backbone Architecture LayerConfiguration Fully ConnectedLinear(N in , 512) ReLU Fully ConnectedLinear(512, 120) ReLU Fully ConnectedLinear(120, N out ) where the gather operator maps valuation scores to indexed body atoms: gather(x, Y)[j,k,l] = x[Y[j,k,l]].(7) The resulting value b (t) i,j,k ∈ [0, 1] reflects the conjunction of subgoal valuations under the k-th substitution of existential variables, used to derive the j-th candidate fact from the i-th meta-rule. Logical conjunction is implemented via element-wise product, modeling the "and" over the rule body. To integrate the effects of multiple groundings of a meta-rule C i , we apply a smooth approximation of logical or across all possible substitutions. Specifically, we compute the aggregated valuation c (t) i,j ∈ [0, 1] as: c (t) i,j = softor γ (b (t) i,j,1 ,...,b (t) i,j,S ),(8) where softor γ denotes a differentiable relaxation of disjunction. This operator is defined as: softor γ (x 1 ,...,x n ) = γ log n X i=1 exp(x i /γ),(9) with temperature parameter γ > 0 controlling the smoothness of the approximation. This formula- tion closely resembles a softmax over valuations and serves as a continuous surrogate for the logical max, following the log-sum-exp technique commonly used in differentiable reasoning (Cuturi & Blondel, 2017). (i) Weighted Aggregation Across Meta-Rules. We compute a weighted combination of meta rules using the learned soft selections: h (t) j,m = C X i=1 w m,i · c (t) i,j,(10) where h (t) j,m ∈ [0, 1] represents the intermediate result for the j-th fact contributed by the m-th slot. Here, w m,i is the softmax-normalized score over the i-th meta-rule: w ∗ m,i = exp(w m,i ) P i ′ exp(w m,i ′ ) , w m,i = w m [i]. Finally, we consolidate the outputs of the M softly selected rule components using a smooth dis- junction: r (t) j = softor γ (h (t) j,1 ,...,h (t) j,M ),(11) which yields the t-step valuation for fact j. This mechanism allows the model to integrate M soft rule compositions from a larger pool of C candidates in a fully differentiable way. 18 Table 12: H 2 RL outperforms other compared baselines in the Continuous Atari learning envi- ronment (CALE). Reward comparison of H 2 RL against other baseline methods which can work in the continuous action space versions of Kangaroo and DonkeyKong. H 2 RLPPOhPPOhReason Kangaroo (cont.)84665 ±49767 1785 ±72 19854 ±18586 557 ±167 DonkeyKong (cont.)10818 ±7431 3836 ±530 991.0 ±446 542 ±975 (i) Iterative Forward Reasoning. We iteratively apply the forward reasoning procedure for T steps. At each step t, we update the valuation of each fact j by softly merging its newly inferred value r (t) j with its previous valuation: v (t+1) j = softor γ (r (t) j ,v (t) j ).(12) This recursive update mechanism approximates logical entailment in a differentiable form, enabling the model to perform T -step reasoning over the evolving fact valuations. The whole reasoning computation Eqs. 6–12 can be implemented using efficient tensor operations. F Reward Comparison We provide a comparison of rewards across different methods pre-trained with the H 2 RL architec- ture and its counterpart in Tab. 1. Table 13: H 2 RL outperforms other compared baselines in the classical Atari Learning en- vironment. Although PPO and hPPO achieve relatively high scores in Kangaroo, their improve- ments stem from policy misalignment. We evaluate H 2 RL against other methods in Kangaroo, Seaquest, and DonkeyKong. ∗ Due to instability during evaluation, we report the maximum perfor- mance achieved by hDQN during training as a representative metric. H 2 RL ++ H 2 RL + H 2 RLPPONUDGE Seaquest4759 ±1004 1802 ±400 2812 ±1477 3247 ±881 63 ±18 Kangaroo131842 ±1221 2754 ±2626 5148 ±3950 14592 ±491 404 ±230 DonkeyKong216793 ±125655 87780 ±32786 33690 ±14565 4536 ±296 21 ±43 BlendRLOption-critichDQNhPPOhReason Seaquest117 ±62 38 ±19 112 ∗ 1906 ±628 1186 ±930 Kangaroo1482 ±1343 61 ±89 322 ∗ 10601 ±915 1701 ±1061 DonkeyKong85 ±82 —434 ∗ 418 ±139 860 ±1301 G Options pretraining The options for the HHRL framework were created by training multiple PPO agents in modified Atari Environments, each with its own custom reward function. The reward function, modifications, and the number of options were obtained from domain experts. More details about the modification can be found in Tab. 14, which modification was used to train certain options in Tab. 15, and the reward functions in Tab. 16. 19 Table 14: A list of modifications for each environment. Environment ModificationDescription Seaquestrandom_startrandomize the starting position of the player random_surfacingset the player to a random position when reaching the surface random_laneset the starting position of the player in one of the 4 lanes where enemies and divers spawn disable_underwater_enemiesprevent all underwater enemies from appearing in the game disable_enemiesprevent all enemies from appearing in the game unlimited_collected_diveralways set the amount of collected divers to 1 empty_divers_automaticallyreset the amount of collected divers to 0 if 5 or more divers were collected enable_surface_enemiesenable the surface submarine unlimited_oxygenprevent the oxygen bar from decreasing Kangaroodisable_coconutkeep falling coconuts on the top of the screen disable_thrown_coconutremove coconuts thrown by the monkeys from the game disable_high_thrown_coconut thrown coconuts are always thrown along the ground disable_low_thrown_coconut thrown coconuts are always thrown in the air disable_monkeysremove enemy monkeys from the game plattform_checkpointsplace the player on the last plattform they stood on after losing a life randomize_kangaroo_position randomize the starting position of the player teleport_killset the player to a random position after defeating a monkey always_falling_coconutdirectly spawn another falling coconut above the player after another one has already fallen below the player change_level_0always set the current level to 0 Donkey Kong skip_start_gameremove the need to press space to start the game spawn_barrel_in_random_lane randomize the starting position of barrels remove_near_barrelremove a barrel if the player is near it spawn_barrel_near_playerplace one barrel in front of the player at the start and after the player has jumped over it no_hammerremove the hammer from the game shorten_timelimitshorten the timelimit to reach the princess to half the original amount random_startrandomize the starting position of the player end_hammerend the episode after the hammer has been used spawn_near_hammerset the starting position of the player randomly near the hammer position change_level_0always set the current level to 0 Bankheistrandom_city_startrandomize the city the player starts in random_startrandomize the starting position of the player two_police_carsalways have two police cars and one bank on the field delete_near_policeremove a police car if it is near the player 20 Table 15: A list of modifications used when training options in each environment. EnvironmentOptionApplied modifications Seaquestget_air, deliver_diverrandom_start disable_underwater_enemies unlimited_collected_diver empty_divers_automatically random_surfacing enable_surface_enemies get_diverrandom_start unlimited_oxygen emtpy_divers_automatically shoot_enemy/wait_in_linerandom_lane unlimited_oxygen Kangarooascenddisable_coconut disable_thrown_coconut plattform_checkpoints change_level_0 deal_with_enemiesrandomize_kangaroo_position disable_coconut teleport_kill change_level_0 avoid_thrown_coconutsrandomize_kangaroo_position disable_coconut change_level_0 avoid_low_thrown_coconutsrandomize_kangaroo_position disable_coconut disable_high_thrown_coconut change_level_0 avoid_high_thrown_coconutsrandomize_kangaroo_position disable_coconut disable_low_thrown_coconut change_level_0 Donkey Kongclimbremove_near_barrel skip_start_game change_level_0 use_hammerrandom_start spawn_barrel_near_player no_hammer shorten_timelimit skip_start_game change_level_0 jump_barrelend_hammer spawn_near_hammer skip_start_game change_level_0 21 Table 16: A list of reward functions used when training options in each environment. Environment OptionReward function Seaquestget_air, deliver_diver+1.5 when player is at the surface get_diver+1 when player collects a diver shoot_enemy/wait_in_line+1 when the player shoots an enemy -10 when the player moves up or down Kangarooascend+20 when player reaches certain positions deal_with_enemies+1 when the player obtains a score from punching an enemy -1 when the player loses a life avoid_thrown_coconuts/ avoid_high_thrown_coconuts +1 once when a thrown coconut is on the left side of the player -1 reward when the player moves left or right avoid_low_thrown_coconuts+1 when player has jumped over a thrown coconut -1 when the player moves left or right Donkey Kong climb+20 when player reaches certain positions use_hammer10 when player grabs the hammer +1 when player obtains score points with the ham- mer jump_barrel+1 when player jumped over a barrel 22 H Logic Rules Seaquest Logic manager rules: get_air(X) :- oxygen_low(B), collected_at_least_one_diver(X), not_below_enemy(X). deliver_diver(X) :- full_divers(X), oxygen_high(B). shoot_enemy(X) :- same_depth_enemy(P,E), same_depth_diver(P,D), not_full_divers(X). wait_in_lane(X) :- below_enemy(X), not_same_depth_diver(X). wait_in_lane(X) :- above_enemy(X), not_same_depth_diver(X), oxygen_high(B). wait_in_lane(X) :- no_visible_divers(D), oxygen_high(B), not_full_divers(X). get_diver(X) :- visible_diver(D), not_same_depth_enemy(X), not_full_divers(X), oxygen_high(B). Logic gating rule: logic_agent(X) :- oxygen_low(B). logic_agent(X) :- full_divers(X). logic_agent(X) :- visible_diver(D). neural_agent(X) :- oxygen_high(B), no_visible_divers(X), not_full_divers(X). Kangaroo Logic manager rules (discrete): ascend(X) :- nothing_around(X). ascend(X) :- on_ladder(P,L). deal_with_enemy(X) :- close_by_monkey(P,M), not_below_fallingcoconut(P,FC). avoid_thrown_coconuts(X) :- close_by_throwncoconut(P,TC), not_on_ladder(X). Logic manager rules (continuous): ascend(X) :- nothing_around(X). ascend(X) :- on_ladder(P,L). deal_with_enemy(X) :- close_by_monkey(P,M), not_below_fallingcoconut(P,FC). avoid_low_thrown_coconuts(X) :- close_by_low_throwncoconut(P,TC), not_on_ladder(X). avoid_high_thrown_coconuts(X) :- close_by_high_throwncoconut(P,TC), not_on_ladder(X). Logic gating rule: logic_agent(X) :- low_on_time(T). logic_agent(X) :- nothing_around(X). neural_agent(X) :- anything_around(X), plenty_time(T). Donkey Kong Logic manager rules: climb(X) :- nothing_around(X). use_hammer(X) :- hammer_on_level(P,H). jump_barrel(X) :- close_by_barrel(P,B). Logic gating rule: logic_agent(X) :- hammer_on_level(P,H). logic_agent(X) :- nothing_around(X). neural_agent(X) :- close_by_barrel(P,B). 23