Paper deep dive
PRISM: Parallel Reward Integration with Symmetry for MORL
Finn van der Knaap, Kejiang Qian, Zheng Xu, Fengxiang He
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 10:21:21 PM
Summary
The paper introduces PRISM, a Parallel Reward Integration with Symmetry algorithm for Multi-Objective Reinforcement Learning (MORL) that addresses heterogeneous reward sparsity. It utilizes ReSymNet to align sparse and dense reward frequencies and SymReg to enforce reflectional equivariance, thereby reducing hypothesis complexity and improving sample efficiency and Pareto coverage on MuJoCo benchmarks.
Entities (7)
Relation Signals (7)
SymReg → enforces → Reflectional Equivariance
confidence 95% · SymReg, a reflectional equivariance regulariser that enforces agent mirroring
PRISM → uses → SymReg
confidence 95% · We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring
PRISM → uses → ReSymNet
confidence 95% · PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives
ReSymNet → addresses → reward sparsity
confidence 90% · To address the challenge of heterogeneous reward objectives, PRISM first transforms sparse rewards into dense, per-step signals.
PRISM → evaluatedon → MuJoCo
confidence 90% · Across MuJoCo benchmarks, PRISM consistently outperforms
PRISM → outperforms → Sparse-reward baseline
confidence 90% · PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards
CAPQL → backboneof → PRISM
confidence 85% · using Concave-Augmented Pareto Q-learning (CAPQL) ... as the backbone for PRISM.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100\% over the baseline and up to 32\% over the oracle. The code is at \href{this https URL}{this https URL}.
Tags
Links
- Source: https://arxiv.org/abs/2602.18277v1
- Canonical: https://arxiv.org/abs/2602.18277v1
Trouble viewing inline? Open PDF directly →
Full Text
117,496 characters extracted from source content.
Expand or collapse full text
PRISM: Parallel Reward Integration with Symmetry for MORL Finn van der Knaap Kejiang Qian Zheng Xu Fengxiang He Abstract This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100% over the baseline and up to 32% over the oracle. The code is at https://github.com/EVIEHub/PRISM. heterogeneous multi-objective reinforcement learning, reflection equivariance, reward shaping 1 Introduction Reinforcement Learning (RL) has been approaching human-level capabilities in many decision-making tasks, such as playing Go (Silver et al., 2017), autonomous vehicles (Kiran et al., 2021), robotics (Tang et al., 2025a), and finance (Hambly et al., 2023). Multi-Objective Reinforcement Learning (MORL) extends this framework to handle multiple reward channels simultaneously, allowing agents to balance competing objectives efficiently (Liu et al., 2014; Hayes et al., 2022). For example, a self-driving car must constantly balance multiple goals, such as minimising travel time while maximising passenger safety and energy efficiency. Prioritising speed would compromise the safety objectives, introducing the need for flexible and robust policies that can optimise across diverse and sometimes conflicting goals. Figure 1: Reflectional symmetry in a two-legged agent. The left panel shows a transition from state s to s′s under action a, whereas the right panel shows the reflected transition, where states and actions are transformed by LgL_g and KgK_g, respectively. This paper considers an important, yet premature, setting where reward channels exhibit considerable heterogeneity in facets such as sparsity. Dense objectives can overshadow their sparse and long-horizon counterparts, steering policies toward short-term gains, while neglecting the objectives that are harder to optimise but potentially more important. A straightforward approach is to employ reward shaping methods to align the reward channels. However, existing algorithms, such as intrinsic curiosity (Pathak et al., 2017; Aubret et al., 2019) and attention-based exploration (Wei et al., 2025), are developed for single-objective cases and have significant deficiencies: separately shaping individual objectives can distort the Pareto front and structures between objectives. This highlights a critical gap in the literature: MORL requires a reward shaping method that enables efficient integration of the parallel but heterogeneous reward signals, leveraging their intrinsic structure, in order to improve sample efficiency. To this end, we propose Parallel Reward Integration with Symmetry for MORL (PRISM), a method that structurally shapes the reward channels and leverages the reflectional symmetry in agents in heterogeneous MORL problems. We design a Reward Symmetry Network (ReSymNet) that predicts the reward given the state of the system and any available performance indicators (e.g., dense rewards in this work). The available sparse rewards are used as supervised targets. In ReSymNet, residual blocks are employed to approximate the ‘scaled opportunity value’, which has been proven to help accelerate training, decrease the approximation error, while maintaining the optimal solution of the native reward signals (Laud, 2004). After proper training, our ReSymNet can be a plug-and-play technique, compatible with any off-the-shelf MORL algorithm in an iterative refinement cycle, where the agent observes the shaped rewards to improve its policy and the reward model observes better trajectories from the updated policy to improve the approximated reward function. To exploit the structural information across reward signals, we design a Symmetry Regulariser (SymReg) to enforce reflectional equivariance of the objectives, which provably reduces the hypothesis complexity. Intuitively, incorporating reflectional symmetry as an inductive bias allows an agent to generalise experience from one situation to its mirrored counterpart. The complementary components of PRISM synergise as follows. Heterogeneous reward structures cause asymmetric policy learning that violates the agent’s physical symmetry: when dense objectives provide immediate gradients while sparse objectives only signal at the end of an episode, the policy may overfit to the denser objectives in specific states, failing to respect reflectional symmetry. ReSymNet eliminates temporal heterogeneity by aligning objectives to the same frequency, whereas SymReg enforces reflectional symmetry by preventing asymmetric learning dynamics. We prove that PRISM constrains the policy search into a subspace of reflection-equivariant policies. This subspace is a projection of the original policy space, induced by the reflectional symmetry operator, provably of reduced hypothesis complexity, measured by covering number (Zhou, 2002) and Rademacher complexity (Bartlett and Mendelson, 2002). This reduced complexity is further translated to improved generalisation guarantees. In practice, this means that by encouraging policies to respect natural symmetries, the agent searches over a smaller, more structured hypothesis space, reducing overfitting and improving sample efficiency. We conduct extensive experiments on the MuJoCo MORL environments (Todorov et al., 2012; Felten et al., 2023), using Concave-Augmented Pareto Q-learning (CAPQL) (Lu et al., 2023) as the backbone for PRISM. Sparse rewards are constructed by releasing cumulative rewards at the end of an episode. PRISM achieves hypervolume gains of over 100% against the baseline operating on sparse signals, and even up to 32% over the oracle (full dense rewards), also indicating a substantially improved Pareto front coverage. These gains are echoed in distributional metrics, confirming that PRISM learns a set of policies that are also better balanced and more robust. Comprehensive ablation studies further confirm that both ReSymNet and SymReg are critical. The code is at https://github.com/EVIEHub/PRISM. 2 Related Work Multi-Objective Reinforcement Learning. MORL algorithms typically fall into three categories: (1) single-policy methods that optimise user-specified scalarisations (Moffaert et al., 2013; Lu et al., 2023; Hayes et al., 2022); (2) multi-policy methods that approximate the Pareto front by solving multiple scalarisations or training policies in parallel (Roijers et al., 2015; Van Moffaert and Nowé, 2014; Reymond and Nowé, 2019; Lautenbacher et al., 2025); and (3) meta-policy and single universal policy methods that learn adaptable policies given some preferences (Chen et al., 2019; Yang et al., 2019; Basaklar et al., 2023; Mu et al., 2025; Liu et al., 2025). While these works have advanced Pareto-optimal learning, less attention has been given to heterogeneity in reward structures. Reward Shaping. A large volume of literature tackles sparse rewards through reward shaping. Potential-based shaping (Ng et al., 1999) ensures policy invariance but requires hand-crafted potentials. However, this method’s reliance on a manually designed potential function proved limiting. Intrinsic motivation methods reward novelty or exploration (Pathak et al., 2017; Burda et al., 2019), while self-supervised methods predict extrinsic returns from trajectories (Memarian et al., 2021; Devidze et al., 2022; Holmes and Chi, 2025). Recent advances utilise statistical decomposition to address sparsity (Gangwani et al., 2020; Ren et al., 2022), or capture complex reward dependencies using transformers (Tang et al., 2024, 2025b). These approaches improve sample efficiency in single-objective RL, but do not extend naturally to MORL, where heterogeneous sparsity and scale can distort learning dynamics and Pareto-optimal trade-offs. Reflectional Equivariance. To incorporate reflectional symmetry, a possible method is data augmentation, which adds mirrored transitions to the replay buffer but doesn’t guarantee a symmetric policy and increases data processing costs (Lin et al., 2020). Mondal et al. (2022) propose latent space learning that encourages a symmetric representation through specialised loss functions. Another line of research focuses on equivariant neural networks (van der Pol et al., 2020; Mondal et al., 2020; Wang et al., 2021). For example, Wang et al. (2022) design a stronger inductive bias via architecture-level symmetry, which hard-codes equivariance into the model for instantaneous generalisation. However, Park et al. (2025) show that strictly equivariant architectures can be too rigid for tasks where symmetries are approximate rather than perfect. Building on this insight, our framework helps overcome the limitations of strictly equivariant architectures through tunable flexibility whilst being model-agnostic. 3 Preliminaries Multi-Objective Markov Decision Process. Formally, we define an MORL problem via the Multi-Objective Markov Decision Process (MOMDP) model, as a tuple ℳ=(,,,,γ)M=(S,A,P, r,γ): an agent at state s from a finite or continuous state space S, taking action a from a finite or continuous action space A, moves herself according to a transition probability function :×′→[0,1]P:S×A×S →[0,1], also denoted as P(s′|s,a)P(s |s,a). The agent receives a reward via an L-dimensional vector-valued reward function :×→ℝL r:S×A ^L, where L is the reward channel number, which decays by a discount factor γ∈[0,1)γ∈[0,1). The goal in MORL is to find a policy π:→π:S that optimises the expected cumulative vector return, defined as (π)=π[∑t=0∞γtt] J(π)=E_π [ _t=0^∞γ^t r_t ]. This paper addresses episodic tasks, where each interaction sequence has a finite horizon and concludes when the agent reaches a terminal state, at which point the environment is reset. Episodes τi _i are i.i.d. draws from the behaviour distribution D, which describes the probability of observing different possible trajectories under the policy being followed. Reward Sparsity. Reward sparsity can be modelled as releasing the cumulative reward accumulated since the last non-zero reward with probability prelp_rel at each timestep. When prel=0p_rel=0, this reduces to the most extreme case: the agent receives rewards from dense channels =d1,d2,…,dDDC=\d_1,d_2,…,d_D\ with observable rewards rtdir_t^d_i at every timestep, but the sparse channel is revealed only once at the end of the episode as RTsp=∑t=1TrtspR_T^sp= _t=1^Tr_t^sp. The central challenge is to recover instantaneous sparse rewards rtspr_t^sp for each (st,at)(s_t,a_t) using only the cumulative observation RTspR_T^sp and correlations with dense channels. Formally, given a trajectory τ=(s1,a1),…,(sT,aT)τ=\(s_1,a_1),…,(s_T,a_T)\ with cumulative sparse reward Rsp(τ)R^sp(τ), the task is to infer sp=[r1sp,…,rTsp]⊤ r^sp=[r_1^sp,…,r_T^sp] , where rtspr_t^sp is the sparse reward at timestep t, such that ∑t=1Trtsp≈Rsp(τ) _t=1^Tr_t^sp≈ R^sp(τ). For prel>0p_rel>0, an episode decomposes into sub-trajectories where the same formulation applies. Generalisability and Hypothesis Complexity. A generalisation gap, at the episodic level, characterises the generalisability from a good empirical performance to its expected performance on new data (Wang et al., 2019). It depends on the hypothesis set’s complexity, which is measured in this work by covering number (Zhou, 2002) and Rademacher complexity (Bartlett and Mendelson, 2002). Definition 3.1 (l∞,1l_∞,1 distance). Let X be a feature space and ℱF a space of functions from X to ℝnR^n. The l∞,1l_∞,1-distance on the space ℱF is defined as l∞,1(f,g)=maxx∈(∑i=1n|fi(x)−gi(x)|)l_∞,1(f,g)= _x ( _i=1^n|f_i(x)-g_i(x)| ). Definition 3.2 (covering number). The covering number, denoted ∞,1(ℱ,r)N_∞,1(F,r), is the minimum number of balls of radius r required to completely cover the function space ℱF under the l∞,1l_∞,1-distance. Definition 3.3 (Rademacher complexity). Let ℱF be a class of real-valued functions on a feature space X, and let τ1,…,τN _1,…, _N be i.i.d. samples from a distribution over X. The empirical Rademacher complexity of ℱF is ℜ^N(ℱ)=σ[supf∈ℱ1N∑i=1Nσif(τi)] R_N(F)=E_σ[ _f 1N _i=1^N _if( _i)], where σ1,…,σN _1,…, _N are independent Rademacher random variables taking values ±1± 1 with equal probability. The Rademacher complexity of ℱF is the expectation over the sample set. 4 Parallel Reward Integration with Symmetry This section introduces our algorithm PRISM. 4.1 ReSymNet: Reward Symmetry Network To address the challenge of heterogeneous reward objectives, PRISM first transforms sparse rewards into dense, per-step signals. We frame this as a supervised learning problem, inspired by but distinct from inverse reinforcement learning, as we do not assume access to expert demonstrations (Ng and Russell, 2000; Arora and Doshi, 2021). The goal is to train a reward model, ℛpredR_pred, parametrised by ψ, that learns to map state-action pairs to individual extrinsic rewards. We hope to train the reward shaping model on a dataset collected by executing a purely random policy, ensuring broad state-space coverage. For each timestep t, we construct a feature vector t=[st,at,tdense] h_t=[s_t,a_t, r^dense_t], where sts_t is the state, ata_t is the action, and dense,t r_dense,t are the dense rewards obtained from taking action ata_t at state sts_t, which crucially leverages the information from already-dense objectives to help predict the sparse ones. Figure 2 visualises the ResNet-like architecture. Remark 4.1. Residual connections in ℛpredR_pred are inspired by the theory of scaled opportunity value (Laud, 2004), whose additive corrections preserve optimal policies, shorten the effective reward horizon, and improve local value approximation (see Appendix B). Figure 2: Overview of ReSymNet. The network is optimised by minimising the mean squared error between the sum of its per-step predictions over a trajectory and the true cumulative sparse reward observed for that trajectory: ℒ(ψ)=∑τ∈(∑t∈τℛpred(t;ψ)−Rsp(τ))2.L(ψ)= _τ ( _t∈τR_pred( h_t;ψ)-R^sp(τ) )^2. (1) To ensure the learned reward function is robust and adapts to the agent’s improving policy, we incorporate two techniques: (1) we train an ensemble of reward models to reduce variance and produce a more stable shaping signal, and (2) we employ iterative refinement: the reward model is periodically updated using new, on-policy data collected by the agent. This allows the reward model to correct for the initial distribution shift and remain accurate as the agent’s behaviour evolves from random exploration to expert execution, as outlined in Algorithm 1 in Appendix B. 4.2 SymReg: Enforcing Reflectional Equivariance However, aligning reward frequencies alone is insufficient, as heterogeneous rewards cause the policy to learn asymmetrically across objectives, violating the agent’s physical symmetry. To address this, we leverage reflectional symmetry as an inductive bias to prevent asymmetric policy learning. For example, for legged agents, flexing a leg is essentially the mirror image of extending it. Standard policies must learn both motions separately, wasting data. By encoding symmetry as an inductive bias, experience from one motion can be reused for its mirror, improving sample efficiency and robustness. We formalise this physical intuition using group theory, specifically the reflection group G=ℤ2G=Z_2. This group consists of two transformations: the identity and a negation/reflection operator, g. Let ⊆ℝdsS ^d_s and ⊆ℝdaA ^d_a denote the state and action spaces, respectively, where dsd_s is the dimension of the state space and dad_a of the action space. We define index sets Iasyms⊂1,…,dsI_asym^s⊂\1,…,d_s\ and Isyms⊂1,…,dsI_sym^s⊂\1,…,d_s\ such that Iasyms∩Isyms=∅I_asym^s∩ I_sym^s= and Iasyms∪Isyms=1,…,dsI_asym^s∪ I_sym^s=\1,…,d_s\. This partitions the state vector as s=(sasym,ssym)s=(s_asym,s_sym) where sasym=sIasymss_asym=s_I_asym^s and ssym=sIsymss_sym=s_I_sym^s. We first partition the state vector s into an asymmetric part, sasyms_asym (e.g., the torso’s position), and a symmetric part, ssyms_sym (e.g., the leg’s relative joint angles and velocities in Figure 1). The state transformation operator, Lg:→L_g:S , reflects the symmetric part of the state as follows: Lg(s)=(sasym,−ssym)L_g(s)=(s_asym,-s_sym). Similarly, we define index sets IasymaI_asym^a and IsymaI_sym^a for the action space, and the action space is split up into an asymmetric part, aasyma_asym, and a symmetric part, asyma_sym. The action transformation operator, Kg:→K_g:A , reflects the symmetric part of the action (e.g., the leg torques): Kg(a)=(aasym,−asym)K_g(a)=(a_asym,-a_sym). The goal is to learn a policy, π, that is equivariant in terms of the aforementioned transformation. A policy π is reflectional-equivariant if it satisfies the following condition for all states s∈s : π(Lg(s))=Kg(π(s))π(L_g(s))=K_g(π(s)). This property means that the action for a reflected state is the same as the reflection of the action for the original state. To enforce this, we introduce a Symmetry Regulariser (SymReg) that explicitly penalises deviations from the desired symmetry property. During training, for each observation s, we compute both the standard policy output π(a|s;ϕ)π(a|s;φ), parameterised by ϕφ, and the output for the reflected state π(a|Lg(s);ϕ)π(a|L_g(s);φ). The equivariance loss is then defined as: ℒeq=s∼,a∼πϕ[∥π(a|Lg(s);ϕ)−Kg(π(a|s;ϕ))∥12].L_eq=E_s ,a _φ [\|π(a|L_g(s);φ)-K_g(π(a|s;φ))\|^2_1 ]. SymReg measures the deviation between the policy’s actual response to a reflected state and the expected reflected response. The training objective combines the standard policy gradient loss, Jπ(ϕ)J_π(φ), with SymReg: ℒtotal=Jπ(ϕ)+λℒeqL_total=J_π(φ)+ _eq, where λ is a hyperparameter controlling SymReg. 5 Theoretical Analysis This section presents theoretical guarantees of PRISM’s generalisability. Let Π be the full hypothesis space of policies represented by ReSymNet, R(π;τ)R(π;τ) is the cumulative return for a single trajectory τ obtained following policy π. Remark 5.1. As the backbone of the whole method, the hypothesis complexity and generalisability of ReSymNet contribute significantly to the generalisability of the whole algorithm. Due to space limit, we present Theorem B.8 in the appendices for the covering number of ReSymNet’s hypothesis space. The theory relies on these assumptions: Assumption 5.2 (bounded returns). For all policies π and trajectories τ, 0≤R(π;τ)≤B0≤ R(π;τ)≤ B. Assumption 5.3 (Lipschitz-continuous return). There exists LR>0L_R>0 such that for all π,π~∈Ππ, π∈ and any trajectory τ, |R(π;τ)−R(π~;τ)|≤LRd(π,π~)|R(π;τ)-R( π;τ)|≤ L_Rd(π, π), where d(π,π~):=sups∈‖π(s)−π~(s)‖1d(π, π):= _s \|π(s)- π(s)\|_1. Assumption 5.4 (compact spaces). The state space S and action space A are compact metric spaces. Assumption 5.5 (bounded policy). Policies π∈Ππ∈ have bounded inputs and weights. Assumption 5.6 (episode sampling). The behaviour distribution D has state marginal lower-bounded by pmin>0p_ >0 on the state support of interest (finite-support or density lower-bound assumption). The Assumptions are reasonably mild. (Bartlett et al., 2017) prove that feedforward ReLU are Lipschitz functions; since our policies are implemented as ReLU networks, this ensures bounded sensitivity of the policy outputs to perturbations. Assuming further that the return function is Lipschitz in the policy outputs, it follows that returns are Lipschitz in the policies themselves, as stated in Assumption 5.3. Assumption 5.6 ensures that all relevant states are sufficiently sampled under the behaviour policy, which is, in practice, reasonable because policy exploration mechanisms prevent the policy from collapsing onto a subset of states. 5.1 Generalisability of Reflection-Equivariant Subspace Let G=ℤ2G=Z_2 act on states and actions via Lg,KgL_g,K_g. An orbit-averaging operator (π)(s)=12(π(s)+Kg(π(Lg(s))))Q(π)(s)= 12 (π(s)+K_g(π(L_g(s))) ) maps any policy to a reflection-equivariant subspace (Qin et al., 2022). The regulariser ℒeq=s‖π(Lg(s))−Kg(π(s))‖12L_eq=E_s\|π(L_g(s))-K_g(π(s))\|_1^2 encourages convergence to the fixed-point subspace, defined as follows. Definition 5.7 (reflection-equivariant subspace). We define reflection-equivariant subspace as Πeq:=π:π(Lg(s))=Kg(π(s)) _eq:=\π:π(L_g(s))=K_g(π(s))\. We prove that Q is reflectional equivariant, a projection, and that its image coincides with the set of equivariant policies in Lemmas C.4, C.5, and C.6 in Appendix C.3, respectively. Thus, Q is surjective onto Πeq _eq. To prove that the subspace Πeq _eq is less complex, we show that the projection Q is non-expansive, which implies its image has a covering number no larger than the original space. Theorem 5.8. The space Πeq _eq has a covering number less than or equal to that of Π . Let ∞,1(ℱ,r)N_∞,1(F,r) be the covering number of a function space ℱF under the l∞,1l_∞,1-distance. Then, ∞,1(Πeq,r)≤∞,1(Π,r)N_∞,1( _eq,r) _∞,1( ,r). The l∞,1l_∞,1-distance between two policies πϕ _φ and πθ _θ is d(πϕ,πθ)=sups‖πϕ(s)−πθ(s)‖1d( _φ, _θ)= _s\| _φ(s)- _θ(s)\|_1. The distance between their projections, d((πϕ),(πθ))d(Q( _φ),Q( _θ)), is no larger using the fact that KgK_g is a norm-preserving isometry, ‖Kg(a)‖1=‖a‖1\|K_g(a)\|_1=\|a\|_1, and that LgL_g is a bijection, which implies that the supremum over s equals the supremum over Lg(s)L_g(s). Hence Q is non-expansive, and a non-expansive surjective map cannot increase the covering number. Following Lemma C.6, (Πeq,r)≤(Π,r)N( _eq,r) ( ,r). A detailed proof can be found in Appendix C.4. The symmetrisation technique is fundamental in empirical process theory that reduces the problem of bounding uniform deviations to analysing Rademacher complexity (Bartlett and Mendelson, 2002). Corollary 5.9. For any class ℱF of functions bounded in [0,B][0,B], the expected supremum of empirical deviations satisfies: [supf∈ℱ|1N∑i=1N(f(τi)−[f])|]≤2[ℜN(ℱ)],E [ _f | 1N _i=1^N(f( _i)-E[f]) | ]≤ 2E[ R_N(F)], where ℜN(ℱ)=σ[supf∈ℱ1N∑i=1Nσif(τi)] R_N(F)=E_σ [ _f 1N _i=1^N _if( _i) ] is the Rademacher complexity and σi _i are independent Rademacher random variables taking values ±1± 1. This bound transforms the original centred empirical process into a symmetrised version that is often easier to analyse. We now prove a high-probability uniform generalisation bound over the reflection-equivariant subspace. A detailed proof can be found in Appendix C.5. We recognise that PRISM does not necessarily converge to it, which will be discussed in the following subsection. Theorem 5.10. With ℛΠeq=τ↦R(π;τ):π∈ΠeqR_ _eq=\τ R(π;τ):π∈ _eq\, fix any accuracy parameter r∈(0,B)r∈(0,B) and confidence δ∈(0,1)δ∈(0,1). Then with probability at least 1−δ1-δ, supπ∈Πeq|J(π)−J^N(π)|≤C(∫rBlog∞,1(ℛΠeq,ε)Nε)+8rN+Blog(2/δ)2N, split& _π∈ _eq -3.99994pt|J(π)- J_N(π)|\\ ≤&C ( _r^B _∞,1(R_ _eq, )N\,d )\\ &+ 8r N+B (2/δ)2N, split where C is an absolute numeric constant, J(π)J(π) is the population expected return and J^N(π)=1N∑i=1NR(π;τi) J_N(π)= 1N _i=1^NR(π; _i) is the empirical return on N i.i.d. episodes τ1,…,τN _1,…, _N. Corollary 5.11. Under the same assumptions as Theorem 5.10, for any r∈(0,B)r∈(0,B) and δ∈(0,1)δ∈(0,1), the upper bound in Theorem 5.10 for Πeq _eq is at most the same bound obtained by replacing Πeq _eq with Π . By Lemma C.8, the return-class covering numbers can be bounded by those of the policy class with radius scaled by 1/LR1/L_R. Mathematically, following Theorem 5.8, for every ε>0 >0, log∞,1(Πeq,ε/LR)≤log∞,1(Π,ε/LR), _∞,1 ( _eq, /L_R )≤ _∞,1 ( , /L_R ), (2) hence the upper bound in Theorem 5.10 is no larger when evaluated on Πeq _eq. The equivariance regulariser projects policies onto a smaller fixed-point subspace Πeq _eq, which provably has covering numbers no larger than Π . The return class inherits this reduction via the Lipschitz map, so the Dudley entropy integral for Πeq _eq is bounded by that of Π . As such, the upper bound on the generalisation gap is no larger for Πeq _eq compared to Π . 5.2 Generalisability of PRISM We now study the generalisability of PRISM, which does not necessarily converge to the reflection-equivariant subspace exactly. Rather, PRISM might converge to an approximately reflection-equivariant class. Using the orbit averaging Q, we quantify this effect below. Definition 5.12 (approximately reflection-equivariant class). Approximately reflection-equivariant class is defined as Πapprox(εeq):=π∈Π:ℒeq≤εeq _approx( _eq):=\π∈ :L_eq≤ _eq\. Theorem 5.13. Let ξ:=12εeq/pminξ:= 12 _eq/p_ . Then for every policy π∈Ππ∈ , |J(π)−J(Q(π))|≤LR⋅d(π,Q(π))≤LRξ.|J(π)-J(Q(π))|≤ L_R· d(π,Q(π))≤ L_Rξ. (3) Then every π∈Πapprox(εeq)π∈ _approx( _eq) lies in the sup-ball of radius ξ around Πeq _eq. Consequently, for any target covering radius r>ξr>ξ, we have: ∞,1(Πapprox(εeq),r)≤∞,1(Πeq,r−ξ).N_∞,1 ( _approx( _eq),r ) _∞,1 ( _eq,r-ξ ). (4) By Lipschitzness of returns, the expected return of a policy and its projection differ by at most LRd(π,Q(π))L_Rd(π,Q(π)). The mismatch Δπ _π controls this distance, and Lemma C.10 bounds its supremum by ξ, giving the first inequality. Geometrically, Πapprox(εeq) _approx( _eq) is contained in a ξ-tube around Πeq _eq. Hence any (r−ξ)(r-ξ)-cover of Πeq _eq yields an r-cover of Πapprox(εeq) _approx( _eq), proving the covering-number relation (see Appendix C.6 for a detailed proof). Theorem 5.14. With ℛΠeq=τ↦R(π;τ):π∈ΠeqR_ _eq=\τ R(π;τ):π∈ _eq\, fix any accuracy parameter r∈(0,B)r∈(0,B) and confidence δ∈(0,1)δ∈(0,1). Then with probability at least 1−δ1-δ, supπ∈Πapprox(εeq)|J(π)−J^N(π)|≤C(∫rBlog∞,1(ℛΠeq,ε)Nε)+8rN+Blog(2/δ)2N+2LRξ. split& _ 25.0ptπ∈ _approx( _eq) 5.0pt|J(π)- J_N(π)|\\ ≤&C ( _r^B _∞,1(R_ _eq, )Nd )\\ &+ 8r N+B (2/δ)2N+2L_Rξ. split For π∈Πapprox(εeq)π∈ _approx( _eq), decompose the generalisation error relative to its projection Q(π)∈ΠeqQ(π)∈ _eq. The differences in population returns |J(π)−J(Q(π))||J(π)-J(Q(π))| and in empirical returns |J^N(π)−J^N(Q(π))|| J_N(π)- J_N(Q(π))| are bounded by LRξL_Rξ (Theorem 5.13). The middle term |J(Q(π))−J^N(Q(π))||J(Q(π))- J_N(Q(π))| is the generalisation error of an equivariant policy. Taking supremum, an equivariant bound is obtained (Theorem 5.10) plus 2LRξ2L_Rξ. Detailed proofs are in Appendix C.6. Corollary 5.15. Under the same assumptions as Theorem 5.14, for any r∈(0,B)r∈(0,B) and δ∈(0,1)δ∈(0,1), the upper bound in Theorem 5.14 for Πapprox(εeq) _approx( _eq) is at most the same bound obtained by replacing Πapprox(εeq) _approx( _eq) with Π . By Lemma C.8, the return-class covering numbers can be bounded by those of the policy class with radius scaled by 1/LR1/L_R. For any target covering radius r>ξr>ξ, we have log∞,1(Πapprox(εeq),r/LR) _∞,1 ( _approx( _eq),r/L_R ) ≤ ≤ log∞,1(Πeq,(r−ξ)/LR) _∞,1 ( _eq,(r-ξ)/L_R ) ≤ ≤ log∞,1(Π,(r−ξ)/LR). _∞,1 ( ,(r-ξ)/L_R ). (5) Hence the upper bound in Theorem 5.14 is no larger when evaluated on Πeq _eq. The covering relation incurs a slack of size ξ, leading to bounds of the form N(Πapprox(εeq),r)≤N(Πeq,r−ξ)≤N(Π,r−ξ)N( _approx( _eq),r)≤ N( _eq,r-ξ)≤ N( ,r-ξ) . By contrast, in Corollary 5.11, this slack disappears. Thus, the exact case guarantees a strict reduction in complexity, whereas the approximate case trades a ξ-shift in the radius for retaining proximity to the equivariant subspace. 6 Experiments We conduct extensive experiments to verify PRISM. The code is at https://github.com/EVIEHub/PRISM. 6.1 Experimental Settings Environments. Four MuJoCo (Todorov et al., 2012) environments are used: mo-hopper-v5, mo-walker2d-v5, mo-halfcheetah-v5, and mo-swimmer-v5. Table 3 in Appendix D displays the environments and their dimensions, highlighting the diversity in space complexity. As a result, a method must be able to find general solutions applicable to various MORL challenges, instead of being just tailored to one specific type of problem. Furthermore, the division of asymmetric and symmetric state and action spaces to model equivariance is detailed in Appendix D. Baselines. PRISM is adaptable to any off-the-shelf MORL algorithm. In this work, CAPQL (Lu et al., 2023) is used as a backbone model, which is a method that trains a single universal network to cover the entire preference space and approximate the Pareto front. We produce (1) oracle: instead of artificially setting a reward channel to be sparse, this baseline model can be seen as the gold standard, and (2) baseline: instead of utilising the proposed reward shaping model, this method uses CAPQL (Lu et al., 2023) and only observes the sparse rewards. Evaluation. We use hypervolume (HV), Expected Utility Metric (EUM), and one distributional metric, Variance Objective (VO) (Cai et al., 2023), for evaluation. The used hyperparameters, together with a detailed explanation of evaluation metrics, can be found in Appendix E. 6.2 Empirical Results Reward Sparsity Sensitivity. Figure 3 illustrates the sensitivity of MORL agents to varying levels of reward sparsity. Across all environments, we observe a sharp decline in HV when one objective is made extremely sparse, with reductions ranging from 20 to 40% relative to the dense setting. These results confirm that sparse objectives worsen policy quality, as agents tend to neglect long-term sparse signals in favour of denser objectives. For the rest of the paper, we continue with the most difficult setting where extreme sparsity is imposed on the first reward objective. (a) Mo-hopper-v5 (b) Mo-walker2d-v5 (c) Mo-halfcheetah-v5 (d) Mo-swimmer-v5 Figure 3: The obtained hypervolume for various levels of sparsity amongst various dimensions. Return Distribution of Policy. Figure 4 illustrates the impact of mixed sparsity on MORL across the considered environments. Each subplot compares the approximated Pareto fronts obtained when objective one is dense (blue dots) versus when it is made sparse (orange dots), while keeping all other objectives dense. Extreme sparsity is imposed, where the sparse reward is released at the end of an episode. The results demonstrate a consistent pattern across all environments: when objective one becomes sparse, agents systematically fail to discover high-performing solutions along this dimension, instead concentrating their learning efforts on the remaining dense objectives. (a) Mo-hopper-v5 (b) Mo-walker2d-v5 (c) Mo-halfcheetah-v5 (d) Mo-swimmer-v5 Figure 4: The approximated Pareto front for dense rewards (blue dots) and sparse rewards (orange dots) for the first reward objective. Comparison Experiments. Table 1 reports the obtained results for HV, EUM, and VO. The results are averaged over 10 trials, with the standard deviations shown in grey. PRISM consistently outperforms both the oracle and baseline across environments. For mo-hopper-v5, PRISM improves hypervolume by 21.5% over the oracle (1.58×1071.58× 10^7 compared to 1.30×1071.30× 10^7) and 88% over the baseline. Similar gains are observed for mo-walker2d-v5, where PRISM achieves a 13% HV improvement over oracle and 43% over the baseline. Notably, in mo-halfcheetah-v5, PRISM yields a 32% improvement in HV compared to the oracle (2.25×1042.25× 10^4 against 1.70×1041.70× 10^4) and more than doubles the sparse result. These improvements imply that PRISM not only restores solutions lost under sparsity but also expands the range of trade-offs accessible to the agent. Improvements in EUM follow the same trend, with increases of up to 50% compared to the baseline. The concurrent increase in EUM demonstrates that these solutions provide higher expected utility, confirming that PRISM learns policies that are both diverse and practically useful. On distributional metrics, PRISM delivers more consistent performance than both the oracle and baseline. VO in mo-hopper-v5 increases from 43.36 (baseline) and 59.07 (oracle) to 66.66 under PRISM, and mo-walker2d-v5 shows a 51% gain over the baseline. These gains are crucial because they indicate that PRISM does not simply maximise HV by focusing on extreme solutions, but also produces Pareto fronts that are better balanced, robust, and fair across objectives. Figure 6 in Appendix F, which shows the approximated Pareto fronts, aligns with these results. Environment Metric Oracle Baseline PRISM Mo-hopper-v5 HV (×107× 10^7) 1.30 ± 0.13 0.84 ± 0.05 1.58 ± 0.05 EUM 129.04 ± 7.96 97.64 ± 4.18 147.43 ± 2.61 VO 59.07 ± 3.45 43.36 ± 1.61 66.66 ± 1.40 Mo-walker2d-v5 HV (×104× 10^4) 4.21 ± 0.11 3.34 ± 0.16 4.77 ± 0.07 EUM 107.58 ± 2.86 82.13 ± 4.34 120.43 ± 1.64 VO 53.22 ± 1.39 39.18 ± 2.49 59.35 ± 0.80 Mo-halfcheetah-v5 HV (×104× 10^4) 1.70 ± 0.20 0.97 ± 0.00 2.25 ± 0.18 EUM 81.29 ± 21.85 -1.46 ± 0.27 89.94 ± 15.33 VO 36.84 ± 10.06 -1.01 ± 0.20 40.72 ± 7.02 Mo-swimmer-v5 HV (×104× 10^4) 1.21 ± 0.00 1.09 ± 0.02 1.21 ± 0.00 EUM 9.41 ± 0.12 4.10 ± 0.80 9.44 ± 0.14 VO 4.22 ± 0.08 1.58 ± 0.40 4.24 ± 0.07 Table 1: Experimental results. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font. (a) Full-episode stability (b) Signal optimisation Figure 5: The dense (blue line) and shaped rewards (orange line) over time for mo-walker2d-v5 and the first reward objective. We provide two distinct examples to analyse the behaviour of the learned reward signals compared to the oracle for mo-walker2d-v5. Figure 5(a) illustrates a full 1000-step episode. The shaped reward is highly correlated with the dense reward throughout the entire trajectory. The alignment of peaks and troughs confirms that ReSymNet captures the dynamics of the environment, ensuring accurate credit assignment without temporal drift. Figure 5(b) highlights a key theoretical advantage of ReSymNet. In high-performance regions (e.g., steps 250–270), the shaped reward amplifies the signal, exceeding the magnitude of the oracle. By creating steeper gradients for desirable behaviours, the shaped reward can provide more effective guidance than the raw environmental signal, explaining why PRISM is capable of outperforming the oracle. Ablation Study. We analyse the following ablation models (w/o is the abbreviation for without), which remove several aspects of the reward shaping model or the equivariance loss: (1) PRISM: This is the proposed method, involving all components, (2) w/o residual: This ablation model removes the two residual blocks from the reward shaping model, (3) w/o dense rewards: We remove the dense rewards as input features to the reward model, (4) w/o ensemble: We remove the ensemble of reward shaping models, and only employ one, (5) w/o refinement: Rather than updating the reward shaping model with expert trajectories, this approach merely trains the reward shaping model using the random trajectories collected at first, and (6) w/o loss: We remove the equivariance loss term and merely use the reward shaping model. We also include two ablation studies that remove ReSymNet from PRISM and replace the reward shaping model as follows: (7) uniform: Distributes the episodic sparse reward Rsp(τ)R^sp(τ) equally across all T timesteps, and (8) random: Samples random weights αt∼(−1,1) _t (-1,1) for each timestep, normalises to sum to one, and scales by the total reward. The ablation results in Tables 10 and 11 in Appendix G highlight the contribution of individual components. Removing residual connections reduces HV and EUM across all environments (e.g., mo-hopper-v5 EUM falls from 147.43 to 128.40), showing their importance for scaled opportunity value. Excluding dense reward features or ensembles also lowers performance, but only moderately, suggesting that state–action features already contain substantial signal. Interestingly, removing iterative refinement barely reduces performance; in some cases, such as mo-halfcheetah-v5, HV, and EUM remain comparable or even slightly higher than the full model. This implies that shaping rewards from a broad set of random trajectories is already highly effective. Removing the symmetry loss reduces performance across environments, indicating that the loss term successfully reduces the search space. Similar patterns are observed for VO. Considering ReSymNet, uniform achieves moderate performance by providing per-step gradients and leveraging SymReg, while random performs poorly due to noisy, misleading rewards. PRISM consistently outperforms both by learning reward decomposition with ReSymNet and enforcing structural consistency via SymReg, enabling accurate credit assignment in complex multi-objective tasks. 7 Conclusion This work proposes Parallel Reward Integration with reflectional Symmetry for Multi-objective reinforcement learning (PRISM), a framework designed to tackle sample inefficiency in heterogeneous multi-objective reinforcement learning, particularly in environments with sparse rewards. Our approach is centred around two key contributions: (1) ReSymNet, a theory-inspired reward model that leverages residual blocks to align reward channels by learning a refined ‘scaled opportunity value’, and (2) SymReg, a novel regulariser that enforces reflectional symmetry as an inductive bias in the policy’s action space. We prove that PRISM restricts policy search to a reflection-equivariant subspace, a projection of the original policy space with provably reduced hypothesis complexity; in this way, the generalisability is rigorously improved. Extensive experiments on MuJoCo benchmarks show that PRISM consistently outperforms even a strong oracle with full reward access in terms of a wide range of metrics, including HV, EUM, and VO. Acknowledgements K. Qian was supported in part by the UKRI Grant EP/Y03516X/1 for the UKRI Centre for Doctoral Training in Machine Learning Systems (https://mlsystems.uk/). References L. N. Alegre, A. L. C. Bazzan, D. M. Roijers, A. Nowé, and B. C. da Silva (2023) Sample-efficient multi-objective learning via generalized policy improvement prioritization. In 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023), p. 2003–2012. Cited by: §H.3. S. Arora and P. Doshi (2021) A survey of inverse reinforcement learning: challenges, methods and progress. Artificial Intelligence 297, p. 103500. Cited by: §4.1. A. Aubret, L. Matignon, and S. Hassas (2019) A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.06976. Cited by: §1. P. L. Bartlett, D. J. Foster, and M. J. Telgarsky (2017) Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems 30. Cited by: §B.2, §5. P. L. Bartlett and S. Mendelson (2002) Rademacher and Gaussian complexities: risk bounds and structural results. Journal of Machine Learning Research 3, p. 463–482. Cited by: §C.1, §1, §3, §5.1. T. Basaklar, S. Gumussoy, and Ü. Y. Ogras (2023) PD-MORL: Preference-driven multi-objective reinforcement learning algorithm. In Eleventh International Conference on Learning Representations (ICLR 2023), Cited by: §2. Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov (2019) Exploration by random network distillation. In 7th International Conference on Learning Representations (ICLR 2019), Cited by: §2. X. Cai, P. Zhang, L. Zhao, J. Bian, M. Sugiyama, and A. Llorens (2023) Distributional Pareto-optimal multi-objective reinforcement learning. In 37th International Conference on Neural Information Processing Systems (NIPS 2023), Vol. 36, p. 15593–15613. Cited by: Appendix E, §6.1. X. Chen, A. Ghadirzadeh, M. Björkman, and P. Jensfelt (2019) Meta-learning for multi-objective reinforcement learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), p. 977–983. Cited by: §2. R. Devidze, P. Kamalaruban, and A. Singla (2022) Exploration-guided reward shaping for reinforcement learning under sparse rewards. In 36th International Conference on Neural Information Processing Systems (NIPS 2022), Cited by: §2. R. M. Dudley (1967) The sizes of compact subsets of Hilbert space and continuity of Gaussian processes. Journal of Functional Analysis 1 (3), p. 290–330. Cited by: §C.1. F. Felten, L. N. Alegre, A. Nowé, A. L. C. Bazzan, E. Talbi, G. Danoy, and B. C. da Silva (2023) A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In 37th International Conference on Neural Information Processing Systems (NIPS 2023), Cited by: Appendix E, §1. C. M. Fonseca, L. Paquete, and M. López-Ibáñez (2006) An improved dimension-sweep algorithm for the hypervolume indicator. In IEEE International Conference on Evolutionary Computation (CEC 2006), p. 1157–1163. Cited by: Appendix E. T. Gangwani, Y. Zhou, and J. Peng (2020) Learning guidance rewards with trajectory-space smoothing. In 33rd Annual Conference on Neural Information Processing Systems 2020 (NIPS 2020, Cited by: §2. B. Hambly, R. Xu, and H. Yang (2023) Recent advances in reinforcement learning in finance. Mathematical Finance 33 (3), p. 437–503. Cited by: §1. C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, et al. (2022) A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems 36 (1), p. 26. Cited by: Appendix E, §1, §2. F. He, T. Liu, and D. Tao (2020) Why resnet works? residuals generalize. IEEE Transactions on Neural Networks and Learning Systems 31 (12), p. 5349–5362. Cited by: §B.2. K. He, X. Zhang, S. Ren, and J. Sun (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV 2015), p. 1026–1034. Cited by: Table 8. I. Holmes and M. Chi (2025) Attention-based reward shaping for sparse and delayed rewards. arXiv preprint arXiv:2505.10802. Cited by: §2. B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez (2021) Deep reinforcement learning for autonomous driving: a survey. IEEE transactions on intelligent transportation systems 23 (6), p. 4909–4926. Cited by: §1. A. D. Laud (2004) Theory and application of reward shaping in reinforcement learning. University of Illinois at Urbana-Champaign. Cited by: §B.1, §1, Remark 4.1. T. Lautenbacher, A. Rajaei, D. Barbieri, J. Viebahn, and J. L. Cremer (2025) Multi-objective reinforcement learning for power grid topology control. arXiv preprint arXiv:2502.00040. Cited by: §2. Y. Lin, J. Huang, M. Zimmer, Y. Guan, J. Rojas, and P. Weng (2020) Invariant transform experience replay: data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters 5 (4), p. 6615–6622. Cited by: §2. C. Liu, X. Xu, and D. Hu (2014) Multiobjective reinforcement learning: a comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems 45 (3), p. 385–398. Cited by: §1. E. Liu, Y. Wu, X. Huang, C. Gao, R. Wang, K. Xue, and C. Qian (2025) Pareto set learning for multi-objective reinforcement learning. In AAAI Conference on Artificial Intelligence, p. 18789–18797. Cited by: §2. H. Lu, D. Herman, and Y. Yu (2023) Multi-objective reinforcement learning: convexity, stationarity and Pareto optimality. In Eleventh International Conference on Learning Representations (ICLR 2023), Cited by: §1, §2, §6.1. C. McDiarmid et al. (1989) On the method of bounded differences. Surveys in Combinatorics 141 (1), p. 148–188. Cited by: §C.1. F. Memarian, W. Goo, R. Lioutikov, S. Niekum, and U. Topcu (2021) Self-supervised online reward shaping in sparse-reward environments. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021), p. 2369–2375. Cited by: §2. K. V. Moffaert, M. M. Drugan, and A. Nowé (2013) Scalarized multi-objective reinforcement learning: novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2013), p. 191–199. Cited by: §2. A. K. Mondal, V. Jain, K. Siddiqi, and S. Ravanbakhsh (2022) EqR: equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning (ICML 2022), PMLR, Vol. 162, p. 15908–15926. Cited by: §2. A. K. Mondal, P. Nair, and K. Siddiqi (2020) Group equivariant deep reinforcement learning. arXiv preprint arXiv:2007.03437. Cited by: §2. N. Mu, Y. Luan, and Q. Jia (2025) Preference-based multi-objective reinforcement learning. IEEE Transactions on Automation Science and Engineering. Cited by: §2. A. Y. Ng, D. Harada, and S. Russell (1999) Policy invariance under reward transformations: theory and application to reward shaping. In Sixteenth International Conference on Machine Learning (ICML 1999), p. 278–287. Cited by: §2. A. Y. Ng and S. Russell (2000) Algorithms for inverse reinforcement learning. In Seventeenth International Conference on Machine Learning (ICML 2000), p. 663–670. Cited by: §4.1. J. Y. Park, S. Bhatt, S. Zeng, L. L. S. Wong, A. Koppel, S. Ganesh, and R. Walters (2025) Approximate equivariance in reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTATS 2025), PMLR, Vol. 258, p. 4177–4185. Cited by: §2. D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-driven exploration by self-supervised prediction. In 34th International Conference on Machine Learning (ICML 2017), PMLR, Vol. 70, p. 2778–2787. Cited by: §1, §2. T. Qin, F. He, D. Shi, W. Huang, and D. Tao (2022) Benefits of permutation-equivariance in auction mechanisms. 36th International Conference on Neural Information Processing Systems (NIPS 2022) 35, p. 18131–18142. Cited by: §5.1. Z. Ren, R. Guo, Y. Zhou, and J. Peng (2022) Learning long-term reward redistribution via randomized return decomposition. In Tenth International Conference on Learning Representations (ICLR 2022), Cited by: §2. M. Reymond and A. Nowé (2019) Pareto-DQN: approximating the Pareto front in complex multi-objective decision problems. In Adaptive and Learning Agents Workshop (ALA 2019), Cited by: §2. D. M. Roijers, S. Whiteson, and F. A. Oliehoek (2015) Computing convex coverage sets for faster multi-objective coordination. Journal of Artificial Intelligence Research 52, p. 399–443. Cited by: §2. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. (2017) Mastering the game of go without human knowledge. Nature 550 (7676), p. 354–359. Cited by: §1. C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martín, and P. Stone (2025a) Deep reinforcement learning for robotics: a survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 28694–28698. Cited by: §1. Y. Tang, X. Cai, Y. Ding, Q. Wu, G. Liu, and M. Sugiyama (2025b) Reinforcement learning from bagged reward. Transactions on Machine Learning Research. Cited by: §2. Y. Tang, X. Cai, J. Pang, Q. Wu, Y. Ding, and M. Sugiyama (2024) Beyond simple sum of delayed rewards: non-markovian reward modeling for reinforcement learning. arXiv preprint arXiv:2410.20176. Cited by: §2. E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. Cited by: §1, §6.1. E. van der Pol, D. E. Worrall, H. van Hoof, F. A. Oliehoek, and M. Welling (2020) MDP homomorphic networks: group symmetries in reinforcement learning. In 33st Annual Conference on Neural Information Processing Systems (NIPS 2020), Cited by: §2. K. Van Moffaert and A. Nowé (2014) Multi-objective reinforcement learning using sets of Pareto dominating policies. The Journal of Machine Learning Research 15 (1), p. 3483–3512. Cited by: §2. D. Wang, R. Walters, and R. Platt (2022) SO(2)-equivariant reinforcement learning. In Tenth International Conference on Learning Representations (ICLR 2022), Cited by: §2. D. Wang, R. Walters, X. Zhu, and R. P. Jr. (2021) Equivariant Q learning in spatial action spaces. In 5th Conference on Robot Learning, PMLR, Vol. 164, p. 1713–1723. Cited by: §2. H. Wang, S. Zheng, C. Xiong, and R. Socher (2019) On the generalization gap in reparameterizable reinforcement learning. In 36th International Conference on Machine Learning (ICML 2019), Vol. 97, p. 6648–6658. Cited by: §3. W. Wei, H. Li, S. Zhou, B. Li, and X. Liu (2025) Attention with system entropy for optimizing credit assignment in cooperative multi-agent reinforcement learning. IEEE Transactions on Automation Science and Engineering 22, p. 14775–14787. Cited by: §1. R. Yang, X. Sun, and K. Narasimhan (2019) A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In 33rd International Conference on Neural Information Processing Systems (NIPS 2019), p. 14610–14621. Cited by: §2. D. Zhou (2002) The covering number in learning theory. Journal of Complexity 18 (3), p. 739–767. Cited by: §1, §3. L. M. Zintgraf, T. V. Kanters, D. M. Roijers, F. Oliehoek, and P. Beau (2015) Quality assessment of MORL algorithms: a utility-based approach. In 24th Annual Machine Learning Conference of Belgium and the Netherlands, Cited by: Appendix E. Appendix A Notation Table 2: Notation. Symbol Description S State space A Action space P(s′|s,a)P(s |s,a) Transition probability (s,a)∈ℝL r(s,a) ^L Vector-valued reward with L objectives γ∈[0,1)γ∈[0,1) Discount factor π:→π:S Policy mapping (π)=π[∑t=0∞γtt] J(π)=E_π\! [ _t=0^∞γ^t r_t ] Expected cumulative vector return D Behaviour distribution to sample episodes from DC=d1,…,dDDC=\d_1,…,d_D\ Dense reward channels rtdir^d_i_t Reward from dense channel did_i at timestep t rtspr^sp_t Sparse reward at timestep t τ=(s1,a1),…,(sT,aT)τ=\(s_1,a_1),…,(s_T,a_T)\ Trajectory Rsp(τ)R^sp(τ) Cumulative sparse reward in episode τ prelp_rel Probability of releasing sparse reward ht=[st,at,tdense]h_t=[s_t,a_t, r^dense_t] Input feature vector for ReSymNet ℛpredR_pred ReSymNet rtshr^sh_t Shaped reward at timestep t Lg,KgL_g,K_g Reflection operators on states and actions Δπ(s)=π(Lg(s))−Kg(π(s)) _π(s)=π(L_g(s))-K_g(π(s)) Equivariance mismatch ℒeqL_eq Equivariance regularisation loss Π Hypothesis space of policies Πeq=π:π(Lg(s))=Kg(π(s)) _eq=\π:π(L_g(s))=K_g(π(s))\ Reflection-equivariant subspace Πapprox(εeq) _approx( _eq) Approximate equivariant policies with tolerance εeq _eq Appendix B Additional Details and Theory of ReSymNet We give additional details of ReSymNet as well as the theoretical motivation behind its architecture in this appendix. B.1 Theoretical Motivation via Scaled Opportunity Value The use of residual connections in ℛpredR_pred is motivated by the theory of scaled opportunity value (Laud, 2004). Definition B.1 (Opportunity value). Let M be an MDP with native reward function R. The opportunity value of a transition (s,a,s′)(s,a,s ) is defined as the difference in the optimal value of successor and current states: OPV(s,a,s′)=γVM(s′)−VM(s),OPV(s,a,s )=γ V^M(s )-V^M(s), where VMV^M is the optimal state-value function under MDP M. Definition B.2 (Scaled opportunity value). For a scale parameter k>0k>0, the scaled opportunity value shaping function augments the native reward with a scaled opportunity correction: OPVk(s,a,s′)=Fk(s,a,s′)=k(γVM(s′)−VM(s))+(k−1)R(s,a).OPV_k(s,a,s )=F_k(s,a,s )=k(γ V^M(s )-V^M(s))+(k-1)R(s,a). Lemma B.3. Let M be an MDP with reward function R and optimal policy π⋆π . With k sufficiently large, the MDP with shaped reward FkF_k satisfies: (1) policy invariance, π⋆π remains optimal under FkF_k; (2) horizon reduction, the effective reward horizon is reduced to 11; and (3) improved local approximation, the additive term increases the separability of local utilities, reducing approximation error in value estimation. Residual blocks mirror the additive structure of scaled opportunity value: each block refines its input prediction via: ℛpred(i)(t;ψ)=ℛpred(i−1)(t;ψ)+Δi(t;ψ)R_pred^(i)( h_t;ψ)=R_pred^(i-1)( h_t;ψ)+ _i( h_t;ψ), where Δi _i is a learned correction. A single block can be viewed as approximating a scaled opportunity-value transformation of its input, while stacking multiple blocks implements iterative refinement: each stage reduces the residual error left by the previous one. This residual formulation both stabilises training and aligns with the principle of scaled opportunity value, gradually shaping per-step predictions into horizon-1 signals that remain consistent with the sparse episodic return Rsp(τ)R^sp(τ). B.2 Generalisability of ReSymNet We extend the theoretical justification of ReSymNet from optimisation to generalisation. Following the stem–vine decomposition of He et al. (2020), we prove that residual connections do not increase hypothesis complexity, and derive a high-probability bound. Notation and Assumptions. ReSymNet maps feature vectors t∈ℝd0h_t ^d_0 to sparse reward predictions rtsp∈ℝr^sp_t through a residual network. We decompose the network into: • A stem: the main feedforward pathway consisting of K layers, each with a weight matrix i∈ℝdi−1×diA_i ^d_i-1× d_i and nonlinearity σi:ℝdi→ℝdi _i:R^d_i ^d_i for i=1,…,Ki=1,…,K. • A collection of vines: residual connections (skip connections) indexed by triples (s,t,i)(s,t,i) where s is the source vertex (where the connection starts), t is the target vertex (where it reconnects), and i distinguishes multiple vines between the same pair of vertices. We denote the set of all vine indices as ℐVI_V. We denote vertices in the network as N(t)N(t), where t indexes the position in the computational graph. Each vine (s,t,i)V(s,t,i) is itself a small feedforward network with weight matrices 1s,t,i,…,Ks,t,is,t,iA^s,t,i_1,…,A^s,t,i_K_s,t,i and nonlinearities σ1s,t,i,…,σKs,t,is,t,iσ^s,t,i_1,…,σ^s,t,i_K_s,t,i, where Ks,t,iK_s,t,i is the number of layers in that vine. The output at vertex N(t)N(t) is: Ft()=FtS()+∑(s,t,i)∈ℐVFs,t,iV(),F_t(X)=F^S_t(X)+ _(s,t,i) _VF^V_s,t,i(X), where FtS()F^S_t(X) is the stem’s output at vertex t and the sum runs over all vines that reconnect at vertex t. Assumption B.4 (Bounded parameters). Each stem weight matrix satisfies ‖i‖σ≤si\|A_i\|_σ≤ s_i for i=1,…,Ki=1,…,K, where ∥⋅∥σ\|·\|_σ denotes the spectral norm. Each vine weight matrix satisfies ‖js,t,i‖σ≤sjs,t,i\|A^s,t,i_j\|_σ≤ s^s,t,i_j. All nonlinearities are ρi _i-Lipschitz continuous: for any 1,2x_1,x_2 in the domain, ‖σi(1)−σi(2)‖2≤ρi‖1−2‖2.\| _i(x_1)- _i(x_2)\|_2≤ _i\|x_1-x_2\|_2. Input features satisfy ‖t‖2≤Bh\|h_t\|_2≤ B_h, network per-step outputs satisfy |Rpred(t;ψ)|≤Bpred|R_pred(h_t;ψ)|≤ B_pred for all t, and sparse rewards satisfy |Rsp(τ)|≤Br|R^sp(τ)|≤ B_r for all trajectories τ. Trajectories have length bounded by TmaxT_ . Lemma B.5. Let ∈ℝn×dX ^n× d be a data matrix with n samples and d features, satisfying ‖2≤B\|X\|_2≤ B. Consider the hypothesis space formed by all linear transformations with bounded spectral norm: ℋA=:∈ℝd×m,‖σ≤s.H_A=\XA:A ^d× m,\ \|A\|_σ≤ s\. Then the ε -covering number satisfies: log∞,2(ℋA,ε)≤⌈s2B2m2ε2⌉log(2dm), _∞,2(H_A, )≤ s^2B^2m^2 ^2 (2dm), where m is the output dimension. This lemma (Bartlett et al., 2017) shows that the complexity of a single linear layer scales with the square of its spectral norm and input norm. Lemma B.6. For an K-layer feedforward network with hypothesis space ℋfH_f, the covering number satisfies: ∞,2(ℋf,ε)≤∏i=1Ksup1,…,i−1i,N_∞,2(H_f, )≤ _i=1^K _A_1,…,A_i-1N_i, where iN_i is the covering number of layer i (viewed as a function of its input) when the preceding layers 1,…,i−1A_1,…,A_i-1 are held fixed. The supremum is taken over all choices of the preceding weight matrices within their respective spectral norm bounds. This result shows that the covering number of a deep network is the product of the covering numbers of its individual layers. For residual networks, where outputs are sums of stem and vine contributions, we require: Lemma B.7. Let ℱF and G be two function classes. If FW_F is an εF _F-cover of ℱF (meaning every f∈ℱf is within distance εF _F of some element in FW_F), and GW_G is an εG _G-cover of G, then the set F+G=f+g:f∈F,g∈GW_F+W_G=\f+g:f _F,\ g _G\ is an (εF+εG)( _F+ _G)-cover of the sum class ℱ+=f+g:f∈ℱ,g∈F+G=\f+g:f ,g \, and ∞,2(ℱ+,εF+εG)≤∞,2(ℱ,εF)∞,2(,εG).N_∞,2(F+G, _F+ _G) _∞,2(F, _F)\,N_∞,2(G, _G). Proof. For any f+g∈ℱ+f+g +G, there exist wf∈Fw_f _F and wg∈Gw_g _G such that ‖f−wf‖2≤εF\|f-w_f\|_2≤ _F and ‖g−wg‖2≤εG\|g-w_g\|_2≤ _G. By the triangle inequality: ‖(f+g)−(wf+wg)‖2≤‖f−wf‖2+‖g−wg‖2≤εF+εG.\|(f+g)-(w_f+w_g)\|_2≤\|f-w_f\|_2+\|g-w_g\|_2≤ _F+ _G. The covering number bound follows since there are at most |F|⋅|G||W_F|·|W_G| distinct pairs (wf,wg)(w_f,w_g). ∎ Theorem B.8. Under Assumption B.4, let εjj=1K\ _j\_j=1^K be tolerances for each stem layer and εs,t,i(s,t,i)∈ℐV\ _s,t,i\_(s,t,i) _V be tolerances for each vine, satisfying ∑j=1Kεj+∑(s,t,i)∈ℐVεs,t,i≤ε. _j=1^K _j+ _(s,t,i) _V _s,t,i≤ . Then the covering number of ReSymNet’s hypothesis space ℋresH_res satisfies: ∞,2(ℋres,ε)≤∏j=1K∞,2(ℋj,εj)∏(s,t,i)∈ℐV∞,2(ℋs,t,iV,εs,t,i),N_∞,2(H_res, )≤ _j=1^KN_∞,2(H_j, _j)\, _(s,t,i) _VN_∞,2(H^V_s,t,i, _s,t,i), where ℋjH_j is the hypothesis space of stem layer j and ℋs,t,iVH^V_s,t,i is the hypothesis space of vine (s,t,i)V(s,t,i). Applying Lemma B.5 to each weight matrix, this yields: log∞,2(ℋres,ε)≤ℛε2, _∞,2(H_res, )≤ R ^2, where the complexity measure ℛR is: ℛ=∑i=1Ksi2‖Fi−1()‖22εi2log(2di2)+∑(s,t,i)∈ℐV(s,t,i)2‖Fs()‖22εs,t,i2log(2ds,t,i2).R= _i=1^K s_i^2\|F_i-1(X)\|_2^2 _i^2 (2d_i^2)+ _(s,t,i) _V (s^s,t,i)^2\|F_s(X)\|_2^2 _s,t,i^2 (2d_s,t,i^2). Here, Fi−1()F_i-1(X) denotes the output of the network after layer i−1i-1 (the input to layer i), and did_i is the dimension at layer i. Proof. We proceed by analysing how residual connections compose with the stem. Consider vertex N(t)N(t) where one or more vines reconnect. The output is: Ft()=FtS()+∑(s,t,i)∈ℐVFs,t,iV().F_t(X)=F^S_t(X)+ _(s,t,i) _VF^V_s,t,i(X). Let tW_t be an εt _t-cover of ℋtH_t (all possible stem outputs at vertex t). For each vine (s,t,i)V(s,t,i) that reconnects at t, let s,t,iVW^V_s,t,i be an εs,t,i _s,t,i-cover of ℋs,t,iVH^V_s,t,i (all possible outputs of that vine). By repeated application of Lemma B.7, the set: t′=WS+∑(s,t,i)∈ℐVWs,t,iV:WS∈t,Ws,t,iV∈s,t,iVW _t= \W_S+ _(s,t,i) _VW^V_s,t,i:W_S _t,W^V_s,t,i ^V_s,t,i \ is an (εt+∑(s,t,i)∈ℐVεs,t,i) ( _t+ _(s,t,i) _V _s,t,i )-cover of ℋt′H _t (the combined outputs at vertex t), with covering number: ∞,2(ℋt′,εt′)≤∞,2(ℋt,εt)⋅∏(s,t,i)∈ℐV∞,2(ℋs,t,iV,εs,t,i),N_∞,2(H _t, _t) _∞,2(H_t, _t)· _(s,t,i) _VN_∞,2(H^V_s,t,i, _s,t,i), where εt′=εt+∑(s,t,i)∈ℐVεs,t,i _t= _t+ _(s,t,i) _V _s,t,i. Each vine (s,t,i)V(s,t,i) is itself a chain-like feedforward network, so Lemma B.6 applies to bound ∞,2(ℋs,t,iV,εs,t,i)N_∞,2(H^V_s,t,i, _s,t,i). For identity vines (containing no trainable parameters), we have s,t,iV=1N^V_s,t,i=1 since there is only one function in the class. Propagating this argument through all K stem layers yields: ∞,2(ℋres,ε)≤∏j=1K∞,2(ℋj,εj)∏(s,t,i)∈ℐV∞,2(ℋs,t,iV,εs,t,i).N_∞,2(H_res, )≤ _j=1^KN_∞,2(H_j, _j)\, _(s,t,i) _VN_∞,2(H^V_s,t,i, _s,t,i). The bound on ℛR follows by applying Lemma B.5 to each weight matrix. For the stem, layer i contributes: logi≤si2‖Fi−1()‖22di2εi2log(2di−1di)≈si2‖Fi−1()‖22εi2log(2di2), _i≤ s_i^2\|F_i-1(X)\|_2^2d_i^2 _i^2 (2d_i-1d_i)≈ s_i^2\|F_i-1(X)\|_2^2 _i^2 (2d_i^2), where we simplify by assuming similar dimensions. Summing over all stem layers and all vine layers gives ℛR. ∎ Corollary B.9. Let ℋfH_f be the hypothesis space of feedforward networks with the same total number of weight matrices Ktotal=K+∑(s,t,i)∈ℐVKs,t,iK_total=K+ _(s,t,i) _VK_s,t,i as ReSymNet. Then for any ε>0 >0, ∞,2(ℋres,ε)≤∞,2(ℋf,ε).N_∞,2(H_res, ) _∞,2(H_f, ). Proof. Both covering numbers have the product form ∏k=1Ktotalk _k=1^K_totalN_k, where each factor kN_k corresponds to a single weight matrix. By Lemma B.5, each kN_k depends only on the spectral norm sks_k of that weight matrix and the norm of its input ‖Fk−1()‖2\|F_k-1(X)\|_2, regardless of whether the matrix appears in the stem or a vine. Therefore, when the total number of weight matrices and their norms are held fixed, the covering numbers are bounded identically. ∎ B.3 Algorithm Chart Algorithm 1 ReSymNet with any MORL algorithm 1: Input: Release probability prelp_rel, number of initial episodes N, number of expert episodes E, dense channels DC, MORL algorithm, timesteps per cycle M, ensembles K, refinements IRIR, val split, patience 2: Output: Trained reward ensemble ℰ=ℛpred,ψ1,…,ℛpred,ψKE=\R_pred, _1,…,R_pred, _K\, trained MORL policy 3: 4: # Collecting random experiences 5: for i=1i=1 to N do 6: Execute random policy to collect τ=(s0,a0),…,(sT,aT)τ=\(s_0,a_0),…,(s_T,a_T)\ 7: Set l=0l=0 8: for t∈Tt∈ T do 9: With prob. prelp_rel, release Rtsp=∑s=ltrsspR_t^sp= _s=l^tr_s^sp; Set l=tl=t if released 10: end for 11: Segment τ into sub-trajectories τj\ _j\ based on released rewards 12: for all sub-trajectory τj _j do 13: for all (st,at)∈τj(s_t,a_t)∈ _j do 14: Compute features: t=[st,at,tdense] h_t=[s_t,a_t, r^dense_t] 15: end for 16: Add datapoint (tt∈τj,Rsp(τj)) (\ h_t\_t∈ _j,R^sp( _j) ) to dataset D 17: end for 18: end for 19: 20: # Ensemble training 21: for k=1k=1 to K do 22: Split D into trainD_train and valD_val 23: Train ℛpred,ψkR_pred, _k via Eq. 1 with early stopping: 24: ℒ(ψk)=∑τ∈train(∑t∈τℛpred(t;ψk)−Rsp(τ))2L( _k)= _τ _train ( _t∈τR_pred( h_t; _k)-R^sp(τ) )^2 25: end for 26: 27: # RL training with iterative refinement 28: timestep=1timestep=1 29: for cycle=1cycle=1 to IRIR do 30: for t=timestept=timestep to M+timestepM+timestep do 31: Observe st,ats_t,a_t and compute features t h_t 32: rt(k)←ℛpred(t;ψk)r_t^(k) _pred( h_t; _k) for k=1,…,Kk=1,…,K 33: rtsh←1K∑k=1Krt(k)r_t^sh← 1K _k=1^Kr_t^(k) 34: Update RL algorithm using rtshr_t^sh and dense rewards 35: end for 36: # Iterative refinement 37: Collect E expert trajectories newD_new using new policy 38: for all ℛpred,ψk∈ℰR_pred, _k do 39: Update ℛpred,ψkR_pred, _k using new data newD_new 40: end for 41: timestep=ttimestep=t 42: end for Appendix C Proofs This appendix collects all proofs omitted from the main text. C.1 Lemmas This section introduces the general lemmas used to obtain an upper bound on the generalisation gap. Dudley Entropy Integral. The Rademacher complexity can be bounded through the metric entropy of the function class using Dudley’s entropy integral (Dudley, 1967; Bartlett and Mendelson, 2002). Lemma C.1 (Dudley Entropy Integral). For any coarse-scale parameter r∈(0,B)r∈(0,B), the empirical Rademacher complexity satisfies: ℜ^N(ℱ)≤C(∫rBlog∞,1(ℱ,r)Nε)+4rN, R_N(F)≤ C ( _r^B _∞,1(F,r)Nd )+ 4r N, where C>0C>0 is an absolute constant, and ∞,1(ℱ,r)N_∞,1(F,r) is the covering number of ℱF in ℓ∞ _∞ at scale r with respect to N samples This inequality connects the probabilistic complexity (Rademacher complexity) to the geometric complexity of the function class and covering numbers. McDiarmid’s Concentration Inequality. To convert expectation bounds into high-probability statements, we employ McDiarmid’s bounded difference inequality (McDiarmid and others, 1989). Lemma C.2 (McDiarmid’s Concentration Inequality). If each trajectory’s replacement can change any empirical average by at most B/NB/N, then for any t>0t>0: Pr(|supf∈ℱ1N∑i=1N(f(τi)−[f])−[supf∈ℱ1N∑i=1N(f(τi)−[f])]|≥t)≤2exp(−2Nt2B2). ( | _f 1N _i=1^N(f( _i)-E[f])-E [ _f 1N _i=1^N(f( _i)-E[f]) ] |≥ t )≤ 2 (- 2Nt^2B^2 ). This concentration result allows us to bound the deviation between the random supremum and its expectation, completing the pipeline from covering numbers to high-probability uniform generalisation gaps. C.2 Generalisation of Scalarised Returns This section shows that generalisation for an arbitrary scalar return implies guarantees for the scalarised components of the Pareto front. Corollary C.3. Let Π be a policy class equipped with a metric d(⋅,⋅)d(·,·), and let (π;τ)∈ℝL R(π;τ) ^L denote the vector-valued return of policy π on trajectory τ. Following Assumption 5.2: supτ‖(π;τ)−(π~;τ)‖∞≤LRd(π,π~)for all π,π~∈Π. _τ\| R(π;τ)- R( π;τ)\|_∞≤ L_R\,d(π, π) all π, π∈ . For any weight vector ω∈ℝLω ^L define the scalarised return Rω(π;τ)=ω⊤(π;τ)R^ω(π;τ)=ω R(π;τ) and let ℛΠωR_ ^ω be the class of scalarised returns induced by Π . Then for any ε>0 >0, ∞,1(ℛΠω,ε)≤∞,1(Π,ε/LRω),where LRω:=‖ω‖1LR.N_∞,1(R_ ^ω, ) _∞,1\! ( ,\; /L_R^ω ), L_R^ω:=\|ω\|_1\,L_R. In particular, when ‖ω‖1=1\|ω\|_1=1 we have LRω=LRL_R^ω=L_R and the scalarised return class has covering numbers no larger than those of the policy class. Consequently, any complexity reduction obtained by projecting Π to an equivariant subspace (e.g. Πeq _eq) is inherited by the scalarised objective class ℛΠωR_ ^ω. Proof. Fix ω∈ℝLω ^L and let π,π~∈Ππ, π∈ . For any trajectory τ, |Rω(π;τ)−Rω(π~;τ)|=|ω⊤((π;τ)−(π~;τ))|≤∑j=1L|ωj||Rj(π;τ)−Rj(π~;τ)|. |R^ω(π;τ)-R^ω( π;τ) |= |ω ( R(π;τ)- R( π;τ) ) |≤ _j=1^L| _j|\, |R_j(π;τ)-R_j( π;τ) |. Using maxj|Rj(π;τ)−Rj(π~;τ)|=‖(π;τ)−(π~;τ)‖∞, _j|R_j(π;τ)-R_j( π;τ)|=\| R(π;τ)- R( π;τ)\|_∞, we obtain |Rω(π;τ)−Rω(π~;τ)|≤‖ω‖1‖(π;τ)−(π~;τ)‖∞. |R^ω(π;τ)-R^ω( π;τ) |≤\|ω\|_1\,\| R(π;τ)- R( π;τ)\|_∞. Taking the supremum over trajectories and applying the vector Lipschitz assumption yields supτ|Rω(π;τ)−Rω(π~;τ)|≤‖ω‖1LRd(π,π~)=LRωd(π,π~). _τ |R^ω(π;τ)-R^ω( π;τ) |≤\|ω\|_1\,L_R\,d(π, π)=L_R^ω\,d(π, π). Thus the scalarised return map π↦Rω(π;⋅)π R^ω(π;·) is Lipschitz with constant LRω=‖ω‖1LRL_R^ω=\|ω\|_1L_R. Following Lemma C.8, for any ε>0 >0, ∞,1(ℛΠω,ε)≤∞,1(Π,ε/LRω).N_∞,1(R_ ^ω, ) _∞,1\! ( ,\; /L_R^ω ). This proves the displayed inequality. The special case ‖ω‖1=1\|ω\|_1=1 follows immediately. Finally, since the inequality holds for any policy class Π , replacing Π by the equivariant subspace Πeq _eq shows that any complexity reduction ((Πeq,⋅)N( _eq,·) is directly inherited by the scalarised return class. ∎ C.3 Projection to Reflection-Equivariant Subspace Let the full hypothesis space of policies be Π=πϕ:ϕ∈Φ =\ _φ:φ∈ \, where ϕφ represents the neural network parameters and Φ represents the parameter space. The reflection group G=ℤ2=e,gG=Z_2=\e,g\ acts on the state and action spaces via operators LgL_g and KgK_g, respectively. We can map any policy to its equivariant counterpart using an orbit averaging operator :Π→ΠQ: → , defined as: (πϕ)(s) ( _φ)(s) =1|G|∑h∈Gρ(h)πϕ(h−1⋅s) = 1|G| _h∈ Gρ(h) _φ(h^-1· s) =1|G|∑h∈GKh(πϕ(Lh(s))) = 1|G| _h∈ GK_h ( _φ(L_h(s)) ) =12(πϕ(s)+Kg(πϕ(Lg(s)))). = 12 ( _φ(s)+K_g( _φ(L_g(s))) ). (6) Here, ρ(h)ρ(h) is the abstract representation in the action space, and h−1⋅sh^-1· s is the abstract action in the state space. In the second line we replace ρ(h)ρ(h) with the action transformation KhK_h, and h−1⋅sh^-1· s with the state transformation Lh(s)L_h(s). For the reflection group G=ℤ2=e,gG=Z_2=\e,g\, since g=g−1g=g^-1 we may drop the inverse without ambiguity. This operator averages a policy’s output with its reflected-transformed equivalent. The regulariser, ℒeq=s[‖πϕ(Lg(s))−Kg(πϕ(s))‖12]L_eq=E_s[\| _φ(L_g(s))-K_g( _φ(s))\|^2_1], encourages policies to become fixed points of this operator, thereby learning policies within the subspace of equivariant functions, denoted Πeq _eq. The operator Q and the subspace Πeq _eq have several crucial properties, which we state in the following lemmas. Lemma C.4. For any π∈Ππ∈ , the function (π)Q(π) is reflectional equivariant: (π)(Lg(s))=Kg((π)(s)),∀s∈.Q(π)(L_g(s))=K_g(Q(π)(s)), ∀ s . Proof. By direct calculation: (π)(Lg(s)) (π)(L_g(s)) =12(π(Lg(s))+Kg(π(Lg(Lg(s))))) = 12 (π(L_g(s))+K_g(π(L_g(L_g(s)))) ) =12(π(Lg(s))+Kg(π(s))), = 12 (π(L_g(s))+K_g(π(s)) ), Kg(π)(s) K_gQ(π)(s) =12(Kg(π(s))+KgKg(π(Lg(s)))) = 12 (K_g(π(s))+K_gK_g(π(L_g(s))) ) =12(Kg(π(s))+π(Lg(s))), = 12 (K_g(π(s))+π(L_g(s)) ), since KgK_g and LgL_g are involutions. Thus, the two expressions coincide. Therefore (π)Q(π) is equivariant. ∎ Lemma C.5. The operator Q is a projection, meaning it is idempotent: ((π))=(π)Q(Q(π))=Q(π) for any π∈Ππ∈ . Proof. We apply the operator to its own output: ((π))(s) (Q(π))(s) =12((π)(s)+Kg((π)(Lg(s)))). = 12 (Q(π)(s)+K_g(Q(π)(L_g(s))) ). First, evaluating the second term, (π)(Lg(s))Q(π)(L_g(s)): (π)(Lg(s)) (π)(L_g(s)) =12(π(Lg(s))+Kg(π(Lg(Lg(s))))) = 12 (π(L_g(s))+K_g(π(L_g(L_g(s)))) ) =12(π(Lg(s))+Kg(π(s))). = 12 (π(L_g(s))+K_g(π(s)) ). Substituting this back: ((π))(s) (Q(π))(s) =12((π)(s)+Kg[12(π(Lg(s))+Kg(π(s)))]) = 12 (Q(π)(s)+K_g\! [ 12(π(L_g(s))+K_g(π(s))) ] ) =12(π)(s)+14(Kg(π(Lg(s)))+Kg(Kg(π(s)))) = 12Q(π)(s)+ 14 (K_g(π(L_g(s)))+K_g(K_g(π(s))) ) =12(π)(s)+14(Kg(π(Lg(s)))+π(s)) = 12Q(π)(s)+ 14 (K_g(π(L_g(s)))+π(s) ) =12(π)(s)+12(12(π(s)+Kg(π(Lg(s))))) = 12Q(π)(s)+ 12 ( 12(π(s)+K_g(π(L_g(s)))) ) =12(π)(s)+12(π)(s) = 12Q(π)(s)+ 12Q(π)(s) =(π)(s). =Q(π)(s). Thus Q is idempotent. ∎ Lemma C.6. The image of the operator Q coincides with the set of equivariant policies: Im()=(π):π∈Π=ΠeqIm(Q)=\Q(π):π∈ \= _eq. Proof. We establish set equality by showing inclusion in both directions. First inclusion (Im()⊆ΠeqIm(Q) _eq): By Lemma C.4, for any π∈Ππ∈ , the output (π)Q(π) is equivariant. Therefore, every element in the image of Q belongs to Πeq _eq. Second inclusion (Πeq⊆Im() _eq (Q)): Let πeq _eq be any equivariant policy, so πeq∈Πeq _eq∈ _eq. We need to show that πeq _eq can be expressed as (π)Q(π) for some π∈Ππ∈ . Since πeq _eq is equivariant, it satisfies πeq(Lg(s))=Kg(πeq(s)) _eq(L_g(s))=K_g( _eq(s)) for all s. Therefore: (πeq)(s) ( _eq)(s) =12(πeq(s)+Kg(πeq(Lg(s)))) = 12 ( _eq(s)+K_g( _eq(L_g(s))) ) =12(πeq(s)+Kg(Kg(πeq(s))))(by equivariance) = 12 ( _eq(s)+K_g(K_g( _eq(s))) ) (by equivariance) =12(πeq(s)+πeq(s))(since Kg is an involution) = 12 ( _eq(s)+ _eq(s) ) (since $K_g$ is an involution) =πeq(s). = _eq(s). Therefore, πeq=(πeq)∈Im() _eq=Q( _eq) (Q). This shows that equivariant policies are fixed points of Q, which is consistent with Lemma C.5. Since every equivariant policy is its own image under Q, we have Πeq⊆Im() _eq (Q). Combining both inclusions yields Im()=ΠeqIm(Q)= _eq. Therefore Q is surjective onto Πeq _eq. ∎ C.4 Reduced Hypothesis Complexity of Reflection-Equivariant Subspace To prove that the subspace Πeq _eq is less complex, we show that the projection Q is non-expansive, which implies its image has a covering number no larger than the original space. Theorem C.7. The space Πeq _eq has a covering number less than or equal to that of Π . Let ∞,1(ℱ,r)N_∞,1(F,r) be the covering number of a function space ℱF under the l∞,1l_∞,1-distance. Then, ∞,1(Πeq,r)≤∞,1(Π,r)N_∞,1( _eq,r) _∞,1( ,r). Proof. We show that Q is non-expansive. The l∞,1l_∞,1-distance between two policies πϕ _φ and πθ _θ is d(πϕ,πθ)=sups‖πϕ(s)−πθ(s)‖1.d( _φ, _θ)= _s\| _φ(s)- _θ(s)\|_1. The distance between their projections is: d((πϕ),(πθ)) d(Q( _φ),Q( _θ)) =sups‖12(πϕ(s)+Kg(πϕ(Lg(s))))−12(πθ(s)+Kg(πθ(Lg(s))))‖1 = _s \| 12 ( _φ(s)+K_g( _φ(L_g(s))) )- 12 ( _θ(s)+K_g( _θ(L_g(s))) ) \|_1 =12sups‖(πϕ(s)−πθ(s))+Kg(πϕ(Lg(s))−πθ(Lg(s)))‖1. = 12 _s \|( _φ(s)- _θ(s))+K_g( _φ(L_g(s))- _θ(L_g(s))) \|_1. ≤12sups(‖πϕ(s)−πθ(s)‖1+‖Kg(πϕ(Lg(s))−πθ(Lg(s)))‖1). ≤ 12 _s (\| _φ(s)- _θ(s)\|_1+\|K_g( _φ(L_g(s))- _θ(L_g(s)))\|_1 ). ≤12(sups‖πϕ(s)−πθ(s)‖1+sups‖πϕ(Lg(s))−πθ(Lg(s))‖1). ≤ 12 ( _s\| _φ(s)- _θ(s)\|_1+ _s\| _φ(L_g(s))- _θ(L_g(s))\|_1 ). =12(d(πϕ,πθ)+d(πϕ,πθ))=d(πϕ,πθ), = 12 (d( _φ, _θ)+d( _φ, _θ) )=d( _φ, _θ), where we use the triangle inequality, the fact that KgK_g is a norm-preserving isometry, ‖Kg(a)‖1=‖a‖1\|K_g(a)\|_1=\|a\|_1, and that LgL_g is a bijection, which implies that the supremum over s equals the supremum over Lg(s)L_g(s). Hence Q is non-expansive, and a non-expansive surjective map cannot increase the covering number. Following Lemma C.6, (Πeq,r)≤(Π,r)N( _eq,r) ( ,r). ∎ The following lemma links coverings of the policy class (with metric d) to coverings of the induced return class (supremum over trajectories). This is the deterministic Lipschitz step that makes the entropy of returns comparable to the entropy of the policy class. Lemma C.8. For any policy set ⊆ΠP and any ε>0 >0, ∞,1(τ↦R(π;τ):π∈,ε)≤∞,1(,ε/LR),N_∞,1 (\τ R(π;τ):π \, ) _∞,1 (P, /L_R ), where the left covering number is with respect to the sup-norm over trajectories and the right is with respect to d(⋅,⋅)d(·,·). Proof. Let π1,…,πM\ _1,…, _M\ be an ε/LR /L_R-cover of P under d(⋅,⋅)d(·,·). For any π∈π choose j with d(π,πj)≤ε/LRd(π, _j)≤ /L_R. Then for every trajectory τ, |R(π;τ)−R(πj;τ)|≤LRd(π,πj)≤ε,|R(π;τ)-R( _j;τ)|≤ L_Rd(π, _j)≤ , so the set τ↦R(πj;τ)j=1M\τ R( _j;τ)\_j=1^M is an ε -cover of the return-class. Thus, the covering inequality holds. ∎ C.5 Generalisation of Reflection-Equivariant Subspace We now prove a high-probability uniform bound over the equivariant class. Theorem C.9. With ℛΠeq=τ↦R(π;τ):π∈ΠeqR_ _eq=\τ R(π;τ):π∈ _eq\, fix any accuracy parameter r∈(0,B)r∈(0,B) and confidence δ∈(0,1)δ∈(0,1). Then with probability at least 1−δ1-δ, supπ∈Πeq|J(π)−J^N(π)|≤C(∫rBlog∞,1(ℛΠeq,ε)Nε)+8rN+Blog(2/δ)2N, _π∈ _eq|J(π)- J_N(π)|≤ C ( _r^B _∞,1(R_ _eq, )Nd )+ 8r N+B (2/δ)2N, where C is an absolute numeric constant, J(π)J(π) is the population expected return and J^N(π)=1N∑i=1NR(π;τi) J_N(π)= 1N _i=1^NR(π; _i) is the empirical return on N i.i.d. episodes τ1,…,τN _1,…, _N. Proof. Let ℱ=ℛΠeqF=R_ _eq. Following Corollary 5.9, we have: [supf∈ℛΠeq|1N∑i=1N(f(τi)−[f])|]≤2[ℜN(ℛΠeq)].E [ _f _ _eq | 1N _i=1^N(f( _i)-E[f]) | ]≤ 2E [ R_N(R_ _eq) ]. Applying Lemma C.1, for any r>0r>0: [supf∈ℛΠeq|1N∑i=1N(f(τi)−[f])|]≤C(∫rBlog∞,1(ℛΠeq,ε)Nε)+8rN.E [ _f _ _eq | 1N _i=1^N(f( _i)-E[f]) | ]≤ C ( _r^B _∞,1(R_ _eq, )Nd )+ 8r N. (7) Now apply Lemma C.2 to convert the expectation bound into a high-probability statement, with probability at least 1−δ1-δ: supf∈ℛΠeq|1N∑i=1N(f(τi)−[f])|≤[supf∈ℛΠeq|1N∑i=1N(f(τi)−[f])|]+Blog(2/δ)2N. _f _ _eq | 1N _i=1^N(f( _i)-E[f]) | [ _f _ _eq | 1N _i=1^N(f( _i)-E[f]) | ]+B (2/δ)2N. (8) Combining Equations 7 and 8 yields the claimed inequality. ∎ C.6 Generalisatisability of PRISM Lemma C.10. If a policy π satisfies ℒeq≤εeqL_eq≤ _eq, then sups‖Δπ(s)‖1≤εeqpmin. _s\| _π(s)\|_1≤ _eqp_ . Consequently, the sup–ℓ1 _1 distance between π and its orbit projection Q(π)Q(π) satisfies d(π,Q(π))=sups‖π(s)−Q(π)(s)‖1≤εeqpmin.d(π,Q(π))= _s\|π(s)-Q(π)(s)\|_1≤ _eqp_ . Proof. Assume the state space has density dμds(s)≥pmin dμds(s)≥ p_ on the common support. Let s∗s^* be such that ‖Δπ(s∗)‖1=sups‖Δπ(s)‖1\| _π(s^*)\|_1= _s\| _π(s)\|_1. The expectation is: εeq=μ[‖Δπ(s)‖12]=∫‖Δπ(s)‖12μ(s). _eq=E_μ [\| _π(s)\|_1^2 ]= \| _π(s)\|_1^2dμ(s). For any neighbourhood Bδ(s∗)B_δ(s^*) of s∗s^*: εeq≥∫Bδ(s∗)‖Δπ(s)‖12μ(s). _eq≥ _B_δ(s^*)\| _π(s)\|_1^2dμ(s). By continuity of ‖Δπ(⋅)‖1\| _π(·)\|_1 and the density lower bound: ∫Bδ(s∗)‖Δπ(s)‖12μ(s)≥(‖Δπ(s∗)‖1−ϵ)2∫Bδ(s∗)μ(s)≥(‖Δπ(s∗)‖1−ϵ)2pmin⋅vol(Bδ(s∗)), _B_δ(s^*)\| _π(s)\|_1^2dμ(s)≥ (\| _π(s^*)\|_1-ε )^2 _B_δ(s^*)dμ(s)≥ (\| _π(s^*)\|_1-ε )^2p_ ·vol(B_δ(s^*)), for sufficiently small δ and any ϵ>0ε>0. Taking δ→0δ→ 0 and ϵ→0ε→ 0: εeq≥pmin(sups‖Δπ(s)‖1)2. _eq≥ p_ ( _s\| _π(s)\|_1 )^2. Rearranging gives sups‖Δπ(s)‖1≤εeqpmin _s\| _π(s)\|_1≤ _eqp_ . ∎ We can now translate this approximation to a bound on returns and to a covering-number statement. Theorem C.11. Let ξ:=12εeq/pminξ:= 12 _eq/p_ . Then for every policy π, |J(π)−J(Q(π))|≤LR⋅d(π,Q(π))≤LRξ.|J(π)-J(Q(π))|≤ L_R· d(π,Q(π))≤ L_Rξ. Define the approximately reflection-equivariant class Πapprox(εeq):=π∈Π:Leq(π)≤εeq _approx( _eq):=\π∈ :L_eq(π)≤ _eq\. Then every π∈Πapprox(εeq)π∈ _approx( _eq) lies in the sup-ball of radius ξ around Πeq _eq. Consequently, for any target covering radius r>ξr>ξ: N∞,1(Πapprox(εeq),r)≤N∞,1(Πeq,r−ξ).N_∞,1 ( _approx( _eq),r )≤ N_∞,1 ( _eq,r-ξ ). Proof. The first claim is that |J(π)−J(Q(π))|≤LR⋅d(π,Q(π))≤LRξ|J(π)-J(Q(π))|≤ L_R· d(π,Q(π))≤ L_Rξ. First, we establish the LRL_R-Lipschitz property of the expected return J(π)=τ[R(π;τ)]J(π)=E_τ[R(π;τ)]. Using the property from that the return function R is LRL_R-Lipschitz, we have: |J(π)−J(Q(π))| |J(π)-J(Q(π))| =|τ[R(π;τ)−R(Q(π);τ)]| =|E_τ[R(π;τ)-R(Q(π);τ)]| ≤τ[|R(π;τ)−R(Q(π);τ)|] _τ [|R(π;τ)-R(Q(π);τ)| ] ≤τ[LR⋅d(π,Q(π))]=LR⋅d(π,Q(π)). _τ [L_R· d(π,Q(π)) ]=L_R· d(π,Q(π)). Next, we bound the distance d(π,Q(π))d(π,Q(π)). Using the definition of the projection Q(π)Q(π), we find the distance from π to its projection: d(π,Q(π)) d(π,Q(π)) =sups‖π(s)−Q(π)(s)‖1 = _s\|π(s)-Q(π)(s)\|_1 =sups‖π(s)−12(π(s)+Kg(π(Lg(s))))‖1 = _s \|π(s)- 12 (π(s)+K_g(π(L_g(s))) ) \|_1 =12sups‖π(s)−Kg(π(Lg(s)))‖1. = 12 _s \|π(s)-K_g(π(L_g(s))) \|_1. The term inside the norm is equal to the equivariance mismatch Δπ(s′):=π(Lg(s′))−Kg(π(s′)) _π(s ):=π(L_g(s ))-K_g(π(s )) evaluated at s′=Lg(s)s =L_g(s), since LgL_g is an involution. Δπ(Lg(s))=π(Lg(Lg(s)))−Kg(π(Lg(s)))=π(s)−Kg(π(Lg(s))). _π(L_g(s))=π(L_g(L_g(s)))-K_g(π(L_g(s)))=π(s)-K_g(π(L_g(s))). Since LgL_g is a bijection, sups‖Δπ(Lg(s))‖1=sups′‖Δπ(s′)‖1 _s\| _π(L_g(s))\|_1= _s \| _π(s )\|_1. By Lemma C.10, this supremum is bounded by ξ. Therefore: d(π,Q(π))=12sups′‖Δπ(s′)‖1≤ξ.d(π,Q(π))= 12 _s \| _π(s ) \|_1≤ξ. The second claim is that for any radius r>ξr>ξ, we have N∞,1(Πapprox(εeq),r)≤N∞,1(Πeq,r−ξ)N_∞,1 ( _approx( _eq),r )≤ N_∞,1 ( _eq,r-ξ ). We know that for any π∈Πapprox(εeq)π∈ _approx( _eq), its projection Q(π)∈ΠeqQ(π)∈ _eq satisfies d(π,Q(π))≤ξd(π,Q(π))≤ξ. This implies that the set Πapprox(εeq) _approx( _eq) is contained in a ξ-neighbourhood of Πeq _eq. Let πjj=1M\ _j\_j=1^M be a minimal (r−ξ)(r-ξ)-cover for Πeq _eq, where M=N∞,1(Πeq,r−ξ)M=N_∞,1( _eq,r-ξ). Now, consider any policy π∈Πapprox(εeq)π∈ _approx( _eq). There must exist a centre πj _j from our cover such that d(Q(π),πj)≤r−ξd(Q(π), _j)≤ r-ξ. By the triangle inequality, we can bound the distance from π to this centre πj _j: d(π,πj) d(π, _j) ≤d(π,Q(π))+d(Q(π),πj) ≤ d(π,Q(π))+d(Q(π), _j) ≤ξ+(r−ξ)=r. ≤ξ+(r-ξ)=r. This shows that the set πjj=1M\ _j\_j=1^M is an r-cover for Πapprox(εeq) _approx( _eq). Since we have found a valid cover of size M, the size of the minimal cover must be no larger: N∞,1(Πapprox(εeq),r)≤N∞,1(Πeq,r−ξ).N_∞,1 ( _approx( _eq),r )≤ N_∞,1 ( _eq,r-ξ ). ∎ Theorem C.12. With ℛΠeq=τ↦R(π;τ):π∈ΠeqR_ _eq=\τ R(π;τ):π∈ _eq\, fix any accuracy parameter r∈(0,B)r∈(0,B) and confidence δ∈(0,1)δ∈(0,1). Then with probability at least 1−δ1-δ, supπ∈Πapprox(εeq)|J(π)−J^N(π)|≤C(∫rBlog∞,1(ℛΠeq,ε)Nε)+8rN+Blog(2/δ)2N+2LRξ. _π∈ _approx( _eq)|J(π)- J_N(π)|≤ C ( _r^B _∞,1(R_ _eq, )Nd )+ 8r N+B (2/δ)2N+2L_Rξ. Proof. For any policy π∈Πapprox(εeq)π∈ _approx( _eq), we can decompose the generalisation error using the triangle inequality by introducing its exact-equivariant projection Q(π)∈ΠeqQ(π)∈ _eq: |J(π)−J^N(π)|≤|J(π)−J(Q(π))|+|J(Q(π))−J^N(Q(π))|+|J^N(Q(π))−J^N(π)|.|J(π)- J_N(π)|≤|J(π)-J(Q(π))|+|J(Q(π))- J_N(Q(π))|+| J_N(Q(π))- J_N(π)|. We bound each of the three terms on the right-hand side. From Theorem C.11, we have: |J(π)−J(Q(π))|≤LR⋅d(π,Q(π))≤LRξ.|J(π)-J(Q(π))|≤ L_R· d(π,Q(π))≤ L_Rξ. Since the return function R(⋅;τ)R(·;τ) is LRL_R-Lipschitz: |J^N(Q(π))−J^N(π)| | J_N(Q(π))- J_N(π)| =|1N∑i=1N(R(Q(π);τi)−R(π;τi))| = | 1N _i=1^N (R(Q(π); _i)-R(π; _i) ) | ≤1N∑i=1N|R(Q(π);τi)−R(π;τi)| ≤ 1N _i=1^N |R(Q(π); _i)-R(π; _i) | ≤1N∑i=1NLR⋅d(π,Q(π))≤LRξ. ≤ 1N _i=1^NL_R· d(π,Q(π))≤ L_Rξ. The middle term, |J(Q(π))−J^N(Q(π))||J(Q(π))- J_N(Q(π))|, is the generalisation error for an exactly equivariant policy. Combining the bounds, we get: supπ∈Πapprox(εeq)|J(π)−J^N(π)|≤supπ′∈Πeq|J(π′)−J^N(π′)|+LRγ. _π∈ _approx( _eq)|J(π)- J_N(π)|≤ _π ∈ _eq|J(π )- J_N(π )|+L_Rγ. Applying the high-probability bound from Theorem C.9 to the supremum over Πeq _eq yields the final result. ∎ Appendix D Additional Details of Environments This appendix presents the tables on the environments and how the state space is divided into a symmetric and an asymmetric part. First Table 3 highlights the differences between environments in dimension sizes. Tables 4, 5, 6, and 7 show the division for mo-hopper-v5, mo-walker2d-v5, mo-halfcheetah-v5, and mo-swimmer-v5, respectively. The action space is always divided into an empty set for the asymmetric part, and the complete set for the symmetric part. Table 3: Considered MuJoCo environments. State Space Action Space Reward Space Mo-hopper-v5 ∈ℝ11S ^11 ∈ℝ3A ^3 ℛ∈ℝ3R ^3 Mo-walker2d-v5 ∈ℝ17S ^17 ∈ℝ6A ^6 ℛ∈ℝ2R ^2 Mo-halfcheetah-v5 ∈ℝ17S ^17 ∈ℝ6A ^6 ℛ∈ℝ2R ^2 Mo-swimmer-v5 ∈ℝ8S ^8 ∈ℝ2A ^2 ℛ∈ℝ2R ^2 Table 4: Reflectional symmetry partition for mo-hopper-v5 observation space. Index Observation Component Type Symmetry 0 z-coordinate of the torso position Asymmetric 1 angle of the torso angle Asymmetric 2 angle of the thigh joint angle Symmetric 3 angle of the leg joint angle Symmetric 4 angle of the foot joint angle Symmetric 5 velocity of the x-coordinate of the torso velocity Asymmetric 6 velocity of the z-coordinate of the torso velocity Asymmetric 7 angular velocity of the angle of the torso angular velocity Asymmetric 8 angular velocity of the thigh hinge angular velocity Symmetric 9 angular velocity of the leg hinge angular velocity Symmetric 10 angular velocity of the foot hinge angular velocity Symmetric Table 5: Reflectional symmetry partition for mo-walker2d-v5 observation space. Index Observation Component Type Symmetry 0 z-coordinate of the torso position Asymmetric 1 angle of the torso angle Asymmetric 2 angle of the thigh joint angle Symmetric 3 angle of the leg joint angle Symmetric 4 angle of the foot joint angle Symmetric 5 angle of the left thigh joint angle Symmetric 6 angle of the left leg joint angle Symmetric 7 angle of the left foot joint angle Symmetric 8 velocity of the x-coordinate of the torso velocity Asymmetric 9 velocity of the z-coordinate of the torso velocity Asymmetric 10 angular velocity of the angle of the torso angular velocity Asymmetric 11 angular velocity of the thigh hinge angular velocity Symmetric 12 angular velocity of the leg hinge angular velocity Symmetric 13 angular velocity of the foot hinge angular velocity Symmetric 14 angular velocity of the left thigh hinge angular velocity Symmetric 15 angular velocity of the left leg hinge angular velocity Symmetric 16 angular velocity of the left foot hinge angular velocity Symmetric Table 6: Reflectional symmetry partition for mo-halfcheetah-v5 observation space. Index Observation Component Type Symmetry 0 z-coordinate of the front tip position Asymmetric 1 angle of the front tip angle Asymmetric 2 angle of the back thigh angle Symmetric 3 angle of the back shin angle Symmetric 4 angle of the back foot angle Symmetric 5 angle of the front thigh angle Symmetric 6 angle of the front shin angle Symmetric 7 angle of the front foot angle Symmetric 8 velocity of the x-coordinate of front tip velocity Asymmetric 9 velocity of the z-coordinate of front tip velocity Asymmetric 10 angular velocity of the front tip angular velocity Asymmetric 11 angular velocity of the back thigh angular velocity Symmetric 12 angular velocity of the back shin angular velocity Symmetric 13 angular velocity of the back foot angular velocity Symmetric 14 angular velocity of the front thigh angular velocity Symmetric 15 angular velocity of the front shin angular velocity Symmetric 16 angular velocity of the front foot angular velocity Symmetric Table 7: Reflectional symmetry partition for mo-swimmer-v5 observation space. Index Observation Component Type Symmetry 0 angle of the front tip angle Asymmetric 1 angle of the first rotor angle Symmetric 2 angle of the second rotor angle Symmetric 3 velocity of the tip along the x-axis velocity Asymmetric 4 velocity of the tip along the y-axis velocity Symmetric 5 angular velocity of the front tip angular velocity Asymmetric 6 angular velocity of first rotor angular velocity Symmetric 7 angular velocity of second rotor angular velocity Symmetric Appendix E Additional Details of Experimental Settings Evaluation Measures. For the approximated Pareto front, we consider three well-known metrics that investigate the extent of the approximated front. First, we consider hypervolume (HV) (Fonseca et al., 2006), which measures the volume of the objective space dominated by the approximated Pareto front relative to a reference point. A downside of many evaluation measures is that they require domain knowledge about the true underlying Pareto front, whereas HV only considers a reference point without any a priori knowledge, making it ideal to assess the volume of the front. The reference point is typically set to the nadir point or slightly worse, and following Felten et al. (2023), we set it to −100-100 for all objectives and environments. The HV is defined as follows: HV(CS,)=λ(⋃∈CS∈ℝL:⪯),HV(CS, r)=λ ( _ cs∈ CS x ^L: cs x r ), where CS=1,2,…,nCS= cs_1, cs_2,…, cs_n is the coverage set, or the Pareto front approximation, ∈ℝL r ^L is the reference point, ⪯ cs x means csi≤xics_i≤ x_i for all objectives i=1,…,Li=1,…,L, and λ(⋅)λ(·) denotes the Lebesgue measure. Yet, hypervolume values are difficult to interpret, as they do not have a direct link to any notion of value or utility (Hayes et al., 2022). As such, we also consider the Expected Utility Metric (EUM) (Zintgraf et al., 2015), which computes the expected maximum utility across different preference weight vectors, and is defined as follows: EUM(CS,)=1||∑∈max∈CSU(,),EUM(CS,W)= 1|W| _ ω _ cs∈ CSU( ω, cs), where =1,2,…,kW=\ ω_1, ω_2,…, ω_k\ is a set of weight vectors, |||W| is the cardinality of the weight set, U(,)U( ω, cs) is the utility function, which is set to U(,)=⋅=∑i=1Lωi⋅csiU( ω, s)= ω· cs= _i=1^L _i·cs_i. To specifically assess performance with respect to distributional preferences, we also consider one metric designed to evaluate the optimality of the entire return distribution associated with the learned policies (Cai et al., 2023). To be precise, we consider the Variance Objective (VO), which evaluates how well the policy set can balance the trade-off between maximising expected returns and minimising their variance. A set of M random preference vectors is generated, where each vector specifies a different weighting between the expected return and its standard deviation for each objective. The satisfaction score u(pi,πj)u(p_i, _j) for a policy πj _j under preference pip_i is a weighted sum of the expected return [Z(πj)]E[Z( _j)] and the negative standard deviation −Var[Z(πj)]- Var[Z( _j)]. The final metric is the mean score over these preferences, rewarding policies that achieve high expected returns with low variance: VO(Π,pii=1M)=1M∑i=1Mmaxπj∈Πu(pi,πj).VO( ,\p_i\_i=1^M)= 1M _i=1^M _ _j∈ u(p_i, _j). Hyperparameters. Due to time, computational limitations, and the excessive number of hyperparameters, we do not perform an extensive hyperparameter tuning process. Below are the used hyperparameters. All hyperparameters that are not mentioned below are set to their default value. The probability of releasing sparse rewards prelp_rel is always set to a one-hot vector, where sparsity is imposed on the reward dimension related to moving forward. Since the main goal is to move forward, imposing sparsity on this channel should make it a more difficult task for the reward shaping model. Furthermore, we deal with extreme heterogeneous sparsity, where most channels exhibit regular rewards, but one channel only releases a reward at the end of an episode, making it more difficult for the model to link certain states and actions to the observed cumulative reward. The hyperparameters in Table 8 for ReSymNet are identical for each environment. The advantage of using the same hyperparameters for each environment is that if one configuration performs well everywhere, it could indicate that the proposed method is inherently stable, especially given the noted diversity between the considered environments. However, this does come at a cost of potentially suboptimal performance per environment. Table 8: Hyperparameters for ReSymNet. PRISM Initial collection N 1000 Expert collection E 1000 Number of refinements IRIR 2 Timesteps per cycle M 100,000 Epochs 1000 Learning rate 0.005 Learning rate scheduler Exponential Learning rate decay 0.99 Ensemble size |ℰ||E| 3 Hidden dimension 256 Dropout 0.3 Initialisation Kaiman (He et al., 2015) Validation split 0.2 Patience 20 Batch size 32 The hyperparameter controlling the symmetry loss differs per environment, since some environments require strict equivariance, whereas others require a more flexible approach. Table 9 shows the used values. Table 9: SymReg hyperparameter. Mo-hopper-v5 Mo-walker2d-v5 Mo-halfcheetah-v5 Mo-swimmer-v5 λ 0.01 1 0.01 0.005 Appendix F Pareto Fronts Figure 6 shows the approximated Pareto fronts. The results demonstrate that shaped rewards yield superior performance, covering a wider and more optimal range of the objective space compared to dense and sparse rewards. (a) Mo-hopper-v5 (b) Mo-walker2d-v5 (c) Mo-halfcheetah-v5 (d) Mo-swimmer-v5 Figure 6: The approximated Pareto front for dense rewards (blue dots), sparse rewards (orange dots), and shaped rewards (green dots). Sparsity is imposed on the first reward objective. Appendix G Ablation Study Tables 10 and 11 report the obtained values for the ablation study. Results are again averaged over ten trials, similar to the main experiments. Environment Metric PRISM w/o residual w/o dense rewards w/o ensemble w/o refinement w/o loss Mo-hopper-v5 HV (×107× 10^7) 1.58 ± 0.05 1.29 ± 0.09 1.38 ± 0.11 1.38 ± 0.08 1.55 ± 0.04 1.42 ± 0.07 EUM 147.43 ± 2.61 128.40 ± 6.06 134.67 ± 6.89 135.28 ± 4.91 145.89 ± 2.73 137.85 ± 4.22 VO 66.66 ± 1.40 58.61 ± 2.71 61.21 ± 3.03 61.51 ± 2.19 66.54 ± 1.34 62.71 ± 1.83 Mo-walker2d-v5 HV (×104× 10^4) 4.77 ± 0.07 4.65 ± 0.11 4.66 ± 0.06 4.60 ± 0.08 4.60 ± 0.09 4.58 ± 0.13 EUM 120.43 ± 1.64 114.33 ± 2.48 116.83 ± 1.65 113.79 ± 2.02 114.98 ± 2.84 112.77 ± 3.01 VO 59.35 ± 0.80 56.46 ± 1.21 57.67 ± 0.73 56.19 ± 0.97 57.03 ± 1.42 55.59 ± 1.44 Mo-halfcheetah-v5 HV (×104× 10^4) 2.25 ± 0.18 1.95 ± 0.20 2.08 ± 0.21 1.91 ± 0.19 2.23 ± 0.18 1.90 ± 0.19 EUM 89.94 ± 15.33 73.06 ± 16.57 82.24 ± 16.97 81.60 ± 17.65 92.68 ± 14.79 71.12 ± 16.91 VO 40.72 ± 7.02 32.99 ± 7.65 37.31 ± 7.99 36.76 ± 8.06 42.28 ± 6.85 32.12 ± 7.75 Mo-swimmer-v5 HV (×104× 10^4) 1.21 ± 0.00 1.21 ± 0.00 1.20 ± 0.00 1.20 ± 0.00 1.21 ± 0.00 1.20 ± 0.00 EUM 9.44 ± 0.14 9.39 ± 0.15 9.07 ± 0.11 9.25 ± 0.13 9.46 ± 0.13 9.35 ± 0.14 VO 4.24 ± 0.07 4.20 ± 0.08 4.09 ± 0.05 4.15 ± 0.08 4.24 ± 0.07 4.24 ± 0.07 Table 10: PRISM ablation study results. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. w/o is the abbreviation of without. The largest values are in bold font. Table 11: ReSymNet ablation study results. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. w/o is the abbreviation of without. Environment Metric uniform random Mo-hopper-v5 HV (×107× 10^7) 1.38 ± 0.08 0.49 ± 0.06 EUM 135.19 ± 5.30 65.22 ± 6.63 VO 63.90 ± 2.34 29.62 ± 3.68 Mo-walker2d-v5 HV (×104× 10^4) 4.67 ± 0.07 1.18 ± 0.10 EUM 116.72 ± 2.11 16.52 ± 4.98 VO 56.22 ± 1.01 3.77 ± 2.46 Mo-halfcheetah-v5 HV (×104× 10^4) 0.98 ± 0.00 0.78 ± 0.05 EUM -1.34 ± 0.39 -10.52 ± 2.67 VO -0.85 ± 0.20 -6.51 ± 1.48 Mo-swimmer-v5 HV (×104× 10^4) 1.09 ± 0.01 1.10 ± 0.02 EUM 4.37 ± 0.69 3.75 ± 0.87 VO 1.56 ± 0.33 1.06 ± 0.40 Appendix H Generalisability H.1 Sparsity on Other Objectives We further investigate the robustness of PRISM by inverting the sparsity setting: we maintain the forward velocity reward as dense but make the control cost objective sparse. Table 12 shows that, without hyperparameter tuning, PRISM handles this problem much better than the baselines. Table 12: Experimental results on the control cost objective. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font. Environment Metric Oracle Baseline PRISM Mo-hopper-v5 HV (×107× 10^7) 1.30 ± 0.13 1.19 ± 0.10 1.51 ± 0.11 EUM 129.04 ± 7.96 124.82 ± 7.21 142.89 ± 7.38 VO 59.07 ± 3.45 56.21 ± 3.20 67.58 ± 3.31 Mo-walker2d-v5 HV (×104× 10^4) 4.21 ± 0.11 3.16 ± 0.13 4.59 ± 0.14 EUM 107.58 ± 2.86 85.95 ± 3.27 114.62 ± 2.80 VO 53.22 ± 1.39 41.29 ± 1.49 54.84 ± 1.25 Mo-halfcheetah-v5 HV (×104× 10^4) 1.70 ± 0.20 0.00 ± 0.00 1.72 ± 0.19 EUM 81.29 ± 21.85 -101.49 ± 3.23 76.50 ± 20.85 VO 36.84 ± 10.06 -56.26 ± 1.63 31.27 ± 8.68 Mo-swimmer-v5 HV (×104× 10^4) 1.21 ± 0.00 1.05 ± 0.02 1.21 ± 0.01 EUM 9.41 ± 0.12 1.50 ± 1.00 9.32 ± 0.19 VO 4.22 ± 0.08 -0.61 ± 0.68 3.95 ± 0.08 For mo-hopper-v5, PRISM improves HV by 16% over the oracle (1.51×1071.51× 10^7 compared to 1.30×1071.30× 10^7) and 27% over the baseline. Similar gains are observed for mo-walker2d-v5, where PRISM achieves a 9% HV improvement over the oracle and 45% over the baseline. Notably, in mo-halfcheetah-v5, the baseline suffers a collapse (HV of 0.000.00), whereas PRISM recovers the performance to exceed the oracle (1.72×1041.72× 10^4 against 1.70×1041.70× 10^4). These improvements imply that PRISM effectively reconstructs the dense penalty signal, preventing the agent from exploiting the delay to maximise velocity at the cost of extreme energy inefficiency. Improvements in EUM follow the same trend, with mo-walker2d-v5 showing an increase of roughly 33% compared to the baseline (114.62114.62 vs 85.9585.95). On distributional metrics, PRISM delivers more consistent performance than the baseline. In mo-swimmer-v5, the baseline’s VO drops to −0.61-0.61, indicating high instability, whereas PRISM achieves 3.953.95, comparable to the oracle (4.224.22). These gains are crucial because they indicate that PRISM produces Pareto fronts that are not only high-performing but also balanced and robust, effectively mitigating the high-variance behaviour from the baseline. H.2 Sensitivity to Sparsity Figure 7 demonstrates that PRISM maintains robust performance across varying levels of reward sparsity. While performance is generally consistent, we observe minor fluctuations at intermediate values (e.g., prel=0.2p_rel=0.2 in mo-hopper-v5 and mo-walker2d-v5). Two key factors explain this behaviour: (1) PRISM was hyperparameter-tuned specifically for the extreme sparsity setting (prel=0p_rel=0), which is the most challenging MORL scenario. We utilised a fixed set of hyperparameters across all experiments to demonstrate method stability rather than optimising for each sparsity level, and (2) increasing prelp_rel increases the number of available reward signals (data points) per episode. Since ReSymNet was calibrated for the data-scarce sparse setting, the increase of supervision targets at higher prelp_rel levels changes optimisation dynamics, leading to temporary instability. Despite these factors, PRISM consistently recovers high performance, proving its capability to handle heterogeneous reward structures without requiring specific tuning for denser environments. (a) Mo-hopper-v5 (b) Mo-walker2d-v5 (c) Mo-halfcheetah-v5 (d) Mo-swimmer-v5 Figure 7: The obtained hypervolume for various levels of sparsity for PRISM. H.3 Sensitivity to MORL Algorithms To demonstrate that PRISM is a model-agnostic framework not limited to specific architectures, we evaluated its performance using GPI-PD (Generalised Policy Improvement with Linear Dynamics) (Alegre et al., 2023) as an alternative backbone to CAPQL. Table 13 confirms that PRISM remains highly effective, consistently outperforming the sparse baseline and obtaining near-oracle performance. Table 13: Experimental results of GPI-PD. We report the average hypervolume (HV), Expected Utility Metric (EUM), and Variance Objective (VO) over 10 trials, with the standard error shown in grey. The largest (best) values are in bold font. Environment Metric Oracle Baseline PRISM Mo-hopper-v5 HV (×107× 10^7) 1.65 ± 0.10 0.67 ± 0.04 1.65 ± 0.07 EUM 151.45 ± 5.87 85.87 ± 3.17 148.19 ± 4.26 VO 72.26 ± 2.90 41.21 ± 1.44 70.24 ± 2.51 Mo-walker2d-v5 HV (×104× 10^4) 5.93 ± 0.10 3.20 ± 0.23 5.61 ± 0.10 EUM 141.88 ± 2.38 76.41 ± 6.47 132.67 ± 2.26 VO 67.63 ± 1.17 35.64 ± 3.91 63.19 ± 1.75 Mo-halfcheetah-v5 HV (×104× 10^4) 1.80 ± 0.22 1.00 ± 0.02 2.24 ± 0.16 EUM 164.75 ± 14.21 -1.31 ± 0.54 99.89 ± 8.06 VO 73.90 ± 7.05 -1.14 ± 0.31 40.74 ± 5.17 Mo-swimmer-v5 HV (×104× 10^4) 1.23 ± 0.01 1.12 ± 0.01 1.22 ± 0.00 EUM 9.68 ± 0.17 5.17 ± 0.58 9.56 ± 0.13 VO 4.23 ± 0.14 2.18 ± 0.39 4.37 ± 0.18 In mo-hopper-v5, PRISM achieves an HV of 1.65×1071.65× 10^7, matching the oracle exactly and far exceeding the baseline (0.67×1070.67× 10^7). This trend of near-perfect recovery is consistent across mo-walker2d-v5 and mo-swimmer-v5. This indicates that the shaped rewards generated by ReSymNet are robust enough to guide different policy optimisation mechanisms effectively. In mo-halfcheetah-v5, PRISM achieves a significantly higher HV (2.242.24) compared to the oracle (1.801.80). Notably, these results were obtained with minimal hyperparameter tuning due to computational constraints. While this lack of fine-tuning explains the slight gap in EUM/VO metrics for mo-halfcheetah-v5 compared to the oracle, the method’s ability to achieve such strong results with a completely different backbone highlights PRISM’s inherent stability and generalisability. Appendix I Declaration on Large Language Models Large Language Models (LLMs) were used for (1) polishing the wording of the manuscript for clarity and readability, (2) brainstorming about algorithm names and their abbreviations, and (3) searching for algorithms for consideration in the preliminary stage.