Paper deep dive
It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learning
Liam P. H. Mertens, Lucas N. Alegre, Florent Delgrange, Diederik M. Roijers, Ann Nowé, Peter Vamplew
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/29/2026, 3:23:02 AM
Summary
The paper argues that existing multi-objective reinforcement learning (MORL) approaches—Scalarised Expected Returns (SER), Expected Scalarised Returns (ESR), and Successor Features (SF)—are insufficient because they address non-linear utility on only one timescale. The authors demonstrate that real-world problems, such as toxicity management, involve simultaneous effects across multiple timescales (acute, sub-acute, chronic). They identify a gap in the literature for methods that optimize for these simultaneous, multi-timescale non-linear utility effects.
Entities (10)
Relation Signals (8)
SER → isinsufficientfor → Multi-timescale Problems
confidence 95% · none of them take into account that different effects happening on different timescales can happen within the same decision problem.
Successor Features → isinsufficientfor → Multi-timescale Problems
confidence 95% · none of them take into account that different effects happening on different timescales can happen within the same decision problem.
ESR → isinsufficientfor → Multi-timescale Problems
confidence 95% · none of them take into account that different effects happening on different timescales can happen within the same decision problem.
Toxicity → exhibits → Non-linear Utility
confidence 92% · Toxicity effects typically follow a non-linear dose-response curve, and thus leads to non-linear utility.
Successor Features → addressestimescale → Acute Exposure
confidence 90% · For acute exposure... We could model this using successor features, by directly considering the toxic response—per timestep—as a state feature, corresponding to RSR.
ESR → addressestimescale → Sub-acute Exposure
confidence 90% · For sub-acute exposure... In RL terms this would correspond to expected scalarised returns (ESR)
SER → addressestimescale → Chronic Exposure
confidence 90% · For chronic exposure... this would correspond to multi-objective RL (MORL) under the scalarised expected returns (SER) criterion
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Time is of the essence when dealing with multiple reward signals and non-linear utility. In this paper we argue that the current main approaches in multi-objectiveRL (SER and ESR), and successor features, are insufficient. While each approach deals with non-linear effects on user utility on different timescales, none of them take into account that different effects happening on different timescales can happen within the same decision problem. We motivate that this can indeed be the case by an example, both intuitively and numerically, leading to a new perspective, and a significant and non-trivial gap in the literature.
Tags
Links
- Source: https://arxiv.org/abs/2608.25723v1
- Canonical: https://arxiv.org/abs/2608.25723v1
Trouble viewing inline? Open PDF directly →
Full Text
50,370 characters extracted from source content.
Expand or collapse full text
It’s a matter of timescale: non-linear utility in successor features and multi-objective planning and learning Thanks: Presented at the Multi-Objective Decision Making workshop at IJCAI, 2026 Liam P.H. Mertens Affiliation: AI Lab, Vrije Universiteit Brussel, Belgium Email: liam.phi.h.mertens Lucas N. Alegre Affiliation: Institute of Informatics - Federal University of Rio Grande do Sul Email: florent.delgrange Florent Delgrange Affiliation: AI Lab, Vrije Universiteit Brussel, Belgium Email: diederik.roijers Diederik M. Roijers Affiliation: AI Lab, Vrije Universiteit Brussel, Belgium Email: ann.nowe@vub.be Ann Nowé Affiliation: AI Lab, Vrije Universiteit Brussel, Belgium Email: lnalegre@inf.ufrgs.br Peter Vamplew Affiliation: Federation University Australia Email: p.vamplew@federation.edu.au Abstract Time is of the essence when dealing with multiple reward signals and non-linear utility. In this paper we argue that the current main approaches in multi-objective RL (SER and ESR), and successor features, are insufficient. While each approach deals with non-linear effects on user utility on different timescales, none of them take into account that different effects happening on different timescales can happen within the same decision problem. We motivate that this can indeed be the case by an example, both intuitively and numerically, leading to a new perspective, and a significant and non-trivial gap in the literature. 1 Introduction Planning and reinforcement learning agents, have been applied successfully in many domains. Many such agents rely on a scalar additive reward structure. This can be a highly limiting assumption in domains where distinct trade-offs between conflicting objectives need to be made. Instead, multi-objective planning and reinforcement learning methods consider multiple reward signals as a reward vector, with each component determining an objective that forms a part of a trade-off Hayes et al. (2022); Vamplew et al. (2022b). These reward vectors can then be accrued over time and then scalarised using a possibly non-linear utility function. Multi-objective RL considers the utility of multiple accrued reward components over time. However, there are two ways of doing so: taking the expectation over accrued reward vectors (called the expected return or value) and considering the utility of this value, computed by applying a scalarisation function to the value vector. This is known as scalarised expected returns (SER) White (1982); Roijers et al. (2013), and is perhaps the most prevalent way to consider utility. The underlying assumption here is that there is enough time for the expected returns to materialise, and that the utility is derived from this average. For example, consider the process mapping of a video editing application Piscitelli (2014). A company that processes video editing jobs on their servers would processes large numbers of videos each day, and they would indeed be interested in the expected return of this process that consists of the average power consumption, average throughput, and average latency. The second option is to accrue the rewards, apply the utility function on the resulting returns vector, and then take the expectation. This is known as expected scalarised returns (ESR) Roijers et al. (2018). This would correspond to a scenario where the utility is derived from one trajectory. For example, imagine a patient that is undergoing treatment for an illness. That person will only undergo the treatment once, and therefore be interested in the expected utility of their own treatment outcome (not the utility of the average outcome) and thus will not have enough time to let the expected vector return materialise. For linear utility functions, ESR and SER are equivalent due to the distribution of the expectancy over the weighted sum. This is not the case under non-linear utility. An alternative approach to multi-objective optimisation is the successor feature (SF) approach Barreto et al. (2017); Zhu et al. (2024). Unlike SER and ESR, which apply non-linear utility functions to vector returns only after they have been accrued over one or multiple trajectories, the SF approach evaluates non-linear utility at the level of individual states or transitions before any accumulation occurs, represented as a feature vector with components the non-linear successor features of these states/transitions. This corresponds to a scenario where utility effects happen on an immediate, per-step timescale. The feature vector is then linearly scalarised, which corresponds to planning and learning with a scalarised reward function in terms of multi-objective learning and planning. For example, imagine a car that hits something. The utility of that event may depend non-linearly on the force of impact, i.e., if you hit an obstacle very hard the risk of bodily injury increases non-linearly. These injury risks stack up over time, but it is not so that many subsequent low-force impacts accumulate to a single high-force one. All three perspectives—SER, ESR, and the successor feature approach—make sense, and are motivated by different scenarios where the utility effects happen on different timescales. As such, it seems like we have three sets of instruments to tackle problems that deal with non-linear effects on user utility, each on its own applicable timescale. There is however a key issue with this: the non-linear effects on user utility might be happening on different timescales within the same decision problem. To motivate this we take a (simplified) look at the domain of toxicity in Section 3, where toxic exposure can have adverse effects both immediately and on intermediate and on longer timescales. We work out a simplified numerical example to show that taking an SER, ESR, and SF approach lead to different policies with different outcome distributions. Furthermore, taking the perspective of adverse effects that happen on three different timescales simultaneously for this problem leads to different policies and different outcome distributions still. In our opinion, this indicates that there is a clear gap in the multi-objective planning and reinforcement learning literature. So far we have not optimised for the effects on utility of rewards at different timescales simultaneously. In order to bridge this gap, we believe that we should redefine how we look at multi-objective problems, and develop new methodologies and algorithms to deal with this new perspective. In Section 6 we will discuss possible ways forwards. 2 Background In RL, the framework for decision making is usually formalised as a Markov decision process (MDP). We care about scenarios where rewards are not necessarily scalar and where objectives may be competing and lead to trade-offs, giving rise to the notion of multi-objective optimisation in an MDP. Formally, a multi-objective Markov decision process (MOMDP) is defined as a tuple M=(,,p,,μ,γ)M=(S,A,p, r,μ,γ), where S is the state space, A is the action space, p(⋅∣s,a)∈Δ()p (· s,a )∈ (S ) is the state transition function, (s,a)∈ℝm r(s,a) ^m is a multi-objective reward function with m objectives, μ is initial state distribution, and γ∈[0,1)γ∈[0,1) is a discount factor for future rewards. A policy is a function mapping states to distributions over actions, π:→Δ()π → (A ), which intuitively prescribes which action an agent should select in each situation.11 1 For the different metrics defined in Sec. 2.1, we generally need both memory-based and stochastic policies. Memory can always be encoded into the state space. In terms of Pareto-optimality, deterministic policies may be used as the vertices of the related convex hull Roijers et al. (2013), but stochastic policies are required to fill the gaps on the curve (via a carefully designed mixture of deterministic ones Vamplew et al. (2009) or adding concave terms do the rewards Lu et al. (2022)). We define the multi-objective return of a policy π as a random variable π≔∑t=0∞γtt G^π _t=0^∞γ^t R_t from trajectories to vectors in ℝmR^m, where the trajectories are drawn as S0∼μS_0 μ, At∼π(⋅∣St)A_t π (· S_t ), t=(St,At) R_t= r (S_t,A_t ), and St+1∼p(⋅∣St,At)S_t+1 p (· S_t,A_t ) for all t≥0t≥ 0. The multi-objective value function of a policy π is the expected return, π(s)=π[π∣S0=s] v^π(s)=E_π[ G^π S_0=s]. 2.1 Rewards in MORL Knowing the multi-objective value functions of several policies is not enough, by itself, to decide which policy should be preferred. The value of a policy is a vector: one component may improve while another one gets worse. To turn such trade-offs into a decision, MORL therefore requires two ingredients: a utility function u:ℝm↦ℝu:R^m , which assigns a scalar utility to a reward or return vector, and an optimisation criterion, which specifies when this utility is applied. The placement of u matters. It determines whether we care about the utility of an average outcome, the average utility of whole trajectories, or the cumulative utility of each individual reward. These choices coincide for linear utility functions, but they can lead to different optimal policies when u is non-linear. Two optimisation settings have been widely used in the MORL literature: Scalarised Expected Returns, and Expected Scalarised Returns. 2.1.1 Scalarised Expected Returns In the scalarised expected returns (SER) setting, the goal is to find π∗∈argmaxπu(π)=argmaxπu(π[π]),π^*∈ _πu( v^π)= _πu(E_π[ G^π]), (1) given any utility function u:ℝm↦ℝu:R^m . SER first averages the return vector of a policy and only then applies the utility function. Intuitively, it evaluates a policy by its long-run mean performance across objectives. This is natural when the same policy is used many times and the decision maker mainly cares about aggregate performance over a population of runs. 2.1.2 Expected Scalarised Returns In the expected scalarised returns (ESR) setting, the goal is to find π∗∈argmaxππ[u(π)],π^*∈ _πE_π [u ( G^π ) ], (2) given any utility function u:ℝm↦ℝu:R^m . ESR applies the utility function to the complete return vector of each trajectory and only then averages across trajectories. It therefore captures how good an individual realised outcome is before taking expectations. This distinction is important when the utility function represents risk, fairness, saturation, or any other non-linear preference over complete outcomes. 2.1.3 Returns of Scalarised Rewards A third possible setting is the approach used by default in single-objective RL: apply the utility function directly to the reward vector at each time step, and then maximise the expected return of the resulting scalar rewards. In returns of scalarised rewards (RSR), the scalar return is Guπ=∑t=0∞γtu(t),G_u^π= _t=0^∞γ^tu( R_t), (3) and the goal is to find π∗∈argmaxππ[Guπ].π^*∈ _πE_π [G_u^π ]. (4) RSR treats each time step as a separate trade-off. It is appropriate when the utility of an immediate reward vector is meaningful on its own, for example when one wants to penalise dangerous or undesirable instantaneous combinations of objectives, rather than only their cumulative effect over a full trajectory. 2.1.4 Utility Functions If u is linear (i.e., a simple weighted sum of the components of the reward or return vector), then the same policy will be optimal for SER, ESR, and RSR. However if a non-linear function is used for u (e.g. Chebyshev distance), then a different policy may be optimal for each of these settings. Importantly, this implies that we may have different optimal solution sets for each problem formulation Hayes et al. (2022). 2.2 Rewards in Succesor Features-based RL The Successor Features (SFs) framework Barreto et al. (2017); Barreto et al. (2020) was originally proposed to model multi-task RL problems, where each task is defined by a different reward function. It assumes that the reward function of an MDP is given by a linear function over a set of d features ϕ(s,a)∈ℝd φ(s,a) ^d : r(s,a)=ϕ(s,a)⋅.r_ w(s,a)= φ(s,a)· w. (5) The SFs of a policy π represents the expected sum of the features: π(s,a)=π[∑t=0∞γtϕt|S0=s,A0=a] ψ^π(s,a)=E_π[ _t=0^∞γ^t φ_t|S_0=s,A_0=a]. Intuitively, we can see this setting as an instantiation of the RSR setting where the vector reward t R_t is replaced by the feature vector ϕt _t, and u is a linear function u(ϕt)=ϕt⋅u_ w( _t)= _t· w. Alegre et al. (2022) showed that, in this case, SFs are equivalent to a multi-objective value function, and the SFs framework is equivalent to MORL under linear utility functions. 2.2.1 Beyond Linear Rewards in SFs The expressiveness of SF-based approaches depends heavily on the definition of the features ϕφ. Unlike the vector rewards in MORL that are typically specified by domain experts, the features ϕφ can instead be learned by an agent. For example, these features can be learned via optimization of supervised criteria, e.g., to approximate a set of reward functions Barreto et al. (2018), or via unsupervised objectives, e.g., to approximate the Laplacian of the MDP Chandrasekar and Machado (2025) or to identify clusters of the state space Bagot et al. (2025). Importantly, if the feature vector is a one-hot encoding indicating the current state of the agent in the state space, ϕ(s)=[S1(s),⋯,Sd(s)]⊤φ (s )= [ 1_S_1(s),·s, 1_S_d(s) ] where d=||d=|S|, then any reward function, r, can be represented as a vector ∈ℝ|| w ^|S|, where =[r(s1)…r(s||)]⊤ w=[r(s_1)\ ...\ r(s_|S|)] . This implies that non-linear functions of the reward vector (Section 2.1.3) would be unnecessary. The Forward Backward (FB) representation Touati and Ollivier (2021); Touati et al. (2022) fits in this context by learning a high-dimensional representation that, under some dimensionality constraints/assumptions, can express the optimal policy for any possible reward function. We observe a tendency for SF-based approaches to increase their expressiveness (the family of problems they can solve) by discovering more powerful feature representations, while maintaining a linear utility function. In contrast, in the MORL literature, most works typically consider a low-dimensional reward vector, but expand the family of problems that MORL techniques can solve by employing non-linear utility functions. We believe that, although orthogonal, it makes sense to combine both directions when tackling some real-world problems. That being said, the multi-objective RSR criterion with non-linear u, can be implemented in successor features by simply appending the existing features ϕt _t by a non-linear combination of these features: ϕt←ϕt∪u~(ϕt) _t← _t∪\ u( _t)\.22 2 This assumes all features in ϕt _t are reward components. If this is not the case we would instead have to apply a composite function u∘r~u r, where r~ r is the translation of the successor features to the rewards in each objective. This feature then gets accrued over time as the utility, and a weight of 11 is placed on this new feature. SER and ESR in general however, cannot be implemented using traditional successor feature approaches, as after accruing rewards over time, only linear combinations are allowed. 3 Motivating Example: Toxicity As mentioned above, the optimisation criteria (SER, ESR, and RSR) say something about the impact of—reward or feature—signals on user utility. This happens at different timescales: across multiple episodes for SER, over one episode for ESR, and at a single-timestep for successor features/RSR. All of these make sense in one setting or another. However, we would like to argue here, that they can also make sense all at once. It is just a matter of timescale. Let us turn to the example of exposure to substances and their toxicity. Toxicity effects typically follow a non-linear dose-response curve, and thus leads to non-linear utility. A high dosage in a short time can lead to severe immediate adverse effects. This is known as acute exposure (one dose, or accumulating exposure up to 24 hours), and could correspond to a timestep-based application of a non-linear (utility function) before the rewards/penalties are added up at all. However, just because a substance does not hurt you when you are exposed to it at a lower dosage once, does not mean you can keep getting exposed to it. In fact, in toxicology the following exposure time-lengths are often distinguished sub-acute (24 hours to 28 days), sub-chronic (less than 90 days), and chronic exposure Denny and Stewart (2024). Let’s consider a small company with three certified people to work with toxic substances. These workers have different experience levels, and therefore also different risk exposure risks for different tasks. Let’s imagine we aim to optimise the monthly work schedule, with an employee taking the same work station for a day, doing one task. For simplicity lets imagine there are 3 tasks in total that are the same each day. Let’s take the dose each of the three employees get as three separate reward functions. What happens when we take different types of exposure into account? 3.1 Intuition First let us see what would happen if we would take different types of toxic exposure into account in an intuitive manner, before looking at a numerical example. Chronic exposure. For chronic exposure, we would consider the long-term damage of being exposed to a toxin in small amounts over a long period of time. This would mean we ought to look at the exposure over the course of multiple months or even years. In RL terms this would correspond to multi-objective RL (MORL) under the scalarised expected returns (SER) criterion Hayes et al. (2022). Furthermore, because the response to toxins are typically sigmoid shaped—meaning that the toxic effects start increasing slowly for low dosages and then increase ever faster before flattening off again—this means we probably just have to spread out the average toxic exposure over all employees as much as possible so that each individual’s long-term average dose is low, leading to the least averse effects. Sub-acute exposure. For sub-acute exposure, we would be interested in the total dosage an employee would get over the entire month. This would require us look at the possible outcomes for each month, as a probability distribution, and look at the expected adverse effects of the cumulative dosages for the entire month (again by applying a probably sigmoidal response curve). It would probably also be important to track closely the dosage the employee already received up until a given day, and dynamically adapt our schedule to that. In RL terms this would correspond to expected scalarised returns (ESR) Roijers et al. (2018). Acute exposure. For acute exposure, we would be interested in the immediate effects of a high dose within 24 hours. This would entail considering the dosage an employee gets as a result of a task, and then applying the response curve directly to that. Taking the average of that would correspond to the average risk of acute exposure on a daily basis. We could model this using successor features, by directly considering the toxic response—per timestep—as a state feature, corresponding to RSR. So what would a responsible employer have to do now? In the decision theoretic literature we seem to have the choice between SER, ESR, or RSR as an optimisation criterion. Each of these would imply only one of the different timescales for toxic exposure. But that does not suffice; a responsible employer should have their eye on every type of toxic exposure, not just one. We should optimise for all at once. We argue that this indicates a critical gap in the literature, i.e., different problems may require us to look at (cumulative) rewards on different timescales, but importantly different timescales may have to be taken into account at the same time. 3.2 Numerical example Now we turn our example into a numerical decision problem, and will show that different policies are optimal for different settings. The decision problem consists of 3 employees of varying skill levels and 3 tasks with varying toxicity (table 1). The goal is to assign the employees optimally over time given a possibly non-linear utility function that limits the maximal toxic dosage per employee either per day, per month, over multiple months or over all possible time frames simultaneously. This problem can be modeled as a MOMDP with state space S defined as all permutations of employees, with their positions indicating their assigned task (e.g. [emp1, emp2, emp3] indicates that employee 1 is assigned to task 1, employee 2 to task 2 and employee 3 to task 3) and action space A as the finite discrete set of permutation indices such that ||=3!|A|=3!. Note that the state needs to be augmented with the accrued reward in order to accommodate non-linear utility under ESR and SER Vamplew et al. (2024); Reymond et al. (2023); Roijers et al. (2018). The vectorial reward function is defined as r(st,at)=[−r1(st,at),−r2(st,at),−r3(st,at)]r(s_t,a_t)=[-r_1(s_t,a_t),-r_2(s_t,a_t),-r_3(s_t,a_t)], where rir_i indicates the toxic dose that each employee is exposed to on a given day. These doses are defined as ri=max(0,(1−Si)⋅Tj+ε)r_i= (0,(1-S_i)· T_j+ ), with TjT_j being the toxicity of the task currently assigned to employee i and SiS_i the corresponding skill level of that employee. The doses are slightly stochastic, subject to the noise ε∼(0,10) (0,10). We note that in a successor feature formulation, the received dosages would be appended to the state, rather than given as a reward vector. We consider episodes of 30 days long to simulate work schedules over a month. Table 1: The environment parameters. Each employee initially starts at their respective task (OPENs0=[0,1,2])s_0=[0,1,2]). Skill level Employee 1 0.75 Employee 2 0.60 Employee 3 0.2 Toxicity Task 1 70 Task 2 25 Task 3 10 In order to model and mitigate the three types of toxic exposure highlighted above, we propose three utility functions. The acute utility function determines the limit on acute exposure and as such, should be applied to the reward at each timestep: uacute(r)=∑i=1311+e−.5(ri+25)CLOSEu_acute(r)= _i=1^3 11+e^-.5(r_i+25) (6) The episodic utility function considers sub-acute and chronic exposure (e.g. over one or multiple episodes) and thus should be applied over the returns or values: uepisodic(Gt)=∑i=1311+e−.02(Gt,i+400)u_episodic(G_t)= _i=1^3 11+e^-.02(G_t,i+400) (7) Note that in the case of SER optimization uepisodic(Vπ)u_episodic(V^π) should be computed instead of over the return GtG_t as under ESR. The combined utility function should be a combination of both uacuteu_acute and uepisodicu_episodic (applied to both the returns and the value). This is a novel approach to solving multi-objective sequential decision-making problems using a utility-based approach. 4 Experiments To showcase the need for a unified view we performed experiments on the example highlighted in section 3.2 for the three optimisation criteria: ESR, SER and RSR. Finally, we combined the three criteria through a multi-objective natural evolution strategy (MONES) Glasmachers et al. (2010); Salimans et al. (2017). Note that the experiments serve as an illustrative proof of concept to show distinct differences between optimisation criteria. 4.1 Algorithms In order to show the differences in policies learned under each optimization criterion, we used algorithms that are tailored for those settings 33 3 The code for our experiments is available at https://github.com/liammertens/MORL-timescale/tree/main. EUPG (ESR) Expected Utility Policy Gradient (EUPG) Roijers et al. (2018) is a multi-objective adaptation of the REINFORCE algorithm Williams (1992). It is designed to learn optimal policies under ESR for non-linear utilities and achieves this by incorporating the accrued reward during learning and applying the utility over the sum of the accrued return and future returns. NLPPO (SER) Optimising for non-linear utility under the SER criterion requires computing the value function first before computing the utility function. This is achieved by modifying an offline variant of proximal policy optimisation (PPO) Schulman et al. (2017); Meng et al. (2023) to apply the utility over the critic network’s output, as proposed by Röpke et al. (2025). SFDQN (SF/RSR) We simulated a successor feature (/RSR) approach for this problem by transforming the problem into a linear multi-objective RL problem through the application of uacuteu_acute (Eq. 6), which is a non-linear transformation with a subsequent linear combination using uniform weights (all employees counting equally) (as in Eq. 5). This non-linear transformation of the reward function corresponds to the learning of successor features that are non-linear w.r.t. the states in the problem, given that the original state space were extended with a non-linear utility applied to the reward components. After application of the linear scalarisation on the non-linear features, we are left with a single-objective RL problem that can be solved using deep Q networks (DQN) Mnih et al. (2013). MONES (combined perspective) Optimising for the combined utility over different timesteps requires a novel approach, as directly combining either of the methods above is infeasible because scalarisation happens over different time windows and does not distribute over time due to the non-linearity of the utilities. We propose to use a natural evolution strategy to learn the search distribution over policy parameters that optimises the combined utility per timestep, per episode and on average over multiple episodes. This combination is achieved by multiplying all three utilities to get the fitness function of the parameter population, inspired by the AND operator in fuzzy logic. In order to ensure equal contribution of each utility to the total, we divide uacuteu_acute by 30, the episode length, to get the average acute utility. This leads to equal contribution of each sub-utility to the total due to the normalisation effect of the sigmoidal utility functions. 4.2 Setup The hyperparameters for the algorithms are given in table 2. Each experiment was performed using 3 different seeds on an Apple M5 CPU. Table 2: Hyperparameter configurations for the evaluated algorithms. Hyperparameter Value Shared Discount factor (γ) 1 Episode length 30 Optimizer Adam EUPG (ESR) Learning rate 1e-4 Replay buffer size 1e5 Learning steps 3e6 NLPPO (SER) Actor learning rate 2.5e-4 Critic learning rate 2.5e-4 Clip range (ϵε) 0.2 Entropy coefficient 0.05 GAE λ 0.95 Learning steps 3e6 SFDQN (RSR) Learning rate 1e-4 Replay buffer size 1e5 Target net update freq. 1000 Exploration fraction 0.5 Learning steps 2e6 MONES (Combined) Population size 50 Eval episodes per member 60 Learning rate 1e-4 Noise std. dev. (σ) 0.1 Learning steps (generations) 5e6 4.3 Results Figure 1: Comparison of the return distributions resulting over 50 policy rollouts. When comparing the return distributions of each method in Fig. 1, we could notice differences between assignments. Optimising under ESR led to a switching strategy being learned that kept the toxicity penalty nicely around the threshold of 400 for all employees. Note that this implies exceeding the daily intake limit at times. Looking at the optimal policy under SER, the return distribution has less kurtosis. This is the result of the policy preferring to keep employees 1 and 2 under the threshold by rotating them, but in turn ’sacrificing’ the least experienced employee 3.44 4 Please note that this is a numerical example problem, and we would not recommend this in practice. By maximising the utility for 2 employees instead of nicely balancing, it aims to get a higher total utility. Note however, that the expected return under SER lies close to the expected return under ESR for employee 1 and 2, although with increased variance. This shows why SER optimisation is not optimal in cases where risk-awareness is necessary. The need for sacrificing employee 3 stems from the rollouts where employee 1 and 2 are kept well under the toxicity threshold for the episode, leading to optimal expected utility. Optimisation under RSR leads to a switch-averse policy that prefers to keep employees working on the tasks that correspond to their skill level, as this strategy is certain to keep working and no additional requirements on total toxicity are given. Finally, the combined method learns to shuffle, but mainly prefers to put employee 3 on the most difficult task, even more than under SER, leading to optimisation for the utilities of the two most experienced employees. One explanation for this phenomenon could be that the combination of all 3 utilities allows the agent to maximise one or two of them and neglect the other, leading to the ’sacrifice’ scenario. E.g. the agent is not willing to risk exceeding the daily thresholds for employee 1 and 2 in exchange for the total utility of employee 3, so they are kept on the less intensive tasks. The sacrificing policies being optimal for SER and the combined objective are a clear indication that the problem is too hard, as not all employees can be safe from chronic exposure. Especially in the combined objective, it appears that the safety of the third employee is given up, indicating a very clear safety issue for the employer. However, if you look at the RSR or ESR optimisation, the switching policies do seem to balance the toxicity well, and the employees are safe. As such, there seems to be a lot of value in looking at the different timescales at once, if we want to guarantee the safety of all. Clearly the employer needs more employees, or better hazard mitigation measure, to lower the exposure risks when looking at the combined objective, while they might not think so looking solely at ESR (or RSR). In this—admittedly highly simplified/abstract—example, looking at different timescales at once is thus truly essential. To conclude we observe that there are problems for which looking at different timescales individually leads to very different results than looking at the combined utility effects on different timescales simultaneously. For this we used three standard algorithms for RSR, ESR and SER, and a heuristic approach (MONES) for the combined utility of different timescales. We believe that these results call for further investigation into, and specialised algorithms for, combined timescale utility optimisation. 5 Related Work The majority of MORL algorithms consider only a single optimisation criteria. For example, Cai et al. (2023) shows how to, given an MOMDP under ESR, construct a MDP with augmented state space and scalar reward function that induces the same optimal policy as the original MOMDP. Vamplew et al. (2022a) discussed a scenario where the optimal policy might maximise SER subject to a satisfactorily small level of variation in returns between episodes (reduced-variance SER), but their proposed algorithm is restricted to the specific case of finding SER-optimal deterministic policies for environments with stochastic rewards but deterministic state dynamics. More recently Röpke et al. (2023) introduced the distributional undominated set (DUS), which in theory contains all optimal policies under both SER and ESR criteria. At a theoretical level, studies have examined how the choice of utility function or optimisation criterion impacts on the policies that are available to a RL or MORL agent. Subramani et al. (2024) proposes a hierarchy of RL formalisms, and shows that, in theory, the SER (Outer MORL) formalism can express all policy orderings that ESR (Inner MORL) can express. Rodriguez-Soto et al. (2024) shows a characterisation of the utility functions for which an associated optimal policy exists, and (2) a characterisation of the types of preferences that can be expressed as utility functions. In the literature on multi-task RL and successor measures/features, a few works have explored similar problems. Regarding approaches that learn distributional value functions, Wiltzer et al. (2024) introduces a distributional analogue to the SFs. In particular, they show that it allows agents to perform zero-shot risk-sensitive policy evaluation, which can be seen as a type of non-linear utility. The Forward-Backward (FB) representation Touati and Ollivier (2021); Touati et al. (2022) is another relevant framework that allows to approximate (under some dimensionality constraints) the optimal policy for any scalar reward function, fitting the formalism presented in Section 2.1.3. Recently, Bagatella et al. (2026) studies how to optimize for general non-linear utility functions of the successor measure of a policy. We believe this type of method is a good example of combining expressive reward representations with non-linear utilities. Another relevant line of work that extends the expressivity of SF-based methods is the Option Keyboard (OK) Barreto et al. (2019); Barreto et al. (2020). Instead of following a policy specialized to a fixed preference vector w, an OK agent is capable of dynamically changing this vector (and the associated policy being followed) at each time step, similar to dynamic preferences in MORL Abels et al. (2019). This type of technique has been shown to be capable of optimally solving some classes of non-linear reward functions Alegre et al. (2025). 6 Discussion In this paper, we have shown the need for a utility-based perspective in multi-objective planning and reinforcement learning that works on multiple timescales at once. We have shown this intuitively and then numerically on a toxicity example. In our example---for reasons of simplicity---we left out the sub-chronic toxicity that is also known from the toxicology literature. In terms of timescale, this would fall between SER and ESR; it goes beyond one execution of the policy (encompassing about a month), but would not be quite the average longtime outcome either. There is some discussion over the exact timescale of what it means to be sub-chronic55 5 See e.g., https://tinyurl.com/subchronic., but if we take the 90 days mark it would correspond to roughly three policy executions in our example. We could look at the distribution of the summed doses in three policy executions, which would be a smoother distribution of values. This could be added as a fourth timescale. Although we only used one abstract example in this paper, we do believe that these types of problems are actually quite common. Consider for example companies that are interested in the average time a customer’s order needs to be completed, in order to optimise serving costs. However, they might also be interested in the individual customer’s waiting times at the same time, as if a handful of customers might have to wait very long compared to the main bulk, this can lead to very bad reviews and lead to damage to the company’s reputation. This would suggest at least a combined SER-ESR perspective and corresponds to an average reward RL problem Mahadevan (1996). In the medical domain (possibly AI-assisted) radiation therapy Van der Veen et al. (2019) is spread out over multiple sessions to reduce the radiation damage to surrounding organs, while making sure the accumulated dose over time is enough to kill a tumor. This suggests a combined RSR-ESR perspective. So indeed, we suspect that a lot of critical real-world decision problems exhibit utility properties on multiple timescales. With the numerical example, we have shown that different policies work better for different timescales and the corresponding optimisation criteria. We argued that we need to develop comprehensive methodology and algorithmic approaches to deal with different timescales simultaneously, as policies that work well for SER, ESR, or RSR may not suffice for a combination of the three. We believe that a key aspect of any solution that works on multiple timescales simultaneously will be a distributional one, i.e., to learn a distributional return and distributional reward function. However, we do note that even if we have those, policy optimisation is far from trivial. A change in policy can have effects on multiple timescales, leading to a highly complex utility landscape. The AI community has developed multiple ways to tackle such complex utility landscapes though, including policy gradients, evolutionary strategies, etc. So we are hopeful that our community can and will find good approaches for this issue. Ethical Statement The toxic exposure example in this paper is used purely for illustrative purposes. We are not advocating for organisations to manage employees’ exposure to toxins in this manner. More broadly, we believe that the greater expressivity provided by the use of multiple rewards and optimisation criteria enhances the AI agents’ capacity to include ethical considerations explicitly during decision-making Vamplew et al. (2018). Acknowledgments We acknowledge the following funding: LPHM was supported by the VUB; FD was supported by the Belgian Flemish AI Research Program and the “DESCARTES” iBOF project; DMR was supported by the PEER project, which has received funding from the EU’s Horizon Europe Research and Innovation Programme, under Grant Agreement number #101120406\#101120406. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. This work was supported by Kunumi Institute. The authors thank the institution for its financial support and commitment to advancing scientific research. We would like to thank Siri Willems (of imec AI labs) for her input and feedback on the medical examples. References Abels et al. (2019) A. Abels, D. Roijers, T. Lenaerts, A. Nowé, and D. Steckelmacher Dynamic Weights in Multi-Objective Deep Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, p. 11–20 (en). External Links: ISSN 2640-3498, Link Cited by: §5. Alegre et al. (2025) L. N. Alegre, A. L. C. Bazzan, A. Barreto, and B. C. da Silva Constructing an Optimal Behavior Basis for the Option Keyboard. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §5. Alegre et al. (2022) L. N. Alegre, A. L. C. Bazzan, and B. C. D. Silva Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer. In Proceedings of the 39th International Conference on Machine Learning, p. 394–413. External Links: ISSN 2640-3498 Cited by: §2.2. Bagatella et al. (2026) M. Bagatella, T. Rupf, G. Martius, and A. Krause Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities. arXiv. Note: arXiv:2602.06769 [cs] External Links: Link, Document Cited by: §5. Bagot et al. (2025) L. Bagot, L. N. Alegre, S. Latre, K. Mets, and B. C. da Silva Successor clusters: a behavior basis for unsupervised zero-shot reinforcement learning. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §2.2.1. Barreto et al. (2019) A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, J. hunt, S. Mourad, D. Silver, and D. Precup The Option Keyboard: Combining Skills in Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §5. Barreto et al. (2018) A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, Stockholm, Sweden, p. 501–510. Cited by: §2.2.1. Barreto et al. (2017) A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, Long Beach, CA, USA. Cited by: §1, §2.2. Barreto et al. (2020) A. Barreto, S. Hou, D. Borsa, D. Silver, D. Silver, and D. Precup Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences of the United States of America 117 (48), p. 30079–30087. External Links: Document Cited by: §2.2, §5. Cai et al. (2023) X. Cai, P. Zhang, L. Zhao, J. Bian, M. Sugiyama, and A. J. Llorens Distributional pareto-optimal multi-objective reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, p. 15593–15613. Cited by: §5. Chandrasekar and Machado (2025) S. Chandrasekar and M. C. Machado Towards An Option Basis To Optimize All Rewards. Cited by: §2.2.1. Denny and Stewart (2024) K. H. Denny and C.W. Stewart Chapter 6 - acute, subacute, subchronic, and chronic general toxicity testing for preclinical drug development. In A Comprehensive Guide to Toxicology in Nonclinical Drug Development (Third Edition), A. S. Faqi (Ed.), p. 149–171. External Links: ISBN 978-0-323-85704-8, Document, Link Cited by: §3. Glasmachers et al. (2010) T. Glasmachers, T. Schaul, and J. Schmidhuber A natural evolution strategy for multi-objective optimization. In Parallel Problem Solving from Nature (PPSN XI), Part I, p. 627–636. External Links: ISBN 978-3-642-15843-8, Document Cited by: §4. Hayes et al. (2022) C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers A practical guide to multi-objective reinforcement learning and planning: cf hayes et al.. Autonomous Agents and Multi-Agent Systems 36 (1), p. 26. Cited by: §1, §2.1.4, §3.1. Lu et al. (2022) H. Lu, D. Herman, and Y. Yu Multi-Objective Reinforcement Learning: Convexity, Stationarity and Pareto Optimality. In The Eleventh International Conference on Learning Representations, Cited by: footnote 1. Mahadevan (1996) S. Mahadevan Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine Learning 22 (1), p. 159–195 (en). External Links: ISSN 1573-0565, Link, Document Cited by: §6. Meng et al. (2023) W. Meng, Q. Zheng, G. Pan, and Y. Yin Off-Policy Proximal Policy Optimization. Proceedings of the AAAI Conference on Artificial Intelligence 37 (8), p. 9162–9170 (en). External Links: ISSN 2374-3468, Link, Document Cited by: §4.1. Mnih et al. (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller Playing atari with deep reinforcement learning. External Links: 1312.5602, Link Cited by: §4.1. Piscitelli (2014) R. Piscitelli Pruning techniques for multi-objective system-level design space exploration. Ph.D. Thesis, Universiteit van Amsterdam. Cited by: §1. Reymond et al. (2023) M. Reymond, C. F. Hayes, D. Steckelmacher, D. M. Roijers, and A. Nowé Actor-critic multi-objective reinforcement learning for non-linear utility functions. Autonomous Agents and Multi-Agent Systems 37 (2), p. 23 (en). External Links: ISSN 1573-7454, Link, Document Cited by: §3.2. Rodriguez-Soto et al. (2024) M. Rodriguez-Soto, J. A. Rodriguez-Aguilar, and M. Lopez-Sanchez An analytical study of utility functions in multi-objective reinforcement learning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Vol. 37, Red Hook, NY, USA, p. 77726–77747. External Links: ISBN 979-8-3313-1438-5 Cited by: §5. Roijers et al. (2018) D. M. Roijers, D. Steckelmacher, and A. Nowé Multi-objective reinforcement learning for the expected utility of the return. In Proceedings of the Adaptive and Learning Agents workshop at FAIM, Vol. 2018. Cited by: §1, §3.1, §3.2, §4.1. Roijers et al. (2013) D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley A survey of multi-objective sequential decision-making. J. Artif. Intell. Res. 48, p. 67–113. External Links: Link, Document Cited by: §1, footnote 1. Röpke et al. (2023) W. Röpke, C. F. Hayes, P. Mannion, E. Howley, A. Nowé, and D. M. Roijers Distributional multi-objective decision making. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, E. Elkind (Ed.), p. 5711–5719. External Links: Document Cited by: §5. Röpke et al. (2025) W. Röpke, M. Reymond, P. Mannion, D. M. Roijers, A. Nowé, and R. Rădulescu Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’25, Richland, SC, p. 1774–1783. External Links: ISBN 979-8-4007-1426-9 Cited by: §4.1. Salimans et al. (2017) T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever Evolution strategies as a scalable alternative to reinforcement learning. External Links: 1703.03864, Link Cited by: §4. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §4.1. Subramani et al. (2024) R. Subramani, M. Williams, M. Heitmann, H. Holm, C. Griffin, and J. Skalse On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning. arXiv. External Links: 2310.11840, Document Cited by: §5. Touati and Ollivier (2021) A. Touati and Y. Ollivier Learning One Representation to Optimize All Rewards. In Advances in Neural Information Processing Systems, Vol. 34, p. 13–23. Cited by: §2.2.1, §5. Touati et al. (2022) A. Touati, J. Rapin, and Y. Ollivier Does Zero-Shot Reinforcement Learning Exist?. In The Eleventh International Conference on Learning Representations, Cited by: §2.2.1, §5. Vamplew et al. (2009) P. Vamplew, R. Dazeley, E. Barker, and A. V. Kelarev Constructing stochastic mixture policies for episodic multiobjective reinforcement learning tasks. In AI 2009: Advances in Artificial Intelligence, 22nd Australasian Joint Conference, Melbourne, Australia, December 1-4, 2009. Proceedings, A. E. Nicholson and X. Li (Eds.), Lecture Notes in Computer Science, p. 340–349. External Links: Link, Document Cited by: footnote 1. Vamplew et al. (2018) P. Vamplew, R. Dazeley, C. Foale, S. Firmin, and J. Mummery Human-aligned artificial intelligence is a multiobjective problem. Ethics and information technology 20 (1), p. 27–40. Cited by: Ethical Statement. Vamplew et al. (2022a) P. Vamplew, C. Foale, and R. Dazeley The impact of environmental stochasticity on value-based multiobjective reinforcement learning. Neural Computing and Applications 34 (3), p. 1783–1799. Cited by: §5. Vamplew et al. (2024) P. Vamplew, C. Foale, C. F. Hayes, P. Mannion, E. Howley, R. Dazeley, S. Johnson, J. Källström, G. Ramos, R. Rădulescu, W. Röpke, and D. M. Roijers Utility-based reinforcement learning: unifying single-objective and multi-objective reinforcement learning. External Links: 2402.02665, Link Cited by: §3.2. Vamplew et al. (2022b) P. Vamplew, B. J. Smith, J. Källström, G. Ramos, R. Rădulescu, D. M. Roijers, C. F. Hayes, F. Heintz, P. Mannion, P. J. Libin, et al. Scalar reward is not enough: a response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems 36 (2), p. 41. Cited by: §1. Van der Veen et al. (2019) J. Van der Veen, S. Willems, S. Deschuymer, D. Robben, W. Crijns, F. Maes, and S. Nuyts Benefits of deep learning for delineation of organs at risk in head and neck cancer. Radiotherapy and Oncology 138, p. 68–74. Cited by: §6. White (1982) D. White Multi-objective infinite-horizon discounted markov decision processes. Journal of mathematical analysis and applications 89 (2), p. 639–647. Cited by: §1. Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3), p. 229–256 (en). External Links: ISSN 1573-0565, Link, Document Cited by: §4.1. Wiltzer et al. (2024) H. Wiltzer, J. Farebrother, A. Gretton, Y. Tang, A. Barreto, W. Dabney, M. G. Bellemare, and M. Rowland A distributional analogue to the successor representation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, Vol. 235, Vienna, Austria, p. 52994–53016. Cited by: §5. Zhu et al. (2024) C. Zhu, X. Wang, T. Han, S. S. Du, and A. Gupta Distributional Successor Features Enable Zero-Shot Policy Optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §1.