Paper deep dive
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Haseeb Shah, Lingwei Zhu, Adam White, Martha White
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/18/2026, 10:14:02 AM
Summary
This paper presents a large-scale empirical study of 33,000 experiments to deconstruct actor-critic reinforcement learning algorithms. Using a water treatment control task, the authors analyze how design components—such as policy parameterization, gradient estimation, and update schedules—affect reliability and hyperparameter sensitivity. The study finds that common defaults like Gaussian distributions with pathwise estimators are often unreliable, while bounded distributions with adaptive schedules offer robustness.
Entities (16)
Relation Signals (14)
Actor-Critic Algorithms → includes → MPO
confidence 95% · Maximum a Posteriori Policy Optimization (MPO)...
Actor-Critic Algorithms → includes → SAC
confidence 95% · Soft Actor-Critic combines reparameterized updates...
Actor-Critic Algorithms → includes → DDPG
confidence 95% · Deep Deterministic Policy Gradient (DDPG)...
Actor-Critic Algorithms → includes → GreedyAC
confidence 95% · Greedy Actor-Critic (GreedyAC)...
Actor-Critic Algorithms → includes → PPO
confidence 95% · We focus on several widely-used actor-critic algorithms: Proximal Policy Optimization (PPO)...
Water Treatment Plant → uses → PID Controller
confidence 95% · This PID controller drives the backwashing pump in a water treatment system
Gaussian Distribution → is → Least Reliable Configuration
confidence 90% · Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations
Actor → parameterizes → PID Gain Parameters
confidence 90% · The actor is a probability distribution over PID gain parameters (pt, it, dt)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.
Tags
Links
- Source: https://arxiv.org/abs/2607.13274v1
- Canonical: https://arxiv.org/abs/2607.13274v1
Trouble viewing inline? Open PDF directly →
Full Text
128,819 characters extracted from source content.
Expand or collapse full text
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners Haseeb Shah a,b,∗,† , Lingwei Zhu c,† , Adam White a,b,d , and Martha White a,b,d a Department of Computing Science, University of Alberta, Canada b Alberta Machine Intelligence Institute (amii), Canada c Great Bay University, China d Canada CIFAR AI Chair Reinforcement learning is increasingly being consid- ered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is es- sential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the dis- tribution over actions, how its gradient is estimated, and how often it is updated relative to the value esti- mator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variabil- ity across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distribu- tions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These find- ings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapt- ing actor-critic methods to new real-world control settings. Keywords: reinforcement learning; actor-critic algorithms; process con- trol; algorithm reliability R einforcement Learning (RL) algorithms are increasingly being used in domains ranging from racing games and optimizing computer operations to simulated reactor control (1–6), but real-world deployments are much more rare be- cause current popular algorithms are designed for and tuned to specific academic benchmarks. Deep RL algorithms are actually a collection of specific components that do not gen- erally work well across problems. For example, the Rain- bow agent (7) combines several algorithmic ideas (noisy-net exploration (8), distributional losses (9), prioritized experi- ence replay (10), etc.) to achieve near state-of-the-art on Atari (11), but performs poorly in small-scale classic control environments like Mountain Car (12). Soft Actor-Critic com- bines reparameterized updates, squashed Gaussian policies, and entropy regularization (13) to great effect in MuJoCo tasks (14), but generally performs poorly in the DM Control Suite (15)—both popular benchmarks for continuous-action * To whom correspondence should be addressed.E-mail: hshah1@ualberta.ca † H.S. contributed equally to this work with L.Z. control methods in RL. This lack of generality is a major obstacle to applying these algorithms to new problems and real-world deployments. Actor-critic methods are of particular interest, as they can be applied to both discrete- and continuous-action problems (13, 16–18) and form the backbone of many of the most widely used RL systems. At a high level, these methods share a common template: a policy responsible for select- ing actions (actor) and a learned estimator that evaluates actions (critic), enabling policy updates with lower vari- ance and greater scalability than vanilla policy gradient al- gorithms. Despite this common backbone, these algorithms differ in how they define and implement policy improvement, for example, through conservative, constraint-based updates (19, 20), regularized objectives that encourage exploration (13), or improvement and projection procedures (16). These differences can interact strongly with how the policy is pa- rameterized and optimization choices, resulting in substan- tially different qualitative behaviors even though all these methods are grouped under the same umbrella of “actor- critic" algorithms. Our limited understanding of individual algorithmic compo- nents and the different ways to implement them limits our ability to deploy them effectively. There is a dizzying ar- ray of algorithms to choose from, with insufficient clarity on their strengths, weaknesses, and failure modes. The be- havior of an algorithm is often inseparable from the bundle of implementation tricks and tuning decisions such as nor- malization choices, entropy schedules, clipping and regular- ization strengths, actor and critic update ratios, target net- works, replay buffer sizes and sampling strategies, and learn- ing rate schedules. These choices can significantly change performance and even reverse conclusions across empirical studies. As a result, a common strategy is to adopt com- plete implementations of popular algorithms, such as PPO (20) or DDPG (21), and avoid modifying them, even when the target problem differs from the setting for which the im- plementation was originally designed. This can lead to awk- ward workarounds that preserve code structure rather than adapting the underlying algorithm to the problem at hand. For example, to apply DDPG to bandit problems, researchers have had to rely on awkward state padding to preserve the original code structure (22). The goal of this paper is to provide clarity on the key design components and implementation choices in actor-critic algo- rithms, both conceptually and through an empirical study. Such an analysis can make it easier for both practitioners and Preprint — under review | July 16, 20261 of 38 arXiv:2607.13274v1 [cs.LG] 14 Jul 2026 ALEARNING LOOP BACKWASH PID ENVIRONMENT LEARNING AGENT Parameters: distribution parameters Output: distribution π(A) Actor πθCritic qw Input: action A = pt,it,dt Parameters: MLP = 2 x 64 Output: value = qw(A) qw(A) update actor update critic Replay buffer B with past (action, reward) tuples (p1,i1,d1, r₁), (p2,i2,d2, r₂), (p3,i3,d3, r₃), ... μ σ PID controller Gain Parameters: pt,it,dt Flow rate setpoint: fideal pump speed sj water in water out Water treatment pump flow rate fj observed reward rt Reward Computation tracking the setpoint over multiple internal steps 0 Internal timesteps j flow rate f j overshoot error undershoot error setpoint fideal sampled action pt,it,dt BCOMPONENT DECISIONS Actor πθCritic qw Replay buffer B Mirror descent Gradient ascent vs KL-constrained update Mixing coefficient λ Reward, critic estimate or mixed target Gradient Estimator Likelihood ratio vs pathwise estimator Actor objective mass-covering vs mode-seeking KL, target policy choice Policy parameterization Gaussian Beta Student's t Update ratios Actor and Critic updates per environment step CPOLICY PARAMETERIZATIONSDKL DIVERGENCE BEHAVIOUR π(a) actions a Gaussian Beta Student's t 1. Mass-covering behavior: argminπ KL(μ ǁ π) π(a) actions a Boltzmann μ Gaussian π 2. Mode-seeking behavior: argminπ KL(π ǁ μ) π(a) actions a Gaussian π Boltzmann μ Fig. 1. Schematic of a reinforcement learning system on a water treatment controller. A: Deployment setting and agent internals. The actor is a probability distribution over PID gain parameters (p t ,i t ,d t ), parameterized here by the mean μ and standard deviation σ. An RL agent containing this actor π θ , a critic q w and a replay buffer B of past data, outputs PID gains and writes them to a classical PID controller. This PID controller drives the backwashing pump in a water treatment system by controlling the pump speed s j over j ∈ 1,...,J discrete internal timesteps forJ = 55, where each step represents 1 second of water flow through the pump. The goal is to achieve a water flow rate f j equal to the setpoint f ideal . The tracking error f j − f ideal is fed back into the PID controller, forming a closed feedback loop. Finally, the reward r t is constructed by summarizing the tracking quality overJ steps and the pair (a t = (p t ,i t ,d t ),r t ) stored in the agent’s buffer. The actor and critic sample from this buffer to do their updates, described in more detail below. B: We run over 33,000 experiments and analyze variations within six different components of the learning agent to study the reliability of the system. C: A visualization of some of the policy parameterizations available to the actor. The Student’s t policy can achieve heavier tails compared to the Gaussian, while the beta policy is naturally bounded. D: We later investigate the impact of using the mass-covering KL vs the mode-seeking KL, and so visualize the difference between the two here. The visualization shows the outcome from optimizing a Gaussian policy π to approximate a bimodal Boltzmann target μ. 1. Minimizing the mass-covering KL results in a learned π that is spread over both modes. 2. Minimizing mode-seeking KL instead commits the learned π to a single mode. researchers to select, modify, and build on actor-critic algo- rithms, particularly those who care about achieving reliable performance. Rather than focusing solely on best-case per- formance, we emphasize both the variability of performance across runs and hyperparameter sensitivity, since these fac- tors determine whether an algorithm is usable in practice and can be adapted to a new domain with a reasonable tuning budget and minimal modifications. To isolate the effects of actor-critic algorithms, we study a simplified, stateless PID tuning setting. Tuning PID con- trollers is a popular task in engineered systems where small changes in parameters can have large effects on overall stabil- ity and efficiency. This setting strips away many of the con- founding factors that dominate outcomes in deep RL bench- marks, such as long-horizon credit assignment, environment- specific engineering, and temporal state dependence, to name a few. This simplified setting allows us to explain the key dif- ferences between these algorithms while also making it easier to run comprehensive hyperparameter sweeps that both pro- vide insights for existing algorithms and highlight promising avenues for future study. Our goal is to provide empirical insights that are transferable through understanding rather than performance-based benchmarking. In particular, we investigate three key choices: how the actor (policy) is updated, how the actor is parameterized, and the utility of typical optimization improvements to the update (e.g., mirror descent). We focus on several widely-used actor- critic algorithms: Proximal Policy Optimization (PPO) (20), Soft Actor-Critic (SAC) (13), Deep Deterministic Policy Gra- dient (DDPG) (21), Maximum a Posteriori Policy Optimiza- tion (MPO) (16), Greedy Actor-Critic (GreedyAC) (17) and two simple methods, vanilla Actor-Critic and REINFORCE (23). Throughout, we focus on hyperparameter sensitivity, which is critical for ease of use and for practitioners to fea- sibly use and deploy these algorithms. We introduce a PID tuning environment based on real data from a drinking wa- ter treatment pump controller. Even in this simple bandit setting, we observe differences among methods and identify trends linking stability to policy parameterizations and to how updates are performed. Problem setting and base algorithms We restrict our focus to a non-contextual continuous-action bandit setting, to make the key ideas behind these actor-critic algorithms more accessible and to facilitate a more thorough empirical analysis. In the bandit setting, on each time step t, the agent takes action A t ∈ A and then observes the out- come reward R t ∈ R. 1 The outcome reward is given by R t = r(A t ) +ε for reward function r :A→ R and some inde- pendent noise ε. The agent’s goal is to maximize accumulated rewards, in other words, its average reward 1 t P t i=1 R i , and ultimately find a policy π with high expected reward, E π [R]. 1 In the full reinforcement learning setting, this reward is typically labeled R t+1 , because it might depend on the next state. We chose to use R t to be clear it is the reward from taking action A t . 2 of 38 To do so, the agent learns a parameterized policy π θ : A → [0,∞) with parameters θ. In the continuous action setting, θ is the parameters for a probability density function (pdf), such as a Gaussian (with θ = (μ,σ) for mean μ and variance σ 2 ), a beta distribution (with θ = (α,β) for scale parameters α > 0,β > 0) and other options discussed in more detail in SI Appendix 3. The goal is to find the best parameterized policy in the policy class (given by Θ) max θ∈Θ E π θ [R] = Z A π θ (a)r(a)da(1) where parameter set Θ is bounded to prevent the policy from becoming fully deterministic. For example, for a Gaussian, we might have Θ = (μ,σ) ∈ R 2 | σ 2 > 0.001. The overall learning loop for the bandit task we consider in this paper is depicted in Figure 1(A). A basic algorithm to achieve this goal is REINFORCE (23), which uses a (stochastic) gradient ascent update based on the objective in Equation 1: ∇ θ E π θ [r(A)] =∇ θ Z A π θ (a)r(a)da = Z A r(a)∇ θ π θ (a)da = Z A r(a)π θ (a)∇ θ lnπ θ (a)da = E π θ [r(A)∇ θ lnπ θ (A)] This gradient increases the probability of high-reward actions and decreases it for low-reward actions. To accelerate learn- ing, it is standard to subtract a baseline v from r(A), to center values around zero. It is equivalent to use r(A)−v in the update because ∇ θ E π θ [r(A)− v] =∇ θ E π θ [r(A)]−∇ θ E π θ [v] | z =0 . That is, any baseline that does not depend on the action will have zero contribution to the gradient. The baseline is typically chosen to be v ≈ E π θ [r(A)]. Therefore, we have the advantage of an action in the update r(A)− v. This way, the gradient has a more intuitive meaning: the probabilities of actions with positive values will increase, and those with negative values will decrease. REINFORCE uses a stochastic sample of this update on time step t, taking action A t ∼ π θ , observing reward R t , and updating with θ ← θ + η(R t − v)∇ lnπ θ (A t ) (REINFORCE) v ← (1− η v )v + η v R t for some stepsizes η,η v > 0 and initializing v = 0 at t = 0. REINFORCE, however, is known to suffer from high- variance; instead we can consider actor-critic (AC) meth- ods that estimate q w (a) ≈ r(a) instead of only us- ing sampled rewards. As the agent generates interac- tions (A t ,R t ), it can store them in a replay buffer B = (A 1 ,R 1 ), (A 2 ,R 2 ),..., (A t ,R t ) up to the current time t. It can use a regression algorithm to update q w using the dataset B, with the primary goal of estimating r as accurately as possible. For example, the parameterized q w can be a neural network, with parameters w, updated using mini-batch SGD. The update can use q w instead of a (noisy) sampled R t θ ← θ + η(q w (A t )− v)∇ lnπ θ (A t )(AC) This buffer-based approach forms the basis of the actor-critic algorithms we consider, as depicted in Figure 1 (A). However, this update can be biased, since it is likely that q w (a) ̸= E[R t |A t = a]. To mitigate this, a parameter λ ∈ [0, 1] can be introduced that mixes between the unbiased, but high variance R t and the low variance, but potentially biased q w (a) θ ← θ+η(λR t + (1−λ)q w (A t )− v)∇ lnπ θ (A t ) (AC(λ)) When λ = 1, we recover REINFORCE. These two base algorithms, REINFORCE and AC(0), pro- vide two canonical policy optimization approaches. REIN- FORCE is a policy gradient method, using an (unbiased) stochastic gradient update on the policy objective. AC(0), on the other hand, can be interpreted as approximate policy iteration (24). It alternates between approximate evaluation (updating q w ) and approximate greedification (updating π θ ) to concentrate on high-value actions according to q w . In the bandit setting, approximate evaluation is simply a regres- sion problem, and we opt to use the same strategy across methods (see Alg. 1). However, the core ideas behind dif- ferent actor updates (approximate greedification) persist in the bandit setting, as they greedify differently given q w (a). In this work, we focus on understanding the differences in these actor updates across variants of algorithms that build on AC(0), and also present experiments contrasting the use of algorithms like REINFORCE with AC(0). Key Design Decisions in the Actor Update In this section, we outline several key design decisions for the actor update, summarized in Figure 1 (B). We ground this discussion by focusing on five algorithms that are mod- ifications of the basic REINFORCE and AC(0) algorithms described above: Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), Deep Deterministic Policy Gradi- ent (DDPG), Maximum a Posteriori Policy Optimization (MPO), and Greedy Actor-Critic (GreedyAC). These algo- rithms are chosen because they cover several key differences in the actor update. PPO can be seen as a modification of REINFORCE that uses only estimates of v without learn- ing q w . The remaining four algorithms can be viewed as modifications of AC(0) that rely heavily on q w . We provide pseudocode and details for each algorithm in SI Appendix 4 and here describe the high-level conceptual differences. Using λ > 0 or λ = 0. This is a key distinction between al- gorithms that are typically called on-policy and off-policy. When we set λ = 0, we use only q w and can counterfactually (off-policy) reason about any action. This decouples the actor update from the sampled action A t and allows us to lever- age buffers of stored data to improve sample efficiency. This partially explains why there are so many off-policy AC algo- rithms, including the ones we consider here: SAC, DDPG, MPO and GreedyAC. These algorithms rely on estimates of the value of each action, and their key differences lie in trying to develop an improved actor update (approximate greedifi- cation). Once we pick λ > 0, we use R t that resulted from sampling A t , giving an on-policy update. REINFORCE, AC(λ) and PPO all use such updates. The REINFORCE update above use only the most recent sample, whereas PPO performs mul- tiple epochs over a recently sampled trajectory. The up- date becomes slightly off-policy after each update because 3 of 38 the policy has changed, and PPO introduces an importance- sampling correction to account for this. However, these up- dates remain nearly on-policy, as the policy does not change significantly before this data is discarded and a new trajec- tory is generated under the current policy (due to clipping and other measures). Using Mirror Descent and Trust-Regions. Another key choice in many actor-critic algorithms is to replace this standard gradi- ent ascent update with a mirror ascent update. The gradient ascent update is obtained by using a local Taylor series ap- proximation around the objective J (θ) . = E π θ [r(A)] for the current point θ t , with an Euclidean distance (25) ˆ J (θ) . = J (θ t ) + (θ− θ t ) ⊤ ∇J (θ t )− 1 2η ∥θ− θ t ∥ 2 2 θ t+1 = arg max θ ˆ J (θ) = θ t + η∇J (θ t ) We get a closed-form update when maximizing ˆ J (θ) because ˆ J (θ) is quadratic. But an Euclidean distance is not always the right choice. For learning policies, it is more sensible to use the distance between distributions rather than be- tween policy parameters. A small change in Euclidean dis- tance between policy parameters might not be reflective of the change in the distributions. We can replace the Eu- clidean distance with a KL divergence between the policies, KL(π θ ||π θ t ) . = R A π θ (a) ln π θ (a) π θ t (a) da, giving instead 2 ˆ J KL (θ) . = J (θ t ) + (θ− θ t ) ⊤ ∇J (θ t )− 1 2η KL(π θ ||π θ t ) θ t+1 = arg max θ ˆ J KL (θ) This argmax no longer has a closed-form solution; instead, we use multiple steps of ascent on ˆ J KL (θ). Many algorithms have been shown (27) to use updates that are more similar to mirror ascent than to gradient ascent. These include TRPO (19), MDPO (28), MPO (16), MD- MPI (27), PPO (20) and FMA-PG (29) and variants (26). TRPO was originally motivated as a trust-region method, inspired by Conservative Policy Iteration (CPI) (30), which had guaranteed improvement if the policy did not change too much after each greedification step. This update used a KL divergence to the previous policy and spurred a line of work further developing such algorithms, ultimately with unifica- tions highlighting the connection to mirror ascent. For that reason, this update has taken a variety of forms. We include the mirror-descent variant of SAC in Alg. S5, leveraging re- cent work that developed such a mirror-descent version (26). Actor Objectives using the KL. It is also possible to use different objectives for approximate greedification, namely the actor update. This change is, in fact, what primarily distinguishes SAC, MPO, and GreedyAC from AC(0). The key idea behind these methods is to choose a target policy π target , and then the approximate greedification step involves reducing the KL divergence to that target policy. As was recently shown (24), SAC and many other actor-critic methods can be seen as 2 This exact formula is only true for J(θ) . =E π θ [r(A)]; it is slightly different for the mirror descent variants of other objectives J for actor-critic methods. This is thoroughly described in a recent work (26). We leverage the derivations from that work, but keep the explanation here simpler. minimizing a mode-seeking KL divergence to the Boltzmann policy: KL(π θ ||π ent ) for π ent (a)∝ exp(q w (a)/β) for entropy parameter β. This Boltzmann policy is the solu- tion to the entropy regularized greedification step: π ent . = arg max π E π [q w (A)] + βH(π) where H(π) = E π [− logπ(A)] is the entropy of policy. As β approaches zero, entropy regularization is removed, and we recover the greedy policy. Note that to optimize this objec- tive, we only need to sample from π θ , and never explicitly form the policy π ent . This KL objective was introduced for the Soft Actor-Critic (SAC) (13), but many other algorithms adopt a similar loss in their approximate greedification step (24, Table 2). In fact, even the standard AC update can be seen as an instance of this update, because as β approaches zero, the negative gradient of this KL objective approaches 3 1 β ∇ θ E π θ [q w (A)]. Most methods use the mode-seeking KL written above, but there are a handful that use the mass-covering KL (24, Ap- pendix A3-A4). This distinction comes from the fact that the KL is not symmetric: D KL (π||π θ )̸= D KL (π θ ||π), resulting in different preferences on the solution, as depicted in Figure 1 (D). The forward KL D KL (π||π θ ) is mass-covering, whereas the reverse KL D KL (π θ ||π) is mode-seeking. Consider that we want to approximate a multi-modal Boltzmann policy us- ing a unimodal Gaussian. One must weigh between covering the most significant mode of the Boltzmann or covering as much of the probability mass as possible. This is depicted in Figure 1 (D). In practice, mass covering can cause the policy to more often select actions that do not correspond to any mode of a multimodal policy, resulting in poor performance (31). It is also possible to pick other target policies and so get other updates. In MPO, the target policy is chosen to be π kl . = arg max π E π [q w (A)]− τKL(π||π t ) for π t the previous policy, with a closed-form solution π kl (a)∝ π t (a) exp(q w (a)/τ ). The loss for MPO uses a mass- covering KL loss to target policy π kl . GreedyAC uses a mass- covering KL to a target policy they call a percentile policy π ρ . The algorithm maintains a slower-changing proposal pol- icy ̃π, samples several actions from ̃π, and only keeps the top ρ percentile. It increases the likelihood of only that top per- centile of actions, for both π θ and ̃π. They show that this approach is equivalent to reducing the KL to a percentile policy π ρ that shifts the action distribution in ̃π to put all weight on only the top percentile of actions. Because both MPO and GreedyAC use a mass-covering KL, the algorithms have a similar flavor: sample from a proposal policy (not from π θ ) and increase the likelihood of those actions. Policy Parameterization. The policy π θ (a) is a distribution that can be parameterized in many different ways, which can sig- nificantly impact learning behavior. One of the most common choices is to use a Gaussian distribution, with some form of 3 Expanding the KL gives −∇ θ KL(π θ ||π ent )= 1 β ∇ θ E π θ [q w (A)] − ∇ θ E π θ [logπ θ (A)], and as β → 0, the first term dominates and the update direction aligns with the standard policy gradient (AC) updateE π θ [q w (A)]. 4 of 38 squashing or clipping to keep the actions within a bounded range. However, other densities have been considered, in- cluding beta (32) and Student’s t (33, 34), as depicted in Figure 1 (C). One could consider discretizing the continuous action space to use the softmax policy. Recently, there has been more investigation into more expressive policies, includ- ing mixture models (31) and diffusion policies (35–37). The best-performing choice typically depends on the problem and other algorithmic components. Understanding the im- pact of the choice remains largely unresolved. There are sev- eral criteria to consider when making the choice. The actor slowly greedifies around high-valued actions according to the critic, but also has a role to provide sufficient exploration. 4 All of these parameterizations introduce stochasticity to pro- mote exploration, initialized with higher variance, with learn- ing typically decreasing stochasticity while enforcing a lower bound on variance. Even deterministic policies, such as in DDPG (21) and TD3 (38), add stochastic (Gaussian) noise for exploration. In fact, this approach of adding Gaussian noise with variance σ to a deterministic policy is equivalent to using a Gaussian parameterization in which the variance is fixed at σ and not learned. It is not clear if more expres- sive policies, like diffusion policies, facilitate either of the two goals of sufficient exploration and greedification. However, some evidence suggests they provide minimal to no advan- tages in the case of mixture policies (31) and unclear evidence in the case of diffusion policies (39). In this work, we focus on comparing these simpler parameterized policies, including variants of the Gaussian, beta, and Student’s t distributions; see SI Appendix 3 for detailed descriptions. Update-to-data Ratios. The update-to-date (UTD) ratio is the number of updates used per environment step. 5 If the UTD for the actor is 10, the actor network is updated 10 times on every environment step, on average. A higher UTD can make the agent more sample-efficient, but care must be taken to avoid overfitting to the data seen so far. As the dataset grows, this issue becomes less of a concern during interaction, but in early learning, a high UTD could cause problems. This has been observed as a phenomenon called primacy bias (40), and common wisdom is that it can be difficult to raise the UTD in reinforcement learning without careful adjustments to the architecture and hyperparameters (41–43). Furthermore, there is an interaction between the UTD for the actor and critic. If the UTD is higher for the critic than for the actor, we take more policy evaluation steps to obtain a more accurate critic estimate before greedifying. If the UTD is higher for the actor than the critic, then the actor is likely to be more greedy, and the critic evaluates a greedier policy. The seminal actor-critic theory assumes the first scenario: that the critic updates more quickly than the actor (44–46). Gradient Estimators. A key distinction in actor updates is how the policy gradient ∇ θ J (θ) is estimated. The likelihood- ratio (LR or score function estimator) and pathwise (PW or reparameterized estimator) are two common unbiased Monte Carlo estimators of the gradient (47, 48). The LR gradi- ent estimator computes the gradient of q w (A) using the log- 4 These two roles can be separated, but the most common choice is for the actor to be the same for the action selected in the bootstrap target and for behavior 5 In online RL, the replay ratio, i.e., how often we sample a mini-batch per envi- ronment step, is used to discuss UTD. The concept is the same, but UTD is more general. beta gaussian student Fig. 2. Mixing immediate reward with value estimates improves stability and performance For each policy parameterization, we fix all the hyperparameters to their default values (Table S2) and vary the value reward mixing coefficient λ ∈ 0, 0.3, 0.9, 1 in AC(λ). The actor update weights the gradient∇ lnπ θ (A t ) by λR t + (1−λ)q w (A t )−v, interpolating between the critic’s action-value estimate (λ = 0) and the observed reward (λ = 1). Solid lines show mean reward (higher is better, left axis) averaged over 10 seeds while the dashed lines show coefficient of variation (CV = σ/|μ|, lower is better, right axis). The curves are drawn using a shape-preserving cubic interpolation without altering pointwise values. Intermediate values of λ achieve the best combined performance and variability across all policy parameterizations. derivative trick: ∇ θ J (θ) =∇ θ E A∼π θ [q w (A)] = E A∼π θ [q w (A)∇ θ lnπ θ (A)]. Adding in the baseline, q w (A) − v, yields the familiar AC update. This estimator can have a high variance, but it can be applied to any differentiable policy distribution, including discrete ones. The PW estimator can be used when the policy allows a differentiable reparameterization. This is achieved by decou- pling the stochasticity in action selection from the policy. For example, if the policy is a univariate Gaussian, we can sample ε∼N (0, 1) to produce the action: a t = g θ (ε) = μ θ + σ θ ε. In this setting, we can rewrite the gradient applying the chain rule directly through the critic and into the actor: ∇ θ J (θ) = E ε∼p [∇ θ q w (g θ (ε))] = E ε∼p [∇ A q w (A)∇ θ g θ (ε)] A = g θ (ε), where ε ∼ p is the noise sampled from a distribution in- dependent of θ. This estimator enables end-to-end back- propagation through the sampled actions and often yields a lower-variance estimate. However, this requires that q w (A) be differentiable with respect to A. Additionally, PW is not directly applicable to discrete distributions unless we use a continuous relaxation (49). Neither gradient estimator is strictly better than the other; each has its own advantages. Although PW is often shown to have lower variance than LR (50), this is not guaranteed. For example, it has been shown that a long chain of non-linear computations in model-based RL can cause the PW gradients to explode, whereas LR gradients remain robust (51). The LR estimator can always be used, whereas the PW esti- mator is restricted to algorithms that learn q w with a mode- seeking KL. REINFORCE and PPO use the LR gradient 5 of 38 0.000120.000370.00110.00330.01 Learning rate η 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Coefficient of Variation (CV) Impact of critic size on variability Critic Size small large Fig. 3. Increasing the critic size reduces run-to-run variability . For each policy parameterization and learning rate, we compare AC(λ) with a small critic (2 hidden layers of 64 units) and a large critic (2 hidden layers of 256 units), with all the hy- perparameters fixed at their default values (Table S2) and λ = 0. The bar height represents the coefficient of variation (CV) across N = 10 seeds, where lower values indicate more stable runs. Larger critics consistently reduce CV across the entire learning-rate range for all policy parameterizations. typically with an advantage estimator, importance weighting and clipping. DDPG and TD3 can be viewed as determinis- tic special cases of the PW gradient. SAC typically uses PW in continuous action spaces and LR in discrete ones. MPO and GreedyAC use the mass-covering KL, where the expecta- tion is wrt to a proposal policy rather than π θ , side-stepping this issue. However, the updates for both of these algorithms can be best interpreted as an LR update as they apply a log-likelihood update on the sampled actions. Results In this section, we evaluate both the performance and stabil- ity of various component choices. The performance is defined as the mean reward across seeds (higher is better), whereas the stability refers to the run-to-run variability, measured by the coefficient of variation across seeds (lower is better). A reliable method should do well on both metrics. All the al- gorithms are tested on the Backwashing PID environment, constructed from data from a real water treatment system, depicted in Fig. 1 (A), with a detailed description of the construction in the Materials and Methods section. Unless otherwise stated, all experiments use the adaptive critic UTD ratios as described in Alg. 1. We present summarized conclu- sions in the main text, and additionally provide detailed run- to-run and hyperparameter variability analysis in the Supple- ment 5. Conclusion 1. Mixing immediate reward with value esti- mates (λ > 0) improves stability and performance. AC(λ) interpolates between the critic’s action-value estimate (λ = 0) and the observed reward (λ = 1) in the policy gradi- ent. In our experiments, relying entirely on the critic (λ = 0) led to high run-to-run variability across all policy parameter- izations, with beta policies being particularly affected. This is likely due to value misspecification, introducing bias into beta gaussian student squashed gaussian squashed student Performance-stability frontier over top-10 hyperparameters per setting Fig. 4. Performance-stability frontier across algorithms and policy parame- terizations. For each algorithm, policy combination, we select the top-10 hyper- parameter configurations by mean reward and aggregate over their 100 runs (10 configurations x 10 seeds). The x-axis shows the mean reward (higher is better) averaged across these runs. The y-axis shows the mean coefficient of variation (lower is better), computed by taking the CV across seeds within each configura- tion and then averaging across 10 configurations: (CV mean = 1 N P i CV i with N = 10). This separates within-configuration variability (run-to-run noise at fixed hyperparameters) from cross-configuration variability (sensitivity to hyperparameter choice). Error bars are 95% bootstrap confidence intervals (n=2000 resamples). Marker shape indicates the algorithm, and color indicates the policy parameteriza- tion. The dashed Pareto frontier connects configurations for which no other plotted point achieves both higher mean reward and lower mean CV, and is computed di- rectly from the plotted aggregate points. The best region of the frontier, with high mean reward and low variability, consists entirely of beta configurations, with five of the six algorithms clustering together. Student’s t and Gaussian appear on the frontier only at intermediate performance levels, while squashed policies are strictly dominated by multiple other configurations. Additionally, unlike other algorithms, which are spread apart, PPO configurations are relatively similar in performance across multiple policies. the policy update. Introducing even a moderate reward mix- ing (λ = 0.3) recovers most of the performance gap while significantly reducing variability, with the best effect consis- tently observed at λ = 0.9. Completely removing the critic contribution (λ = 1) tends to either reduce performance or increase variability for Gaussian and Student’s t policies. See Fig. 2 for the effect of λ on performance and variability and Fig. S8 for a more detailed breakdown across different hy- perparameter and environmental noise configurations. Conclusion 2. Critic inaccuracy is a major contributor to instability. Inaccurate critics increase run-to-run variability: some runs achieve good performance, whereas others fail, thereby reduc- ing stability. There are several lines of evidence supporting this conclusion. Firstly, increasing λ in AC(λ) reduces the critic’s contribution to the policy gradient. Fig. 2 shows that higher λ values decrease variability across all policy pa- rameterizations. Secondly, increasing the critic size increases the critic’s representational capacity, reducing critic error as long as the critic is trained sufficiently to exploit this added capacity. Because our adaptive critic update (Alg. 1) trains the critic until its batch error falls below a fixed threshold, the larger network actually realizes this lower error. Consis- tent with this, Fig. 3 shows that larger critic networks reduce 6 of 38 SAC (PW) SAC (LR) PPO GreedyAC MPO 0 500 1k 1.5k 2k SAC (PW) SAC (LR) PPO GreedyAC MPO Convergence speed over top-10 hyperparameters with = 1 beta gaussian student squashed gaussian unbounded-support policies bounded-support policies squashed student mean bootstrap 95% CI Steps until convergence Convergence speed over top-10 hyperparameters with = 10 Fig. 5. Bounded-support policies converge slower than unbounded-support across algorithms. For each algorithm, policy combination, we report the number of steps until convergence, defined as the first step at which the return reaches within 90% of the policy’s own final 5% average performance. Note that this is a within- policy convergence metric, as the convergence does not guarantee reaching some predefined global threshold. Results are aggregated by pooling all runs from the top-10 hyperparameter configurations for each combination, then computing the mean (height of the lighter bar) and 95% bootstrap confidence intervals (dark bars, n=2000 resamples). Left: at default actor UTD (UTD actor = 1), the policies with bounded support converge slower than unbounded support in most cases, with the beta policy often requiring 2-3x more steps than Gaussian. Right: at UTD actor = 10, this ordering largely collapses and all policies converge at comparable rates. 131030 Actor update scaling factor 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 Critic LR � = 0.03 131030 Actor update scaling factor Critic LR � = 0.0033 131030 Actor update scaling factor Critic LR � = 0.00037 Learning rate �× UTD ratio × Scaling actor learning rate vs actor UTD ratio beta gaussian Fig. 6. Actor update-to-data ratio is a safer tuning knob compared to the actor learning rate. For each policy (Gaussian, beta), we compare two strategies for scaling the actor update magnitude by a factor k ∈1, 3, 10, 30 i.e. scaling the actor learning rate η actor × k (solid lines) or scaling the actor UTD ratio UTD actor × k (dashed lines), across three representative critic learning rates η critic (different panels) and default SAC hyperparameters (Tables S2, S3). Before scaling is applied, the actor and critic learning rates are equivalent, i.e., η actor = η critic . Results are shown as mean reward across 10 seeds, and the shaded bands represent 95% bootstrap confidence intervals with n=1000 resamples. For beta policies, the dashed and solid lines overlap across all learning rates, indicating that the two scaling strategies are almost identical. For Gaussian policies, they are equivalent to low critic learning rates but start diverging at high η critic = 0.03 (left), where scaling actor LR causes collapse at k ≥ 10 while scaling actor UTD remains relatively stable. This suggests that the Gaussian policy’s instability at high actor LR arises due to big updates to both the actor and critic networks at the same time, as opposed to large actor parameter movements alone. variability across all policy parameterizations in AC(λ). Fi- nally, note that PPO in our non-contextual bandit setting does not require learning an action-value critic and, corre- spondingly, its configurations are among the most favorable in the performance-stability trade-off. Three of the four PPO configurations are Pareto efficient in Fig. 4 while the remain- ing one lies close to the Pareto frontier. Conclusion 3.Beta policies achieve the best performance-stability tradeoff. We aggregate results over the top-10 hyperparameters for each algorithm, including allowing different UTD per algo- rithm. In Fig. 4 we observe two patterns. Firstly, the best region of the performance-stability frontier consists entirely of beta configurations, with five different algorithms cluster- ing tightly around near-optimal performance. This region denotes configurations that have high mean reward and low variability across runs and hyperparameters. Secondly, ex- cept for AC(λ), no non-beta policy parameterization dom- inates any beta configuration. The variants of AC(λ) that perform worse are likely due to not sweeping over the actor UTD ratio in that setting. Some Student’s t and Gaussian configurations also lie on the frontier, but only in intermedi- ate regions where they achieve lower mean reward but com- petitive variability. In contrast, squashed Gaussian, which is the widely-used default in SAC implementations, performs poorly. For a more detailed breakdown, see SAC with PW estimator, SAC with LR estimator, PPO, AC(λ), GreedyAC in Figs. S3, S4, S5, S8 and S7 respectively. Conclusion 4. Bounded-support policy parameterizations are slower to converge compared to unbounded-support ones. We group the squashed and beta policy parameterizations as bounded-support, and the clipped Gaussian and Student’s t as unbounded-support. From Fig. 5 (middle), we observe that the effect depends on how strongly the action-space con- straints are enforced by the policy parameterization. With the default actor UTD ratio of 1, the beta policy consistently learns the slowest among all algorithms, followed by squashed 7 of 38 131030 Actor UTD ratio 0.6 0.7 0.8 0.9 1 1.1 1.2 131030131030 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Mean Coefficient of Variation Impact of Actor UTD ratio on Gaussian policy Impact of Actor UTD ratio on Beta policy GreedyAC GreedyAC PPO PPO SAC SAC Impact of Actor UTD ratio on variability Actor UTD ratioActor UTD ratio Fig. 7. Higher actor update-to-data ratio improves beta policies but destabilizes Gaussian policies. We computed results using top-5 learning rates for every policy− UTD actor combination while fixing the other hyperparameters to their default values from Table S3. Left and middle: Mean rewards obtained by GreedyAC, SAC and PPO when changing the actor UTD, where higher is better. Each dot is the average over all selected runs. The shaded area represents 95% bootstrap confidence interval over 50 runs with n=2000 resamples. The curves are drawn using a quadratic B-spline interpolation for visual clarity without altering pointwise values. With a Gaussian policy, the impact of changing UTD actor on the average performance is minimal: PPO and GreedyAC decrease, whereas SAC increases. On the other hand, the average performance of the beta policy improves very significantly as UTD actor is increased. Right: Mean variability for different Actor UTD ratios, where lower is better. Since we are aggregating results over multiple hyperparameter configurations, we compute the variability (CV i = σ i /|μ i |) separately for each (top-5) hyperparameter configuration and average over these variabilities to give us a single number (CV mean = 1 N P i CV i where N = 10 configurations). As we increase UTD actor , the variability increases for Gaussian and decreases for beta policy. With the best hyperparameter configurations, the beta policy achieves higher mean reward and lower mean variability than the Gaussian policy across all three learning algorithms. 0.00010.0010.010.1 0.0 0.1 0.2 0.3 0.4 Pathwise estimator squashed gaussian squashed student beta gaussian student 0.00010.0010.010.1 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Coefficient of Variation 0.00010.0010.010.1 5% relative variability Variability with pathwise (PW) gradient estimator Variability with likelihood ratio (LR) gradient estimator Mean reward gap between gradient estimators squashed gaussian squashed student Fig. 8. Pathwise gradient estimators are often unreliable with clipped and squashed policy parameterizations. We compare pathwise (PW / reparameterized) and likelihood ratio (LR / non-reparameterized) gradient estimators in Soft Actor-Critic by picking the same hyperparameters for each policy parameterization, and display their performance across different entropy values β. Left: absolute difference in mean reward obtained between the two gradient estimators, computed as ∆ log 10 (Reward) = log 10 (Reward LR )− log 10 (Reward PW ). The curves are drawn using a shape-preserving cubic interpolation to improve visual clarity without altering the pointwise values. The dashed baseline line represents the mean performance of the reparameterization trick variant. The shaded region represents the bootstrap confidence intervals based on n=2000 resamples. A point above the dashed line implies that the corresponding setting has a higher mean performance when used with an LR estimator. We observe that the policies with clipped sampling (Student’s t and Gaussian) obtain a higher mean reward with an LR estimator, whereas the policies with natural boundedness (beta) and squashed sampling (squashed Gaussian) behave similarly under both estimators. Middle and right: coefficient of variation (CV = σ/|μ|) of the total reward across training runs for the LR and PW versions, respectively. Bars show the relative variability across all policies at different entropy values, with smaller values indicating a more stable setting. The squashed Gaussian policies are the least stable under both estimators, with a very high variability when used with the PW estimator. policies and then clipped policies. The clipped policies are based on unbounded densities and therefore can be optimized in a relatively unconstrained parameter space. The squashed policies enforce bounds through a smooth non-linear trans- formation, which introduces additional curvature and sensi- tivity near the boundaries. In contrast, the beta policy en- forces bounds through its density, and its parameters couple together to control the location, shape, and skewness of the distribution simultaneously, making optimization more diffi- cult. Once we increase the actor UTD ratio (Fig. 5 (right)), the differences in convergence speed among the various policy parameterizations largely disappear. However, if the critic is not sufficiently accurate, a high actor UTD could amplify the critic’s errors and steer the policy towards poor solutions. Conclusion 5. Actor update-to-date ratio (UTD) is a safer tuning knob compared to the actor learning rate. Scaling the actor learning rate η actor by k and the actor UTD actor by k are both ways to control how much the param- eters are updated. We find scaling either has a very similar effect for the beta policy across the three critic learning rates η critic depicted in Figure 6. For a Gaussian policy, however, the behavior is different at high critic learning rates: increas- ing the actor learning rate produces a significantly stronger performance collapse compared to increasing the actor UTD ratio. This suggests that the Gaussian policy’s instability stems from simultaneous fast updates to both the critic and actor networks, rather than from actor parameter movements alone. This indicates that the actor UTD ratio is a safer tun- ing knob. Conclusion 6. Higher actor update-to-data ratio im- proves beta policies but destabilizes Gaussian policies. UTD actor has different effects depending on the policy param- 8 of 38 110100 Critic UTD ratio -0.25 -0.20 -0.15 -0.10 -0.05 0.00 0.05 0.10 Adaptive110100 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Coefficient of Variation Adaptive110100 10% relative variability 30% relative variability Variability with Actor UTD ratio Variability with Actor UTD ratio Adaptive critic UTD Mean reward gap between adaptive and fixed critic UTD ratios Critic UTD ratioCritic UTD ratio Fig. 9. Adaptive critic updating is more reliable than a fixed critic update-to-data ratio. For each combination of policy, actor UTD ratio UTD actor and critic UTD ratio UTD critic , we select the best-performing learning rate and aggregate over its 10 seeds. Left: Reward gap between the fixed critic UTD ratio UTD critic and adaptive critic UTD ratio, computed as ∆ log 10 (Reward) = log 10 (Reward fixed )− log 10 (Reward adaptive ), so values above the dashed reference line at zero (adaptive UTD performance) indicate that the fixed ratio outperforms adaptive, and values below indicate the opposite. Solid and dashed curves correspond to UTD actor = 1 and UTD actor = 10, respectively. Shaded bands represent 95% bootstrap confidence intervals over 50 runs with n=2000 resamples, and curves use shape-preserving cubic interpolation between measured points for visual clarity. Middle and right: coefficient of variation for UTD actor = 1 and UTD actor = 10, respectively, with adaptive UTD shown as the leftmost pair of bars in each panel for direct comparison. Horizontal dotted lines mark 10% and 30% relative variability as fixed reference points to ease comparison across panels and figures. Fixed critic UTD ratios are competitive with adaptive only for the beta policy at low actor UTD UTD actor = 1. In every other configuration, fixed ratios either yield lower mean reward (left panel, points below baseline) or substantially higher variability (middle and right panels). The Gaussian policy at UTD critic = 1 is the most extreme failure mode, with variability rising to roughly 11× the adaptive value when UTD actor = 10. eterization (Fig. 7). For beta policies, increasing UTD actor from 1 to 30 significantly improves the mean reward and lowers run-to-run variability consistently across PPO, SAC and GreedyAC. In contrast, for Gaussian policies, the mean reward changes minimally while the variability increases sig- nificantly, rising by a factor of four for SAC and roughly doubling for GreedyAC and PPO. Conclusion 7. Pathwise gradient estimators are often unreliable with clipped or squashed policies. Most continuous-control codebases use the pathwise (PW) gradient estimator by default, often without exposing it as a tunable choice. Whether this is a good default, however, depends on the policy parameterization. PW gradients flow through the sampled action, so any transformation that dis- torts this path such as clipping or squashing can negatively affect the gradient signal. In contrast, the likelihood ratio (LR) gradients are unaffected by this issue, since they do not differentiate through the sampled action. Fig. 8 shows three regimes. Firstly, clipped policies (Gaus- sian and Student’s t) suffer a significant drop in performance at higher entropies under PW, but remain robust under LR. This is likely because the high entropy pushes more sampled actions to the clipping boundary, where they contribute zero PW gradient. Secondly, squashed policies achieve similar mean rewards under both estimators, but a lower run-to-run variability under the LR estimator. Finally, the beta policy is essentially insensitive to the choice of estimator in both mean performance and variability, likely because its bounds are en- coded in the density rather than controlled by a transforma- tion, so there is no clipped or squashed path for the gradient to flow through. The practical implication is to treat PW as a choice rather than a default whenever the policy involves clipped or squashing, and to consider LR in those settings. Conclusion 8. Adaptive critic updating is more reliable than a fixed critic update-to-data ratio. Comparing the adaptive critic UTD ratio (Alg. 1) with the commonly used fixed critic UTD strategy (Alg. S1), we observe that adaptive updating largely avoids the fail- ure modes of fixed critic UTD ratios (Fig. 9). The worst case is UTD critic = 1, which produces 11x higher run-to- run variability for Gaussian policies and 5x for beta under UTD actor = 10, as well as lower mean reward across configu- rations. Higher fixed ratios of UTD critic = 10, 100 perform comparably; however, they require the practitioner to either know or tune the appropriate value in advance. The one case where the fixed ratio can outperform adaptive is beta with UTD actor = 1; however, both perform poorly compared to higher UTD actor = 10. Overall, we find adaptive updating replaces the more sensi- tive UTD critic hyperparameter with an easier-to-interpret er- ror threshold ω, which can be set based on the magnitude of the critic loss. In principle, a threshold ω defined relative to a normalized reward scale should transfer across environments more naturally than a raw update count, whose appropriate value depends on batch size, critic’s representational capac- ity, and the complexity of the environment. Discussion We organize the discussion around common themes that emerge from these results and their implications for the full sequential reinforcement learning setting. Common empirical themes. The first theme is that critic ac- curacy controls actor-critic stability, and many of the differ- ences in run-to-run variability we observed are connected to it. Mixing immediate reward into the policy gradient (AC(λ), with λ > 0) reduced the run-to-run variability across ev- ery policy parameterization we tested. A larger critic, with greater representational capacity to learn the value surface, reduced run-to-run variability across the entire learning-rate range. Adaptive critic updating, which keeps the critic error below a threshold before each actor update, outperformed every fixed update-to-data ratio we tried, except in one con- figuration that corresponded to a suboptimal agent. Most im- 9 of 38 portantly, PPO, which in a bandit setting requires no learned critic, produced some of the most favorable points on the performance-stability frontier. The common reason is that when the critic is wrong, the ac- tor inherits that error at every gradient step, and the result- ing instability manifests more prominently as high run-to-run variability rather than low average reward. For deployment, this means that instead of only considering whether to use a specific algorithm as a whole, we should consider whether the chosen implementation has sufficient critic capacity, sufficient evaluation steps before each greedification, and a convenient way to trade off bootstrapped estimates against the observed signal. Adaptive critic updating, in particular, is a general method that can be added to most actor-critic implementa- tions. The second theme is that policy parameterization is a domi- nant factor driving differences between agents. The same al- gorithm with two different policy parameterizations behaved more differently than two algorithms with the same parame- terization. Gaussian and beta policies react oppositely in re- sponse to the actor UTD: increasing the actor UTD improved beta’s mean reward and reduced its run-to-run variability, whereas the same change destabilized Gaussian policies. Gra- dient estimator choice interacts similarly with parameteriza- tion: the clipped Gaussian and Student’s t policies suffered significant performance degradation at higher entropies un- der pathwise gradients but remained robust under likelihood- ratio gradients, whereas beta was insensitive to the choice of gradient estimator. The squashed Gaussian policy, as the standard default in continuous-control SAC setups, sat in a strictly dominated region of the performance-stability fron- tier and exhibited high run-to-run variability under pathwise gradients. The takeaway is that switching the policy parame- terization within an algorithm can change the outcome more than switching the algorithm itself, a fact often overlooked in the standard practice of selecting an algorithm and adopting its default parameterization. Finally, many current default choices exhibit poor run-to-run variability, which is not easy to detect when only looking at average performance across runs. The pathwise gradient es- timator is the default in most continuous-control codebases, but can have high run-to-run variability under the standard choices of clipped and squashed policies. Fixed critic UTD ratios are usually set to the values that worked for the al- gorithm’s original set of benchmark environments, but this choice seems sensitive. The squashed Gaussian policy is a default for SAC, but it performed much worse than a beta policy. Each of these component-level decisions contributes significantly to the deployment reliability. Implications beyond the bandit setting. We can consider out- comes in the more general Markov decision process (MDPs) setting considered in reinforcement learning. In an MDP, the agent’s actions affect a state that evolves over time, and the consequences of an action are distributed over a long horizon of future rewards rather than realized immediately. Mov- ing to the general MDP setting introduces several new ques- tions to investigate, such as the effect of bootstrapped tar- gets in the critic, exploration operating over the state space rather than the action space alone, more significant distri- bution shifts in the replay as the policy improves, and the tighter coupling between actor and critic as now the actor impacts the critic’s bootstrapped target. Several conclusions in this paper are based on per-step mech- anisms that should transfer to the MDP setting. Clipped and squashed policy parameterizations having distorted gra- dients under pathwise estimators is a property of how a single sampled action contributes to a gradient step; states and tra- jectories do not impact this. The same reasoning applies to the beta policy’s robustness to estimator choice, which stems from its bounds being imposed by the density itself rather than by a transformation. Adaptive critic updating can also be used in the MDP setting, and should have similar ben- efits. There could be new outcomes, however, because the measured error (the temporal difference error) is now a sur- rogate for the value error, unlike the bandit setting, where the squared reward error is a clear signal of critic accuracy. There are some findings that are likely to be more notably different in the MDP setting compared to the bandit set- ting. The optimal AC(λ) mixing point in our experiments was around λ = 0.9, which placed most of the weight on the observed reward rather than the critic. In MDPs, the alter- native to a bootstrapped critic estimate is not a single-step reward but a multi-step or λ-return—equivalently a gener- alized advantage estimator in PPO—whose variance grows with the horizon and the discount factor. The mixing param- eter trades off the bias of an inaccurate critic and the variance of an unbiased target. This trade-off is the same in MDPs, but the variance is much larger. Furthermore, if the data is off-policy, we need to incorporate importance-sampling ratios into the λ-return, which further increases variance. The use of λ = 0, as in SAC, MPO, and GreedyAC, is more typical when using buffers in the MDP setting, as this avoids these is- sues with off-policy λ-returns. In the MDP setting, on-policy algorithms like PPO do not leverage large replay buffers and are often less sample efficient than algorithms like SAC; we expect this sample efficiency difference to be more significant in the MDP setting than it was in our bandit setting. Fur- ther, in the MDP setting, changes to the actor impact the critic update by changing the bootstrap target, whereas in the bandit setting it only impacts the data that is gathered. With a higher actor UTD in MDPs, the critic now has to track a faster changing actor, and so we might find a more complex relationship between actor UTD and critic UTD. Future directions. The most direct next step is to test which of our conclusions transfer to the standard MDP setting. Each conclusion is falsifiable through a focused empirical study. Another direction is to extend the analysis to non-stationary settings, which are common in many real-world control tasks. Industrial controllers must adapt as the equipment degrades, as the operating conditions change, and as the system’s oper- ating points change. The component choices that dominate reliability in our stationary setting may not necessarily be the ones that dominate reliability under non-stationarity, and identifying which ones matter would be valuable in practice. Materials and Methods Evaluation Criteria. What makes a good learning algorithm? We want an algorithm that achieves 1) High overall mean reward during deploy- ment, 2) Low run-to-run variability, and 3) Low sensitivity 6 to different hy- perparameters. Accordingly, we analyze performance distributions using 6 In RL, hyperparameter sensitivity typically is used to mean one of two things. An algorithm’s hyperparameters are sensitive if: 1) good performance in two different environments requires tuning the hyperparameters for each, 2) Small changes to the 10 of 38 the coefficient of variation (CV = σ/|μ| whereσ is the standard deviation and μ is the mean of reward r) as a scale-invariant measure of run-to-run variability. A CV of 0.1 represents 10% relative variability regardless of the reward scale, allowing direct comparisons across algorithms, hyper- parameter configurations, and, in principle, across environments. Addi- tionally, we report bootstrap confidence intervals on aggregate statistics to quantify uncertainty. The figures in the main text are presented as evi- dence for the conclusions and are summarized from the extended results in Appendix 5. ExperimentalSetup. Weconductedallexperimentsinour Backwashing-PID Environment.Each hyperparameter configura- tion was trained using 10 random seeds for T = 5000 environment interaction steps. At every step, we sampled a triplet of PID gains: p,i,d ∈ [0, 20] from the policy, which were scaled appropri- ately depending on policy parameterization.Each experiment is swept over the entire range of learning rates for the actor and critic 0.00012, 0.00037, 0.0011, 0.0033, 0.01, 0.03, 0.09,and both are optimized with Adam (β 1 ,β 2 ) = (0.9, 0.999). For the main sweeps, we fix actor and critic learning rates to be equal to reduce the hy- perparameter search space, and vary them independently in specific experiments (Fig 6). The critic is a 2-hidden-layer MLP with 64 units per layer. Adaptive critic UTD follows Algorithm 1 with tolerance ω = 10 −3 and a maximum of 100 updates per interaction step. Replay buffer sampling begins after the first step; the mini-batch size is min(512, |B|) where |B| is the current buffer size. We rank the hyperparameters by their mean reward across the entire training run (area under the curve). Since the rewards are normalized to [−1, 0], we report the performance as− log 10 (|Reward|) so that higher values indicate better performance. For brevity, figure axes display this as log 10 (Reward) without negation and absolute value. See Table S4 for an overview of all the algorithm-specific hyperparameters considered under this study, Table S1 for environment hyperparameters, and Tables S2, S3 for the fixed hyperparameters. Backwashing-PID Environment. We evaluate actor–critic methods on a real-world control task: tuning a PID controller (52) that regulates water flow for backwashing for drinking water treatment. Our motivation is to explore algorithmic behavior across a range of component choices and hyperparameters in a realistic setting, since tuning controllers on live sys- tems is costly and often impractical. To this end, we built a simulator by fitting a second-order polynomial to empirical data collected from the drinking water treatment plant in Drayton Valley, Alberta (pump speed s and flow rate f ). We will describe our simulator in this section. This is a continuing problem with no episodes or state resets. Our simulator is defined over discrete internal (hidden to the agent) timesteps j ∈1,...,J, corresponding to a single control cycle. Each of these internal timesteps represents 1 second of water flow through the pump. At each agent–environment interaction step t, the agent se- lects PID gainsp t ,i t ,d t , which are held constant for that control cycle. Then, the environment steps through the internal timesteps j as follows: 1. Flow generation: Given current pump speed s j (with s 0 = 20), the simulator produces the noisy flow rate: f j = 6.0553× 10 −4 × s 2 j − 2.530755× 10 −2 × s j + 0.26727398 + ε (f ) j (2) where ε (f ) j ∼ N (0,σ 2 f ) represents the measurement noise with the fixed standard deviation of σ f = 0.003 in all our experiments. 2. Error and PID control: We compute the flow error E j = f j −f ideal and update the pump speed: s j+1 = p t E j + i t j X k=1 E k ∆t + d t E j − E j−1 ∆t (3) s j+1 = clip (s j ,s min ,s max )(4) where ∆t = 1 second is the time period between each internal timestep, s min = 0 is the minimum pump speed, s max = 100 the maximum pump speed, and E 0 = 0 is the initial error. hyperparameters cause significant changes to performance in one environment. In this work, we are concerned with #2. After completing a control cycle, we can compute the reward r t based on the performance of the controller. The goal in this environment is to adjust the PID controller parameters such that the flow rate f j over the control cycle 1...J matches the setpoint value f ideal as quickly and stably as possible. To this end, we first define an asymmetric deviation term that penalizes overshooting more heavily than the undershooting: v j = 5(f j − f ideal ), if f j > 1.05f ideal , f j − f ideal ,otherwise. (5) Using Eq. 5, we compute the total deviation vector⃗v = (v 1 ,...,v J ) and compute the reward for timestep t as: r t = min 0, − ∥⃗v∥ 2 110 + ε (r) t , ε (r) t ∼N (0,σ 2 r )(6) where∥⃗v∥ 2 denotes the Euclidean norm of the deviation vector, and the additive ε (r) term models fixed environmental noise using σ r = 0.01. The constant 110 and the clipping ensure that the rewards are roughly normalized to the [−1, 0] range. All key environment hyperparameters are summarized in Table S1. Fig. S1 illustrates the reward surfaces. Adaptive critic updates. Unless otherwise stated, we use an adaptive critic UTD strategy (Alg. 1), rather than the commonly used fixed critic UTD strategy (Alg. S1). At each environment step, we perform critic up- dates until the batch error falls below the threshold ω or we reach the maximum number of updates n steps . This typically keeps the critic error below ω, subject to the ceiling set by n steps , before the critic is used to up- date the actor, thereby reducing one confounding source when studying actor-critic design choices. Additionally, keeping the critic error bounded before each update is consistent with two-timescale convergence analy- sis of actor-critic methods, which require the critic to update on a faster timescale than the actor (44–46). This method introduces two hyperpa- rameters (ω and n steps ), but we find the threshold ω is easier to set than a fixed UTD ratio because it is interpretable relative to the reward scale. n steps can be set to a reasonably high value depending on the available compute. See Fig. 9 for a detailed comparison against fixed UTD ratios. Algorithm 1 Action-value Update with Adaptive Critic UTD ratio 1: Input: action-value parameters w, replay buffer B 2: Hyperparameters: error tolerance ω (default 0.001), maximum number of critic updates n steps (default 100), batch size |B| 3: Initialize error err =∞, steps = 0 4: while steps < n steps and err > ω do 5:steps ← steps +1 6:Sample a mini-batch B from buffer B 7:Compute mean-squared error: err ← 1 |B| X (a,r)∈B r− q w (a) 2 8:Update parameters by gradient descent: w ← w− α∇ w err 9: end while Data, Materials, and Software Availability. All the study data are in- cluded in this article and the SI Appendix, including the original data from the water treatment plant used to construct our simulator. The code is released at https://github.com/haseebs/deconstruct-ac. Acknowledgements. This work was supported by NSERC Discovery Grants (M.W., A.W.), Canada CIFAR AI Chairs Program (M.W., A.W.), and a Canada Research Chair (M.W.). Computational resources were provided by the Digital Research Alliance of Canada. References 1. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550:354–359, 2017. 11 of 38 2. Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610:47–53, 2022. 3. Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Sub- ramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Dür, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602:223–228, 2022. 4. Jonas Degrave, Federico Felici, and et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602:414–419, 2022. 5. Shuo Feng, Haowei Sun, and et al. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, pages 620–627, 2023. 6. Dong Wang, Hanmo You, Lingwei Zhu, Kaiwei Lin, Zheng Chen, Chen Yang, Junji Yu, Zan Wang, and Junjie Chen. A survey of reinforcement learning for software engineering, 2025. 7. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 8. Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Confer- ence on Learning Representations, 2018. 9. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on rein- forcement learning. In International conference on machine learning, pages 449–458. Pmlr, 2017. 10. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. 11. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research, 47:253–279, 2013. 12. Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pages 1373–1383. PMLR, 2021. 13. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pages 1861–1870, 2018. 14. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012. 15. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. 16. Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. 17. Samuel Neumann, Sungsu Lim, Ajin George Joseph, Yangchen Pan, Adam White, and Martha White. Greedy actor-critic: A new conditional cross-entropy method for policy im- provement. In The Eleventh International Conference on Learning Representations, 2023. 18. Lingwei Zhu, Matthew Schlegel, Han Wang, and Martha White. Offline reinforcement learn- ing with tsallis regularization. Transactions on Machine Learning Research, 2024. 19. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1889–1897, 2015. 20. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017. 21. Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. 22. Eslam G. Al-Sakkari, Ahmed Ragab, Mohamed Ali, Hanane Dagdougui, and Daria C. Boffito. Simulate intelligently: Causal incremental reinforcement learning for streamlined industrial chemical process design optimization. Journal of Environmental Chemical Engineering, 13(6):120167, 2025. ISSN 2213-3437. doi: https://doi.org/10.1016/j.jece.2025. 120167. 23. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine Learning, page 229–256, May 1992. 24. Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23(253):1–79, 2022. 25. Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003. 26. Samuel Neumann, Jiamin He, Adam White, and Martha White. Investigating the utility of mirror descent in off-policy actor-critic. In Reinforcement Learning Journal, 2025. 27. Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of regularization in rl. In Advances in Neural Information Processing Systems 33, pages 1–12, 2020. 28. Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. In International Conference on Learning Representations, 2022. 29. Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C. Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 8619–8649, 2022. 30. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learn- ing. In 19th International Conference on Machine Learning (ICML), pages 267–274, 2002. 31. Jiamin He, Samuel Neumann, Jincheng Mei, Adam White, and Martha White. Revisiting mixture policies in entropy-regularized actor-critic. In Second Workshop on Aligning Rein- forcement Learning Experimentalists and Theorists of NeurIPS 2025, 2026. 32. Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradi- ents in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning, pages 834–843. PMLR, 2017. 33. Taisuke Kobayashi. Student-t policy in reinforcement learning to acquire global optimum of robot control. Applied Intelligence, 49(12):4335–4347, 2019. 34. Lingwei Zhu, Haseeb Shah, Han Wang, Yukie Nagai, and Martha White. q-exponential policy optimization. In International Conference on Learning Representations (ICLR), 2025. 35. Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023. 36. Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023. 37. Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy opti- mization. Advances in Neural Information Processing Systems, 37:53945–53968, 2024. 38. Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018. 39. Vineet Jain, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. Sampling from energy- based policies using diffusion. Reinforcement Learning Journal, 6:2291–2307, 2025. 40. Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, pages 16828–16847, 2022. 41. Mansi Maheshwari, John C Raisbeck, and Bruno Castro da Silva. Addressing the plasticity- stability dilemma in reinforcement learning. arXiv preprint arXiv:2512.01034, 2025. 42. Pierluca D’Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep reinforcement learning workshop neurIPS 2022, 2022. 43. Claas Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand, and Igor Gilitschen- ski. Mad-td: Model-augmented data stabilizes high update ratio rl. In International Confer- ence on Learning Representations, volume 2025, pages 71619–71649, 2025. 44. Vivek S Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5):291–294, 1997. 45. Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999. 46. Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor–critic algorithms. Automatica, 45(11):2471–2482, 2009. 47. Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75–84, 1990. 48. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 49. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax. arXiv preprint arXiv:1611.01144, 2016. 50. Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction properties of the reparameterization trick. In The 22nd international conference on artificial intelligence and statistics, pages 2711–2720. PMLR, 2019. 51. Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4065–4074. PMLR, 2018. 52. Karl J Astrom. Pid controllers: theory, design, and tuning. The international society of measurement and control, 1995. 53. Lingwei Zhu, Han Wang, and Yukie Nagai. Fat-to-thin policy optimization: Offline rl with sparse policies. In International Conference on Learning Representations (ICLR), 2025. 54. Yana Suchikova, Natalia Tsybuliak, Jaime A Teixeira da Silva, and Serhii Nazarovets. Gaidet (generative ai delegation taxonomy): A taxonomy for humans to delegate tasks to generative artificial intelligence in scientific research and publishing. Accountability in Research, 33(3): 2544331, 2026. 12 of 38 Supplementary Materials Contents 1 Reward Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Additional Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3 Policy Parameterizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4 Actor-Critic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5 Extended Variability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 6 Reproducibility and Disclosures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 List of Algorithms S1 Action-value update with fixed critic UTD ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 S2 Actor update for AC(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 S3 Actor update for DDPG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 S4 Actor update for PPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 S5 Actor update for Soft Actor-Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 S6 Actor update for MPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 S7 Actor update for Greedy Actor-Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 List of Figures S1 Reward surfaces for the Backwashing-PID environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 S2 Mirror descent does not consistently improve Soft Actor-Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 S3 Analysis of Soft Actor-Critic with reparameterization trick (pathwise gradient) . . . . . . . . . . . . . . . . . 23 S4 Analysis of Soft Actor-Critic without reparameterization trick (log-likelihood gradient) . . . . . . . . . . . . 24 S5 Analysis of actor UTD ratio in Proximal Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 S6 Analysis of Soft Actor-Critic with mirror descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 S7 Analysis of Greedy Actor-Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 S8 Analysis of AC(λ) and changing environmental noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 S9 Analysis of Maximum a Posteriori Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 S10 Analysis of on-policy batch size in Proximal Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . . . 30 S11 Analysis of Deep Deterministic Policy Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 S12 Analysis of AC(λ) and Soft Actor-Critic rewards with Gaussian policy . . . . . . . . . . . . . . . . . . . . . . 32 S13 Analysis of AC(λ) and Soft Actor-Critic rewards with beta policy . . . . . . . . . . . . . . . . . . . . . . . . . 33 List of Tables S1 Environment hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 S2 Fixed values for hyperparameters common across the tested algorithms . . . . . . . . . . . . . . . . . . . . . . 36 S3 Fixed values for algorithm-specific hyperparameters across all the tested algorithms . . . . . . . . . . . . . . 37 S4 Overview of experiments and swept hyperparameters and component choices. . . . . . . . . . . . . . . . . . . 38 Supplementary Material13 of 38 Supplementary Note 1: Reward Surfaces Fig. S1. The Backwashing-PID environment is non-trivial despite its simplicity. The reward is the clipped negative norm of the per-step flow-rate deviation from the setpoint, with overshoots penalized more heavily than undershoots. Each panel fixes the derivative gain d and shows the reward as a function of the proportional gain p (x-axis) and integral gain i (y-axis), while d increases across panels. All three gains (actions) are bounded to [0, 20]. Three properties make this environment challenging for an actor-critic agent. First, most of the action space is dominated by low-reward plateaus, while the high-reward region is a narrow area that the policy must concentrate on without overshooting. Second, the three gains couple non-trivially: changing d reshapes the entire (p,i) surface, so the agent cannot learn each dimension independently. Third, the agent must learn under two independent sources of noise, i.e., the measurement noise ε (f) ∼N (0,σ 2 f ) in the flow simulator and reward noise ε (r) ∼N (0,σ 2 r ) in the observed return. Supplementary Material14 of 38 Supplementary Note 2: Additional Conclusions Fig. S2. Mirror descent does not consistently improve Soft Actor-Critic.The y-axis shows ∆ log 10 (Reward) = log 10 (Reward MD ) − log 10 (Reward Baseline ), where the rewards on the R.H.S. are the rewards averaged over the top three learning rates per hyperparameter configura- tion. The columns represent changing entropy levels β, the x-axis represents the Mirror Descent (MD) period η md , and the colours represent different MD step sizes λ = 1/τ . The curves are drawn using a quadratic B-spline interpolation across MD periods without altering pointwise values. Values above the zero line represent higher average rewards under MD, while the values under represent performance worse than the Gradient Descent (GD) baseline. At the two lower entropies (β = 0.001, 0.01), all MD configurations perform worse than GD. The worst case is at 1/τ = 0.5 and high η md , where the performance drops by roughly 5x compared to GD. At the highest entropy (β = 0.1), we observe a small improvement at the lowest MD step sizes (represented by squares), but the absolute performance at this entropy is still significantly lower than the lower entropies (Fig. S6). Conclusion 9. Mirror descent adds complexity without consistent empirical gains In Fig. S2, we observe that increasing the mirror descent period η md generally resulted in similar or significantly worse performance. Increasing the mirror descent step size λ (where λ = 1/τ) resulted in worse mean rewards at smaller entropies (β ∈ 0.001, 0.01) but higher at the largest tested entropy (β = 0.1). However, the mean reward at the larger entropy is still significantly lower than smaller entropies (Fig. S6). Overall, the best performance is achieved at the smallest values of mirror descent stepsize λ and period η md . This indicates the stronger the mirror descent, the lower is the resulting performance. Conclusion 10. Beta policy is more robust to higher learning rates when the critic’s contribution is minimized PPO (Fig. S10) and AC(λ > 0.9) (Fig. S8) show improved robustness to higher learning rates when coupled with the beta policy. In the bandit setting, PPO has no critic, and we observe that the beta policy at lower actor UTD ratios can perform just as well as at higher ratios when the learning rate is tuned. In all the other settings, the beta policy struggled to learn unless the actor UTD ratio was increased. Conclusion 11. Actor UTD reshapes learning-rate sensitivity The relationship between learning rate and mean performance is typically similar to an upside-down parabola: with the best setting in the middle, and a poorer performance at the higher and lower extremes. We observe that increasing the actor UTD ratio can alter the shape of this parabola, and in some cases, reduce the sensitivity to learning rate hyperparameter. This implies that in certain settings, an arbitrarily chosen learning rate from a reasonable range is more likely to perform well. We observe this reduction in sensitivity with all the policy losses that we used to analyze UTD actor : SAC with both gradient estimators (Figs. S3-S4), PPO (Fig. S5), GreedyAC (Fig. S7) and MPO (Fig. S9). However, the run-to-run variability depends on the policy parameterization. As η actor is increased, the Gaussian, Student’s t, and squashed policies generally exhibit higher variabilities, while the variability of the beta policy is reduced. Due to this, the reduced learning-rate sensitivity is useful only with the beta policy parameterization. Conclusion 12. Averaging over more candidate actions stabilizes policy updates Across multiple policy losses, we observe that increasing the number of samples used to compute the policy gradient increases the performance and/or reduces overall run-to-run variability. In PPO, increasing the on- policy batch size increases mean performance, particularly at higher entropies (Fig. S10). We do not observe an improvement in variability for PPO because it is already too small. In GreedyAC, increasing the number of pro- posal actions η top both improves the performance and reduces variability (Fig. S7). Although the improvements are monotonic within our tested ranges, we surmise that there can be diminishing returns or adverse effects if these hyperparameters are increased further. Supplementary Material15 of 38 Supplementary Note 3: Policy Parameterizations Gaussian Policy: One of the most common policy parameterizations in the literature is based on the Gaussian distribution, which assumes that the ideal actions are normally distributed around a learned mean. In the scalar bandit setting, a policy using this reparameterization can be written as: π θ (a) . = 1 σ θ √ 2π exp − (a− μ θ ) 2 2σ 2 θ (7) where μ θ ∈ R is the mean and σ θ ∈ R + is the standard deviation. To ensure positivity, the standard deviation is typically parameterized as an exponential of an unconstrained parameter, e.g., σ θ = exp(σ raw ). The Gaussian distribution has infinite support, which is not practical for policy gradient algorithms. We will discuss two methods of restricting the support and sampling from a Gaussian: 1) Clipped Gaussian Sampling. In this approach, the raw mean μ raw is passed through a tanh nonlinearity and scaled to fit within the action bounds: μ = a min + a max − a min 2 (tanh(μ raw ) + 1)(8) a = μ + σ· ε, ε∼N (0, 1)(9) a scaled = clip(a,a min ,a max )(10) 2) Squashed Gaussian Sampling. An alternative approach is to sample from the Gaussian and apply a smooth tanh squashing function: ̃a = μ + σ· ε, ε∼N (0, 1), a = tanh( ̃a)(11) followed by rescaling: a scaled = a min + a max − a min 2 (a + 1)(12) The squashed Gaussian applies tanh directly to the sampled action, which alters the distribution’s support and requires a log-determinant Jacobian correction to maintain accurate log-probability estimates (13). Beta Policy The beta distribution belongs to non-location-scale family of distributions and has a re- stricted support in the interval [0, 1], which ensures that no probability density falls outside the boundary and eliminates the need for any clipping or squashing. This distribution offers flexibility in modelling peaked, skewed, or bi-modal distributions within a bounded domain. A policy using the beta parameterization can be defined as: π θ (a) . = a α θ −1 (1− a) β θ −1 B(α θ ,β θ ) , a∈ [0, 1](13) where α θ ,β θ ∈ R + are shape parameters, and B(α θ ,β θ ) is the beta function defined as: B(α θ ,β θ ) = Γ(α θ )Γ(β θ ) Γ(α θ + β θ ) (14) where Γ(·) is the gamma function. In a beta policy, the actions are sampled and scaled as: a∼ B(α θ ,β θ ) a scaled = a min + (a max − a min )· a (15) Similar to the Gaussian policy, the beta policy also supports the reparameterization trick. In this work, we restrict α θ > 1;β θ > 1, which ensures that the policy is unimodal. Student’s t Policy The Student’s t distribution allows for heavier tails and is more robust to outliers than the Gaussian distribution. A policy using the Student’s t parameterization is given by: π θ (a) . = Γ ν θ +1 2 √ ν θ πσ θ Γ ν θ 2 1 + (a− μ θ ) 2 ν θ σ 2 θ − ν θ +1 2 (16) where μ θ ∈ R is the location, σ θ ∈ R + is the scale, and ν θ ∈ R + is the degrees of freedom. The mean exists only when ν θ > 1, and the variance exists only when ν θ > 2. The sampling and bounding of support is done similarly to the clipped Gaussian, as described by Eq. 8,9,10. Supplementary Material16 of 38 Compared to the Gaussian policy, the Student’s t policy produces heavier-tailed action distributions, which can promote better exploration-exploitation (34, 53). An important special case is ν θ = 1, which corresponds to the Cauchy distribution, an extremely heavy-tailed distribution without a defined mean or variance. As ν θ → ∞, the Student’s t distribution converges to a relatively lighter-tailed Gaussian, recovering the standard Gaussian policy as a limiting case. In practice, a Student’s t policy with ν θ ≥ 30 is sufficiently similar to the Gaussian policy. Deterministic Policy A deterministic policy directly outputs an action: a = μ θ .(17) This approach is used in deterministic actor-critic methods such as DDPG. Although it eliminates action stochas- ticity, external noise (e.g., Ornstein-Uhlenbeck or Gaussian noise) is typically added during training to encourage exploration. Supplementary Material17 of 38 Supplementary Note 4: Actor-Critic Algorithms Critic Update. All of the algorithms that we discuss share the same adaptive critic update as described in the main text. In practice, however, most implementations utilize a fixed critic UTD ratio UTD critic , which involves repeatedly sampling from a replay buffer and updating the critic network using regression for UTD critic steps. This procedure is described in Alg. S1. Algorithm S1 Action-value update with fixed critic UTD ratio 1: Input: Action-value parameters w, replay buffer B 2: Hyperparameters: critic UTD ratio UTD critic , batch size |B| 3: for steps = 1, 2, 3,..., UTD critic do 4:Sample a mini-batch B from buffer B 5:Compute mean-squared error: err ← 1 |B| X (a,r)∈B r− q w (a) 2 6:Update parameters by gradient descent: w ← w− α∇ w err 7: end for Actor Critic λ. On-policy AC(λ) allows us to range from an unbiased, higher variance REINFORCE update, with λ = 1, to a more biased, lower-variance actor-critic algorithm with λ = 0 that only uses q w with no reward sample in the actor update. The algorithm is naturally on-policy, because we update with the reward from the action that the agent took. If we wanted to update from (a,r) stored in the replay buffer, then we would have to use importance sampling to correct for the action distribution. It is rare to do so, and instead, to use λ-returns, it is typically more common to simply run on-policy algorithms like PPO. Note that on-policy AC(λ) is like PPO in this way, but does not have all the extra tricks related to clipping. The complete algorithm is described in Alg. S2. Algorithm S2 Actor update for AC(λ) 1: Initialize: policy parameters θ, action-value parameters w, empty replay buffer B, value baseline v = 0 2: Hyperparameters: Value-reward mixing coefficient λ∈ [0, 1], value baseline decay rate η v (default 0.1), actor and critic stepsizes 3: for t = 1, 2, 3... do 4:Sample action A t ∼ π θ 5:Execute action A t , observe reward R t 6:Update value baseline: v ← (1− η v )v + η v R t 7:Store (A t ,R t ) in replay buffer B 8:Update q w using Algorithm S1 9:Update π θ by gradient ascent using gradient δ∇ θ lnπ θ (a) for δ . = (λR t + (1− λ)q w (A t ))− v 10: end for Supplementary Material18 of 38 Deep Deterministic Policy Gradient. DDPG learns a deterministic policy. For a bandit, this means it’s policy parameters θ correspond to an estimate of the best action. DDPG is an off-policy method that learns the greedy policy (θ), and adds stochastic noise for exploration. In particular, we use Gaussian noise, with clipping to ensure the action remains in the bounded range. For consistency in the pseudo-code, we still write A t ∼ π θ , which in this case means we sample ε t ∼N (0,σ) for noise variance σ > 0, and set A t = clip A (θ + ε t ). The update for DDPG modifies the policy parameters by taking the gradient ∇ θ q w (π θ ), to adjust the policy parameters to get a higher value on q w (to get an action where q w (a) is higher). Using the chain rule, we take ∇ a q w (a) and then look at how the action changes as θ changes. In the bandit setting, θ actually equals the action, so the update for the current parameters θ t uses ∇ θ q w (θ t ) which is the same as ∇ a q w (θ t ). Algorithm S3 Actor update for DDPG 1: Initialize: policy parameters θ, action-value parameters w, value baseline v = 0 2: Hyperparameters: Gaussian noise variance σ, Gaussian noise decay rate ζ, batch size b 3: for t = 1, 2, 3... do 4:Sample A t = clip A (θ + ε t ) with ε t ∼N (0,σ) 5:Execute action A t , observe reward R t 6:Store (A t ,R t ) in replay buffer B 7:Update q w using Algorithm S1 8:Update θ with gradient ascent step using gradient ∇ θ q w (θ) 9:Anneal noise using σ = σ∗ ζ 10: end for Proximal Policy Optimization. At each iteration t, PPO performs multiple steps of SGD on the objective function of the following unconstrained optimization problem: max π E a∼π θ t min π θ (a) π θ t (a) (R− v), clip π θ (a) π θ t (a) , 1− ε, 1 + ε (R− v) . Recall that for our bandit setting there is no state and therefore the advantage function degenerates to one- sample Monte Carlo estimate (R t −v), where v is a scalar baseline maintained as an exponential moving average of observed rewards. The hyper-parameter ε determines how the policy ratio π θ /π θ t is clipped. Compared to other actor-critic algorithms, the clipping stands out as it allows the trust region to be always satisfied. Algorithm S4 Actor update for PPO 1: Initialize: policy parameters θ, value baseline v = 0, empty on-policy replay buffer B 2: Hyperparameters: number of actor updates UTD actor (default 1), entropy coefficient β, clipping threshold ε, on-policy minibatch size b, baseline learning rate η v , entropy sample count M 3: for t = 1, 2, 3... do 4:Sample action A t ∼ π θ 5:Execute action A t , observe reward R t 6:Update baseline v ← (1− η v )v + η v R t 7:Store (A t ,R t , lnπ ̄ θ (A t )) in on-policy replay buffer B 8: if mod(|B|,b) == 0 then 9:for steps = 1, 2, 3,..., UTD actor · b do 10:Get samples a i 1:b , r i 1:b , lnπ ̄ θ (a i ) 1:b 11:Normalize rewards r i ← (r i − ̄r)/(σ r + 10 −7 ) 12:Compute advantage ˆ δ i = r i − v 13:Evaluate new log-probabilities: lnπ θ (a i ) 1:b 14:Compute ratio ρ i = exp(lnπ θ (a i )− lnπ ̄ θ (a i )) 15:Policy loss L θ =− 1 b P i min ρ i ˆ δ i , Clipρ i , 1− ε, 1 + ε ˆ δ i 16:Estimate entropy H(π θ )≈− 1 M P M j=1 lnπ θ ( ̃a j ) with ̃a j ∼ π θ 17:Update θ using gradient descent on L θ − β·H(π θ ) 18:end for 19:Empty on-policy replay buffer B 20: end if 21: end for Supplementary Material19 of 38 Soft Actor-Critic. As introduced before, SAC (13) encourages exploration by maximizing the Shannon entropy augmented objective that leads to the Boltzmann policy. This leads to minimizing a reverse KL loss L SAC (θ) : = KL (π θ ||π ent ) = KL π θ exp (q w (a)/β) Z = E π θ lnπ θ − q w (a)− v β , The parametrized policy π θ is projected to be close to the Boltzmann policy. In practice, π θ is chosen to be the Gaussian policy by default, but potentially a more exploring policy like the Student’s t could lead to better performance. Depending on action values, π ent can have multiple modes and heavy tails. The Gaussian may not be able to fully capture these characteristics. A mirror descent version of SAC penalizes going too far from the previous policy π t by adding a KL penalty KL(π θ ||π t ). Note that the reference policy here π t is different from π ent : L SAC-MD (θ) : = E π θ lnπ θ − q w (a)− v β − τ ln π θ (a) π t (a) , where τ denotes the weighing coefficient. Algorithm S5 Actor update for Soft Actor-Critic 1: Initialize: policy parameters θ, action-value parameters w, empty replay buffer B, value baseline v = 0 2: Hyperparameters: number of actor updates UTD actor (default 1), actor and critic stepsizes, number actions to sample m, entropy coefficient β, value baseline decay rate η v (default 0.1), policy from the last update (for mirror descent) π t 3: for t = 1, 2, 3... do 4:Sample action A t ∼ π θ 5:Execute action A t , observe reward R t 6:Update value baseline: v ← (1− η v )v + η v R t 7:Store (A t ,R t ) in replay buffer B 8:Update q w using Algorithm S1 9: for steps = 1, 2, 3,..., UTD actor do 10:Sample m actions from policy a 1 ...a m ∼ π θ 11:Update policy by gradient ascent using: 1 m X a∈a 1 ...a m (q w (a)− v)− β lnπ θ (a) ∇ θ lnπ θ (a) 12:or Mirror Descent: 1 m X a∈a 1 ...a m (q w (a)− v)− β lnπ θ (a)− τ ln π θ (a) π t (a) ∇ θ lnπ θ (a) 13:π t ← π θ 14: end for 15: end for Supplementary Material20 of 38 Maximum a Posteriori Policy Optimization. MPO is also derived from the KL regularization perspective: π kl . = arg max π E π [q(A)]− αKL(π||π t ) where the reference policy π t denotes the last policy distribution. The solution is π kl (a)∝ π t (a) exp (q w (a)/τ ). MPO leverages this fact in minimizing the KL loss between π θ and π kl : min π θ KL(π kl ||π θ ) = min π θ E a∼π kl [lnπ kl (a)− lnπ θ (a)] = min π θ E a∼π t exp q w (a) τ (lnπ kl (a)− lnπ θ (a)) = min π θ E a∼π t − exp q w (a) τ lnπ θ (a) where the last equation is because the term lnπ kl does not depend on the optimization variable θ. Therefore, the MPO actor objective can be understood as maximizing the log-likelihood weighted by the exponential of action value. The original MPO algorithm adds another KL penalty to the loss above and makes the coefficient α learnable: min α>0 max π θ E a∼π t exp q w (a) τ lnπ θ (a) + α (ε− KL (π t ||π θ )), where ε denotes the trust region threshold, α denotes the weighing coefficient. The complete algorithm is described in Alg. S6. Algorithm S6 Actor update for MPO 1: Initialize: policy parameters θ, action-value parameters w, value baseline v, policy from last update π t 2: Hyperparameters: number of actor updates UTD actor (default 1), number of action proposals m, KL coefficient τ, trust region threshold ε, dual variable learning rate η α , value baseline decay rate η v (default 0.1) 3: for t = 1, 2, 3... do 4:Sample action A t ∼ π θ 5:Execute action A t , observe reward R t 6:Update value baseline: v ← (1− η v )v + η v R t 7:Store (A t ,R t ) in replay buffer B 8:Update q w using Algorithm S1 9: for steps = 1, 2, 3,..., UTD actor do 10:Sample m actions from last policy a 1 ...a m ∼ π t 11:Compute scale exp q w (a 1:m )−v τ 12:Update policy by minimizing the loss: 1 m X a∈a 1 ...a m − exp q w (a)− v τ lnπ θ (a)− α ε− ln π t (a) π θ (a) 13:Update coefficient α by minimizing the following loss using learning rate η α ∇ α 1 m X a∈a 1 ...a m α ε− ln π t (a) π θ (a) 14:Clamp α← max(α, 0) 15:π t ← π θ 16: end for 17: end for Supplementary Material21 of 38 Greedy Actor-Critic. GreedyAC (17) aims to learn from unbiased rewards but at the same time enjoys expedited exploration from Shannon entropy augmented rewards. To this end, GreedyAC maintains an additional proposal policy for exploration by maximizing Shannon entropy augmented rewards. Its actor policy maximizes unbiased reward and learns from the high-quality actions generated by the proposal policy. To simplify notations, we use I to denote the set of high quality actions. L GreedyAC, prop (θ ′ ) := E a∈I [− lnπ θ ′ (a)−H (π θ ′ )], L GreedyAC, actor (θ) := E a∈I [− lnπ θ (a)]. GreedyAC maximizes log-likelihood of the actor and entropy-augmented likelihood for the proposal policy. Note that when π θ is a non-standard policy like the beta or Student’s t, its Shannon entropy H (π θ ) may not have a closed-form expression. Therefore, we can use log-probabilities as a surrogate just like in SAC. Algorithm S7 Actor update for Greedy Actor-Critic 1: Initialize: policy parameters θ, proposal policy parameters θ ′ , action-value parameters w, empty replay buffer B 2: Hyperparameters: number of actor updates UTD actor (default 1), actor and critic stepsizes, number actions to sample m 3: // Use top 0.1 percentile of actions 4: n top . =⌈0.1m⌉ 5: for t = 1, 2, 3... do 6:Sample action A t ∼ π θ 7:Execute action A t , observe reward R t 8:Store (A t ,R t ) in replay buffer B 9:Update q w using Algorithm S1 10: for steps = 1, 2, 3,..., UTD actor do 11:Sample m actions from proposal policy a 1 ...a m ∼ π θ ′ 12:Sort a 1 ...a m according to q w (a 1 )...q w (a m ) 13:Set I = top n top actions, I ′ = top 2n top actions 14:Update policy by gradient ascent using top n top actions 1 n top X a∈I ∇ θ lnπ θ (a) 15:Update proposal policy by gradient ascent using top 2n top actions 1 2n top X a∈I ′ ∇ θ ′ lnπ θ ′ (a) 16: end for 17: end for Supplementary Material22 of 38 Supplementary Note 5: Extended Variability Analysis 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian Entropy coefficient β= 0.0001Entropy coefficient β= 0.001Entropy coefficient β= 0.01Entropy coefficient β= 0.1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Gaussian 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate SAC with Pathwise Gradient Estimator Actor UTD ratio UTD actor = 1Actor UTD ratio UTD actor = 3Actor UTD ratio UTD actor = 10Actor UTD ratio UTD actor = 30 Fig. S3. Analysis of Soft Actor-Critic with reparameterization trick (pathwise gradient). Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing entropy levels β, the rows represent different policy parameterizations, and the colours represent different actor UTD ratios UTD actor . By comparing the Gaussian (row 1) and the squashed Gaussian (row 2), we observe that the squashed Gaussian has higher run-to-run variability and is more sensitive to the learning rate, while being less sensitive to the entropy regularization hyperparameter. Looking at any colour and increasing entropy (going left-to-right), we observe that performance decreases significantly as entropy is increased. The beta policy (row 3) is the most robust, while Student’s t (row 4) suffers from highest degradation in performance. If we analyze the actor UTD ratios (changing colours), we observe that the Gaussian, squashed Gaussian, Student’s t and squashed Student’s t policies become increasingly unstable as the ratio UTD actor increases. On the other hand, the beta policy performs poorly at the lowest UTD actor and better at higher ratios. The highest mean performance and lowest variability are achieved with a beta policy parameterization and a high actor UTD ratio. Supplementary Material23 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian Entropy coefficient β= 0.0001Entropy coefficient β= 0.001Entropy coefficient β= 0.01Entropy coefficient β= 0.1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Gaussian 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate SAC with Likelihood Ratio Gradient Estimator Actor UTD ratio UTD actor = 1Actor UTD ratio UTD actor = 3Actor UTD ratio UTD actor = 10Actor UTD ratio UTD actor = 30 Fig. S4. Analysis of Soft Actor-Critic without reparameterization trick (log-likelihood gradient). Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing entropy levels β, the rows represent different policy parameterizations, and the colours represent different actor UTD ratios UTD actor . We observe that the Gaussian (row 1) and Student’s t (row 4) are the most robust to changes in the entropy coefficient hyperparameter, a behavior which is significantly different from the pathwise estimate (Fig. S3). If we analyze the actor UTD ratios (changing colours), we observe similar behavior: all policies other than beta (row 3) become increasingly unstable as the ratio UTD actor increases. In fact, we observe that the beta policy with the log-likelihood estimator can perform almost as well as the pathwise estimator shown in Fig. S3 when the hyperparameters are matched, while other policies are more sensitive to the gradient estimator choice. Supplementary Material24 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian Entropy coefficient β = 0.001Entropy coefficient β = 0.01Entropy coefficient β = 0.1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Gaussian 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate PPO Actor UTD ratio UTD actor = 1Actor UTD ratio UTD actor = 3Actor UTD ratio UTD actor = 10Actor UTD ratio UTD actor = 30 Fig. S5. Analysis of actor UTD ratio in Proximal Policy Optimization. Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing entropy levels β, the rows represent different policy parameterizations, and the colours represent different actor UTD ratios UTD actor . In this experiment, the on-policy batch size is fixed at b = 10. For all policies other than beta, increasing the entropy reduces the maximum achieveable performance. Increasing the actor UTD ratio (different colours) increases variability across runs for the Gaussian, squashed Gaussian and Student’s t. In the case of beta policy, we observe that increasing UTD actor shifts the performance across all learning rates upwards, including the best learning rate. Additionally, we observe that the beta policy’s performance drops off less significantly at the highest learning rate as compared to other policies. Supplementary Material25 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) MD stepsize 1/ τ : 0.001 Entropy coefficient β= 0.001Entropy coefficient β= 0.01Entropy coefficient β= 0.1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) MD stepsize 1/ τ : 0.01 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) MD stepsize 1/ τ : 0.1 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) MD stepsize 1/ τ : 0.5 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 Learning Rate 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 4e-041e-033e-03 0.010.03 Learning Rate SAC with Mirror Descent MD updates η md : 3MD updates η md : 10MD updates η md : 100MD updates η md : 1000 Fig. S6. Analysis of Soft Actor-Critic with mirror descent. Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing entropy levels β, the rows represent different mirror descent stepsize λ = 1/τ , and the colours represent different mirror descent periods η md . If we look at the lowest entropy settings β = 0.001, 0.01 (first two columns), we observe that as we increase 1/τ (by going down) or η md (by changing colour), the performance at every learning rate is reduced and run-to-run variability is increased. At the highest entropy setting β = 0.1 (rightmost column), the same trends hold but are less severe, with stronger MD settings reaching slightly higher peak rewards. Overall, the smallest MD step size 1/τ = 0.001 at small η md matches the baseline SAC most closely. See Fig. S2 for the summarized comparison. Supplementary Material26 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian Proposal Actions Selected η top = 1Proposal Actions Selected η top = 3Proposal Actions Selected η top = 10 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Gaussian 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate GreedyAC Actor UTD ratio UTD actor = 1Actor UTD ratio UTD actor = 3Actor UTD ratio UTD actor = 10Actor UTD ratio UTD actor = 30 Fig. S7. Analysis of Greedy Actor-Critic. Each dot represents the average reward of a single training run for a given learning rate. The columns represent the changing number of selected proposal actions UTD actor , the rows represent different policy parameterizations, and the colours represent different actor UTD ratios η top . Increasing UTD actor increases the run-to-run variability for all policy parameterizations, except for beta, in which case the performance improves and variability reduces. The Gaussian, squashed Gaussian, and Student’s t policies perform better at low to intermediate actor UTD ratios, whereas the beta policy performs better at a high ratio. Supplementary Material27 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian Reward Noise σ r = 0Reward Noise σ r = 0.01Reward Noise σ r = 0.1Reward Noise σ r = 0.3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate AC(λ) across Reward Noise Value-reward coefficient λ= 0Value-reward coefficient λ= 0.3Value-reward coefficient λ= 0.9Value-reward coefficient λ= 1 Fig. S8. Analysis of AC(λ) and changing environmental noise. Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing reward noise levels in the environment σ r , the rows represent different policy parameterizations, and the colours represent different value-reward mixing coefficients λ. All the other experiments in this paper have used a fixed reward noise of σ r = 0.01. At λ = 1, the critic is not used in the loss function, whereas at λ = 0, only the critic is used. Across all three policies, increasing λ shifts the mean performance upwards and reduces the run-to-run variability. We observe that for Gaussian and Student’s t, the best performance is obtained at λ = 0.9, whereas for beta, it is at λ = 0.9, 1. Additionally, unlike other policies, the beta policy’s performance does not drop off at the highest learning rate when coupled with a high λ value. Increasing environmental noise increases the variability of the Gaussian and Student’s t policies, while having no noticeable effect on the beta policy. Supplementary Material28 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Gaussian KL coef τ= 0.01KL coef τ= 0.05KL coef τ= 0.1KL coef τ= 0.5 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Squashed Gaussian 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Beta 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 log 10 (Reward) Student-t 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate MPO Actor UTD ratio UTD actor = 1Actor UTD ratio UTD actor = 3Actor UTD ratio UTD actor = 10Actor UTD ratio UTD actor = 30 Fig. S9. Analysis of Maximum a Posteriori Policy Optimization. Each dot represents the average reward of a single training run for a given learning rate. The columns represent the KL coefficient τ , the rows represent the different policy parameterizations, and the colours represent different actor UTD ratios UTD actor . For all policies, only a narrow range of hyperparameters perform well. If we analyze the actor UTD ratios (changing colours), we observe that the Gaussian, squashed Gaussian, Student’s t and squashed Student’s t policies become increasingly unstable as the ratio UTD actor increases. On the other hand, the beta policy performs poorly at the lowest UTD actor and better at higher ratios. Supplementary Material29 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ( ) Gaussian Entropy coefficient = 0.001Entropy coefficient = 0.01Entropy coefficient = 0.1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ( ) Beta 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ( ) Softmax 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate PPO Actor UTD ratio =Actor UTD ratio =Actor UTD ratio = Fig. S10. Analysis of on-policy batch size in Proximal Policy Optimization. Each dot represents the average reward of a single training run for a given learning rate. The columns represent changing entropy levels β, the rows represent different policy parameterizations, and the colours represent different on-policy batch sizes b. The softmax policy discretizes each continuous action dimension into fixed grid of 20 bins and samples from a categorical distribution over these grid points, with per-action probabilities given by temperature-scaled softmax over learned vector of scores. As PPO is an on-policy algorithm, we freeze the policy for b steps, collect transitions into a buffer, and then update the policy using b gradient descent updates. We observe that both the Gaussian and beta policies suffer a performance drop as entropy regularization increases. In contrast, the Softmax policy does not suffer from a performance drop. Across all policies, increasing the on-policy batch size b improves performance, with the effect more pronounced in the high-entropy setting. Supplementary Material30 of 38 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ( ) Noise variance = 1.1Noise variance = 1.5 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ( ) Noise variance = 2 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 1e-044e-041e-033e-03 0.010.030.09 Learning Rate Noise variance = 3 DDPG Noise decay = .Noise decay = .Noise decay = .Noise decay = Fig. S11. Analysis of Deep Deterministic Policy Gradient. Each dot represents the average reward of a single training run for a given learning rate. The 2×2 grid of panels represents different initializations for noise variance σ and the colours represent different noise decay rates ζ. The noise is decayed at every step as σ = σ∗ ζ. We observe that DDPG has lower stability than other algorithms. There are very few hyperparameter combinations that lead to a higher mean performance with low variability across runs, and matching the performance of SAC or PPO is difficult to achieve without more extensive tuning of these hyperparameters. Supplementary Material31 of 38 Fig. S12. Analysis of AC(λ) and Soft Actor-Critic rewards with Gaussian policy. Each row represents an experiment with a different learning rate η ∈ 0.01, 0.03, 0.09, and each column represents a different algorithm: AC(λ) at four values of λ followed by SAC. Faint lines represent individual seeds and the bold curve represents the mean. At the smallest learning rate η = 0.01, every algorithm reaches a mean reward around 1.0 with low seed spread. At η = 0.03, which is the default learning rate used throughout this paper (Table S2), we observe early signs of failure: AC(λ = 0) shows individual seeds dropping below 0.5, and AC(λ = 1) becomes highly variable across seeds. In contrast, AC(λ = 0.9) and SAC remain near 1.0 with low spread. The behaviour at η = 0.09 (bottom row) shows the same trends in extreme form, with AC(λ = 0) collapsing and SAC also becoming highly variable. This supports the finding that mixing reward with value estimates improves both performance and variability. Supplementary Material32 of 38 stable at high learning rate Fig. S13. Analysis of AC(λ) and Soft Actor-Critic rewards with beta policy. Each row represents an experiment with a different learning rate η ∈ 0.01, 0.03, 0.09, and each column represents a different algorithm: AC(λ) at four values of λ followed by SAC. Faint lines represent individual seeds and the bold curve represents the mean. We observe a different behaviour as compared to the Gaussian policy in Fig. S12. AC(λ = 0) collapses more strongly under the beta policy, with the mean reward dropping to near 0.2 at the default learning rate η = 0.03 (Table S2). In contrast, AC(λ = 0.9) and AC(λ = 1) consistently reach a high mean reward with low run-to-run variability at every learning rate. SAC with beta policy can reach the same high reward at a lower learning rate η = 0.01, but becomes highly unstable at higher learning rates, which shows that mixing immediate reward with the critic estimates is useful for lowering run-to-run variability. Additionally, Unlike Gaussian policy (Fig. S12), beta has a very low run-to-run variability at the highest learning rate with λ =0.9, 1. Supplementary Material33 of 38 Supplementary Note 6: Reproducibility and Disclosures Generative AI Use Declaration. The authors declare the use of generative AI tools in the research and writing process. Following the GAIDeT taxonomy (54), the following tasks were delegated to AI tools under full human supervision: • Code generation • Proofreading and editing Tools used: Grammarly; Claude Opus 4.7; Gemini 3.5. Scope. Grammarly was used to proofread and fix grammatical errors in the manuscript. AI assistance with code was limited to adapting existing scripts written by the authors, such as modifying variables, changing data-fetch paths, and providing debugging assistance. All AI-assisted outputs were reviewed and verified by the authors before inclusion. Experiments. All experiments use the Backwashing-PID environment with the shared defaults listed in Tables S1, S2, and S3. The codebase uses Python 3.10 with PyTorch as the ML library. All sources of randomness are controlled by a single global seed per run. The determinism is enforced by torch.use_deterministic_algorithms(True), torch.manual_seed and np.random.seed. Replay buffer sam- pling uses Python’s random.sample, and all environment stochasticity (such as measurement noise ε (f ) ∼ N (0, 0.003 2 ) and reward noise ε (r) ∼ N (0, 0.01 2 )) is drawn via np.random.normal. Given the same seed, runs are therefore reproducible. We release the code for all experiments reported in this paper at https: //github.com/haseebs/deconstruct-ac. Supplementary Material34 of 38 Table S1. Environment hyperparameters ParameterValue Cycle length J55 Timestep interval ∆t1 Pump speed bounds [s min ,s max ] [0, 100] Measurement noise std σ f 0.003 Reward noise std σ r 0.01 Deviation norm divisor110 Action space rangep,i,d∈ [0, 20] Initial pump speed s 0 20 Flow-rate setpoint f ideal 0.6309013 Supplementary Material35 of 38 Table S2. Fixed values for hyperparameters common across the tested algorithms HyperparameterValue Gaussian initializationμ: 0, σ: 1 Beta initializationα: 5, β: 5 Student’s t initializationμ: 0, σ: 1, ν: 3 Mini-batch size512 Critic MLP layers2 Critic MLP features64 Adaptive critic UTD error tolerance ω0.001 Adaptive critic UTD max updates per step n steps 100 Value baseline decay rate η v 0.1 Actor and critic learning rate η actor , η critic 0.03 Actor and critic optimizerAdam β : (0.9, 0.999) Supplementary Material36 of 38 Table S3. Fixed values for algorithm-specific hyperparameters across all the tested algorithms HyperparameterDefault(s) PPO Clipping threshold ε0.2 On-policy minibatch size b10 SAC Entropy coefficient β0.001 Number of actions to sample m 30 MPO Number of actions to sample m 30 DDPG Everything is swept GreedyAC Number of actions to sample m 30 Number of best actions n top 0.1∗ m AC(λ) Everything is swept Supplementary Material37 of 38 Table S4. Overview of experiments and swept hyperparameters and component choices. Every configuration is run over 10 random seeds. The learning rate grid η ∈ 1.2× 10 −4 , 3.7× 10 −4 , 1.1× 10 −3 , 3.3× 10 −3 , 10 −2 , 3× 10 −2 , 9× 10 −2 is swept in every experiment unless noted otherwise. Unless a hyperparameter is listed as swept, it is fixed to the default in Tables S2, S3. ExperimentPolicy parameteri- zations Swept hyperparameters SAC, main sweepGaussian, squashed Gaussian, beta, Stu- dent’s t, squashed Student’s t Entropy coefficient β ∈ 10 −5 , 10 −4 , 10 −3 , 10 −2 , 0.1; ac- tor UTD UTD actor ∈ 1, 3, 10, 30; gradient estimator ∈ pathwise, likelihood-ratio SAC, fixed critic UTDGaussian, betaCritic UTD UTD critic ∈ 1, 10, 100; actor UTD UTD actor ∈ 1, 10 SAC, η actor vs. UTD actor scaling Gaussian, betaScaling factor k ∈ 1, 3, 10, 30; strategy ∈ scale η actor , scale UTD actor ; critic LR η critic ∈ 3.7 × 10 −4 , 3.3 × 10 −3 , 3 × 10 −2 (LR axis not swept independently) SAC, mirror descentGaussianEntropy β ∈ 10 −3 , 10 −2 , 0.1; MD step size 1/τ ∈ 10 −3 , 10 −2 , 0.1, 0.5; MD period η md ∈ 3, 10, 100, 1000; LR sub-grid of 5 values AC(λ), main sweepGaussian, beta, Stu- dent’s t Value–reward mixing coefficient λ∈0, 0.3, 0.9, 1 AC(λ), reward-noise sweepGaussian, beta, Stu- dent’s t λ∈0, 0.3, 0.9, 1; reward noise std σ r ∈0, 0.01, 0.1, 0.3 AC(λ), critic capacityGaussian, beta, Stu- dent’s t Critic hidden units per layer ∈64, 256 (2 hidden layers); λ = 0 PPO, actor-UTD sweepGaussian, squashed Gaussian, beta, Stu- dent’s t Entropy coefficient β ∈ 10 −3 , 10 −2 , 0.1; actor UTD UTD actor ∈1, 3, 10, 30; on-policy batch size b = 10 (fixed) PPO, batch-size sweepGaussian, beta, soft- max Entropy coefficient β ∈ 10 −3 , 10 −2 , 0.1; on-policy batch size b∈1, 3, 10; actor UTD UTD actor = 1 (fixed) MPOGaussian, squashed Gaussian, beta, Stu- dent’s t KL coefficient τ ∈ 0.01, 0.05, 0.1, 0.5; actor UTD UTD actor ∈1, 3, 10, 30 GreedyACGaussian, squashed Gaussian, beta, Stu- dent’s t Actor UTD UTD actor ∈1, 3, 10, 30; number of top proposal actions η top ∈1, 3, 10; proposal samples m = 30 (fixed) DDPGDeterministic policy with Gaussian explo- ration noise Initial noise variance σ ∈1.1, 1.5, 2.0, 3.0; noise decay ζ ∈ 0.9995, 0.9997, 0.9999, 1.0 Supplementary Material38 of 38