Paper deep dive
Delightful Policy Gradient
Ian Osband
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:11:39 AM
Summary
The paper introduces the Delightful Policy Gradient (DG), a novel reinforcement learning algorithm that improves upon standard policy gradients by gating gradient updates with 'delight'—the product of advantage and action surprisal. DG effectively amplifies rare successful actions (breakthroughs) and suppresses rare failures (blunders), leading to improved directional accuracy in single contexts and a more balanced gradient allocation across multiple contexts, effectively shifting the expected gradient closer to the supervised cross-entropy oracle.
Entities (5)
Relation Signals (3)
Delightful Policy Gradient → outperforms → REINFORCE
confidence 95% · Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines
Delightful Policy Gradient → outperforms → PPO
confidence 95% · Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines
Delightful Policy Gradient → testedon → MNIST
confidence 95% · Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the \textit{Delightful Policy Gradient} (DG), which gates each term with a sigmoid of \emph{delight}, the product of advantage and action surprisal (negative log-probability). For $K$-armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks.
Tags
Links
- Source: https://arxiv.org/abs/2603.14608v1
- Canonical: https://arxiv.org/abs/2603.14608v1
Trouble viewing inline? Open PDF directly →
Full Text
65,936 characters extracted from source content.
Expand or collapse full text
Delightful Policy Gradient Ian Osband Keywords: policy gradient, reinforcement learning, gradient estimation, variance reduction, scaling Summary Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the Delightful Policy Gradient (DG), which gates each term with a sigmoid of delight, the product of advantage and action surprisal (negative log-probability). For K-armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks. Contribution(s) 1. We introduce the Delightful Policy Gradient (DG), which gates each sampled gradient term by a sigmoid of delight, the product of advantage and action surprisal (negative log probability). DG amplifies rare successes and suppresses rare failures, yielding a simple drop-in replacement for standard policy gradients that requires no importance ratios. Context: Prior variance-reduction methods (baselines, control variates (kool2019buy)) reduce variance but leave the expected gradient direction unchanged. Trust-region methods (schulman2015trust; schulman2017proximal) clip importance ratios to limit policy change, while advantage-weighted methods (peng2019advantage; abdolmaleki2024preference) reweight by advantage alone. Neither uses action surprisal to reshape the update. 2. In tabular K-armed bandits, we prove that DG improves policy-gradient updates through two distinct mechanisms. Within a single decision context, DG reduces directional variance by suppressing noise from low-probability negative-advantage actions (Proposition 1). Across multiple contexts, DG changes the bias of the expected gradient, shifting it strictly closer to the supervised cross-entropy oracle, even in the infinite-sample limit (Proposition 2). Context: Standard policy gradients allocate gradient budget in proportion to current success probability, creating a self-reinforcing dynamic in which easy contexts dominate while harder ones stall (williams1992simple). DG instead redistributes budget toward harder contexts, identifying a more balanced allocation rule for long-run learning under bandit feedback. 3. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control. The gains grow with problem difficulty: on Token Reversal, DG exhibits a smaller empirical scaling exponent than all baselines. Context: Together, these experiments show that DG’s mechanism is not a tabular artifact: MNIST diagnoses gradient geometry, Token Reversal tests scaling in sequential learning with Transformers, and the DeepMind Control Suite (tassa2018deepmind) tests density-based surprisal in continuous action spaces. Abstract Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the Delightful Policy Gradient (DG), which gates each term with a sigmoid of delight, the product of advantage and action surprisal (negative log-probability). For K-armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks. 1 Introduction Policy gradient methods underpin some of the most consequential systems in modern AI, from superhuman game play (silver2017mastering; vinyals2019grandmaster) to large language model alignment (ouyang2022training). In deep learning, optimizers often normalize gradients or constrain effective step size (kingma2014adam; you2019large), making update direction a primary determinant of learning progress. Most prior work therefore asks how to estimate the policy-gradient direction with lower variance or follow it more safely. We ask a different question: is the policy-gradient direction itself the right one to follow? We show that, in many settings, it is not. Policy gradients weight each per-sample term by advantage, regardless of how probable the sampled action was under the current policy (williams1992simple; sutton1999policy). Within a single decision context, a rare negative-advantage action can disproportionately distort the update direction, even though the policy already avoids it. Across contexts, the problem is deeper: the expected policy-gradient direction systematically over-allocates gradient budget to contexts the policy already solves well. A classifier allocates more gradient budget to an image it already gets right 99% of the time than to one it gets right 50%; likewise, an easy prompt dominates a harder one simply because the model already succeeds on it. This bias is not a finite-sample artifact: it persists even with infinite data. We propose a simple fix: gate each gradient term with a sigmoid of delight, the product of advantage and action surprisal, where surprisal is the negative log-probability of the sampled action under the current policy. The resulting estimator is the Delightful Policy Gradient (DG): one sigmoid, one multiply, and a temperature η that we fix to 11 throughout. Within a single context, DG suppresses perpendicular noise from unlikely negative-advantage actions (Prop. 1), improving directional accuracy; this variance effect vanishes as batch size grows. Across contexts, DG shifts the expected gradient strictly closer to the supervised cross-entropy oracle (Prop. 2); this directional effect persists even in the infinite-sample limit. We build this case progressively across theory and experiments. In tabular K-armed bandits, we isolate the two mechanisms analytically (Section 4). On MNIST contextual bandits, we directly confirm both effects and show that DG closes roughly half the gap to supervised cross-entropy (Section 3). On token reversal with transformers, DG’s advantage compounds with difficulty, yielding a smaller scaling exponent than all baselines (Section 5). On continuous control across 28 environments, DG matches or exceeds baselines without task-specific tuning (Section 6). 2 Delightful Policy Gradient We consider the standard episodic reinforcement learning setting. At each timestep t, the agent observes a history ℋtH_t, samples an action At∼πθ(⋅∣ℋt)A_t _θ(· _t), and receives reward RtR_t. Standard policy gradients form per-sample updates gt=Ut∇θlogπθ(At∣ℋt),g_t=U_t _θ _θ(A_t _t), where UtU_t denotes an advantage estimate. Thus each score term is weighted by advantage alone, regardless of how likely the sampled action was under the current policy. DG modulates each term by a second quantity: action surprisal. 2.1 Definitions The action surprisal is ℓt=−logπθ(At∣ℋt), _t=- _θ(A_t _t), which is large when the chosen action is unlikely under the current policy in that decision context. This surprisal is policy-relative: it measures how unlikely the action was under the policy, not how common that action is in the environment. It is also action-relative rather than outcome-relative: ℓt _t depends only on the probability the policy assigned to the sampled action, not on whether the resulting reward was good or bad. We define delight as χt=Utℓt, _t=U_t _t, the product of advantage and action surprisal, which is large when an unlikely action has high advantage.111We write χ for the Greek chara (delight). The gate is wt=σ(χt/η)w_t=σ( _t/η) for the sigmoid σ(x)=11+e−x,σ(x)= 11+e^-x, where η>0η>0 controls sharpness. DG therefore replaces the standard term gtg_t with the gated term wtgtw_t\,g_t. In other words, DG keeps the standard policy-gradient term but rescales it according to delight. This gate weights score terms asymmetrically. When a rare action has positive advantage (χt≫0 _t 0), the gate opens and wt≈1w_t≈ 1; we call such events breakthroughs. When a rare action has negative advantage (χt≪0 _t 0), the gate closes and wt≈0w_t≈ 0; we call such events blunders. For actions the policy already favors, the surprisal is small, so the gate stays near 12 12. The intuition for this asymmetry is simple. A blunder—a low-probability action with negative advantage—is already being avoided; pushing it down further has limited value. A breakthrough—a low-probability action with positive advantage—is a discovery the policy should exploit, and increasing its probability also creates more opportunities to learn from it in the future. Standard policy gradients treat these two cases symmetrically; DG preserves breakthroughs while attenuating blunders. The sigmoid gate also arises from a local entropy-regularized objective over gate values (Appendix A). Figure 1 visualizes the resulting effective coefficient on the gradient ∇θlogπ _θ π. (a) Coefficient ω vs. advantage U. (b) Coefficient ω vs. action probability p. Figure 1: Effective coefficient ω=w⋅Uω=w· U weighting ∇θlogπ _θ π. DG amplifies breakthroughs (rare successes) and suppresses blunders (rare failures); PG (dashed) is probability-blind. 2.2 Estimator and Implementation Operationally, DG simply multiplies each standard policy-gradient term gt=Ut∇θlogπθ(At∣ℋt)g_t=U_t _θ _θ(A_t _t) by the gate wt=σ(χt/η)w_t=σ( _t/η). This is a drop-in replacement for standard policy gradients: Δθ∝∑t∈ℬwtgt=∑t∈ℬσ(χt/η)Ut∇θlogπθ(At∣ℋt). θ\; \; _t w_t\,g_t\;=\; _t σ( _t/η)\,U_t\, _θ _θ(A_t _t). (1) Because the gate reweights samples, DG changes the expected update direction, not merely its variance. At an optimal policy, on-policy advantages vanish, so delight vanishes and optimal policies are stationary points of DG for all η>0η>0 (Appendix A). We use η=1η=1 throughout and find it robust across experiments. Algorithm 1 gives pseudocode; relative to PG, DG adds one sigmoid and one multiply per sample. Algorithm 1 Delightful Policy Gradient (discrete actions) 1:Batch ℬB, policy πθ _θ, temperature η=1η=1 2:Δθ←0 θ← 0 3:for t∈ℬt do 4: ℓt←−logπθ(At∣ℋt) _t←- _θ(A_t _t) ⊳ Surprisal 5: χt←Ut⋅ℓt _t← U_t· _t ⊳ Delight 6: wt←σ(χt/η)w_t←σ( _t/η) ⊳ Gate 7: Δθ←Δθ+wtUt∇θlogπθ(At∣ℋt) θ← θ+w_t\,U_t\, _θ _θ(A_t _t) 8:return Δθ θ For continuous actions, density-based surprisal makes χ=Uℓχ=U sensitive to action scaling. In our control experiments, clipping log-densities to [−10,10][-10,10] was sufficient (Algorithm 2 in Appendix E.1); when action scales vary substantially across dimensions, one can also whiten χ~t=(χt−μχ)/(σχ+ϵ) χ_t=( _t- _χ)/( _χ+ε) before gating. 3 MNIST Diagnostic We cast MNIST classification as a one-step contextual bandit. Given an image, the learner predicts a digit and observes only whether that prediction was correct, not the label itself. This makes MNIST a clean test of whether policy-gradient updates recover the gradient geometry that standard supervised learning gets from full labels, without the additional complications of sequential control. Formally, given an image X, the agent samples A∈0,…,9A∈\0,…,9\ from the policy and receives reward R=A=YR=I\A=Y\, where Y is the true label. The label itself is never revealed to the learner. We train a two-layer ReLU network with Adam on batches of B=100B=100 images. To diagnose gradient quality, we compare each batch update against two oracle directions, both computed from the true labels and therefore unavailable to the learner: gPG∗ g^*_PG =∑x∈ℬp(x)∇θlogπθ(Y∣x) = _x p(x)\, _θ _θ(Y x) (PG oracle), gCE∗ g^*_CE =∑x∈ℬ∇θlogπθ(Y∣x) = _x _θ _θ(Y x) (cross-entropy oracle), where p(x):=πθ(Y∣x)p(x):= _θ(Y x) is the probability assigned to the correct label. The PG oracle weights each image by its current success probability, whereas the cross-entropy oracle weights all images equally. Equivalently, these directions correspond to maximizing ∑p(x)Σ p(x) and ∑logp(x)Σ p(x), respectively. All results average over 100 seeds; shaded regions show ± 1 standard error. Figure 2 shows that DG learns faster than PG, closing roughly half the gap to supervised cross-entropy (CE) on this configuration. CE requires labels; PG and DG see only rewards. The architecture and optimizer are identical across methods; the only difference is how each method weights its gradients. Figure 2: MNIST classification error. Supervised CE requires labels; PG and DG do not. To test whether this advantage is merely variance reduction, Figure 3 varies both the baseline and the number of independent action samples drawn per image, denoted S. The advantage takes the form U=R−bU=R-b, and we consider four baselines: b=0b=0 (zero), b=0.5b=0.5 (constant), b=^[R∣x]b= E[R x] using the agent’s own probability estimate (expected), and b=[R∣x]b=E[R x] using the true label probability (oracle). As S increases, PG converges from above to the error floor set by the exact PG oracle gPG∗g^*_PG (dashed line in Figure 3). DG surpasses this floor for every baseline: with the expected baseline, DG at S=1S=1 already matches the level that PG approaches only as S→∞S→∞. Because this gain persists at large S, it cannot be explained by variance reduction alone: DG changes the expected gradient direction itself. Figure 3: Classification error at T=10kT=10k vs. samples per image S, faceted by baseline. Dashed line: error from the exact PG-oracle gradient gPG∗g^*_PG, PG’s best achievable direction. Figure 4 separates the two mechanisms by measuring alignment to both oracles at S=1S=1 and S=100S=100. Against gPG∗g^*_PG (a), DG’s advantage shrinks at S=100S=100: this is the within-context variance effect disappearing as sampling noise is reduced. Against gCE∗g^*_CE (b), DG’s advantage persists at S=100S=100: this is the cross-context directional effect, which remains even when sampling noise is negligible. The only difference between the two oracles is how they weight images: the PG oracle allocates more budget to images the model already classifies correctly, whereas the cross-entropy oracle treats every image equally. DG compresses these weights toward equality, reallocating gradient budget from well-solved images to harder ones. Section 4 formalizes both mechanisms in settings where the true gradients are analytically tractable. (a) Misalignment to gPG∗g^*_PG. (b) Misalignment to gCE∗g^*_CE. Figure 4: Gradient misalignment over training at S=1S=1 (solid) and S=100S=100 (dashed). (a) DG’s advantage diminishes with S (variance). (b) DG’s advantage persists at S=100S=100 (directional). Appendix B provides extensive ablations showing that DG’s gains persist across learning rates, batch sizes, network widths, and baselines. Validation error tracks training error closely, indicating no overfitting. The multiplicative form χ=U⋅ℓχ=U· also outperforms additive alternatives and entropy regularization. Taken together, these results suggest that DG addresses a core policy-gradient mismatch that appears even in canonical classification under bandit feedback. 4 Tabular Analysis The MNIST diagnostics revealed two effects: DG denoises the gradient estimate (Figure 4(a)) and rotates its expected direction toward the cross-entropy oracle (Figure 4(b)). To isolate these mechanisms analytically, we move to the tabular setting, replacing the neural network with explicit tables over actions, and study K-armed bandits under simple assumptions: symmetry, orthogonality, and normalized steps. The MNIST, transformer, and control experiments violate all of these assumptions; the tabular analysis isolates mechanism rather than modeling the full applications. We parameterize policies by logits z, so ϕ(a)=∇zlogπ(a)=ea−πφ(a)= _z π(a)=e_a-π.222Elsewhere ϕφ denotes the score in general θ-space; under logits, it simplifies to ea−πe_a-π. To model direction-limited optimization, each step takes the form z←z+αg/‖g‖z← z+α g/\|g\|: every update has norm α, so better direction is the only way to learn faster. This is a useful idealization of large-scale training regimes in which adaptive optimization and gradient clipping can attenuate differences in raw gradient norm, making update direction increasingly important for progress (Appendix C.1). 4.1 Single context: variance reduction We begin with a single context: a K-armed bandit with one correct action y∗y^* among K≥3K≥ 3 choices, like classifying a single MNIST image but with the neural network replaced by an explicit probability table. The reward is R=A=y∗R=I\A=y^*\. In any single context of this form, the reward and cross-entropy oracles coincide. The score identity ∑aπ(a)ϕ(a)=0 _aπ(a)\,φ(a)=0 implies [gPG]=π(y∗)ϕ(y∗)∝gCE∗E[g_PG]=π(y^*)\,φ(y^*) g^*_CE. The only way to improve gradient quality is therefore to reduce perpendicular sampling noise. We write Π⟂ _ for projection orthogonal to ∇J∇ J and Var⟂(g)=‖Π⟂(g)‖2Var_ (g)=E\| _ (g)\|^2 for the perpendicular variance. Symmetry yields closed-form equalities. We specialize to a policy with π(y∗)=1−επ(y^*)=1- and π(a)=ε/(K−1)π(a)= /(K-1) for each incorrect action, with baseline b∈(0,1)b∈(0,1). The DG gate then takes a common value w+w_+ on the correct action and w−w_- on all incorrect actions, with w−<12<w+w_-< 12<w_+.333Explicitly: w+=σ((1−b)log11−ε/η)w_+=σ\! ((1-b) 11- /η ) and w−=σ(−blogK−1ε/η)w_-=σ\! (-b K-1 /η ). Proposition 1 (Variance reduction in symmetric bandits). In the bandit above, for any η>0η>0: (i) DG preserves the expected gradient direction: [gDG]=s⋅gPG∗E[g_DG]=s· g^*_PG, where s=(1−b)w++bw−>0s=(1-b)\,w_++b\,w_->0. (i) DG reduces perpendicular variance by exactly w−2w_-^2: Var⟂(gDG)=w−2⋅Var⟂(gPG)Var_ (g_DG)=w_-^2·Var_ (g_PG). (i) For the batch mean g¯=1B∑i=1Bgi g= 1B _i=1^Bg_i, in the regime where g¯ g concentrates around its mean, 1−[cos(g¯DG,gPG∗)]1−[cos(g¯PG,gPG∗)]≈w−2s2< 1. 1-E[ ( g_DG,\,g^*_PG)]1-E[ ( g_PG,\,g^*_PG)]\;≈\; w_-^2s^2\;<\;1. (2) DG always reduces the alignment gap; both methods converge to cosine 11 as B→∞B→∞. The ratio w−2/s2w_-^2/s^2 measures the fraction of PG’s cosine gap that DG retains. Since σ(−x)≤e−xσ(-x)≤ e^-x, the gate satisfies w−≤ε/(K−1)w_-≤ /(K-1) for b=12b= 12 and η=1η=1, giving w−2/s2≤16ε/(K−1)w_-^2/s^2≤ 16\, /(K-1). Evaluating the exact ratio confirms that the reduction is substantial: for K=100K=100 at ε=0.5 =0.5, DG retains only 4%4\% of PG’s cosine gap; at ε=0.1 =0.1, only 1%1\%. Beyond symmetry. Without uniform incorrect-action probabilities, each incorrect action receives a different gate, so [gDG]E[g_DG] is no longer exactly collinear with ∇J∇ J. However, tail suppression still holds: w−(a)≤π(a)b/ηw_-(a)≤π(a)^b/η, so rare actions are damped more aggressively than under PG. As the policy concentrates (ε→0 → 0), the directional bias vanishes and the variance bound tightens (Appendix C.3). We validate this picture with K=100K=100, B=100B=100, α=0.1α=0.1, and η=1η=1, averaged over 100 seeds. Despite identical step magnitudes, DG converges faster (Figure 5a) and maintains lower misalignment throughout training (Figure 5b). Late in training, incorrect actions become rare but each delivers a large perpendicular kick; PG’s misalignment rebounds while DG’s stays suppressed. (a) Error ε=1−π(y∗) =1-π(y^*). (b) Misalignment 1−cos(g,gPG∗)1- (g,\,g^*_PG). Figure 5: Symmetric bandit, normalized steps (K=100K=100, B=100B=100, α=0.1α=0.1). 4.2 Multiple contexts: directional improvement For a single context, the reward and cross-entropy oracles coincide (Section 4.1), so DG can only reduce variance. In practice, however, a single gradient step must improve many contexts at once: each MNIST image is a separate K=10K=10 classification problem, and the update must allocate learning across all of them. With N independent contexts at each step (a contextual bandit), the two oracles diverge. PG allocates gradient budget proportional to pnp_n, the probability of the correct action in context n; CE allocates equally, making direction itself a degree of freedom. MNIST confirmed this: DG surpasses PG’s performance floor even at large S (Figure 3). Formally, suppose the agent faces N independent contexts at each step, sums their gradients, and takes a single normalized step. Each context n contributes an orthogonal gradient component vnv_n; let pn:=πn(yn)p_n:= _n(y_n) denote the probability of the correct action. With baseline b=0b=0, only correct actions contribute, and the three gradient directions differ only in how they weight contexts:444With baseline b>0b>0, DG additionally suppresses incorrect-action variance (Prop. 1). Setting b=0b=0 isolates the cross-context reweighting for this subsection. gCE∗ g^*_CE =∑nvn = _nv_n (cross-entropy: equal weight), gPG∗ g^*_PG =∑npnvn = _np_n\,v_n (PG: weight ∝pn p_n), (3) [gDG] [g_DG] =∑npnσ((−logpn)/η)vn = _np_n\,σ\! ((- p_n)/η )\,v_n (DG: weights compressed by gate). PG weights each context by pnp_n, so well-solved contexts dominate the gradient. The gate σ((−logpn)/η)σ((- p_n)/η) is larger when pnp_n is small and smaller when pnp_n is large, partially cancelling the pnp_n weighting and redistributing budget toward hard contexts (Figure 4(b)). These three directions are not arbitrary. Under normalized steps, cn∝pnc_n p_n is greedy-optimal for Δ(∑pn) (Σ p_n) and cn∝1c_n 1 for Δ(∑logpn) (Σ p_n) (Lemma 1). PG’s direction is myopically optimal but self-reinforcing: since ∇pn=pnvn∇ p_n=p_nv_n, improving an easy context makes its gradient larger, attracting even more budget next step. DG moves weights toward cn∝1c_n 1, rebalancing budget to hard contexts where each step yields more long-run progress. As with the Kelly criterion (kelly1956new), greedy single-step optimality does not imply optimal long-run compounding. Lemma 1 (Greedy directions under normalized steps). Under a normalized step with g=∑ncnvng= _nc_nv_n: the maximizer of Δ(∑pn) (Σ p_n) is cn∝pnc_n p_n, and the maximizer of Δ(∑logpn) (Σ p_n) is cn∝1c_n 1. Proposition 2 (Directional improvement toward cross-entropy). Let N=2N=2 with p1≠p2p_1≠ p_2 and η>1/2η>1/2. Then cos([gDG],gCE∗)>cos(gPG∗,gCE∗). \! (E[g_DG],\;g^*_CE )\;>\; \! (g^*_PG,\;g^*_CE ). DG’s expected direction is strictly closer to the cross-entropy oracle than PG’s, for any batch size, including the limit B→∞B→∞. Proof sketch. The cosine between c1v1+c2v2c_1v_1+c_2v_2 and v1+v2v_1+v_2 is maximized at ratio r=c1/c2=1r=c_1/c_2=1 (Appendix C.5). The DG weight function h(p)=pσ((−logp)/η)h(p)=p\,σ((- p)/η) is increasing for η>1/2η>1/2, and the sigmoid factor is decreasing in p, so DG compresses PG’s ratio: 1<rDG<rPG1<r_DG<r_PG, achieving higher cosine. Appendix C.6 extends to arbitrary N via a path argument; Figure 6 confirms the compression at N=100N=100. We validate with N=100N=100 independent contexts, K=10K=10 actions each, (0,1)N(0,1) logit init, α=0.1α=0.1, averaged over 100 seeds. At each step we compute exact population gradients (no sampling noise), so PG follows gPG∗g^*_PG exactly. Despite this, DG converges faster (Figure 6a): its rebalancing toward hard contexts compounds over many steps. The cosine gap (Figure 6(b)) persists throughout training, confirming a directional effect rather than variance reduction. (a) Average error 1−p¯1- p. (b) Misalignment to gCE∗g^*_CE. Figure 6: N=100N=100 independent contexts, K=10K=10 actions, (0,1)N(0,1) init, exact gradients. DG converges faster (a) by rebalancing gradient budget to hard contexts (b). The two mechanisms map onto a bias–variance decomposition. For a single context, gPG∗=gCE∗g^*_PG=g^*_CE, so DG’s advantage is pure variance reduction and vanishes as B→∞B→∞ (Prop. 1). Across multiple contexts, Proposition 2 shows something stronger: DG improves the expected update itself, not just its noisy estimate. The induced bias is beneficial, rotating the gradient toward cross-entropy and rebalancing learning toward hard contexts. This means DG’s advantage is not a finite-sample artifact: even with exact gradients, standard policy gradients can allocate update budget poorly, and DG corrects that defect. 5 Transformer Sequence Modeling The MNIST and bandit experiments isolated DG’s effect on update geometry in single-step problems. We now test the same mechanism in a sequential setting with memory, autoregressive generation, and temporal credit assignment. We introduce the Token Reversal task (Figure 7). An input sequence x1:Hx_1:H of length H is drawn uniformly from a vocabulary of size M, and a decoder-only Transformer autoregressively predicts the reverse sequence, y^t=xH−t+1 y_t=x_H-t+1. This is a controlled model of token-space reasoning: the learner must attend to the input, retain sequence structure, and generate a coherent output autoregressively. Because the output space has size MHM^H, fully correct sequences become exponentially rarer as horizon and vocabulary grow. We report sequence error, the fraction of output tokens predicted incorrectly; full details are in Appendix D. We compare DG against REINFORCE, PPO (schulman2017proximal), and PMPO (abdolmaleki2024preference), each tuned over its key hyperparameters (Appendix D.1). All methods share the same Transformer architecture, optimizer, and compute budget; results average over 30 seeds. On a default configuration (H=10H=10, M=2M=2, K=1,000K=1,000 episodes), Figure 8 shows that DG converges faster and reaches lower sequence error than all baselines. Multiplicative advantage × surprisal gating provides a qualitatively different signal from additive entropy bonuses or trust-region constraints. UCB-style mixtures (1−α)U+αℓ(1-α)U+α improve over REINFORCE but do not match DG (Appendix D.4). Figure 7: Token Reversal task. Figure 8: Sequence error on Token Reversal. To test whether DG’s advantage grows with difficulty, we increase H and M while holding the training budget fixed at K=10kK=10k episodes. Figure 9a–b shows final sequence error; Figure 9c–d shows cumulative error on log-log axes. A sharp contrast emerges: baseline performance deteriorates rapidly beyond a complexity threshold, whereas DG degrades much more gracefully. The log-log plots reveal approximate power-law scaling, with DG achieving a smaller exponent; its advantage compounds with difficulty. (a) Final error vs. H (b) Final error vs. M (c) Avg error vs. H (d) Avg error vs. M Figure 9: Scaling with task complexity. (a,b) Final sequence error after K=10kK=10k episodes; baselines degrade beyond a threshold while DG degrades more gracefully. (c,d) Log-log cumulative error reveals approximate power-law scaling; DG achieves a smaller exponent. The scaling advantage reflects the mechanism identified in the bandit analysis: as MHM^H grows, reallocating gradient weight from high-surprisal failures to high-surprisal successes becomes increasingly valuable. Appendix D.3 tests eight task variants; DG leads in every configuration, with larger margins in harder settings. If these trends transfer to larger-scale sequence generation, where long horizons and large vocabularies make correct outputs increasingly rare, the advantage of rebalancing gradient weight could be substantial. 6 Continuous Control Token Reversal showed that DG’s advantage grows with task complexity in a discrete sequential setting. We now test whether the same mechanism transfers to continuous control, where the policy defines an action density rather than a discrete distribution. We evaluate on the DeepMind Control Suite (tassa2018deepmind): 28 environments, 10 million steps, and 3 seeds per environment. All methods share the same actor–critic architecture, critic algorithm (Retrace (munos2016safe) with replay), and optimizer; only the policy update rule differs. DG is applied only to the actor, reweighting on-policy score terms without importance ratios. For continuous actions, delight uses clipped log-densities (Section 2.2). DG matches or exceeds the best baseline on a majority of environments and avoids catastrophic failures: PPO collapses on humanoid:run, while REINFORCE fails on hopper:hop. On humanoid:run, DG discovers a successful gait while all baselines plateau (Figure 11). Across all 84 runs, DG is never the worst method (Figure 10) and achieves the lowest average regret throughout training (Figure 12). It also remains competitive with highly tuned MPO (abdolmaleki2018maximum) and SAC (haarnoja2018soft) despite using no task-specific tuning (Appendix E.4). These results show that delight-based reweighting remains effective with continuous action densities. Figure 10: Average reward across 28 Control Suite environments. DG (red) is never the worst method. Figure 11: Learning curve on humanoid:run. Figure 12: Aggregate regret across 28 tasks. 7 Related Work Many policy-gradient methods can be viewed through the effective score weight ωt _t multiplying ∇θlogπθ _θ _θ. This lens makes clear which methods distinguish rare from common actions and which do not (Figure 13). For standard PG, ωt=Ut _t=U_t is linear in advantage and blind to action probability. For DG, ωt=σ(Utℓt/η)Ut _t=σ(U_t _t/η)\,U_t depends on both advantage and surprisal, allowing rare successes and rare failures to be treated asymmetrically. Trust-region methods. The natural policy gradient (kakade2001natural) and its trust-region successors TRPO, PPO (schulman2015trust; schulman2017proximal), and GRPO (shao2024deepseekmath) constrain or precondition the update, keeping ωt _t in a narrow band around UtU_t. This limits large updates but also attenuates rare high-advantage events. DG instead modulates ωt _t using only the current policy and requires no importance ratios. Variance reduction. Leave-one-out baselines (kool2019buy), generalized advantage estimation (schulman2016high), and other control-variate methods reduce gradient variance without changing the expected direction. DG’s single-context mechanism (Prop. 1) achieves a related effect through gating rather than control variates, but its cross-context rebalancing (Prop. 2) changes the expected direction itself—an effect that persists even with perfect variance reduction. Advantage-weighted methods. AWR (peng2019advantage) and PMPO (abdolmaleki2024preference) set ωt=f(Ut) _t=f(U_t) for some increasing function of advantage, but remain blind to surprisal. Gradient budget therefore continues to flow toward predictable actions until their advantage is driven toward zero. DG makes ωt _t depend on surprisal directly, so rare and common actions are weighted differently even at the same advantage. Entropy and intrinsic motivation. Entropy regularization (haarnoja2018soft) and curiosity-based methods (pathak2017curiosity) change the learning signal indirectly by modifying the reward or objective. DG instead leaves the reward unchanged and modifies the policy-gradient coefficient itself, reallocating gradient budget toward surprising task-relevant outcomes. Its distinct behavior relative to entropy bonuses is confirmed empirically in the MNIST experiments (Section 3). Figure 13: Score weight ω vs. advantage U for common (π=0.9π=0.9) and rare (π=0.01π=0.01) actions. DG treats rare successes and rare failures asymmetrically; PG and PMPO are blind to action probability, while PPO clips both tails through importance-ratio constraints. 8 Conclusion This paper identifies a basic mismatch in standard policy-gradient learning. In the K-armed bandits we analyze, rare failures inject disproportionate noise within a context, and the expected gradient over-allocates budget to contexts the policy already handles well. MNIST, transformer, and control experiments suggest both effects persist beyond the tabular setting. DG corrects both effects by gating each score term with delight, the product of advantage and surprisal, amplifying rare successes and suppressing rare failures. The first effect, variance reduction, vanishes with infinite samples; the second, a beneficial bias toward the cross-entropy oracle, does not. We characterize both analytically and confirm them empirically across bandits, MNIST, transformer sequence modeling, and continuous control. More broadly, these results suggest that delight is a useful primitive for decision learning under evaluative feedback. DG does more than reduce variance: it changes how finite update budget is allocated across samples and contexts. From this perspective, the method reveals that standard policy gradients use a suboptimal weighting rule for learning from evaluative feedback. Formal convergence guarantees remain open, as does the question of how far this mechanism transfers to sparse-reward settings, offline RL, and large-scale transformer training and RLHF. Acknowledgements We thank Ben Van Roy, Satinder Singh, Tor Lattimore, and Jincheng Mei for detailed reviews and feedback on earlier drafts. John Aslanides, Yotam Doron, Georg Ostrovski, Hubert Soyer, and Blanca Huergo contributed to the codebase and helped shape the experimental infrastructure. We are grateful to Raia Hadsell, Zoubin Ghahramani, Demis Hassabis, and Satinder Singh for fostering the research environment that made this work possible. Finally, we thank Dan Zigmond and Barry Kerzin for inspiration from Buddhist philosophy and feedback on the wider research into delightful learning. References Supplementary Materials The following content was not necessarily subject to peer review. Appendix A The Delightful Gate: Derivation and Properties We derive the sigmoid gate and establish basic properties. Entropy-regularized gate selection. Treat each sampled score term gtg_t as a candidate update that is applied with weight w∈[0,1]w∈[0,1]. For fixed delight χ, we choose w to maximize a linear reward plus an entropy cost: maxw∈[0,1]χw+ηH(w),H(w)=−wlogw−(1−w)log(1−w). _w∈[0,1]\;χ\,w+η\,H(w), H(w)=-w w-(1-w) (1-w). (4) Differentiating and setting to zero: ∂w[χw+ηH(w)]=χ−ηlog(w1−w)=0⟹w∗=σ(χη). ∂ w [χ w+η H(w) ]=χ-η \! ( w1-w )=0 w^*=σ\! ( χη ). (5) Since H is strictly concave, this is the unique global maximizer. Softplus potential. Substituting w∗w^* into the objective yields the optimal value: Ψη(χ)=maxw∈[0,1]χw+ηH(w)=ηlog(1+eχ/η)=ηsoftplus(χ/η). _η(χ)= _w∈[0,1] \χ w+η H(w) \=η \! (1+e^χ/η )=η\,softplus(χ/η). (6) This softplus potential provides a local variational interpretation of the DG gate. It saturates for χ≪0χ 0 (suppressing updates from disfavored actions) and grows linearly for χ≫0χ 0 (preserving updates from surprising successes). Temperature interpolation. The temperature η controls gate sharpness and interpolates between two regimes. In the hedonic limit (η→∞η→∞), w∗→1/2w^*→ 1/2 for bounded χ, recovering standard PG up to a constant: the gate ignores surprisal and responds only to advantage. In the enlightened limit (η→0η→ 0), w∗→χ>0w^* \χ>0\, a hard gate that passes all positive-delight samples equally and fully suppresses negative ones. We use η=1η=1 throughout, which provides meaningful asymmetry without hard thresholding. Stationarity at optimal policies. At an optimal policy π∗π^*, on-policy advantages vanish on the support. Since the DG update is w⋅U⋅∇θlogπθw· U· _θ _θ and U=0U=0 on supported actions, the update vanishes. Optimal policies are therefore stationary points of DG for all η>0η>0. Appendix B MNIST Experimental Details We provide supplementary details and robustness analyses for the MNIST contextual bandit experiments in Section 3. Setup. We parameterize the policy πθ _θ as a two-layer MLP with ReLU activations and hidden width 100. Training proceeds in online batches: each step, the agent collects B=100B=100 trajectories, computes gradients, and updates parameters with Adam (learning rate 10−310^-3). We train for K=10,000K=10,000 episodes unless otherwise noted. B.1 Generalization A natural concern is that prioritizing high-surprisal events might cause overfitting. Figure 14 shows validation error tracks training error closely for both PG and DG, and the optimal temperature η≈1η≈ 1 is consistent across splits. DG’s gains persist on held-out data. (a) Learning curves (train vs. val). (b) Temperature sweep (train vs. val). Figure 14: Generalization on MNIST. (a) Validation error tracks training error; DG achieves lower error on both. (b) The optimal temperature η≈1η≈ 1 is consistent across train and validation. B.2 Comparison with Entropy Regularization A natural alternative is additive entropy regularization, augmenting the objective with αH(πθ)α H( _θ). Figure 15 sweeps α for PG and η for DG. Entropy regularization is sensitive to its coefficient: small α provides negligible benefit, while large α collapses performance by forcing excessive stochasticity. DG achieves a lower error floor than any entropy-regularized baseline and maintains its advantage across a wide range of η. Figure 15: Entropy regularization vs. DG temperature. Large entropy coefficients degrade accuracy; DG is robust across η and consistently achieves lower error. B.3 Robustness to Learning Rate Figure 16 sweeps learning rate α∈[10−5,10−2]α∈[10^-5,10^-2]. DG consistently outperforms PG across the effective range, with a wider basin of low error around α∈[10−4,10−3]α∈[10^-4,10^-3]. Figure 16: Learning rate sensitivity. DG (red) outperforms PG (blue) across all baselines and learning rates. B.4 Robustness to Batch Size Figure 17 sweeps batch size B∈[1,1000]B∈[1,1000]. Larger batches reduce gradient variance and improve both methods, but DG maintains a consistent advantage. This confirms that DG’s benefit is not merely variance reduction—it provides a distinct signal that complements larger batches. Figure 17: Batch size sensitivity. DG maintains its advantage across all batch sizes. B.5 Robustness to Network Width Figure 18 varies hidden width W∈[1,2048]W∈[1,2048]. Both methods exhibit a sweet spot in capacity, but DG achieves lower error at every width. Figure 18: Network width sensitivity. DG outperforms PG across all capacity levels. B.6 Alternative Definitions of Delight We investigate whether alternative functional forms could improve on χ=U⋅ℓχ=U· . Additive mixtures. Inspired by UCB, we consider χαUCB=(1−α)U+αℓχ^UCB_α=(1-α)U+α , interpolating between pure advantage (α=0α=0) and pure surprisal (α=1α=1). Figure 19(a) shows additive mixtures consistently underperform the multiplicative form. Surprisal exponent. We generalize to χβ=U⋅ℓβ _β=U· ^β. Figure 19(b) shows the optimum is exactly β=1β=1; deviations in either direction degrade performance. Within this family of alternatives, the simple product χ=U⋅ℓχ=U· performs best. (a) Additive mixtures. (b) Surprisal exponent β. Figure 19: Alternative definitions of delight. (a) Additive mixtures underperform the multiplicative form (dotted). (b) The simple product (β=1β=1) is optimal. Appendix C Proofs for Tabular Analysis This section collects the proofs for Section 4. We first justify the normalized-step model, then prove Propositions 1 and 2 and their extensions. C.1 Cosine Controls Progress The tabular analysis assumes normalized steps z←z+αg/‖g‖z← z+α g/\|g\|. The following proposition shows that under this update rule, expected improvement scales linearly with cos(g,∇J) (g,∇ J), so a higher-cosine gradient estimator translates directly into faster learning. Proposition 3 (Cosine controls progress). Let J have L-Lipschitz gradient. For the normalized update z+=z+αg/‖g‖z^+=z+α g/\|g\|, [J(z+)−J(z)∣z]≥α‖∇J(z)‖⋅[cos(g,∇J(z))∣z]−L2α2.E[J(z^+)-J(z) z]≥α\|∇ J(z)\|·E[ (g,∇ J(z)) z]- L2α^2. Proof. By smoothness: J(z+)≥J(z)+⟨∇J,αg/‖g‖⟩−L2α2J(z^+)≥ J(z)+ ∇ J,α g/\|g\| - L2α^2. Taking expectations and using ⟨∇J,g/‖g‖⟩=‖∇J‖cos(g,∇J) ∇ J,g/\|g\| =\|∇ J\| (g,∇ J). ∎ C.2 Proof of Proposition 1 (Variance Reduction) We use the symmetric bandit setup from Section 4.1: K actions, correct action y∗y^*, π(y∗)=1−επ(y^*)=1- , π(a)=q:=ε/(K−1)π(a)=q:= /(K-1) for a≠y∗a≠ y^*, baseline b, and gate values w+w_+ (correct) and w−w_- (incorrect). The proof proceeds in three parts: (i) a symmetry lemma shows that all expected gradients are collinear, (i) the perpendicular variance factors exactly, and (i) a Taylor expansion converts the variance ratio into a cosine-gap ratio. Part (i): Direction preservation. Let u:=ϕ(y∗)=ey∗−πu:=φ(y^*)=e_y^*-π. We show ∑a≠y∗ϕ(a) _a≠ y^*φ(a) is proportional to u, which forces the expected DG gradient to be collinear with ∇J=(1−ε)u∇ J=(1- )\,u. Lemma 2 (Symmetry identity). ∑a≠y∗ϕ(a)=−(K−1)(1−ε)εϕ(y∗). _a≠ y^*φ(a)=- (K-1)(1- ) \,φ(y^*). Proof. Since ∑a=1Kϕ(a)=∑a(ea−π)=−Kπ _a=1^Kφ(a)= _a(e_a-π)=1-Kπ, we have ∑a≠y∗ϕ(a)=−Kπ−ϕ(y∗) _a≠ y^*φ(a)=1-Kπ-φ(y^*). We verify this is proportional to ϕ(y∗)φ(y^*) component by component. In component y∗y^*: 1−K(1−ε)−ε=−(K−1)(1−ε)1-K(1- )- =-(K-1)(1- ). In component a≠y∗a≠ y^*: 1−Kq−(−q)=1−(K−1)q=1−ε1-Kq-(-q)=1-(K-1)q=1- . Since ϕ(y∗)φ(y^*) has component ε in position y∗y^* and −q=−ε/(K−1)-q=- /(K-1) in other positions, the ratio is −(K−1)(1−ε)/ε-(K-1)(1- )/ in both components. ∎ Now compute the expected DG gradient: [gDG] [g_DG] =(1−ε)w+(1−b)ϕ(y∗)+∑a≠y∗qw−(−b)ϕ(a) =(1- )\,w_+\,(1-b)\,φ(y^*)+ _a≠ y^*q\,w_-\,(-b)\,φ(a) =(1−ε)w+(1−b)ϕ(y∗)+w−(−b)q⋅(−(K−1)(1−ε)ε)ϕ(y∗) =(1- )\,w_+\,(1-b)\,φ(y^*)+w_-\,(-b)\,q· (- (K-1)(1- ) )φ(y^*) =(1−ε)[(1−b)w++bw−]ϕ(y∗) =(1- ) [(1-b)\,w_++b\,w_- ]\,φ(y^*) =s⋅∇J, =s·∇ J, where s=(1−b)w++bw−s=(1-b)\,w_++b\,w_- and we used q(K−1)=εq(K-1)= . Since w+,w−>0w_+,w_->0 and b∈(0,1)b∈(0,1), we have s>0s>0, confirming the direction is preserved. Part (i): Perpendicular variance is exactly w−2⋅Var⟂(gPG)w_-^2·Var_ (g_PG). Since ∇J∝ϕ(y∗)∇ J φ(y^*), the projection Π⟂ϕ(y∗)=0 _ φ(y^*)=0. Therefore the correct-action sample contributes zero perpendicular energy under both PG and DG. For incorrect actions, gDG(a)=w−⋅gPG(a)g_DG(a)=w_-· g_PG(a) pointwise, so Π⟂(gDG(a))=w−⋅Π⟂(gPG(a)) _ (g_DG(a))=w_-· _ (g_PG(a)). Note that [Π⟂(g)]=Π⟂([g])=0E[ _ (g)]= _ (E[g])=0 for both PG and DG (from Part (i), both means are parallel to ∇J∇ J), so Var⟂Var_ equals the second moment. The perpendicular variance is: Var⟂(gDG) _ (g_DG) =‖Π⟂(gDG)‖2=∑a≠y∗π(a)w−2b2‖Π⟂(ϕ(a))‖2 =E\,\| _ (g_DG)\|^2= _a≠ y^*π(a)\,w_-^2\,b^2\,\| _ (φ(a))\|^2 =w−2∑a≠y∗π(a)b2‖Π⟂(ϕ(a))‖2=w−2⋅Var⟂(gPG). =w_-^2 _a≠ y^*π(a)\,b^2\,\| _ (φ(a))\|^2=w_-^2·Var_ (g_PG). Part (i): Alignment gap ratio. When g¯ g concentrates near μ:=[g¯]μ:=E[ g], write g¯=μ+ξ g=μ+ξ with ξ small. Since μ is parallel to ∇J∇ J (Part (i)), cos(g¯,∇J)=cos(μ+ξ,μ) ( g,∇ J)= (μ+ξ,μ). Taylor-expanding to second order in ξ: 1−cos(μ+ξ,μ)≈‖ξ⟂‖22‖μ‖2,1- (μ+ξ,μ)≈ \| _ \|^22\|μ\|^2, where ξ⟂=Π⟂(ξ) _ = _ (ξ) is the perpendicular component. Taking expectations: 1−[cos(g¯,∇J)]≈Var⟂(g¯)2‖[g¯]‖2=Var⟂(g)2B‖[g¯]‖2.1-E[ ( g,∇ J)]≈ Var_ ( g)2\,\|E[ g]\|^2= Var_ (g)2B\,\|E[ g]\|^2. For PG: ‖[g¯PG]‖=‖∇J‖\|E[ g_PG]\|=\|∇ J\|. For DG: ‖[g¯DG]‖=s‖∇J‖\|E[ g_DG]\|=s\,\|∇ J\|. Using Part (i): 1−[cos(g¯DG,∇J)]1−[cos(g¯PG,∇J)]≈w−2⋅Var⟂(gPG)/(s2‖∇J‖2)Var⟂(gPG)/‖∇J‖2=w−2s2. 1-E[ ( g_DG,∇ J)]1-E[ ( g_PG,∇ J)]≈ w_-^2·Var_ (g_PG)/(s^2\|∇ J\|^2)Var_ (g_PG)/\|∇ J\|^2= w_-^2s^2. C.3 Extension to Non-Symmetric Bandits The symmetric assumption in Proposition 1 gives clean equalities, but the noise-suppression mechanism extends to general policies. The coincidence of oracles does not require symmetry: in any single-context bandit, the score identity ∑aπ(a)ϕ(a)=0 _aπ(a)\,φ(a)=0 forces [gPG]=π(y∗)ϕ(y∗)∝gCE∗E[g_PG]=π(y^*)\,φ(y^*) g^*_CE. The directional improvement of Proposition 2 is therefore a fundamentally multi-context phenomenon; in a single context, DG can only reduce variance. Variance suppression. For any incorrect action a with negative advantage (U<0U<0) and surprisal ℓ(a)=−logπ(a) (a)=- π(a), the DG gate satisfies w(a)=σ(−bℓ(a)/η)≤e−bℓ(a)/η=π(a)b/η.w(a)=σ(-b\, (a)/η)\;≤\;e^-b\, (a)/η=π(a)^b/η. For rare actions (π(a)≪1π(a) 1), this bound is small: the gate suppresses their contribution by at least π(a)b/ηπ(a)^b/η. In a general (non-symmetric) bandit, each incorrect action a contributes π(a)w(a)2b2‖Π⟂(ϕ(a))‖2π(a)\,w(a)^2\,b^2\,\| _ (φ(a))\|^2 to Var⟂(gDG)Var_ (g_DG). Using the bound above: Var⟂(gDG)≤∑a≠y∗π(a)1+2b/ηb2‖Π⟂(ϕ(a))‖2.Var_ (g_DG)\;≤\; _a≠ y^*π(a)^1+2b/η\,b^2\,\| _ (φ(a))\|^2. When b/η>0b/η>0, the exponent 1+2b/η>11+2b/η>1 ensures that rare actions are suppressed more aggressively than under PG (which has exponent 11). The exact equality of Proposition 1(i) becomes an inequality, but the qualitative conclusion—DG suppresses perpendicular noise from rare incorrect actions—holds without symmetry. Directional bias. Without symmetry, each incorrect action receives a different gate value, so [gDG]E[g_DG] is no longer exactly parallel to ∇J∇ J. This introduces a small perpendicular bias—a directional cost, since the two oracles already coincide in a single context. The bias vanishes as the policy concentrates on the correct action (ε→0 → 0): all incorrect-action contributions shrink, and the gate values converge. C.4 Proof of Lemma 1 (Greedy Directions) We derive the greedy-optimal direction for each objective by Cauchy–Schwarz. Let an:=‖vn‖2a_n:=\|v_n\|^2 then for an update z←z+αg/‖g‖z← z+α\,g/\|g\| with g=∑ncnvng= _nc_nv_n and ‖g‖2=∑ncn2an\|g\|^2= _nc_n^2a_n (by orthogonality): Arithmetic objective. Δ(∑npn)=∑n⟨∇znpn,αcnvn/‖g‖⟩ ( _np_n)= _n _z_np_n,α c_nv_n/\|g\| . Since ∇znpn=pnvn _z_np_n=p_nv_n, each term is αcnpnan/‖g‖α c_np_na_n/\|g\|. So Δ(∑pn)=α∑ncnpnan/∑mcm2am (Σ p_n)=α _nc_np_na_n/ _mc_m^2a_m. To maximize over c, set xn=cnanx_n=c_n a_n and yn=pnany_n=p_n a_n. The ratio ∑xnyn/‖x‖Σ x_ny_n/\|x\| is maximized by Cauchy–Schwarz when x∝yx y, i.e., cn∝pnc_n p_n. Log objective. Δ(∑nlogpn)=∑n⟨∇znlogpn,αcnvn/‖g‖⟩ ( _n p_n)= _n _z_n p_n,α c_nv_n/\|g\| . Since ∇znlogpn=vn _z_n p_n=v_n, each term is αcnan/‖g‖α c_na_n/\|g\|. Maximized when cn∝an/an=1c_n a_n/a_n=1 (again by Cauchy–Schwarz with yn=any_n= a_n). ∎ C.5 Proof of Proposition 2 (N=2N=2) The key tool is a monotonicity lemma: the cosine between a weighted sum and the equal-weight sum is uniquely maximized when the ratio of weights equals one. Lemma 3 (Two-vector cosine monotonicity). For v1,v2v_1,v_2 orthogonal with norms a1,a2>0a_1,a_2>0, define C(r):=cos(rv1+v2,v1+v2)C(r):= (r\,v_1+v_2,\;v_1+v_2) for r>0r>0. Then C(r)C(r) is uniquely maximized at r=1r=1 and strictly decreasing as r moves away from 11. Proof. By orthogonality: C(r)=ra1+a2(r2a1+a2)(a1+a2).C(r)= r\,a_1+a_2 (r^2a_1+a_2)(a_1+a_2). Since C(r)>0C(r)>0 on (0,∞)(0,∞), C is maximized where C(r)2C(r)^2 is. Define Q(r):=C(r)2⋅(a1+a2)=(ra1+a2)2/(r2a1+a2)Q(r):=C(r)^2·(a_1+a_2)=(r\,a_1+a_2)^2/(r^2a_1+a_2). Differentiating by the quotient rule and simplifying the numerator: Q′(r)=2a1a2(1−r)(ra1+a2)(r2a1+a2)2.Q (r)= 2a_1a_2(1-r)(r\,a_1+a_2)(r^2a_1+a_2)^2. Since a1,a2>0a_1,a_2>0 and ra1+a2>0r\,a_1+a_2>0, we have Q′(r)>0Q (r)>0 for r<1r<1 and Q′(r)<0Q (r)<0 for r>1r>1. Thus Q (and hence C) is uniquely maximized at r=1r=1. ∎ Proof of Proposition 2. The PG population direction has ratio rPG=p1/p2r_PG=p_1/p_2. The DG population direction has ratio rDG=h(p1)/h(p2)r_DG=h(p_1)/h(p_2), where h(p)=pσ((−logp)/η)h(p)=p\,σ((- p)/η). Assume WLOG p1>p2p_1>p_2, so rPG>1r_PG>1. Since h is increasing, rDG=h(p1)/h(p2)>1r_DG=h(p_1)/h(p_2)>1. Since the gate σ((−logp)/η)σ((- p)/η) is decreasing, rDG=(p1/p2)⋅[σ((−logp1)/η)/σ((−logp2)/η)]<rPGr_DG=(p_1/p_2)·[σ((- p_1)/η)/σ((- p_2)/η)]<r_PG. Thus 1<rDG<rPG1<r_DG<r_PG, and by Lemma 3, C(rDG)>C(rPG)C(r_DG)>C(r_PG). ∎ C.6 Directional Improvement for General N The N=2N=2 proof (Appendix C.5) uses direct ratio compression. For general N, we interpolate continuously between PG and DG weights and show the squared cosine strictly increases along the path. Proposition 4 (Directional improvement, general N). Let N≥2N≥ 2 contexts with orthogonal score vectors vnv_n of norms an>0a_n>0. Let η>1/2η>1/2, and suppose the pnp_n are not all equal. Then cos([gDG],gCE)>cos([gPG],gCE). \! (E[g_DG],\;g_CE )\;>\; \! (E[g_PG],\;g_CE ). Proof. Define the interpolated weights cn(t)=pn[σ((−logpn)/η)]tc_n(t)=p_n\,[σ((- p_n)/η)]^t for t∈[0,1]t∈[0,1]. At t=0t=0, cn(0)=pnc_n(0)=p_n (PG weights); at t=1t=1, cn(1)=pnσ((−logpn)/η)c_n(1)=p_n\,σ((- p_n)/η) (DG weights). Define the squared-cosine proxy Φ(t):=(∑ncn(t)an)2∑ncn(t)2an, (t):= ( _nc_n(t)\,a_n )^2 _nc_n(t)^2\,a_n, which is proportional to cos2(∑cn(t)vn,∑vn) ^2\! (Σ c_n(t)\,v_n,\;Σ v_n ). We show Φ′(t)>0 (t)>0 for all t∈[0,1)t∈[0,1). Setting λn:=logσ((−logpn)/η)<0 _n:= σ((- p_n)/η)<0 (since σ<1σ<1 on (0,1)(0,1)), we have cn′(t)=cn(t)λnc_n (t)=c_n(t)\, _n. Differentiating Φ=A2/B =A^2/B with A=∑cnanA=Σ c_na_n and B=∑cn2anB=Σ c_n^2a_n: Φ′(t)=2AB2[A′B−AB′/2]∝∑n,mcncm2anam(λn−λm). (t)= 2AB^2 [A \,B-A\,B \!/2 ]\; \; _n,mc_n\,c_m^2\,a_n\,a_m\,( _n- _m). Antisymmetrizing: Φ′(t)∝∑n<m(λn−λm)anamcncm(cm−cn). (t)\; \; _n<m( _n- _m)\,a_n\,a_m\,c_n\,c_m\,(c_m-c_n). (7) We check the sign of each term. Since η>1/2η>1/2, the function p↦p[σ((−logp)/η)]tp p\,[σ((- p)/η)]^t is increasing for all t∈[0,1]t∈[0,1].555Write pσ((−logp)/η)t=p/(1+p1/η)tp\,σ((- p)/η)^t=p/(1+p^1/η)^t. Its derivative has the sign of 1+(1−t/η)p1/η1+(1-t/η)\,p^1/η, which at worst (t=1t=1, p→1p→1) equals 2−1/η>02-1/η>0. Therefore cn(t)c_n(t) preserves the ordering of the pnp_n. Since λn _n is strictly decreasing in pnp_n (as σ((−logp)/η)σ((- p)/η) is decreasing), for each pair n<mn<m with pn≠pmp_n≠ p_m: • pn>pm⟹cn>cmp_n>p_m c_n>c_m and λn<λm _n< _m, so (λn−λm)(cm−cn)>0( _n- _m)(c_m-c_n)>0. • pn<pm⟹cn<cmp_n<p_m c_n<c_m and λn>λm _n> _m, so (λn−λm)(cm−cn)>0( _n- _m)(c_m-c_n)>0. Each factor anamcncma_n\,a_m\,c_n\,c_m is positive, so every non-degenerate term in (7) is strictly positive. Since not all pnp_n are equal, Φ′(t)>0 (t)>0 for all t∈[0,1)t∈[0,1), giving Φ(1)>Φ(0) (1)> (0). Both cosines are positive, so cos([gDG],gCE)>cos([gPG],gCE) (E[g_DG],g_CE)> (E[g_PG],g_CE). ∎ On the η>1/2η>1/2 condition. The condition η>1/2η>1/2 ensures that the DG weight function p↦pσ((−logp)/η)p p\,σ((- p)/η) is monotonically increasing, which the path argument requires to preserve the ordering of weights. Since we use η=1η=1 throughout, this condition is always satisfied. For η≤1/2η≤ 1/2 the weight function can be non-monotone near p=1p=1, and Proposition 2 may fail; exploring this regime is an interesting direction for future work. Appendix D Transformer Sequence Modeling We provide experimental details and robustness checks for the Token Reversal experiments in Section 5. Experimental Setup. The agent is a decoder-only Transformer with causal attention, model dimension dmodel=64d_model=64, 2 layers, and 2 attention heads. We use a distributed actor-learner architecture with 10 parallel actors, each collecting batches of 10 trajectories, yielding 100 episodes per gradient step. All agents are trained with Adam using a standard empirical mean baseline for variance reduction. Unless otherwise noted, the training budget is K=1,000K=1,000 episodes. D.1 Baseline Tuning To ensure fair comparison, we tuned hyperparameters for PPO and PMPO. For PPO, we swept the clipping parameter ϵε and KL penalty β; Figure 20a shows the algorithm is insensitive to ϵε, and the KL penalty does not help on this deterministic task. For PMPO, we swept the weighting threshold α and KL penalty β; Figure 20b shows boundary values (α≈0α≈ 0 or 11) outperform intermediate values. Even the best-tuned PPO (Regret≈0.03Regret≈ 0.03) and PMPO (Regret≈0.08Regret≈ 0.08) lag behind DG (Regret<0.01Regret<0.01). (a) Tuning PPO (ϵε, β). (b) Tuning PMPO (α, β). Figure 20: Hyperparameter tuning for baselines. Neither PPO nor PMPO matches DG despite extensive sweeps. D.2 Robustness We validate robustness on the default task (H=10H=10, M=2M=2). Figure 21a shows regret at K=1,000K=1,000 across learning rates; DG consistently outperforms baselines over a wide effective range. Figure 21b extends training to K=10,000K=10,000 episodes; baselines do not catch up, confirming the advantage is not due to faster early learning alone. (a) Learning rate sensitivity. (b) Extended training (K=10kK=10k). Figure 21: DG’s advantage is robust to learning rate (a) and persists asymptotically (b). D.3 Task Variations To ensure findings generalize beyond reversal, we test four target logics: • Copy: yi=xiy_i=x_i • Flip: yi=1−xiy_i=1-x_i • Reverse Copy: yi=xH−i+1y_i=x_H-i+1 (the default) • Reverse Flip: reverse and negate We also vary reward structure. Bag-of-Tokens gives credit for each correct token regardless of position; Sequential gives credit only up to the first mistake, making credit assignment harder. Figure 22 illustrates the difference. Figure 23 shows learning curves for all eight configurations. DG achieves the lowest regret in every setting. The gap is often larger in the harder Sequential settings, where baselines struggle with truncated reward streams. (a) Bag-of-Tokens (dense). (b) Sequential (strict). Figure 22: Reward structures. Bag-of-Tokens credits all correct tokens; Sequential stops at the first error. Figure 23: Learning curves across 8 task variants. Top: Bag-of-Tokens. Bottom: Sequential. DG (red) achieves the lowest error in all configurations. D.4 Multiplicative vs. Additive Delight We compare DG to UCB-style additive mixtures: χαUCB=(1−α)U+αℓ.χ UCB_α=(1-α)U+α . (8) Figure 24 sweeps α∈[0,1.25]α∈[0,1.25] and η∈0.2,0.5,1,2,5η∈\0.2,0.5,1,2,5\. Additive mixtures can outperform REINFORCE (black dashed) but never approach DG (red dashed). The best additive achieves regret ≈0.1≈ 0.1 versus DG’s ≈0.04≈ 0.04. Additive bonuses treat surprisal symmetrically, encouraging the policy to chase high-surprisal actions even when advantage is negative. DG’s multiplicative gate suppresses such blunders, filtering them from the update. Figure 24: Additive mixtures (colored lines) vs. multiplicative DG (red dashed). No additive configuration matches DG. Appendix E Detailed Control Suite Results We provide experimental details and extended results for Section 6. E.1 Experimental Setup Architecture. All methods use the same actor-critic architecture: a 2-layer MLP with 256 hidden units for both actor and critic. The actor outputs mean and diagonal covariance of a Gaussian policy; the critic estimates state-action values. We use the Retrace algorithm (munos2016safe) for off-policy correction. Optimization. All methods use Adam with learning rate 3×10−43× 10^-4. We train for 10 million environment steps with 4 parallel actors, a replay buffer of size 2×1062× 10^6, and batch size 256. Target networks are updated every 100 learner steps. DG-specific details. For continuous actions, surprisal ℓ=−logπ(a∣s) =- π(a s) can take large values. We clip surprisal to [−10,10][-10,10] before computing delight. We also normalize rewards using an exponential moving average (decay 0.999) for critic stability; this normalization is applied to all methods. The gate temperature is η=1η=1, consistent with all other experiments. Algorithm 2 provides the continuous-action variant used in all control experiments. Algorithm 2 Delightful Policy Gradient (continuous actions) 1:Batch ℬB, Gaussian policy πθ(⋅∣s) _θ(· s), temperature η=1η=1, clip bound C=10C=10 2:Δθ←0 θ← 0 3:for t∈ℬt do 4: ℓt←clip(−logπθ(At∣ℋt),−C,C) _t (- _θ(A_t _t),\;-C,\;C) 5: χt←Ut⋅ℓt _t← U_t· _t ⊳ Delight 6: wt←σ(χt/η)w_t←σ( _t/η) ⊳ Gate 7: Δθ←Δθ+wtUt∇θlogπθ(At∣ℋt) θ← θ+w_t\,U_t\, _θ _θ(A_t _t) 8:return Δθ θ Baselines. We compare against three baselines within our codebase: • PG: Standard policy gradient with Retrace critic and no gating. • PPO: Clipped surrogate objective with ϵ=0.2ε=0.2. • MPO: Softmax-weighted updates with temperature η=1.0η=1.0. All baselines use identical architecture, optimizer, and replay settings. E.2 Per-Environment Learning Curves Figure 25 displays individual learning curves for all 28 Control Suite environments. DG (red) consistently matches or exceeds baseline performance, with notable improvements on exploration-heavy tasks such as acrobot:swingup, finger:turn_hard, and humanoid:run. Figure 25: Learning curves for 28 Control Suite environments. DG (red) consistently matches or exceeds the hedonic baseline (purple), PPO (blue), and MPO (green). E.3 Baseline Hyperparameter Sensitivity To ensure fair comparison, we verified that default hyperparameters are reasonable for PPO and MPO. Figure 26 shows sensitivity sweeps on cartpole:swingup. For PPO, we sweep clip parameter ϵ∈[0.01,100]ε∈[0.01,100]; for MPO, we sweep temperature η∈[0.001,1000]η∈[0.001,1000]. Neither sweep reveals a clear optimum that substantially outperforms the defaults (ϵ=0.2ε=0.2, η=1.0η=1.0). We therefore use these standard values throughout. (a) PPO clip ϵε. (b) MPO temperature η. Figure 26: Hyperparameter sensitivity on cartpole:swingup. Default values (dashed) perform comparably to alternatives. E.4 Comparison to Tuned External Implementations We also benchmark DG against highly-optimized external implementations: MPO (Tuned) with adaptive temperature and SAC (Tuned) with automatic entropy adjustment. These use separate codebases with extensive per-task tuning. This comparison is especially stringent because regret is defined relative to the best performance achieved by any method: Regretk=Rbest−RkRegret_k=R_best-R_k. Figure 27 shows aggregate regret over training. Even against these optimized baselines, DG achieves the lowest average regret. Figure 28 breaks down final performance by environment, confirming that DG’s advantage is broad-based rather than driven by outliers. DG achieves low regret across domains ranging from cartpole to humanoid and dog. Figure 27: Aggregate regret against tuned SOTA baselines. DG (pink) outperforms tuned MPO (gold) and SAC (blue). Figure 28: Per-environment regret at 10M steps. DG (pink) consistently achieves low regret across the suite.