Paper deep dive
Portfolio Reinforcement Learning with Scenario-Context Rollout
Vanya Priscillia Bendatu, Yao Lu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 6:58:52 AM
Summary
The paper introduces Scenario-Context Rollout (SCR), a macro-conditioned method for portfolio reinforcement learning that generates plausible next-day multivariate return scenarios under stress events. It addresses the reward-transition mismatch in temporal-difference learning by constructing counterfactual next states from rollout-implied continuations, stabilizing critic training and improving Sharpe ratio and drawdown metrics across U.S. equity and ETF portfolios.
Entities (9)
Relation Signals (6)
Scenario-Context Rollout → uses → Reinforcement Learning
confidence 95% · We propose macro-conditioned scenario-context rollout (SCR) ... in training RL agents for portfolio management.
Scenario-Context Rollout → addresses → Reward-Transition Mismatch
confidence 93% · incorporating scenario-based rewards from rollouts introduces a reward–transition mismatch in temporal-difference learning... We fix the gap here
Counterfactual Continuation → stabilizes → Critic Training
confidence 92% · Doing so stabilizes the learning and provides a viable bias-variance tradeoff.
Scenario-Context Rollout → improves → Sharpe Ratio
confidence 90% · our method improves Sharpe ratio by up to 76%
Scenario-Context Rollout → reduces → Maximum Drawdown
confidence 90% · reduces maximum drawdown by up to 53%
Hybrid Bellman Operator → causes → Fixed-Point Bias
confidence 85% · this hybrid operator generally has a distinct fixed point from the scenario-consistent objective, creating an inherent fixed-point bias.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Market regime shifts induce distribution shifts that can degrade the performance of portfolio rebalancing policies. We propose macro-conditioned scenario-context rollout (SCR) that generates plausible next-day multivariate return scenarios under stress events. However, doing so faces new challenges, as history will never tell what would have happened differently. As a result, incorporating scenario-based rewards from rollouts introduces a reward--transition mismatch in temporal-difference learning, destabilizing RL critic training. We analyze this inconsistency and show it leads to a mixed evaluation target. Guided by this analysis, we construct a counterfactual next state using the rollout-implied continuations and augment the critic agent's bootstrap target. Doing so stabilizes the learning and provides a viable bias-variance tradeoff. In out-of-sample evaluations across 31 distinct universes of U.S. equity and ETF portfolios, our method improves Sharpe ratio by up to 76% and reduces maximum drawdown by up to 53% compared with classic and RL-based portfolio rebalancing baselines.
Tags
Links
- Source: https://arxiv.org/abs/2602.24037v1
- Canonical: https://arxiv.org/abs/2602.24037v1
Trouble viewing inline? Open PDF directly →
Full Text
66,686 characters extracted from source content.
Expand or collapse full text
Portfolio Reinforcement Learning with Scenario-Context Rollout Vanya Priscillia Bendatu National University of Singapore vanyabendatu@u.nus.edu Yao Lu National University of Singapore luyao@comp.nus.edu.sg Abstract Market regime shifts induce distribution shifts that can degrade the performance of portfolio rebalancing policies. We propose macro-conditioned scenario-context rollout (SCR) that generates plausible next-day multivariate return scenarios under stress events. However, doing so faces new challenges, as history will never tell what would have happened differently. As a result, incorporating scenario-based rewards from rollouts introduces a reward–transition mismatch in temporal-difference learning, destabilizing RL critic training. We analyze this inconsistency and show it leads to a mixed evaluation target. Guided by this analysis, we construct a counterfactual next state using the rollout-implied continuations and augment the critic agent’s bootstrap target. Doing so stabilizes the learning and provides a viable bias-variance tradeoff. In out-of-sample evaluations across 31 distinct universes of U.S. equity and ETF portfolios, our method improves Sharpe ratio by up to 76% and reduces maximum drawdown by up to 53% compared with classic and RL-based portfolio rebalancing baselines. 1 Introduction The financial market is non‑stationary. Over time, the expected returns, volatilities, and cross‑asset dependency change as the macroeconomic structure and the market dynamics evolve. In portfolio management and rebalancing applications, regime changes may cause degraded returns due to the dependencies during downturns [23, 4, 1]. A real-world portfolio policy must be resilient to distribution shifts and synchronized sell‑offs [22, 12, 18]. Risk management traditionally addresses robustness through scenario analysis and stress testing. Agent-based macro modeling and simulation [31, 41] push this idea further such that policies can be evaluated under counterfactual scenarios while preserving economically meaningful structures. On the other hand, reinforcement learning (RL) enables portfolio rebalancing and treats the asset allocation problem as a Markov decision process (MDP), aiming to maximize the return while following trading limits and constraints [26, 15, 10, 36]. Recent policy‑gradient methods, such as proximal policy optimization (PPO) [34], stabilize training with clipped objectives and bootstrapped value targets. In this paper, we draw inspiration from agent-based macro modeling and explore this idea in training RL agents for portfolio management. To improve next-state estimation, we propose Scenario-Context Rollout (SCR), a leak-safe feedback mechanism to produce a distribution of next-day joint returns under potential economic shocks. However, doing so poses a key technical challenge: while outcomes can be evaluated under plausible simulated scenarios, the agent observes only the realized transitions from the market history and cannot interact with the market to test what‑if actions [22]. As a result, alternative trajectories are only hypothetical. This problem worsens in distribution shifts; the policy may generate state and action pairs that are not supported by economic structures [17, 18]. To deepen our comprehension of this problem, we in theory formalize scenario-conditioned learning with historical market data and show that coupling scenario-based rewards with tape-based (realized history) continuations induces hybrid Bellman operator [32, 42] under outcome-dependent state memory and causes a reward–transition mismatch in temporal-difference (TD) learning. Accordingly, we use the Wasserstein-form operator bounds [38, 9] to prove that this hybrid operator generally has a distinct fixed point from the scenario-consistent objective, creating an inherent fixed-point bias. This divergence is not only a nuisance but also exposes a bias–variance tradeoff in critic training. Based on this analysis, we propose to augment the critic’s bootstrap target with a counterfactual continuation that is constructed from the SCR returns. The resulting target interpolates between a tape and a scenario bootstrap; we show that doing so improves training stability and robustness while adhering to economic structure and historical observations. We evaluate our solution on day-to-day portfolio rebalancing across 31 distinct universes of U.S. equity and ETF portfolios. We compare against classic portfolio rebalancing algorithms, tape-only PPO, risk-penalized, and rollout-based baselines. Results show that our solution improves out-of-sample Sharpe ratio by up to 76% and reduces maximum drawdown by over 53%, while providing a more stable RL training. Contributions of this paper can be summarized as follows: • We introduce SCR for RL-based portfolio management. SCR is a macroeconomics-guided feedback mechanism that produces a distribution of returns under economic shock scenarios. • We analyze in theory that scenario rewards together with tape continuations induce a hybrid Bellman operator and yield a fixed-point bias characterized by Wasserstein-form bounds. This implies a bias-variance tradeoff in critic training. We design our solution based on this analysis. • Our solution shows empirical gains using historical market data over state-of-the-art portfolio rebalancing algorithms and RL models. 2 Method Rebalancing close-to-close trading portfolios is a sequential decision problem. At the close of every trading day t, we pick a vector of weights t∈ℝNw_t ^N across N assets. To calculate the performance of the portfolio, we use the asset return vector t+1∈ℝNr_t+1 ^N. Each individual element rt+1,ir_t+1,i of this vector is defined as rt+1,i:=pt+1,i−pt,ipt,i,i=1,…,N,r_t+1,i:= p_t+1,i-p_t,ip_t,i, i=1,…,N, (1) where pt,ip_t,i represents the closing price of asset i on day t. The final realized net portfolio return is then given by Rp,t+1=t⊤t+1−costt,R_p,t+1=w _tr_t+1-cost_t, (2) where costtcost_t is a term that increases with portfolio turnover [28, 37]. Figure 1: Overview of our RL paradigm. SCR produces a conditional scenario distribution over next-day joint return vectors. The critic is trained using a bootstrap target that augments the realized continuation bootstrap with a counterfactual continuation. Figure 1 summarizes our scenario-conditioned RL pipeline. We follow the standard PPO actor-critic training paradigm [34] and utilize a learned value function together with a critic bootstrap target. Our objective is to enhance robustness in the face of regime shifts and changes in distribution. Specifically, we gather regime/exposure features which serve as a retrieval descriptor denoted by ψt _t. Crucially, ψt _t excludes portfolio-history components (e.g., previous weights or internal memory states), ensuring that scenario retrieval relies solely on market conditions instead of the agent’s past trajectory. Therefore, Scenario-Context Rollout (SCR) estimates a conditional distribution Pscen(⋅∣ψt)P_scen(· _t) over next-day joint return vectors induced by macro scenarios. Let t+1∼Pscen(⋅∣ψt)R_t+1 P_scen(· _t) denote a generic macro scenario draw. To approximate expectations, we sample independent and identically distributed (i.i.d.) macro scenarios t+1(s)∼i.i.d.Pscen(⋅∣ψt),s=1,…,S.R^(s)_t+1 .i.d. P_scen(· _t), s=1,…,S. We define ¯t+1scen r^scen_t+1 as a compact representation used in the counterfactual continuation (Sec. 2.2). This joint-return, conditioned by the prevailing regime, captures the dependencies and tail risks that become prominent in times of stress. At each decision time t, the policy observes an input state ϕt=(zt,ht) _t=(z_t,h_t) containing only information available by time t. In this context, ztz_t encompasses market features and portfolio history and hth_t provides longer-horizon context including SCR outputs. The policy model generates an action ata_t. To ensure that the final portfolio is valid, we apply a set of operational constraints tW_t, covering box, leverage, and turnover constraints: t=Πt(at)∈t,w_t= _t(a_t) _t, (3) where Πt _t denotes the projection onto tW_t. All constraint budgets are treated as held identical across all strategies. Let r(ϕ,;)r(φ,w;x) denote the per-step reward evaluated at state ϕφ using weights w and based on an input return vector x. We train PPO to maximize a discounted objective where rewards are evaluated under SCR macro scenarios, which involve a conditional distribution of plausible next-day joint returns: Jscen(π):=π,Pscen[∑t=0T−1δtr(ϕt,t;t+1)].J_scen(π):=E_π,P_scen\! [ _t=0^T-1δ^t\,r( _t,w_t;R_t+1) ]. (4) The concrete form of reward r(ϕt,t;t+1)r( _t,w_t;R_t+1) is defined in Sec. 2.2. We use a discount factor δ∈(0,1)δ∈(0,1) to ensure stability in temporal-difference (TD) bootstrapping. That is, in discounted MDPs, the Bellman operator acts as a contraction with a modulus of δ. This enhances the conditioning of value targets and mitigates sensitivity to noise arising from long-horizon estimations (e.g., [36, 3]). It is important to recognize that a logged market tape (i.e., a fixed, non-rewinding historical record) provides only a single realized next-day return vector t+1r_t+1 at each time t, and thus only a single realized continuation. By utilizing executed weights tw_t, we update the realized next state through function ϕt+1=Upd(ϕt,t,t+1) _t+1=Upd( _t,w_t,r_t+1). Conversely, Eq. (4) specifies how actions are scored in macro scenario estimation, but the logged tape determines the realized continuation. This creates a mismatch between (i) scenario-based reward scoring (as in Jscen(π)J_scen(π)) and (i) the realized next-state transition ϕt+1 _t+1. We fix the gap here using the macro scenario conditional mean ¯t+1scen r^scen_t+1 by constructing a counterfactual continuation: ϕt+1cf=Upd(ϕt,t,¯t+1scen)φ^cf_t+1=Upd( _t,w_t, r^scen_t+1), and then mixing the critic bootstraps evaluated at ϕt+1 _t+1 and ϕt+1cfφ^cf_t+1 in the augmented TD target YβcfY_ _cf (see Sec. 2.2). This results in a bootstrap signal that is aligned with the scenario and exhibits low variance, while remaining grounded in the tape. Ultimately, PPO updates the critic based on YβcfY_ _cf and updates the policy accordingly. 2.1 Scenario-Context Rollout (SCR) Regime representation. Each trading day t can be represented by a low-dimensional regime embedding Ut∈ℝkU_t ^k to summarize the market. UtU_t is learned to capture patterns in two sources of information: (i) market–macro behavior (cross-sectional returns and macro shocks) and (i) policy-text signals. Intuitively, nearby points in the shock space correspond to days with similar “macro-financial regimes.” A shock day is hence a day whose regime embedding UtU_t contains an anomaly; this can be learned during training as a binary indicator shockt∈0,1shock_t∈\0,1\. As shocks often happen sequentially, spanning days to months. We cluster subsequent shock days into a shock channel, assigning each shock-day embedding to an existing cluster or creates a new one when it is farther than a distance threshold λ2λ^2 from all existing channel centroids using the running mean embedding of each channel. We keep a ShockLedger, a compact catalog of shock channels that encapsulates the channel’s macro signature and representative movers. To obtain a daily regime characteristic for conditioning, we aggregate recent channel hits into a binary activation pattern, considering the activated channels as active near time t. This yields a channel-activation vector t∈0,1C χ_t∈\0,1\^C, where χt,c=1 _t,c=1 if and only if channel c is assigned to at least one shock day within a brief trailing lookback window concluding at time t. The lookback aggregation reflects the enduring effects of macro shocks over several days. This reduces daily fluctuations in the output of the detector, resulting in a more stable conditioning signal. Scenario retrieval and rollout. Macro scenario retrieval is facilitated through channel activations using ψt _t with the preprocessed regime and exposure data that is accessible at time t (e.g., macro shocks, channel activations t χ_t, and exposure summaries). To uphold leak-safety, ψt _t keeps only necessary features and omits those related to the portfolio history. Hence, SCR maintains a library of return vectors indexed by a series of historical decision times ℐI: :=(~u+1,ψu)u∈ℐ,D:=\( r_u+1, _u)\_u , (5) where u<tu<t denotes a past decision time. The vector ~u+1 r_u+1 is a one-step-ahead macro scenario return that is derived from the macro context through a rolling macro-to-return map, using only information available at time u. At time t, SCR restricts to past indices ℐt:=u∈ℐ:u<tI_t:=\u :\ u<t\ and retrieves nearest neighbors by comparing the current descriptor ψt _t and past descriptors ψuu∈ℐt\ _u\_u _t in D. Let sim(⋅,⋅)sim(·,·) be a similarity score. We define the k nearest-neighbor indices as k(t):=TopKu∈ℐtsim(ψt,ψu).N_k(t):=TopK_u _t\ sim( _t, _u). (6) Therefore, SCR produces the conditional distribution based on the retrieval: Pscen(⋅∣ψt):=1|k(t)|∑u∈k(t)δ~u+1(⋅),P_scen(· _t):= 1|N_k(t)| _u _k(t) _ r_u+1(·), (7) where δx _x denotes a point mass at x. During RL training, we draw S i.i.d. macro scenarios t+1(s)s=1S\R^(s)_t+1\_s=1^S from Pscen(⋅∣ψt)P_scen(· _t). Regime context gtg_t. Beyond scenario sampling, SCR outputs a regime context gtg_t that gates risk-taking at decision time t. It acts as a risk-budget multiplier: gt≈1g_t≈ 1 corresponds to normal conditions, while smaller gtg_t indicates stress and shrinks scenario-evaluated portfolio payoffs, discouraging aggressive exposures. We compute a nonnegative severity score vtv_t via a short-horizon deterministic macro stress rollout driven by the active channels t χ_t, tracking the peak Mahalanobis [24] stress along the rollout. We then normalize vtv_t against a trailing quantile baseline, as it may drift over time. Definition 2.1 (Quantile-normalized regime context gtg_t). Let vt≥0v_t≥ 0 be the severity score at decision time t, and let t:=t−Lg,…,t−1W_t:=\t-L_g,…,t-1\ be a trailing window of past decisions of length LgL_g and a quantile qg∈(0,1)q_g∈(0,1). Define the reference scale qt:=Quantileqg(vu:u∈t)q_t:=Quantile_q_g (\v_u:\ u _t\ ), and the regime context gt:=clip(1−αgvtqt+ε,g¯,1)∈[g¯,1],g_t:=clip\! (1- _g v_tq_t+ , g,1 )∈[ g,1], (8) where αg>0 _g>0 controls sensitivity, g¯∈(0,1) g∈(0,1) sets a minimum value. We apply gtg_t as an exposure gate by scaling scenario portfolio returns u~t+1(s)=gt⟨t,t+1(s)⟩, u_t+1^(s)\;=\;g_t\, _t,R_t+1^(s) , so larger estimated severity (larger vtv_t) yields smaller gtg_t, attenuating the reward signal during periods of stress. In other words, it reduces risk-taking and provides smooth, effective risk management. 2.2 Counterfactual Continuation for Critic Target Augmentation We use a deterministic function Upd:(ϕt,t,t+1)↦ϕt+1Upd:\ ( _t,w_t,x_t+1) _t+1, which advances the current state when fed with an input return vector t+1∈ℝNx_t+1 ^N.111t+1x_t+1 is a placeholder for the return vector; on the tape we observe t+1=t+1x_t+1=r_t+1. At time t, the agent selects an action ata_t and executes feasible portfolio weights t=Πt(at)w_t= _t(a_t). The tape provides the realized return vector t+1r_t+1 and the realized continuation ϕt+1=Upd(ϕt,t,t+1),t+1from tape. _t+1=Upd\! ( _t,w_t,r_t+1 ), _t+1\ from tape. (9) This realized continuation mismatches with macro scenario estimates. We address this problem by constructing a counterfactual continuation and augmenting the critic’s one-step bootstrap target. Specifically, given executed weights tw_t and macro scenario samples t+1(s)s=1S∼Pscen(⋅∣ψt)\R_t+1^(s)\_s=1^S P_scen(· _t), we use the gated outcomes u~t+1(s) u_t+1^(s) defined in Sec. 2.1 and their sample mean u~¯t+1:=1S∑s=1Su~t+1(s). u_t+1\;:=\; 1S _s=1^S u_t+1^(s). Integrating these macro scenario outcomes yields a risk-aware reward: rt:= r_t= u~¯t+1−λρ(Riskη(u~t+1(s)s=1S)+ηε) u_t+1- _ρ (Risk_η\! (\ u_t+1^(s)\_s=1^S )+η\, ) (10) −Reg(t,t−1), -Reg(w_t,w_t-1), where λρ≥0 _ρ≥ 0 regulates the intensity of the tail-risk penalty, RiskηRisk_η represents a functional that is sensitive to tail conditions with a temperature parameter of η>0η>0, and RegReg accounts for trading frictions as well as structural penalties. The conditional mean return is hence approximated through Monte Carlo [14]: ¯t+1scen:=Pscen(⋅∣ψt)[t+1]≈1S∑s=1St+1(s). r^scen_t+1:=E_P_scen(· _t)[R_t+1]≈ 1S _s=1^SR_t+1^(s). (11) Using this approximation, we establish the counterfactual continuation by applying the same update function to ¯t+1scen r^scen_t+1: ϕt+1cf=Upd(ϕt,t,¯t+1scen).φ^cf_t+1=Upd\! ( _t,w_t, r^scen_t+1 ). (12) Notably, this counterfactual continuation is secured against data leakage as it relies solely on quantities available by time t. It is also aligned with macro scenario since ¯t+1scen r^scen_t+1 is derived from the same macro scenario distribution Pscen(⋅∣ψt)P_scen(· _t) utilized for the reward. Augmenting Critic Bootstrap Target. Let Vω(ϕ)V_ω(φ) denote the critic, i.e., a parametric approximation to the state-value function under policy. Let δ∈(0,1)δ∈(0,1) be the RL discount factor. Using the same scenario-based reward rtr_t from Eq. (10), let the one-step targets using (1) the tape-based realized continuation and (2) the counterfactual continuation respectively be Yr:=rt+δVω(ϕt+1),Yc:=rt+δVω(ϕt+1cf).Y_r:=r_t+δ\,V_ω( _t+1), Y_c:=r_t+δ\,V_ω(φ^cf_t+1). Therefore we augment the critic target using Yβcf=(1−βcf)Yr+βcfYc,βcf∈[0,1],Y_ _cf=(1- _cf)\,Y_r+ _cf\,Y_c, _cf∈[0,1], (13) where βcf _cf interpolates between tape-faithful bootstrapping and scenario-aligned bootstrapping. The critic is trained by regressing Vω(ϕt)V_ω( _t) onto YβcfY_ _cf, and the policy model undergoes updates through PPO-Clip with GAE advantages (Alg. 1). Algorithm 1 Actor–critic training using logged tape with SCR macro scenarios and augmented targets 1:Logged tape ϕt,t+1t=0T−1\ _t,r_t+1\_t=0^T-1; SCR module returning (Pscen(⋅∣ψt),gt)(P_scen(· _t),g_t); feasibility map Πt _t; update UpdUpd; discount δ; mixing βcf _cf; scenario sample size S 2:for each iteration of PPO update do 3: for t=0,…,T−1t=0,…,T-1 (traverse through logged dates) do 4: Construct ϕt _t; sample at∼πθ(⋅∣ϕt)a_t _θ(· _t); t=Πt(at)w_t= _t(a_t) 5: Obtain (Pscen(⋅∣ψt),gt)(P_scen(· _t),g_t) from the SCR module 6: Sample t+1(s)s=1S∼Pscen(⋅∣ψt)\R_t+1^(s)\_s=1^S P_scen(· _t) 7: Compute scenario reward rtr_t using t+1(s)\R_t+1^(s)\, gtg_t (Eq. (10)) 8: Compute scenario mean ¯t+1scen=1S∑s=1St+1(s) r^scen_t+1= 1S _s=1^SR_t+1^(s) (Eq. (11)) 9: Read realized t+1r_t+1 from tape; ϕt+1=Upd(ϕt,t,t+1) _t+1=Upd( _t,w_t,r_t+1) 10: Set ϕt+1cf=Upd(ϕt,t,¯t+1scen)φ^cf_t+1=Upd( _t,w_t, r^scen_t+1) (Eq. (12)) 11: Form augmented target YβcfY_ _cf using Eq. (13) 12: end for 13: Update critic by regressing Vω(ϕt)V_ω( _t) onto YβcfY_ _cf 14: Compute Generalized Advantage Estimation A^t A_t 15: Update actor using PPO-Clip with A^t A_t 16:end for 3 Continuation Mismatch and Fixes This section discusses the continuation mismatch at the Bellman operator level [32, 36, 3, 27] and explains why augmenting targets on the critic side is both adequate and well-founded. Hybrid vs. scenario-consistent operators. Let a policy π be defined along with any bounded value function V. According to Sec. 2, ψ be the leak-safe descriptor used by SCR and Pscen(⋅∣ψ)P_scen(· ψ) for the induced scenario return law. Let Preal(⋅∣ϕ)P_real(· φ) denote the (conceptual) real-world conditional return law for the tape outcome. We consider two Bellman-style operators that differ only in their bootstrapping techniques: the hybrid operator evaluates scenario reward but bootstraps on the realized continuation, while the scenario-consistent operator uses scenario outcomes for both reward and continuation: (ThybπV)(ϕ):=[r(ϕ,;)+δV(Upd(ϕ,,))|ϕ],(T_hyb^πV)(φ):=E\! [r(φ,w;R)+δ\,V\! (Upd(φ,w,r) )\ |\ φ ], (14) where a∼π(⋅∣ϕ)a π(· φ), =Π(a)w= (a), ∼Pscen(⋅∣ψ)R P_scen(· ψ), and ∼Preal(⋅∣ϕ)r P_real(· φ). In contrast, (TscenπV)(ϕ):=[r(ϕ,;)+δV(Upd(ϕ,,))|ϕ],(T_scen^πV)(φ):=E\! [r(φ,w;R)+δ\,V\! (Upd(φ,w,R) )\ |\ φ ], (15) with the same action sampling but using the scenario outcome within its continuation. Thus ThybπT_hyb^π corresponds to scenario reward + realized continuation, while TscenπT_scen^π corresponds to scenario reward + scenario continuation. When UpdUpd depends on the newly observed next-day return via the memory component h and Pscen(⋅∣ψ)≠Preal(⋅∣ϕ)P_scen(· ψ)≠ P_real(· φ), these operators typically converge to different fixed points. This is because they take evaluation of the continuation term V(Upd(ϕ,,⋅))V(Upd(φ,w,·)) under different outcome distributions. Lemma 3.1. Fix ϕφ and take any bounded V. We define the function fϕ,a()≔V(Upd(ϕ,Π(a),)).f_φ,a(x) V\! (Upd(φ, (a),x) ). (16) Then for any policy π, (ThybπV−TscenπV)(ϕ) (T_hyb^πV-T_scen^πV)(φ) =δa∼π(⋅∣ϕ)[Δ(ϕ,ψ,a)], =δ\,E_a π(· φ)\! [ (φ,ψ,a) ], (17) Δ(ϕ,ψ,a) (φ,ψ,a) :=∼Preal(⋅∣ϕ)[fϕ,a()] =E_r P_real(· φ) [f_φ,a(r) ] −∼Pscen(⋅∣ψ)[fϕ,a()]. -E_R P_scen(· ψ) [f_φ,a(R) ]. Proposition 3.2. Assume that for each fixed (ϕ,)(φ,w) the map ↦Upd(ϕ,,)x (φ,w,x) is LhL_h-Lipschitz in x through the outcome-updated memory component h, i.e., ‖h(Upd(ϕ,,))−h(Upd(ϕ,,))‖≤Lh‖−‖2\|h(Upd(φ,w,x))-h(Upd(φ,w,y))\|≤ L_h\|x-y\|_2. Also, assume V is LVL_V-Lipschitz in that memory (holding exogenous context fixed). Let o denote the information available at the decision time, ϕ=Φ(o)φ= (o) and ψ=Ψ(o)ψ= (o) be summaries constructed from the same o. Let δ∈(0,1)δ∈(0,1) denote the discount factor used in the operators. Define the mismatch by ΔW:=supoW1(Preal(⋅∣Φ(o)),Pscen(⋅∣Ψ(o))), _W:= _o\ W_1\! (P_real(· (o)),\;P_scen(· (o)) ), (18) where W1W_1 is the 1-Wasserstein distance with ground metric d(,)=‖−‖2d(x,y)=\|x-y\|_2 [30]. Then for any policy π and any such V, the resulting one-step operator deviation is bounded by ‖ThybπV−TscenπV‖∞≤δLVLhΔW. \|T_hyb^πV-T_scen^πV \|_∞\ ≤\ δ\,L_VL_h\, _W. (19) Corollary 3.3. Let VscenπV_scen^π be the fixed point of TscenπT_scen^π and VhybπV_hyb^π be the fixed point of ThybπT_hyb^π, assuming both are δ-contractions on (,∥⋅∥∞)(V,\|·\|_∞). Assume VscenπV_scen^π is LVL_V-Lipschitz in the outcome-updated memory component h, and define LV:=Liph(Vscenπ).L_V\;:=\;Lip_h\! (V_scen^π ). Under the conditions of Proposition 3.2, ‖Vhybπ−Vscenπ‖∞ \|V_hyb^π-V_scen^π\|_∞ ≤δ1−δLVLhΔW, ≤ δ1-δ\,L_VL_h\, _W, (20) with ΔW _W from (18). A conflict inherent to logged tape learning is shown in Corollary 3.3. Applying macro scenarios directly to the reward creates immediate friction with the critic. The fundamental problem is that we are scoring actions under SCR macro scenarios, even though we still depend on the tape for bootstrapping. Consequently, the TD update gradually deviates. It follows the hybrid operator and converges (in expectation) to VhybπV^π_hyb, not to the scenario-consistent value, VscenπV^π_scen, required by the scoring objective JscenJ_scen (Eq. (4)). Lemma 3.1 isolates the core issue, showing that the mismatch stems entirely from the continuation term. This is a crucial detail since it means we can adjust the critic bootstrap without actually interfering with the tape traversal. We resolve the issue by generating a counterfactual continuation state and combining its value with the realized tape bootstrap. This creates an augmented target that reduces the mismatch. As a result, following the logic in Sec. 2.2, we can rectify the critic while ensuring our rollouts strictly faithful to the tape. Assumption 3.4. Let ϕ=(z,h)φ=(z,h), where h is the outcome-updated memory component and z denotes exogenous context. For any fixed (ϕ,)(φ,w), assume the update map ↦Upd(ϕ,,)x (φ,w,x) is LhL_h-Lipschitz in x through h. Likewise, the scenario-consistent value VscenπV^π_scen is LVL_V-Lipschitz in h (holding z fixed). Finally, the mismatch and proxy second moments are assumed to be finite: W22(Preal(⋅∣ϕ),Pscen(⋅∣ψ)) W_2^2\! (P_real(· φ),\,P_scen(· ψ) ) <∞, <∞, [∥−μψ∥22|ψ] \! [\|R- _ψ\|_2^2\, |\,ψ ] <∞, <∞, where ∼Pscen(⋅∣ψ)R P_scen(· ψ) and μψ:=[∣ψ] _ψ:=E[R ψ]. We bound the one-step mean squared error between the augmented target and the scenario-consistent target Y⋆Y in accordance with Assumption 3.4. Theorem 3.5 (One-step mixing bound (Wasserstein form)). Consider a decision time with state ϕφ, descriptor ψ, and a raw action a with executed control =Π(a)w= (a). Let the scenario-consistent one-step target be Y⋆ Y :=r(ϕ,;)+δVscenπ(Upd(ϕ,,)), =r(φ,w;R)+δ\,V^π_scen\! (Upd(φ,w,R) ), ∼Pscen(⋅∣ψ). P_scen(· ψ). Define YβcfY_ _cf as in (13) using VscenπV^π_scen in place of VωV_ω and ϕcf=Upd(ϕ,,μψ)φ^cf=Upd(φ,w, _ψ). Under Assumption 3.4, define Δ2(ϕ,ψ) _2(φ,ψ) :=W22(Preal(⋅∣ϕ),Pscen(⋅∣ψ)), =W_2^2\! (P_real(· φ),\,P_scen(· ψ) ), σscen2(ψ) σ^2_scen(ψ) :=[∥−μψ∥22|ψ]. =E\! [\|R- _ψ\|_2^2\, |\,ψ ]. Then [(Yβcf−Y⋆)2|ϕ,ψ,a]≤2δ2(LVLh)2((1−βcf)2Δ2(ϕ,ψ)+βcf 2σscen2(ψ)).E\! [(Y_ _cf-Y )^2\, |\,φ,ψ,a ]\\ ≤ 2\,δ^2(L_VL_h)^2 ((1- _cf)^2 _2(φ,ψ)+ _cf^\,2σ^2_scen(ψ) ). (21) Corollary 3.6. Let A(ϕ,ψ):=Δ2(ϕ,ψ)A(φ,ψ):= _2(φ,ψ) denote the mismatch and B(ψ):=σscen2(ψ)B(ψ):=σ^2_scen(ψ) the proxy variance. Minimizing the right hand side of (21) yields the optimal mixing weight: βcf⋆(ϕ,ψ)=A(ϕ,ψ)A(ϕ,ψ)+B(ψ)∈[0,1].β _cf(φ,ψ)= A(φ,ψ)A(φ,ψ)+B(ψ)∈[0,1]. (22) Critic target augmentation is a bias–variance tradeoff. By weighing the mismatch error, (1−βcf)2Δ2(1- _cf)^2\, _2, against the proxy variance, βcf2σscen2β^2_cfσ^2_scen, the bound reveals a clear tension. Corollary 3.6 suggests that a ”sweet spot” exists whenever both error sources are significant, mirrors the empirical trends for βcf _cf seen in Sec. 4.4. 4 Experiments We evaluate our method on the daily close-to-close portfolio rebalancing task to demonstrate the effectiveness and robustness of our solution, particularly in regime mixture changes. 4.1 Experimental Setup Datasets and universe construction. We use the FinRL U.S. equities and ETFs dataset covering 2009–2023 [29]. Across all experiments, we enforce a chronological split: Training (2009–2017), Validation (2018–2019), and an out-of-sample Testing (2020–2023). A universe is a fixed set of assets (along with their return panels) that we rebalance. To make sure our findings aren’t an artifact of any one asset set, we evaluate on 31 distinct trading universes, each containing roughly 10–20 assets. We evaluate across four categories: Market-Proxy (1 universe of broad market/sector ETFs), High-Vol (10 distinct high-volatility equity universes), Low-Vol (10 distinct low-volatility equity universes), and General (10 distinct equity universes from the broader cross-section) [2, 43]. Table 1: Evaluations and comparisons of portfolio rebalancing algorithms by universe group. Values are in median [Q1, Q3]. Universe Method Sharpe ↑ Calmar ↑ AnnVol ↓ MaxDD ↓ Turnover High-Vol 1/N 0.618 [0.563, 0.692] 0.912 [0.471, 2.114] 0.635 [0.356, 3.047] 0.445 [0.396, 0.493] 0.0000 [0.0000, 0.0000] Markowitz 0.561 [0.476, 0.699] 0.500 [0.317, 0.767] 0.527 [0.258, 0.780] 0.441 [0.411, 0.461] 0.0256 [0.0215, 0.0283] Inverse-Vol 0.630 [0.576, 0.695] 0.873 [0.411, 2.363] 0.589 [0.316, 2.350] 0.418 [0.404, 0.480] 0.0175 [0.0169, 0.0182] GMV (Ledoit-Wolf) 0.580 [0.533, 0.645] 0.850 [0.305, 1.397] 0.858 [0.518, 1.076] 0.412 [0.365, 0.449] 0.0261 [0.0216, 0.0305] PPO (Historical Replay) 0.615 [0.517, 0.719] 1.127 [0.445, 2.451] 0.569 [0.363, 3.674] 0.416 [0.385, 0.498] 0.298 [0.293, 0.299] BootRollout–PPO 0.598 [0.535, 0.729] 1.112 [0.502, 2.295] 0.647 [0.331, 3.798] 0.414 [0.367, 0.478] 0.2995 [0.2951, 0.2997] SCR–PPO–Full 0.782 [0.628, 0.948] 0.699 [0.519, 1.104] 0.226 [0.199, 0.297] 0.238 [0.187, 0.304] 0.0049 [0.0044, 0.0074] Low-Vol 1/N 0.086 [0.073, 0.232] 0.001 [-0.006, 0.075] 0.171 [0.158, 0.180] 0.340 [0.321, 0.368] 0.0000 [0.0000, 0.0000] Markowitz -0.084 [-0.155, -0.027] -0.058 [-0.093, -0.039] 0.137 [0.123, 0.147] 0.333 [0.300, 0.344] 0.0046 [0.0018, 0.0146] Inverse-Vol 0.105 [0.041, 0.155] 0.014 [-0.015, 0.042] 0.165 [0.143, 0.172] 0.321 [0.276, 0.326] 0.0143 [0.0133, 0.0152] GMV (Ledoit-Wolf) -0.088 [-0.131, -0.055] -0.057 [-0.071, -0.038] 0.112 [0.102, 0.117] 0.278 [0.247, 0.295] 0.0274 [0.0215, 0.0292] PPO (Historical Replay) 0.079 [0.029, 0.181] -0.004 [-0.031, 0.046] 0.174 [0.157, 0.183] 0.337 [0.317, 0.369] 0.164 [0.156, 0.294] BootRollout–PPO 0.090 [0.054, 0.184] 0.000 [-0.018, 0.050] 0.176 [0.159, 0.184] 0.343 [0.302, 0.372] 0.2995 [0.2993, 0.2998] SCR–PPO–Full 0.641 [0.602, 0.707] 0.453 [0.400 0.569] 0.116 [0.109, 0.121] 0.164 [0.150, 0.186] 0.0049 [0.0046, 0.0052] General 1/N 0.444 [0.376, 0.468] 0.182 [0.148, 0.197] 0.231 [0.229, 0.235] 0.423 [0.414, 0.433] 0.0000 [0.0000, 0.0000] Markowitz 0.434 [0.333, 0.554] 0.167 [0.122, 0.305] 0.223 [0.189, 0.267] 0.387 [0.340, 0.427] 0.0044 [0.0018, 0.0095] Inverse-Vol 0.472 [0.440, 0.516] 0.221 [0.200, 0.246] 0.238 [0.222, 0.239] 0.395 [0.370, 0.402] 0.0169 [0.0159, 0.0171] GMV (Ledoit-Wolf) 0.200 [0.082, 0.397] 0.063 [0.008, 0.280] 0.148 [0.134, 0.222] 0.329 [0.260, 0.377] 0.0282 [0.0263, 0.0314] PPO (Historical Replay) 0.435 [0.341, 0.525] 0.188 [0.136, 0.270] 0.238 [0.231, 0.242] 0.409 [0.399, 0.416] 0.194 [0.158, 0.282] BootRollout–PPO 0.419 [0.360, 0.517] 0.188 [0.137, 0.239] 0.232 [0.226, 0.235] 0.400 [0.388 0.420] 0.2996 [0.2995, 0.2997] SCR–PPO–Full 0.900 [0.768, 0.992] 0.972 [0.855, 1.350] 0.171 [0.162, 0.178] 0.167 [0.142, 0.176] 0.0059 [0.0054, 0.0062] Market-Proxy 1/N 0.508 [0.508, 0.508] 0.250 [0.250, 0.250] 0.225 [0.225, 0.225] 0.371 [0.371, 0.371] 0.0000 [0.0000, 0.0000] Markowitz 0.381 [0.381, 0.381] 0.165 [0.165, 0.165] 0.210 [0.210, 0.210] 0.362 [0.362, 0.362] 0.0056 [0.0056, 0.0056] Inverse-Vol 0.494 [0.494, 0.494] 0.241 [0.241, 0.241] 0.217 [0.217, 0.217] 0.361 [0.361, 0.361] 0.0091 [0.0091, 0.0091] GMV (Ledoit-Wolf) 0.569 [0.569, 0.569] 0.303 [0.303, 0.303] 0.208 [0.208, 0.208] 0.335 [0.335, 0.335] 0.0468 [0.0468, 0.0468] PPO (Historical Replay) 0.492 [0.479, 0.523] 0.241 [0.234, 0.268] 0.228 [0.228, 0.228] 0.370 [0.367, 0.372] 0.220 [0.210, 0.267] BootRollout–PPO 0.496 [0.481, 0.509] 0.245 [0.238, 0.255] 0.227 [0.226, 0.228] 0.365 [0.364, 0.371] 0.2926 [0.2893, 0.2947] SCR–PPO–Full 1.004 [1.001, 1.014] 0.971 [0.928, 0.991] 0.175 [0.173, 0.177] 0.179 [0.176, 0.193] 0.0053 [0.0052, 0.0054] Evaluation metrics. In line with recent related work in portfolio rebalancing and risk assessment [39, 40, 6], we evaluate out-of-sample performance using a set of comprehensive metrics. The assessment of risk-adjusted performance employs the Sharpe ratio and the Calmar ratio, which respectively reflect the average excess return per unit of volatility and drawdown-aware performance. To assess the total risk exposure, we measure annualized volatility (AnnVol), while tail risk is analyzed via maximum drawdown (MaxDD), defined as the largest peak-to-trough loss. Additionally, we report turnover as a measure of trading intensity and implementability, computed as the average ℓ1 _1 change in portfolio weights. These metrics, taken together, show how returns and risk behave when the market environment changes and the distribution shifts. Baselines. We compare against both classical portfolio baselines and RL-based solutions. All reported metrics are computed from realized net returns on the held-out out-of-sample test window. • Classic portfolio algorithms, including (i) equal-weight 1/N1/N [7, 13], (i) mean–variance [25, 19], (i) inverse-volatility weighting [5, 35], and (iv) global minimum-variance (GMV) with Ledoit–Wolf shrinkage [20, 21]. • PPO (Historical Replay): PPO trained only on realized market transitions, using PPO-Clip [34] with generalized advantage estimation [33]. • BootRollout–PPO (with historical simulation): Inspired by historical simulation [11], we bootstrap S joint return vectors from a trailing window of realized returns (preserving cross-asset co-movements) and score portfolios with a downside-risk–penalized objective (entropic left-tail) [8]; PPO otherwise trains on the same realized tape transitions with standard bootstrapping. • SCR–PPO–RewardOnly: A variant of our solution in which SCR macro scenarios for reward only, with standard tape bootstrapping and without regularization. • SCR–PPO–NoCF: A variant of our solution in which SCR macro scenarios for reward with regularization, but bootstrapping uses realized tape continuation only. • SCR–-PPO–Full: SCR macro scenarios for reward with regularization stack and augmented critic bootstrap target (βcf>0 _cf>0). Shock Discovery. SCR conditions applied to the channel-activation vector t χ_t confirms the conditioning signal is non-degenerate and exhibits reasonable behaviour in out-of-sample. On Market-Proxy, the detector successfully re-identifies recurring channels in validation and test, producing similar t χ_t patterns. Notably, it identifies March 2020 as out-of-support when the lookback window activates channels that were not encountered during training. In instances of novelty detection, SCR resorts to acceptable nearest-neighbor resampling from the available historical data, thereby avoiding retrieval from unrelated episodes. Table 2 presents representative entries from ShockLedger, verifying that regime summaries via t χ_t are stable and capable of identifying novel instances outside the existing support. Building on this qualitative verification, we turn to quantitative evaluation to test the effect of regime-local scenario conditioning, including generalization and value learning. Table 2: Illustrative ShockLedger channels used in experiments. Episode Type Macro Signature Top Moves Activity Period Recurring Geopolitical/Oil Shock GPR↑,Brent↑,Energy↑GPR ,\ Brent ,\ Energy ++(XLP, XLV); −-(XLF, XLB) 2009–2023 (38 days; 14 in train) Recurring Policy Uncertainty GPR↓,EPU↑,WTI↓GPR ,\ EPU ,\ WTI ++(XLK, XLV); −-(XLE, XLF) 2009–2021 (18 days; 16 in train) Novel (Anomaly) COVID-19 Liquidity Crisis No training-set channel match Flagged by novelty filter Detected as novel episode March 2020 (Consecutive days) 4.2 Evaluation on Out-of-Sample Regimes Table 1 presents the evaluation metrics for High-Vol, Low-Vol, General, and Market-Proxy groups. SCR–PPO–Full enhances risk-adjusted performance and tail behavior out-of-sample. It stays reliable across different regimes and cross-sectional variations. Table 3: Ablation study of our solution. We report median values across multiple runs. Variant Sharpe Calmar AnnVol MaxDD Turnover Gapfinal Resid-AUC PPO (Historical Replay) 0.457 [0.256, 0.543] 0.222 [0.086, 0.335] 0.231 [0.188, 0.347] 0.381 [0.359, 0.414] 0.2320 [0.1600, 0.2983] 0.526 [0.304, 1.882] 0.217 [0.204, 0.231] SCR–PPO–RewardOnly 0.506 [-0.054, 0.650] 0.364 [0.223, 0.584] 0.127 [0.107, 0.205] 0.181 [0.148, 0.330] 0.0050 [0.0049, 0.0051] 0.0003 [0.0001, 0.0016] 0.102 [0.054, 0.115] SCR–PPO–NoCF 0.647 [0.571, 0.810] 0.561 [0.379, 0.833] 0.146 [0.110, 0.217] 0.184 [0.158, 0.198] 0.0045 [0.0042, 0.0049] 0.0001 [0.0000, 0.0012] 0.087 [0.077, 0.098] SCR–PPO–Full 0.805 [0.635, 0.970] 0.737 [0.494, 0.984] 0.171 [0.122, 0.196] 0.179 [0.162, 0.194] 0.0046 [0.0044, 0.0049] 0.0001 [0.0000, 0.0003] 0.075 [0.061, 0.084] Figure 2: We stress test under shocks. The wealths are rebased to 1 at the window start. Top: COVID-19 sell-off window. Bottom: 2021–2022 macro shock window. In comparison to PPO (Historical Replay), Sharpe shows consistent increase: 0.615→0.7820.615→ 0.782 (High-Vol), 0.079→0.6410.079→ 0.641 (Low-Vol), 0.435→0.9000.435→ 0.900 (General), and 0.492→1.0040.492→ 1.004 (Market-Proxy). These enhancements are accompanied by reduced drawdowns (e.g., High-Vol MaxDD 0.416→0.2380.416→ 0.238, Low-Vol 0.337→0.1640.337→ 0.164 General 0.409→0.1670.409→ 0.167, Market-Proxy 0.370→0.1790.370→ 0.179), along with negligible turnover (≈0.004≈ 0.004–0.0060.006) in contrast to PPO’s high turnover (≈0.15≈ 0.15–0.300.30), suggesting reduced churn without sacrificing downside control. Qualitative stress-window evidence. On Market-Proxy, Fig. 2 provides a stringent stress test: during two major crisis windows, SCR–PPO–Full protects capital and keeps drawdowns smaller. It also stays more stable throughout the downturn. When markets rebound, it compounds faster than both RL and classical rebalancing baselines. Overall, it achieves higher cumulative returns under sharp regime shifts. 4.3 Ablations and Scenario-to-Real Analysis To ensure that the scenario–real mismatch and critic stability are observable, we report seed-averaged diagnostics on Market-Proxy. Here, beyond the aggregate metrics in Sec. 4.2, we track diagnostics that correspond to the operator and mixing analysis in Sec. 3. We gauge scenario–real mismatch on held-out dates via scenario-to-real gap Gapfinal=|J^scen−J^real|,Gap_final\;=\; | J_scen- J_real |, where J^scen J_scen is the cumulative average daily return over the test dates under SCR scenario scoring and J^real J_real is the corresponding cumulative average daily return realized on the tape. We summarize critic stability by Residual-AUC, the area under the Bellman-residual curve residℓ2resid_ _2 versus training progress. Table 3 demonstrates the results. Relative to PPO trained on Historical Replay, introducing scenario-conditioned training via SCR–PPO–RewardOnly improves pooled out-of-sample robustness: Sharpe increases (0.457→0.5060.457\!→\!0.506), drawdowns improve (MaxDD 0.381→0.1680.381\!→\!0.168), and turnover drops sharply (0.232→0.0050.232\!→\!0.005). SCR also reduces mismatch (Gapfinal 0.526→3×10−40.526\!→\!3× 10^-4) and lowers critic error (Resid-AUC 0.217→0.1020.217\!→\!0.102), consistent with higher-support scenario conditioning stabilizing value learning on the logged tape. Figure 3: Scenario-to-real validation. Cumulative average daily return under SCR scenario scoring versus realized tape returns. Figure 4: Critic stability under logged-tape mismatch. Bellman residual residℓ2resid_ _2 versus training progress (mean ± 95% CI). Compared to PPO (Historical Replay), SCR–PPO–Full attains smaller residuals, and improved critic stability. Our controlled comparison SCR–PPO–NoCF → SCR–PPO–Full holds the scenario mechanism and regularization stack fixed and changes only the critic bootstrap target, isolating continuation mismatch (Lemma 3.1). Enabling counterfactual continuation mixing improves pooled risk-adjusted performance (Sharpe 0.647→0.8050.647\!→\!0.805, Calmar 0.561→0.7370.561\!→\!0.737) and reduces critic error (Resid-AUC 0.087→0.0750.087\!→\!0.075). On Market-Proxy, we report mean ± 95% confidence intervals for J^scen J_scen and J^real J_real (Fig. 3), and evaluate critic stability via TD-residual contraction (Fig. 4), as motivated by Theorem 3.5. These seed-averaged plots make the mismatch and critic contraction predicted by our theory directly observable (Figs. 3–4). Table 4: Sensitivity to βcf _cf. We report median Sharpe here. Setting High-Vol Low-Vol General Market-Proxy βcf=0.0 _cf=0.0 0.736 0.598 0.640 0.821 βcf=0.25 _cf=0.25 0.529 0.529 0.716 0.823 βcf=0.5 _cf=0.5 0.782 0.641 0.900 1.004 βcf=0.75 _cf=0.75 0.518 0.509 0.671 0.838 βcf=1.0 _cf=1.0 0.563 0.584 0.645 0.842 4.4 Sensitivity Analysis Counterfactual mixing βcf _cf. We show in Table 4 that using a moderate coefficient results in consistent performance across different universes. In contrast, setting the coefficient in both extremes, i.e., βcf=0 _cf=0 uses only continuation mismatch, while βcf=1 _cf=1 relies entirely on counterfactual continuation and can amplify approximation error, resulting in degraded performance. This observation aligns with the bias–variance trade-off in Theorem 3.5. 5 Conclusion To improve the return and robustness of reinforcement learning-based portfolio rebalancing, we propose Scenario-Context Rollout (SCR), which is a macroeconomic-guided feedback mechanism to produce scenario returns. To effectively train the RL agent using historical tapes, we analyzed how combining scenario-based rewards with tape-realized transitions induces a continuation mismatch. To address this issue, we augment the critic target using counterfactual continuation; doing so provides a principled bias–variance trade-off and stabilizes critic updates. Our experiments show that the proposed solution improves out-of-sample Sharpe ratios and reduces drawdowns compared with strong baselines. Meanwhile, our solution remains effective across market regimes. References [1] A. Ang and J. Chen (2002) Asymmetric correlations of equity portfolios. Journal of Financial Economics 63 (3), p. 443–494. Cited by: §1. [2] A. Ang, R. J. Hodrick, Y. Xing, and X. Zhang (2006) The cross-section of volatility and expected returns. The Journal of Finance 61 (1), p. 259–299. External Links: Document Cited by: §4.1. [3] D. P. Bertsekas and J. N. Tsitsiklis (1996) Neuro-dynamic programming. Athena Scientific, Belmont, MA. External Links: ISBN 978-1-886529-10-6 Cited by: §2, §3. [4] R. A. J. Campbell, K. G. Koedijk, and P. Kofman (2002) Increased correlation in bear markets. Financial Analysts Journal 58 (1), p. 87–97. External Links: Document Cited by: §1. [5] D. B. Chaves, J. Hsu, F. Li, and O. Shakernia (2011) Risk parity portfolio vs. other asset allocation heuristic portfolios. The Journal of Investing 20 (1), p. 108–118. External Links: Document Cited by: 1st item. [6] H. Choudhary, A. Orra, and M. Thakur (2025) FinXplore: an adaptive deep reinforcement learning framework for balancing and discovering investment opportunities. Note: arXiv:2509.10531 External Links: 2509.10531, Document Cited by: §4.1. [7] V. DeMiguel, L. Garlappi, and R. Uppal (2009) Optimal versus naive diversification: how inefficient is the 1/n portfolio strategy?. The Review of Financial Studies 22 (5), p. 1915–1953. External Links: Document Cited by: 1st item. [8] Y. Fei, Z. Yang, Y. Chen, and Z. Wang (2021) Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: 3rd item. [9] N. Ferns, P. Panangaden, and D. Precup (2004) Metrics for finite markov decision processes. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (UAI), p. 162–169. Cited by: §1. [10] T. Fischer and C. Krauss (2018) Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research 270 (2), p. 654–669. External Links: Document Cited by: §1. [11] C. Francq and J. Zakoïan (2020) Virtual historical simulation for estimating the conditional var of large portfolios. Journal of Econometrics 217 (2), p. 356–380. External Links: Document Cited by: 3rd item. [12] S. Fujimoto, D. Meger, and D. Precup (2019) Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2052–2062. Cited by: §1. [13] M. Gelmini and P. Uberti (2024) The equally weighted portfolio still remains a challenging benchmark. International Economics 179, p. 100525. External Links: Document Cited by: 1st item. [14] P. Glasserman (2004) Monte carlo methods in financial engineering. Applications of Mathematics, Springer, New York, NY. Cited by: §2.2. [15] Z. Jiang, D. Xu, and J. Liang (2017) A deep reinforcement learning framework for the financial portfolio management problem. External Links: 1706.10059, Link Cited by: §1. [16] L. V. Kantorovich and G. S. Rubinstein (1958) On a space of completely additive functions. Vestnik Leningradskogo Universiteta 13 (7), p. 52–59. Cited by: §A.2. [17] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine (2019) Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, Cited by: §1. [18] A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §1, §1. [19] Z. Lai and H. Yang (2023) A survey on gaps between mean-variance approach and exponential growth rate approach for portfolio optimization. ACM Computing Surveys 55 (2), p. 25:1–25:36. External Links: Document Cited by: 1st item. [20] O. Ledoit and M. Wolf (2004) Honey, i shrunk the sample covariance matrix. The Journal of Portfolio Management 30 (4), p. 110–119. External Links: Document Cited by: 1st item. [21] O. Ledoit and M. Wolf (2020) Analytical nonlinear shrinkage of large-dimensional covariance matrices. The Annals of Statistics 48 (5), p. 3043–3065. External Links: Document Cited by: 1st item. [22] S. Levine, A. Kumar, G. Tucker, and J. Fu (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §1, §1. [23] F. Longin and B. Solnik (2001) Extreme correlation of international equity markets. The Journal of Finance 56 (2), p. 649–676. External Links: Document Cited by: §1. [24] P. C. Mahalanobis (2018) Reprint of: mahalanobis, p.c. (1936) “on the generalised distance in statistics.”. Sankhya A 80 (Suppl 1), p. 1–7. External Links: Document Cited by: §2.1. [25] H. Markowitz (1952) Portfolio selection. The Journal of Finance 7 (1), p. 77–91. External Links: Document Cited by: 1st item. [26] J. E. Moody and M. Saffell (2001) Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12 (4), p. 875–889. External Links: Document Cited by: §1. [27] R. Munos (2005) Error bounds for approximate value iteration. In Proceedings of the 20th National Conference on Artificial Intelligence (AAAI-05), Vol. 2, p. 1006–1011. Cited by: §3. [28] A. V. Olivares-Nadal and V. DeMiguel (2018) Technical note—a robust perspective on transaction costs in portfolio optimization. Operations Research 66 (3), p. 733–739. External Links: Document Cited by: §2. [29] Open-Finance-Lab (2025) Task 1: finrl-deepseek for stock trading (finrl contest 2025 documentation). Note: https://huggingface.co/datasets/Zihan1004/FNSPIDRetrieved July 12, 2025 Cited by: §4.1. [30] G. Peyré and M. Cuturi (2019) Computational optimal transport. Foundations and Trends® in Machine Learning 11 (5–6), p. 355–607. External Links: Document Cited by: Proposition 3.2. [31] S. Poledna, M. G. Miess, C. Hommes, and K. Rabitsch (2023) Economic forecasting with an agent-based model. European Economic Review 151, p. 104306. Cited by: §1. [32] M. L. Puterman (1994) Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons. External Links: ISBN 9780471619772 Cited by: §1, §3. [33] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel (2015) High-dimensional continuous control using generalized advantage estimation. External Links: 1506.02438 Cited by: 2nd item. [34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §1, §2, 2nd item. [35] H. Shimizu and T. Shiohama (2020) Constructing inverse factor volatility portfolios: a risk-based asset allocation for factor investing. International Review of Financial Analysis 68, p. 101438. External Links: Document Cited by: 1st item. [36] R. S. Sutton and A. G. Barto (2018) Reinforcement learning: an introduction. 2 edition, MIT Press, Cambridge, MA. Cited by: §1, §2, §3. [37] D. Vassallo (2022-01) Dynamic models for financial and sentiment time series. Ph.D. Thesis, Scuola Normale Superiore. External Links: Link Cited by: §2. [38] C. Villani (2008) Optimal transport: old and new. Grundlehren der mathematischen Wissenschaften, Vol. 338, Springer. Cited by: §1. [39] J. Wang, M. Hai, and H. Li (2024) A deep reinforcement learning model for portfolio management based on weight adjustment. Procedia Computer Science 242, p. 356–363. External Links: Document Cited by: §4.1. [40] X. Wang and L. Liu (2025) Risk-sensitive deep reinforcement learning for portfolio optimization. Journal of Risk and Financial Management 18 (7), p. 347. External Links: Document Cited by: §4.1. [41] F. Westerhoff and R. Franke (2018) Agent-based models for economic policy design. In The Oxford Handbook of Computational Economics and Finance, S. Chen, M. Kaboudan, and Y. Du (Eds.), p. 603–624. Cited by: §1. [42] H. Yu, A. R. Mahmood, and R. S. Sutton (2018) On generalized bellman equations and temporal-difference learning. Journal of Machine Learning Research 19 (47), p. 1–50. External Links: Link Cited by: §1. [43] Z. Zhu, W. Ding, Y. Jin, and D. Shen (2023) Dissecting the idiosyncratic volatility puzzle: a fundamental analysis approach. Research in International Business and Finance 66, p. 102085. External Links: Document, Link Cited by: §4.1. Appendix A Additional Proofs Definition A.1 (Induced continuation distributions). Fix a decision-time state ϕφ and action a with executed control =Π(a)w= (a). Define the realized continuation distribution as Kreal(B∣ϕ,a) K_real(B φ,a) :=∫EB(ϕ,,)Preal(d∣ϕ), = 1\E_B(φ,w,r)\\,P_real(dr φ), (23) EB(ϕ,,) E_B(φ,w,r) :=(Upd(ϕ,,)∈B). = (Upd(φ,w,r)∈ B ). and the scenario-consistent continuation distribution by Kscen(B∣ϕ,ψ,a) K_scen(B φ,ψ,a) :=∫EB(ϕ,,)Pscen(d∣ψ), = 1\E_B(φ,w,R)\\,P_scen(dR ψ), (24) EB(ϕ,,) E_B(φ,w,R) :=(Upd(ϕ,,)∈B). = (Upd(φ,w,R)∈ B ). for any measurable set B in the state space. A.1 Proof of Lemma 3.1 Proof. Fix ϕφ and a policy π. By definition, (ThybπV)(ϕ)=[r(ϕ,;)+δV(Upd(ϕ,,))|ϕ],(T_hyb^πV)(φ)=E\! [r(φ,w;R)+δ\,V(Upd(φ,w,r))\ |\ φ ], where a∼π(⋅∣ϕ)a π(· φ), =Π(a)w= (a), ∼Pscen(⋅∣ψ)R P_scen(· ψ) and ∼Preal(⋅∣ϕ)r P_real(· φ). Similarly, (TscenπV)(ϕ)=[r(ϕ,;)+δV(Upd(ϕ,,))|ϕ].(T_scen^πV)(φ)=E\! [r(φ,w;R)+δ\,V(Upd(φ,w,R))\ |\ φ ]. Subtracting, the reward terms cancel since both operators evaluate r(ϕ,;⋅)r(φ,w;·) on the same scenario draw R: ΔπV(ϕ):=(ThybπV−TscenπV)(ϕ),ϕ+(ϕ,a,x):=Upd(ϕ,Π(a),x). gathered ^πV(φ):=(T_hyb^πV-T_scen^πV)(φ),\\ φ^+(φ,a,x):=Upd(φ, (a),x). gathered ΔπV(ϕ) ^πV(φ) =δa∼π(⋅∣ϕ)[ΔV(ϕ,ψ,a)], =δ\,E_a π(· φ)\! [ _V(φ,ψ,a) ], (25) ΔV(ϕ,ψ,a) _V(φ,ψ,a) :=V(ϕ+(ϕ,a,))−V(ϕ+(ϕ,a,)). =E_r\,V\! (φ^+(φ,a,r) )-E_R\,V\! (φ^+(φ,a,R) ). Using the induced continuation distributions Kreal(⋅∣ϕ,a)K_real(· φ,a) and Kscen(⋅∣ϕ,ψ,a)K_scen(· φ,ψ,a) from Definition A.1, rewrite V(Upd(ϕ,Π(a),))=∫V(ϕ′)Kreal(dϕ′∣ϕ,a).E_r\,V\! (Upd(φ, (a),r) )= V(φ )\,K_real(dφ φ,a). (26) V(Upd(ϕ,Π(a),)) _R\,V\! (Upd(φ, (a),R) ) (27) =∫V(ϕ′)Kscen(dϕ′∣ϕ,ψ,a). = V(φ )\,K_scen\! (dφ φ,ψ,a ). (28) Therefore πϕ _φ :=π(⋅∣ϕ), =π(· φ), ΔKϕ,ψ,a(dϕ′) K_φ,ψ,a(dφ ) :=Kreal(dϕ′∣ϕ,a)−Kscen(dϕ′∣ϕ,ψ,a). =K_real(dφ φ,a)-K_scen(dφ φ,ψ,a). ⟨V,ΔKϕ,ψ,a⟩ V, K_φ,ψ,a :=∫V(ϕ′)ΔKϕ,ψ,a(dϕ′). = V(φ )\, K_φ,ψ,a(dφ ). where ΔK:=Kreal−Kscen K:=K_real-K_scen. If Preal(⋅∣ϕ)=Pscen(⋅∣ψ)P_real(· φ)=P_scen(· ψ), then Kreal=KscenK_real=K_scen and the difference is zero. If Upd(ϕ,,⋅)Upd(φ,w,·) is outcome-independent, then both induced kernels coincide as well. ∎ A.2 Proof of Proposition 3.2 Proof. Fix ϕφ. Fix an o such that Φ(o)=ϕ (o)=φ, and set ψ:=Ψ(o)ψ:= (o). By Lemma 3.1, mreal(ϕ,a) m_real(φ,a) :=∼Preal(⋅∣ϕ)fϕ,a(), =E_r P_real(· φ)f_φ,a(r), mscen(ϕ,ψ,a) m_scen(φ,ψ,a) :=∼Pscen(⋅∣ψ)fϕ,a(). =E_R P_scen(· ψ)f_φ,a(R). Let ΔπV(ϕ):=(ThybπV−TscenπV)(ϕ). ^πV(φ):=(T_hyb^πV-T_scen^πV)(φ). Then ΔπV(ϕ)=δa∼π(⋅∣ϕ)[mreal(ϕ,a)−mscen(ϕ,ψ,a)] ^πV(φ)=δ\,E_a π(· φ)\! [m_real(φ,a)-m_scen(φ,ψ,a) ] where fϕ,a():=V(Upd(ϕ,Π(a),))f_φ,a(x):=V(Upd(φ, (a),x)). By assumption, ↦Upd(ϕ,Π(a),)x (φ, (a),x) is LhL_h-Lipschitz through the outcome-updated memory component and V is LVL_V-Lipschitz in that component, hence fϕ,af_φ,a is (LVLh)(L_VL_h)-Lipschitz under ∥⋅∥2\|·\|_2. By Kantorovich–Rubinstein duality for W1W_1 (with cost d(,)=‖−‖2d(x,y)=\|x-y\|_2) [16], Prealϕ:=Preal(⋅∣ϕ),Pscenψ:=Pscen(⋅∣ψ).P_real^φ:=P_real(· φ), P_scen^ψ:=P_scen(· ψ). |fϕ,a()−fϕ,a()| |E_rf_φ,a(r)-E_Rf_φ,a(R) | ≤(LVLh)W1(Prealϕ,Pscenψ). ≤(L_VL_h)\,W_1\! (P_real^φ,\,P_scen^ψ ). Taking expectation over a∼π(⋅∣ϕ)a π(· φ), multiplying by δ, and then taking the supremum over o yields ΔW:=supoW1(Preal(⋅∣Φ(o)),Pscen(⋅∣Ψ(o))). _W:= _o\ W_1\! (P_real(· (o)),\;P_scen(· (o)) ). ‖ThybπV−TscenπV‖∞≤δLVLhΔW.\|T_hyb^πV-T_scen^πV\|_∞≤δ L_VL_h\, _W. ∎ A.3 Proof of Corollary 3.3 Proof. Let VhybπV_hyb^π and VscenπV_scen^π be the fixed points of ThybπT_hyb^π and TscenπT_scen^π. Since both operators are δ-contractions in ∥⋅∥∞\|·\|_∞, ‖Vhybπ−Vscenπ‖∞ \|V_hyb^π-V_scen^π\|_∞ =‖ThybπVhybπ−TscenπVscenπ‖∞ = \|T_hyb^πV_hyb^π-T_scen^πV_scen^π \|_∞ ≤‖ThybπVhybπ−ThybπVscenπ‖∞ ≤ \|T_hyb^πV_hyb^π-T_hyb^πV_scen^π \|_∞ +‖(Thybπ−Tscenπ)Vscenπ‖∞. + \|(T_hyb^π-T_scen^π)V_scen^π \|_∞. The first term is at most δ‖Vhybπ−Vscenπ‖∞δ\|V_hyb^π-V_scen^π\|_∞ by contraction. Rearranging gives ‖Vhybπ−Vscenπ‖∞≤11−δ‖(Thybπ−Tscenπ)Vscenπ‖∞.\|V_hyb^π-V_scen^π\|_∞≤ 11-δ\|(T_hyb^π-T_scen^π)V_scen^π\|_∞. Apply Proposition 3.2 with V=VscenπV=V_scen^π to obtain (20). ∎ A.4 Proof of Theorem 3.5 Proof. Fix (ϕ,ψ,a)(φ,ψ,a) and write =Π(a)w= (a). Define f():=Vscenπ(Upd(ϕ,,)).f(x):=V_scen^π\! (Upd(φ,w,x) ). By Assumption 3.4, ↦Upd(ϕ,,)x (φ,w,x) is LhL_h-Lipschitz through the outcome-updated memory component and VscenπV_scen^π is LVL_V-Lipschitz in that component, hence f is (LVLh)(L_VL_h)-Lipschitz under ∥⋅∥2\|·\|_2. Let ∼Pscen(⋅∣ψ)R P_scen(· ψ), ∼Preal(⋅∣ϕ)r P_real(· φ), and μψ=[∣ψ] _ψ=E[R ψ]. Since both Y⋆Y and YβcfY_ _cf use the same reward term r(ϕ,;)r(φ,w;R), the deviation enters only via continuation: Yr−Y⋆ Y_r-Y =δ(f()−f()), =δ (f(r)-f(R) ), Yc−Y⋆ Y_c-Y =δ(f(μψ)−f()). =δ (f( _ψ)-f(R) ). Let β=βcfβ= _cf and α=1−βα=1-β. Then Yβ−Y⋆=α(Yr−Y⋆)+β(Yc−Y⋆).Y_β-Y =α(Y_r-Y )+β(Y_c-Y ). Using (u+v)2≤2u2+2v2(u+v)^2≤ 2u^2+2v^2 yields (Yβ−Y⋆)2≤2α2(Yr−Y⋆)2+2β2(Yc−Y⋆)2.(Y_β-Y )^2≤ 2α^2(Y_r-Y )^2+2β^2(Y_c-Y )^2. Taking conditional expectations, [(Yβ−Y⋆)2∣ϕ,ψ,a] \! [(Y_β-Y )^2 φ,ψ,a ] ≤ 2α2[(Yr−Y⋆)2∣ϕ,ψ,a] ≤ 2α^2\,E\! [(Y_r-Y )^2 φ,ψ,a ] (29) + 2β2[(Yc−Y⋆)2∣ϕ,ψ,a]. +2β^2\,E\! [(Y_c-Y )^2 φ,ψ,a ]. For the realized-continuation term, let γ⋆γ be an optimal coupling of Preal(⋅∣ϕ)P_real(· φ) and Pscen(⋅∣ψ)P_scen(· ψ), so that Δ2(ϕ,ψ)=W22(Preal(⋅∣ϕ),Pscen(⋅∣ψ))=γ⋆[∥−∥22]. _2(φ,ψ)=W_2^2\! (P_real(· φ),P_scen(· ψ) )=E_γ [\|r-R\|_2^2 ]. By Lipschitzness of f, |f()−f()|≤(LVLh)‖−‖2,|f(r)-f(R)|≤(L_VL_h)\|r-R\|_2, hence [(Yr−Y⋆)2∣ϕ,ψ,a] \! [(Y_r-Y )^2 φ,ψ,a ] =δ2[(f()−f())2∣ϕ,ψ,a] =δ^2\,E\! [(f(r)-f(R))^2 φ,ψ,a ] ≤δ2(LVLh)2Δ2(ϕ,ψ). ≤δ^2(L_VL_h)^2\, _2(φ,ψ). For the proxy-continuation term, by Lipschitzness, |f(μψ)−f()|≤(LVLh)‖μψ−‖2,|f( _ψ)-f(R)|≤(L_VL_h)\| _ψ-R\|_2, so [(Yc−Y⋆)2∣ϕ,ψ,a] \! [(Y_c-Y )^2 φ,ψ,a ] =δ2[(f(μψ)−f())2∣ϕ,ψ,a] =δ^2\,E\! [(f( _ψ)-f(R))^2 φ,ψ,a ] ≤δ2(LVLh)2[‖−μψ‖22∣ψ] ≤δ^2(L_VL_h)^2\,E\! [\|R- _ψ\|_2^2 ψ ] =δ2(LVLh)2σscen2(ψ). =δ^2(L_VL_h)^2\,σ^2_scen(ψ). Substitute into (29) to obtain (21). ∎ A.5 Proof of Corollary 3.6 Proof. We derive the optimal mixing weight by minimizing g(β):=(1−β)2A(ϕ,ψ)+β2B(ψ),β∈[0,1].g(β):=(1-β)^2A(φ,ψ)+β^2B(ψ), β∈[0,1]. Upon expansion, we obtain g(β)=A(ϕ,ψ)−2A(ϕ,ψ)β+(A(ϕ,ψ)+B(ψ))β2g(β)=A(φ,ψ)-2A(φ,ψ)β+(A(φ,ψ)+B(ψ))β^2. Next, we differentiate this expression and set the derivative equal to zero: g′(β) g (β) =−2A(ϕ,ψ)+2(A(ϕ,ψ)+B(ψ))β=0, =-2A(φ,ψ)+2 (A(φ,ψ)+B(ψ) )β=0, ⇒β⋆ β =A(ϕ,ψ)A(ϕ,ψ)+B(ψ). = A(φ,ψ)A(φ,ψ)+B(ψ). Given that both A(ϕ,ψ)A(φ,ψ) and B(ψ)B(ψ) are non-negative, it follows that β⋆∈[0,1]β ∈[0,1]. ∎