Paper deep dive
When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game
Siu Tung Wong, Carlo Campajola
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker--trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker's continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validation-selected PPO--FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO--FFNN and PPO--LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker's observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes \(2.22\%\) of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation.
Tags
Links
- Source: https://arxiv.org/abs/2610.03598v1
- Canonical: https://arxiv.org/abs/2610.03598v1
Trouble viewing inline? Open PDF directly →
Full Text
55,040 characters extracted from source content.
Expand or collapse full text
When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker–Trader Game Conference: ACM International Conference on AI in Finance; November 14–17, 2026; Milan, ItalyCCS: Computing methodologies Reinforcement learningCCS: Applied computing Economics Siu Tung Wong Affiliation: Institute of Finance and Technology, University College London, Gower Street, London, WC1E 6BT, United Kingdom email: ucesstw@ucl.ac.uk and Carlo Campajola Affiliation: Institute of Finance and Technology, University College London, Gower Street, London, WC1E 6BT, United Kingdom Affiliation: UZH Blockchain Center, Andreasstrasse 15, Zürich, 8050, Switzerland email: c.campajola@ucl.ac.uk © , 2026 Abstract. Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker–trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker’s continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validation-selected PPO–FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO–FFNN and PPO–LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker’s observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes 2.22%2.22\% of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation. Keywords: optimal execution, stochastic control, proximal policy optimization, partial observability, potential-based reward shaping, residual reinforcement learning, model misspecification, analytical policy initialization, analytical benchmarks 1. Introduction Optimal execution balances execution costs against inventory exposure and adverse price movements. Reinforcement learning (RL) is attractive when richer market dynamics make analytical strategies difficult to obtain. We ask whether RL can recover an analytically known broker strategy from its performance criterion and use that strategy after the environment changes. Our environment follows the continuous-time game of Cartea, Jaimungal, and Sánchez-Betancourt (Cartea et al., 2026), who model the interaction between a broker, an informed trader, and an uninformed trader. The broker internalises client orders and hedges in the lit market. Informed flow contains private information, while rapid trading is costly and slow trading leaves inventory exposed. Their analytical equilibrium gives an action at every state, allowing same-state action comparisons together with realised payoffs and inventory trajectories. Prior studies use RL to adjust an Almgren–Chriss schedule, learn liquidation policies against analytical benchmarks, or structure actor–critic learning with closed-form policy and value functions (Hendricks and Wilcox, 2014; Macrì and Lillo, 2024; Wang et al., 2026). They do not test whether a finite-step reward recovers the known state-dependent strategy in this broker–trader game. Analytical-policy initialisation and adaptation have also been proposed (Ning et al., 2021; Wang et al., 2026). We separate discretisation, policy learning, and hidden-state estimation. We study three research questions: • RQ1: Full-information control. Can PPO–FFNN and PPO–LSTM recover the broker’s analytical trading strategy when every relevant state variable is observed? • RQ2: Partial-information control. How closely can PPO–FFNN and PPO–LSTM approach the certainty-equivalent benchmark (Mania et al., 2019) when the informed trader’s inventory, signal, and current trading rate are hidden? • RQ3: Adapting an analytical policy. When the execution cost differs from the value used to calculate the analytical policy, can PPO learn an adjustment that improves the unchanged policy? Section 2 reviews related work, Section 3 introduces the environment, Section 4 answers RQ1–RQ2, and Section 5 answers RQ3. 2. Related Work Analytical execution and broker games: Analytical execution models relate trading actions to inventory, impact, order flow, and predictive signals (Cartea and Jaimungal, 2016; Casgrain and Jaimungal, 2019). Broker–trader games add strategic interaction because hedging affects prices while client orders change the broker’s inventory and information (Cartea et al., 2026; Wu and Jaimungal, 2024). We use the known broker action as the RL target. RL for optimal execution: RL has learned liquidation schedules through value-based methods, PPO, and continuous-time actor–critic (Nevmyvaka et al., 2006; Lin and Beling, 2020; Ning et al., 2021; Moallemi and Wang, 2022; Koulouris and Campajola, 2026). Hybrid methods adjust or initialise from an analytical schedule, or embed closed-form structure (Hendricks and Wilcox, 2014; Ning et al., 2021; Wang et al., 2026). The broker–trader equilibrium instead permits same-state action comparisons. Although policy gradients have convergence results for linear–quadratic control (Fazel et al., 2018), PPO remains sensitive to implementation and evaluation choices (Engstrom et al., 2020; Andrychowicz et al., 2020; Henderson et al., 2018). Analytical guidance and reward shaping: Analytical equations can guide learning and provide a benchmark. Prior work adds a Hamilton–Jacobi–Bellman loss or closed-form policy and value structure (Mukherjee and Liu, 2023; Wang et al., 2026). Residual RL adds a learned action to an existing controller (Johannink et al., 2019); in Section 5, we use it to adjust the frozen broker action after the execution cost changes. We also test potential-based shaping, which preserves policy ordering (Ng et al., 1999; Sowerby et al., 2022). RL under partial information: With hidden state, an agent can summarise observations recurrently or estimate the hidden variables explicitly (Igl et al., 2018; Lambrechts et al., 2022). We compare feedforward and recurrent PPO with a certainty-equivalent controller that updates such an estimate from observable history. 3. Environment Design 3.1. Broker Execution Game Our baseline is the continuous-time broker–trader model of Cartea, Jaimungal, and Sánchez-Betancourt (Cartea et al., 2026). An informed trader sends orders to the broker at rate ηt _t, uninformed clients generate flow ξt _t, and the broker hedges the resulting inventory in a lit market at rate νt _t. The informed trader responds to a predictive signal process α=(αt)t∈[0,T]α=( _t)_t∈[0,T]. PPO chooses the broker’s trading rate νt _t, while the informed trader follows its analytical feedback rule and the uninformed flow remains exogenous. Let QtBQ_t^B and QtIQ_t^I denote broker and informed-trader inventories, YtY_t the price-impact state, and StS_t the midprice. Their dynamics are dQtB _t^B =(νt−ηt−ξt)dt, =( _t- _t- _t)dt, dQtI _t^I =ηtdt, = _tdt, dYt _t =(hνt−pYt)dt, =(h _t-pY_t)dt, dαt _t =−κααtdt+σαdWtα, =- _α _tdt+ _αdW_t^α, (1) dξt _t =−κξξtdt+σξdWtξ, =- _ξ _tdt+ _ξdW_t^ξ, dSt _t =(αt+hνt−pYt)dt+σSdWtS. =( _t+h _t-pY_t)dt+ _SdW_t^S. The broker’s trade changes inventory, incurs execution cost, and affects the price at which its remaining inventory is valued. The analytical solution gives the feedback controls (2) νt⋆ _t =P11(t)QtB+P12(t)QtI+P13(t)Yt+g1(t)αt+g2(t)ξt, =P_11(t)Q^B_t+P_12(t)Q^I_t+P_13(t)Y_t+g_1(t) _t+g_2(t) _t, (3) ηt⋆ _t =P21(t)QtB+P22(t)QtI+P23(t)Yt+h1(t)αt+h2(t)ξt. =P_21(t)Q^B_t+P_22(t)Q^I_t+P_23(t)Y_t+h_1(t) _t+h_2(t) _t. The time-dependent coefficients Pij(t)P_ij(t), gi(t)g_i(t) and hi(t)h_i(t) follow from the Riccati equations in (Cartea et al., 2026). In the simulator, the informed trader applies Equation (3) at every state generated by the broker’s actions. Therefore even if the PPO agent takes suboptimal-actions, the informed trader continues to use the analytical feedback rule and does not operate in subgame nash equilibrium. We refer to it as the analytical-feedback trader. 1. Information before decision k Broker and market state Sk,QkB,Yk,ξk,τkS_k,Q_k^B,Y_k, _k, _k are observed; νk−1 _k-1 also enters the partial policy Informed trader and signal QkI,αkQ_k^I, _k determine ηk⋆ _k ; full-state only Uninformed clients current flow ξk _k is observed 2. Broker policy Observation vector full: okF=(S,QB,QI,Y,α,ξ,τ)ko_k^F=(S,Q^B,Q^I,Y,α,ξ,τ)_k partial: okP=(S,QB,Y,ξ,τ,νk−1)ko_k^P=(S,Q^B,Y,ξ,τ, _k-1)_k Broker policy PPO–FFNN / PPO–LSTM / analytical policy + PPO adjustment Reference νkref _k^ref evaluation only Executed νk _k clipped actor action 3. Execution and feedback Lit-market execution νk _k creates price impact hνkh _k; client flows update QBQ^B Environment transition produces Sk+1,Qk+1B,Yk+1S_k+1,Q_k+1^B,Y_k+1 and reward rkr_k updated state determines the next informed response ηk+1⋆ _k+1 full statenext decision; past ηk _k is now reconstructible Figure 1. Information and feedback for one broker decision. The policy maps the full- or partial-information observation to a trading rate, which is clipped to [−100,100][-100,100] before the environment updates the state and reward. Red variables are hidden under partial information; past informed flow ηk _k is reconstructible at decision k+1k+1. The reference action (νkref _k^ref) is evaluated at every PPO state.A three-column diagram shows the broker's observed and hidden information, the observation entering an actor, the actor action driving the environment, and the reference action \( _k^ref\) recalculated at the state reached by PPO. The environment uses Gymnasium 1.3.0 and PPO–FFNN is implemented with Stable-Baselines3 2.9.0 and PPO–LSTM with SB3-Contrib 2.9.0. The main parameters are T=1T=1, Δt=2×10−4 t=2× 10^-4, a=0.0012a=0.0012, b=c=h=0.001b=c=h=0.001, p=rB=rI=0p=r_B=r_I=0, φ=ψ=1 =ψ=1, κα=5 _α=5, σα=σS=1 _α= _S=1, and κξ=15 _ξ=15, with σξ∈0,100 _ξ∈\0,100\. Initially S0=100S_0=100 and all other states are zero. Here a is the broker’s lit-market execution-cost coefficient, while b and c determine the transaction prices for informed and uninformed client flow; h,ph,p are impact and resilience; rB,rIr_B,r_I are running inventory penalties; and φ,ψ ,ψ are terminal inventory penalties. Thus p=0p=0 gives zero resilience and rB=0r_B=0 removes the broker’s running, but not terminal, inventory penalty. Table 1. Networks and training settings. The reported zero-flow FFNN uses an initial log standard deviation of 3; the other FFNN and LSTM experiments use 0. Checkpoints are selected by mean payoff on 50 separate validation paths.A compact comparison of the actor and critic architectures, PPO settings, training procedures, and analytical prior used by the feedforward, recurrent, and analytical-policy-adjustment controllers. Setting PPO–FFNN PPO–LSTM Analytical policy with PPO adjustment Actor / critic Actor and critic each use two fully connected 64-unit ReLU hidden layers Actor and critic each use 24-unit LSTM and a fully connected 64-unit Tanh hidden layer Fixed analytical action; two-layer, width-64 Tanh correction network; two-layer, width-128 Tanh critic PPO Section 4 use Block B: ξ=0ξ=0, log std 33; σξ=100 _ξ=100, log std 00. Block C: supervised, ξ=0ξ=0. Block D: σξ=100 _ξ=100, log std 00 Blocks B and D: log std 00. Block C: supervised, ξ=0ξ=0 Not used in Section 4 Adam; rate 10−410^-4; γ=1γ=1; GAE λ=0.99λ=0.99; clip 0.20.2; batch 250; ten epochs; value/entropy coefficients 0.5/00.5/0; gradient norm 0.50.5; normalised advantages; state-dependent exploration sampled every four steps 100 critic-only warm-up and 800 joint updates; four environments × 1,024 steps; actor/critic rates ×10−5/×10−42\!×\!10^-5/2\!×\!10^-4; γ=1γ=1, advantage-estimation λ=0.95λ=0.95, clip 0.10.1, batch 1,024, ten epochs Training / selection 4×1064× 10^6 transitions; five training seeds; activation and checkpoints selected by finite-grid payoff on separate validation paths Five training seeds; checkpoints at updates 50, 100, 200, 400, and 800; the highest mean payoff on 50 common validation paths is selected; final evaluation on 100 previously unused paths Supervised diagnostic ReLU FFNN and Tanh LSTM with the same actor shapes as PPO; Adam at 10−310^-3, 100 epochs; 40/20/40 analytical train/validation/test episodes Not used Let XtBX_t^B be the broker’s cash. The broker maximises (4) JB(ν)=[XTB+QTBST−φ(QTB)2−rB∫0T(QtB)2t].J^B(ν)=E\! [X_T^B+Q_T^BS_T- (Q_T^B)^2-r_B _0^T(Q_t^B)^2dt ]. After substituting the cash and price dynamics, this equals an action-independent constant plus ∫0TgtBtE _0^Tg_t^Bdt, with gtB= g_t^B= −aνt2+bηt2+cξt2 -a _t^2+b _t^2+c _t^2 (5) +QtB[αt+hνt−pYt−2φ(νt−ηt−ξt)−rBQtB]. +Q_t^B\! [ _t+h _t-pY_t-2 ( _t- _t- _t)-r_BQ_t^B ]. Thus JBJ^B is the broker’s total expected payoff, while gtBg_t^B is its action-dependent payoff rate during trading. 3.2. Discrete PPO Environment We divide [0,T][0,T] into N=T/ΔtN=T/ t intervals and use left-endpoint Euler–Maruyama transitions (Higham, 2001). Define uk=νk−ηk−ξku_k= _k- _k- _k, μk=αk+hνk−pYk _k= _k+h _k-pY_k, ρα=1−καΔt _α=1- _α t, and ρξ=1−κξΔt _ξ=1- _ξ t. The finite-grid dynamics are Qk+1B Q_k+1^B =QkB+ukΔt, =Q_k^B+u_k t, Qk+1I Q_k+1^I =QkI+ηkΔt, =Q_k^I+ _k t, (6) Yk+1 Y_k+1 =Yk+(hνk−pYk)Δt, =Y_k+(h _k-pY_k) t, αk+1 _k+1 =ρααk+σαΔWkα = _α _k+ _α W_k^α ξk+1 _k+1 =ρξξk+σξΔWkξ, = _ξ _k+ _ξ W_k^ξ, Sk+1 S_k+1 =Sk+μkΔt+σSΔWkS,. =S_k+ _k t+ _S W_k^S,. The three ΔW W terms are independent Gaussian increments with variance Δt t. We define τk=(T−tk)/T _k=(T-t_k)/T as the fraction of the trading horizon remaining. Full-information PPO receives SkS_k, QkBQ_k^B, QkIQ_k^I, YkY_k, αk _k, ξk _k, and τk _k. Partial-information PPO receives SkS_k, QkBQ_k^B, YkY_k, ξk _k, τk _k, and νk−1 _k-1, while QkIQ_k^I and αk _k are hidden. Before entering either network, S is centred at 100, Y is multiplied by 1,000, and ξ and the previous action are divided by 100; we found these rescaling provides better results as they place the network inputs on comparable numerical scales; the dynamics and reported payoffs remain in the model’s original units. After the action has been executed, the broker observes Qk+1BQ_k+1^B and can reconstruct the informed flow from the inventory transition: (7) ηk=νk−ξk−Qk+1B−QkBΔt. _k= _k- _k- Q_k+1^B-Q_k^B t. The broker therefore cannot use ηk _k when choosing νk _k, but can use the reconstructed value when forming its estimates for decision k+1k+1. 3.3. Finite-step reward Let ΔMkS=σSΔWkS M_k^S= _S W_k^S. The transitions used in the reward calculation are ΔQkB Q_k^B =ukΔt,ΔSk=μkΔt+ΔMkS, =u_k t,\;\;\;\;\; S_k= _k t+ M_k^S, (8) ΔXkB X_k^B =(−Skuk−aνk2+bηk2+cξk2)Δt. = (-S_ku_k-a _k^2+b _k^2+c _k^2 ) t. Applying the exact discrete product rule gives Δ(QBS)k (Q^BS)_k =QkBΔSk+SkΔQkB+ΔQkBΔSk =Q_k^B S_k+S_k Q_k^B+ Q_k^B S_k (9) =(QkBμk+Skuk)Δt+ukμkΔt2+Qk+1BΔMkS, =(Q_k^B _k+S_ku_k) t+u_k _k t^2+Q_k+1^B M_k^S, while the terminal-inventory term satisfies (10) (Qk+1B)2−(QkB)2=2QkBukΔt+uk2Δt2.(Q_k+1^B)^2-(Q_k^B)^2=2Q_k^Bu_k t+u_k^2 t^2. Adding the cash increment cancels −SkukΔt+SkukΔt-S_ku_k t+S_ku_k t. Therefore, define the accumulated finite-grid objective and its one-step increment by k _k =XkB+QkBSk−φ(QkB)2−∑j<krB(QjB)2Δt, =X_k^B+Q_k^BS_k- (Q_k^B)^2- _j<kr_B(Q_j^B)^2 t, (11) k+1−k _k+1-V_k =gkBΔt+(ukμk−φuk2)Δt2+Qk+1BΔMkS. =g_k^B t+ (u_k _k- u_k^2 ) t^2+Q_k+1^B M_k^S. The Δt2 t^2 terms are the cross-products generated by the discrete product rule and squared-inventory increment. The remaining term, Qk+1BΔMkSQ_k+1^B M_k^S, is the change in inventory value caused by the unpredictable price innovation. Its conditional expectation is zero, so the reward supplied to PPO is (12) rk=gkBΔt+(ukμk−φuk2)Δt2.r_k=g_k^B t+ (u_k _k- u_k^2 ) t^2. We write JΔtB(π)=π[∑k=0N−1rk]J_ t^B(π)=E_π[ _k=0^N-1r_k]. Summing Equation (11) shows that this expected return equals the expected cash-and-inventory objective in Equation (4), up to the fixed initial value. On one realised path, the two accumulated quantities differ by the omitted price innovations ∑kQk+1BΔMkS _kQ_k+1^B M_k^S. The analytical payoff is undiscounted, so we set γRL=1 _RL=1. 3.4. Environment Step We use PPO as our main algorithm (Schulman et al., 2017) and compare FFNN and LSTM actors, each trained for four million environment transitions. At time tkt_k, the policy samples a raw action νkπ _k^π (“Raw” denotes the network output before clipping), while the environment executes νk=clip(νkπ,−100,100) _k=clip( _k^π,-100,100), as shown in Figure 1. PPO evaluates the likelihood of the raw action during training. We record how often clipping occurs and the size of the adjustment between the raw and executed actions. The informed trader calculates ηk⋆ _k from the state at time k before either action changes the state. 4. Experimental Results We now answer the research questions introduced in Section 1. Block A verifies the reference action νkref _k^ref used throughout the experiments. Blocks B–C answer RQ1 by testing full-information PPO, actor capacity under supervised learning, the PPO critic’s ability to rank nearby actions, and potential-based reward shaping. Block D answers RQ2 by comparing partial-information PPO with a certainty-equivalent (CE) controller. Section 5 answers RQ3. Table 1 gives the training settings; code and fixed experiment configurations will be released after peer review. Each PPO configuration uses five training seeds and four million transitions per seed. Because PPO performance can deteriorate late in training, we evaluate checkpoints every 0.4 million transitions on 50 validation paths and select the checkpoint with the highest average broker payoff. After checkpoint selection, every trained policy is evaluated on the same 250 test paths, which are not used for training or checkpoint selection. The reported 95% Student-t intervals are calculated across five independently trained policies, all evaluated on the same test paths. 4.1. Block A: Continuous and finite-grid alignment Block A tests whether the comparison with PPO is affected by discretising the continuous-time game. The simulator applies Equation (2) at every step of the Euler–Maruyama simulator. Let xk=(QkB,QkI,Yk,αk,ξk)x_k=(Q_k^B,Q_k^I,Y_k, _k, _k) T. The finite-grid linear–quadratic (LQ) action is KkxkK_kx_k, where KkK_k is the time-dependent feedback vector obtained from the backward recursion below. We use the reference action as shown in Figure 1, νkref=clip(Kkxk,−100,100). _k^ref=clip(K_kx_k,-100,100). This gives a feasible reference in the PPO action range; the unconstrained LQ action is KkxkK_kx_k. To solve the finite-grid problem, write the Euler transition and reward as xk+1 x_k+1 =Akxk+Bkνk+εk+1, =A_kx_k+B_k _k+ _k+1, (13) rk(xk,νk) r_k(x_k, _k) =xkkxk+2νkskxk+Rkνk2, =x_k T Q_kx_k+2 _ks_k Tx_k+R_k _k^2, where xk=(QkB,QkI,Yk,αk,ξk)x_k=(Q_k^B,Q_k^I,Y_k, _k, _k) T. If the next-step value has quadratic coefficient Fk+1F_k+1, moving one step backwards gives Hk H_k =k+AkFk+1Ak, = Q_k+A_k TF_k+1A_k, βk _k =sk+AkFk+1Bk, =s_k+A_k TF_k+1B_k, χk _k =Rk+BkFk+1Bk, =R_k+B_k TF_k+1B_k, Kk K_k =−βk/χk, =- _k T/ _k, (14) Fk F_k =Hk−βkβk/χk. =H_k- _k _k T/ _k. The complete value is Vk⋆(x)=xFkx+ckV_k (x)=x TF_kx+c_k, where FN=cN=0F_N=c_N=0. For noise covariance Σk _k, ck=ck+1+tr(Fk+1Σk)c_k=c_k+1+tr(F_k+1 _k). The constant does not change the optimal action but is used in the reward-shaping experiment below. Since χk<0 _k<0, νkLQ=Kkxk _k^LQ=K_kx_k is the unique unconstrained maximiser. Completing the square gives JΔtB,⋆−JΔtB(π)=π∑k(−χk)(νk−νkLQ(xk))2J_ t^B, -J_ t^B(π)=E_π _k(- _k)( _k- _k^LQ(x_k))^2. We estimate it by (15) LLQ=1M∑i=1M∑k=0N−1(−χk)[νi,kexec−νkLQ(xi,k)]2,L_LQ= 1M _i=1^M _k=0^N-1(- _k) [ _i,k^exec- _k^LQ(x_i,k) ]^2, where M is the number of test paths, and νi,kexec _i,k^exec is the action taken by the agent. We refer to this quantity as the LQ loss. We therefore replay the continuous analytical feedback and the finite-grid LQ feedback on 250 common market paths at three time resolutions. Table 2(a) shows that the action RMSE and LQ loss both decrease as the grid is refined. At the grid used for training, Δt=2×10−4 t=2× 10^-4, they are 0.0033890.003389 and 1.383×10−81.383× 10^-8. The linear dynamics and Gaussian innovations make every unconstrained reference action Gaussian. If its mean and standard deviation at decision k are mkm_k and sks_k, the union bound gives (16) Pr(maxk|νkLQ|>100)≤∑k[Φ(−100−mksk)+1−Φ(100−mksk)]. \! ( _k| _k^LQ|>100 )≤ _k\! [ \! ( -100-m_ks_k )+1- \! ( 100-m_ks_k ) ]. For the zero-flow reference this bound is 1.07×10−51.07× 10^-5, and no crossing occurs on 500 additional simulated paths. Table 2. Summary of Blocks A–D. Action RMSE is the root mean squared difference between the broker action and the stated reference action (νkref _k^ref). (a) compares the continuous analytical action with νkref _k^ref at three time steps. (b) compares full-information PPO with νkref _k^ref when ξ=0ξ=0. (c) evaluates the supervised FFNN and LSTM on analytical test states and then when their actions determine subsequent states. (d) reports payoff, payoff shortfall from the clipped reference, and action RMSE when σξ=100 _ξ=100. In (b) and (d), the reference is recalculated at each state reached by the controller. Brackets are 95% intervals across five training seeds; the LSTM entry 11.85–339.70 is the five-seed range. Higher payoff is better; lower LQ loss, payoff shortfall, and RMSE are better. (a) Block A: Grid alignment Δt t Steps N LQ loss Action RMSE 10−310^-3 1,000 3.462×10−73.462× 10^-7 0.0168350.016835 5×10−45× 10^-4 2,000 8.681×10−88.681× 10^-8 0.0084680.008468 2×10−42× 10^-4 5,000 1.383×10−81.383× 10^-8 0.0033890.003389 (b) Block B: Full information, ξ=0ξ=0 Policy Unconstrained LQ diagnostic Action RMSE FFNN–ReLU 0.045±0.0310.045 ±0.031 5.494±2.1635.494 ±2.163 LSTM–Tanh 0.688±0.6870.688 ±0.687 16.304±2.92116.304 ±2.921 (c) Block C: Direct action supervision Actor Supervised test RMSE Supervised closed-loop RMSE FFNN–ReLU 2.31 [1.99,2.63] 2.06 [1.27,2.85] LSTM–Tanh 2.62 [2.44,2.81] 11.85–339.70 (d) σξ=100 _ξ=100: feasible comparisons Info. Controller JΔtBJ_ t^B Shortfall RMSE Full Clipped reference 0.1303 0 0 Full FFNN–ReLU −7.060-7.060 7.191 20.449 Full LSTM–Tanh −39.194-39.194 39.324 31.284 Partial CE estimate 0.1297 0.00061 0.732 Partial FFNN–ReLU −21.680-21.680 21.810 24.819 Partial LSTM–Tanh −24.108-24.108 24.238 26.585 A separate numerical check verifies Equation (11) to floating-point precision. Together, the grid, action-bound, and reward checks show that these numerical effects are much smaller than the PPO errors reported below. 4.2. Block B: Full-information PPO recovery We first set ξ=0ξ=0, the zero-uninformed-flow model also considered in (Cartea et al., 2026). We then set σξ=100 _ξ=100 to introduce stochastic uninformed flow. Five independently initialised policies per architecture observe the full state and train for four million transitions. We compare each action by the policy with νkref _k^ref. For the zero-flow FFNN, we use an initial log standard deviation of 33. Table 2(b) reports an LQ loss of 0.045±0.0310.045 ±0.031 and an action RMSE of 5.494±2.1635.494 ±2.163 across five independently trained roots. Figure 2 shows that the selected FFNN approaches the zero-flow reference, whereas the LSTM remains inaccurate. Table 2(d) shows that the PPO agents tested with σξ=100 _ξ=100 also remain inaccurate. Thus, for RQ1, PPO–FFNN can approach the full-information strategy when ξ=0ξ=0, but we do not obtain similar performance after introducing stochastic uninformed flow. Seven panels show zero-flow broker trajectories, lower-cost misspecification trajectories, validation broker payoff during learning, PPO actor loss, and critic value loss. Figure 2. Trajectories and PPO diagnostics. Panels (a)–(b) show the zero-flow experiment and (c)–(d) the change from a=0.0012a=0.0012 to 0.00060.0006. Lines are means and bands span the 5th–95th percentiles; (a)–(b) use signed symmetric-log vertical scales. Panel (e) uses separate payoff scales and normalises progress by each training budget; markers denote checkpoints. Panels (f)–(g) show losses smoothed over 25 updates and averaged across five training seeds. PPO–FFNN/LSTM are from Block B and “Analytical policy + PPO” from Section 5. The dotted line marks the end of its 100-update critic warm-up.Seven panels show zero-flow broker trajectories, lower-cost misspecification trajectories, validation broker payoff during learning, PPO actor loss, and critic value loss. 4.3. Block C: Actor capacity and learning diagnostics Block C tests whether the FFNN and LSTM can reproduce νkref _k^ref when that action is supplied directly. With ξ=0ξ=0, we train a ReLU FFNN and a Tanh LSTM through supervised learning, using νkref _k^ref as labels and the supervised learning has the same architectures as the PPO actors. We evaluate them on analytical test trajectories and then allow each actor to control the simulator. Table 2(c) shows that supervised learning substantially reduces the test error of both architectures. The supervised FFNN also remains close to the reference when it controls the simulator, whereas the LSTM result varies considerably across training seeds. Supervised learning therefore shows that both networks can approximate νkref _k^ref during testing. Figure 2 shows the performance of the models. Reward and critic diagnostics: Starting from the full-information baseline (σξ=100 _ξ=100, ReLU FFNN), we change one component at a time to probe the sensitivity of PPO. The analytical broker action and corrected reward do not depend on the midprice SkS_k. We therefore remove SkS_k to test whether this input makes PPO learning more difficult, and varify the effects of the Δt2 t^2 correction by removing it or multiplying the reward by 100. Every condition uses five seeds, four million training transitions per seed, and the same numbers of validation and test paths. Table 3 reports the resulting payoff, action RMSE, and critic explained variance. Removing the Δt2 t^2 correction or the price input SkS_k does not produce a consistent payoff change across seeds. Multiplying the reward by 100 lowers the payoff for every seed, showing that PPO is sensitive to reward scale. Table 3. Full-information ReLU FFNN ablations at σξ=100 _ξ=100. JΔtBJ_ t^B is evaluated using Equation (12), and action RMSE is calculated using νkrefν^ref_k. Explained Variance (EV) is the critic’s value predictions at the selected checkpoint. Relative to the corrected baseline, paired payoff changes (95% intervals) are 1.48[−7.34,10.31]1.48[-7.34,10.31] without the Δt2 t^2 term, −29.24[−33.93,−24.54]-29.24[-33.93,-24.54] after multiplying rewards by 100, and −1.09[−8.56,6.37]-1.09[-8.56,6.37] without SkS_k. Higher payoff and EV, and lower RMSE, are better.Broker payoff and action error for PPO trained with the corrected or first-order reward, with a fixed reward scaling, and without the absolute price input, together with in-sample critic explained variance. Training reward / input JΔtBJ_ t^B Action RMSE EV Corrected / with SkS_k −±4.82-5.75\!±\!4.82 ±4.1519.76\!±\!4.15 ±.07.93\!±\!.07 First order / with SkS_k −±5.59-4.27\!±\!5.59 ±4.6317.95\!±\!4.63 ±.15.79\!±\!.15 Corrected ×100× 100 / with SkS_k −±.78-34.99\!±\!.78 ±.1931.01\!±\!.19 ±.35.42\!±\!.35 Corrected / without SkS_k −±7.79-6.84\!±\!7.79 ±5.6619.13\!±\!5.66 ±.15.84\!±\!.15 PPO learns a state-value function Vθ(s)V_θ(s). Since γRL=1 _RL=1, the Bellman relation gives Qkπ(s,a)=[rk(s,a)+Vk+1π(sk+1)∣sk=s,νk=a].Q_k^π(s,a)=E\! [r_k(s,a)+V_k+1^π(s_k+1) s_k=s, _k=a ]. We test whether the critic correctly ranks two actions near the trained policy. At a test state s that is not used to train or select the critic, let ν¯π(s) ν_π(s) be the deterministic policy action and set a±=clip(ν¯π(s)±10,−100,100)a_±=clip( ν_π(s)± 10,-100,100). Clipping is inactive in these 120 comparisons, so the two actions are 20 units apart. For Monte Carlo sample ℓ , rkℓ(s,a)r_k (s,a) is the immediate reward after taking action a, and sk+1(a,ℓ)s_k+1^(a, ) is the resulting next state. Replacing VπV^π with VθV_θ and averaging over L simulated transitions gives ΔQ^θ,k Q_θ,k =1L∑ℓ=1L[(rkℓ(s,a+)+Vθ(sk+1(a+,ℓ)))⏟Q^θ,kℓ(s,a+)−(rkℓ(s,a−)+Vθ(sk+1(a−,ℓ)))⏟Q^θ,kℓ(s,a−)], = 1L _ =1^L [ (r_k (s,a_+)+V_θ(s_k+1^(a_+, )) )_ Q_θ,k (s,a_+)- (r_k (s,a_-)+V_θ(s_k+1^(a_-, )) )_ Q_θ,k (s,a_-) ], (17) ΔQkMC Q_k^MC =1L∑ℓ=1L[k,ℓπ(a+)−k,ℓπ(a−)], = 1L _ =1^L [G_k, ^π(a_+)-G_k, ^π(a_-) ], where k,ℓπ(a)=rkℓ(s,a)+∑j=k+1N−1rjℓ(sja,ℓ,νjπ),G_k, ^π(a)=r_k (s,a)+ _j=k+1^N-1r_j (s_j^a, , _j^π ), is the sum of the remaining rewards from Monte Carlo sampling after selecting action a at decision k. ΔQ^θ,k Q_θ,k is the action preference predicted by the critic using the immediate reward and next-state value, while ΔQkMC Q_k^MC is the action preference estimated from the complete simulated returns (if ΔQ^θ,k Q_θ,k and ΔQkMC Q_k^MC have the same sign, it indicates that the critic correctly ranks the two actions). The two candidate actions use the same market innovations and future policy noise, and L=32L=32 Monte Carlo simulations are averaged for each comparison. We also compare Vθ(s)V_θ(s) with the Monte Carlo return of the current policy at 24 test states per seed. The critics can distinguish states with generally higher or lower expected remaining returns, but Figure 3 shows that they cannot reliably rank two nearby actions taken from the same state. Inaccurate local action ranking is therefore a plausible source of weak PPO performance in ξ=100ξ=100. Figure 3. Learning diagnostics. Left and centre: each point compares a+a_+ and a−a_- at one of 120 test states. The horizontal axis is their difference in cumulative reward estimated by Monte Carlo; the vertical axis is the difference predicted using the immediate reward and critic value of the next state. Each point is (ΔQkMC,ΔQ^θ,k)( Q_k^MC, Q_θ,k), where both differences subtract the payoff of a−a_- from a+a_+. In the top-right region, both estimates prefer a+a_+; in the bottom-left region, both prefer a−a_-. In the top-left region, Monte Carlo prefers a−a_- but the critic predicts a+a_+, while in the bottom-right region, Monte Carlo prefers a+a_+ but the critic predicts a−a_-. Pearson correlation measures whether the critic difference increases with the Monte Carlo difference, while Spearman correlation measures whether the critic ranks the action comparisons in the same order as Monte Carlo. The pooled Pearson and Spearman correlations are 0.216 and 0.053, and sign accuracy is 51.7%. Horizontal bars are 95% Monte Carlo intervals; open points indicate decisions where a+a_+ and a−a_- appear equally good within this uncertainty. Without including open points the remaining 71 points show a sign accuracy is 49.3%. Right: each seed’s payoff shortfall from the clipped reference, with σξ=100 _ξ=100; diamonds and bars show the mean and 95% Student-t interval. Correlations closer to one and higher sign accuracy are better; lower payoff shortfall is better.The left panel shows weak association between critic and Monte Carlo local action preferences, the centre panel enlarges the dense region near zero, and the right panel shows five seed-level payoff shortfalls for full- and partial-information FFNN and LSTM policies. Potential-based reward shaping: We next test whether the long-horizon broker payoff provides insufficient information about the quality of each action. The potential Vk⋆(x)=xFkx+ckV_k (x)=x TF_kx+c_k is computed from the finite-grid LQ solution, which is used to shape our reward from Equation (12). The PPO is trained with rkshape r_k^shape =rk+Vk+1⋆(xk+1)−Vk⋆(xk), =r_k+V_k+1 (x_k+1)-V_k (x_k), (18) [rkshape∣xk,νk] [r_k^shape x_k, _k] =χk(νk−Kkxk)2≤0. = _k ( _k-K_kx_k )^2≤ 0. Because VN⋆=0V_N =0 and the initial state is fixed, the shaped and original episode returns differ by the same constant. Reward shaping therefore changes the learning signal without changing the ordering of policies (Ng et al., 1999). Since KkxkK_kx_k is calculated from the unconstrained finite-grid LQ problem it can produce trading speeds that lie outside the action range, the squared term measures deviation from the unconstrained LQ action. We train ten FFNN policies, five with the original reward in Equation (12) and 5 with the shaped reward in Equation (18). Each pair has the same initial weights, market paths, and training settings. Shaping increases mean payoff by 1.571.57 (95% interval [-5.00,8.14]) and reduces action RMSE by 1.761.76 (95% interval [-4.50,8.02]); both changes are favourable in three of five seeds. It also increases the per-step reward standard deviation by approximately 75%. Raw training actions cross the feasible range in approximately 0.1% of decisions under either reward. Under this protocol, shaping changes the learning signal but does not reliably recover νkref _k^ref. 4.4. Block D: Partial-information control Block D asks how closely PPO can approach the causal CE controller when both use the same information available before each decision. PPO observes (S,QB,Y,ξ,τ,νk−1)(S,Q^B,Y,ξ,τ, _k-1), while QIQ^I, α, and the current informed-trader rate are hidden before acting. The CE controller uses the game dynamics and the broker’s causal history ℋkB=σ(o0P,ν0,…,νk−1,okP)H_k^B=σ(o_0^P, _0,…, _k-1,o_k^P) to estimate the hidden components QkIQ_k^I and αk _k. It then substitutes these estimates into the finite-grid feedback rule: νkCE=Π(Kkx^k). _k^CE= _A\! (K_k x_k ). Specifically, the CE controller forms Q^k|k−I=[QkI∣ℋkB],α^k|k−=[αk∣ℋkB]. Q^I_k k^-=E[Q_k^I _k^B], α_k k^-=E[ _k _k^B]. After observing Qk+1BQ_k+1^B, the controller reconstructs the informed flow that has already occurred and updates the informed trader’s inventory: (19) ηkrec _k^rec =νk−ξk−Qk+1B−QkBΔt, = _k- _k- Q_k+1^B-Q_k^B t, Q^k+1|k+1−I Q^I_k+1 k+1^- =Q^k|k−I+ηkrecΔt. = Q^I_k k^-+ _k^rec t. Writing Pij,k=Pij(tk)P_ij,k=P_ij(t_k) and hi,k=hi(tk)h_i,k=h_i(t_k), we rearrange the informed trader’s feedback equation to estimate the signal: (20) α^k|k α_k k =(ηkrec−P21,kQkB−P22,kQ^k|k−ICLOSE = ( _k^rec-P_21,kQ_k^B-P_22,k Q^I_k k^- OPEN−P23,kYk−h2,kξk)/h1,k, -P_23,kY_k-h_2,k _k )/h_1,k, α^k+1|(k+1)− α_k+1 (k+1)^- =ραα^k|k. = _α α_k k. The estimated informed-trader inventory and predictive signal are combined with the observed state variables to calculate the CE action νk+1CE _k+1^CE at the next decision. On 250 test paths, the reconstructed informed-trader inventory has negligible error, while the predicted signal and pre-action informed flow have RMSE 0.0141 and 0.873. The remaining error is caused by the new signal innovation, which is unavailable before the broker acts. Table 2(d) shows that the CE controller remains close to νkref _k^ref, while both partial-information PPO agents have substantially lower payoffs and larger action errors. The broker’s observable history therefore contains enough information for accurate control in this environment, but the PPO agents do not learn to use that history as effectively. This answers RQ2. 5. Adapting the Analytical Policy with PPO We now address RQ3 by testing the direction we introduced in Section 1 and proposed in (Ning et al., 2021; Wang et al., 2026): using an analytical policy as the starting point for RL. Our propsed method would be to freeze the analytical broker action and train PPO to correct execution-cost misspecification through a residual adjustment (Johannink et al., 2019). Let ℰ(a)E(a) denote the broker environment with execution-cost coefficient a. We calculate the analytical policy in the original environment ℰ0=ℰ(a0)E_0=E(a_0), where a0=0.0012a_0=0.0012, and train the PPO adjustment in ℰnew=ℰ(atrue)E_new=E(a_true). Frozen analytical policy: In ℰ0E_0, we evaluate and store the analytical coefficients P11,P12,P13P_11,P_12,P_13 and g1,g2g_1,g_2 from (Cartea et al., 2026), with the optimal action (21) νkℰ0=P110(tk)QkB+P120(tk)QkI+P130(tk)Yk+g10(tk)αk+g20(tk)ξk. _k^E_0=P_11^0(t_k)Q_k^B+P_12^0(t_k)Q_k^I+P_13^0(t_k)Y_k+g_1^0(t_k) _k+g_2^0(t_k) _k. Equation (21) has the same broker-feedback form as Equation (2), and the superscript 00 identifies coefficients calculated in ℰ0E_0. On 100 test paths generated at a0a_0, the frozen and replayed analytical trajectories have trading-speed and inventory RMSEs of 6.36×10−76.36× 10^-7 and 3.29×10−93.29× 10^-9, respectively. PPO adjustment after a cost change: In ℰnew=ℰ(atrue)E_new=E(a_true), the analytical action calculated in ℰ0E_0 remains fixed and PPO learns an additive correction: (22) μ(ok)=νkℰ0+δν(ok).μ(o_k)= _k^E_0+δν(o_k). Here, νkℰ0 _k^E_0 is the action generated by the frozen analytical policy and δν(ok)δν(o_k) is the residual adjustment learned by PPO. The residual is a two-layer, width-64 Tanh FFNN of the full-information observation in ℰnewE_new (we also tested the ReLU setup from Block B, the wider exploration improves learning when ξ=0ξ=0, but it lowers payoff for all five residual policies; thus we found this Tanh setup to be the best for our misspecification problem). Its output layer is initialised at zero, so the initial policy mean equals νkℰ0 _k^E_0. PPO then trains the residual without an additional adjustment bound or penalty. Gaussian exploration is centred on the combined mean, and the environment clips sampled actions to [−100,100][-100,100]. Table 1 reports the remaining training settings. All three cost experiments use full information and σξ=100 _ξ=100. We test atrue∈0.0006,0.0012,0.0018a_true∈\0.0006,0.0012,0.0018\ in ℰnewE_new. The informed trader’s coefficients are recomputed at the true cost, while the broker retains the coefficients from ℰ0E_0 and does not observe atruea_true as an input. A separate PPO adjustment is trained for each cost. For evaluation, νkref _k^ref is recomputed at the true cost. For each seed, we evaluate five checkpoints on 50 validation paths and, as in Section 4, select the checkpoint with the highest mean finite-grid broker payoff. The selected policy is then evaluated on 100 common test paths. Table 4. Finite-grid payoff after changing execution cost. “Reference” is recalculated in ℰnewE_new, while “frozen” uses ℰ0E_0. Brackets are 95% Student-t intervals over five training seeds after averaging 100 common paths. Higher payoff and change are better.Mean broker payoff for the changed-cost reference, unchanged analytical policy, and PPO-adjusted policy at three execution costs. atruea_ true Reference Frozen PPO adjusted Change (95% CI) 0.0006 0.53010 0.29632 0.30151 0.00519[0.00296,0.00743]0.00519\;[0.00296,0.00743] 0.0012 0.06882 0.06882 0.06901 0.00019[−0.00041,0.00079]0.00019\;[-0.00041,0.00079] 0.0018 −0.15783-0.15783 −0.16164-0.16164 −0.16145-0.16145 0.00019[−0.00034,0.00071]0.00019\;[-0.00034,0.00071] When the execution cost is halved, all five PPO adjustments improve the test payoff. The mean increase is 0.005190.00519, with a 95% interval of [0.00296,0.00743][0.00296,0.00743] across the five independently trained policies, and closes 2.22%2.22\% of the gap between the unchanged policy and the changed-cost reference. The largest residual action on the lower-cost test paths is 1.41, and final-action clipping occurs on 0.029% of decisions. PPO therefore provides a small, repeatable improvement only in the lower-cost case tested here. 6. Discussion and Conclusion We studied whether PPO can recover and subsequently adapt an analytically known broker strategy. For RQ1, we have shown that PPO–FFNN approaches νkref _k^ref under full information when ξ=0ξ=0, however when ξ=100ξ=100, the policies remain inaccurate. The finite-step reward and grid-refinement results show that these errors are not caused by numerical issues, while supervised learning shows that the FFNN can approximate and execute the reference action on states generated by the analytical policy. The critic diagnostic shows inaccurate local action ranking can contribute to the difficulty of learning from rewards. For RQ2, the certainty-equivalent controller remains close to the reference action νkref _k^ref when the informed trader’s inventory and signal are hidden. Its performance shows that the available history supports accurate control in this environment, while PPO–FFNN and PPO–LSTM do not learn to use that information as effectively. For RQ3, the frozen analytical policy provides an accurate starting strategy after the execution cost changes. Residual PPO produces a small and repeatable payoff improvement when the cost is halved. The analytical solution is therefore useful both as a benchmark for diagnosing learned policies and as a prior for subsequent adaptation, thus providing some evidence of analytical policy initialisation being useful (Ning et al., 2021). The numerical checks and supervised learning make discretisation, reward implementation, and insufficient FFNN capacity unlikely to be the primary limitations. A more plausible explanation is the interaction between exploration and reward-based value estimation. Wider initial exploration improves PPO when ξ=0ξ=0, but performance remains poor after stochastic uninformed flow is introduced, where the critic also shows weak local action ranking. Partial information additionally requires relevant information to be recovered from past observations. When an analytical policy is already available, wider exploration can instead move the adjustment away from a useful starting point, while conservative adjustments produce a small improvement under the lower-cost misspecification. Future work can use analytical value or action-value information to improve the local learning signal, study alternative action and policy parameterisations, and develop structured residual policies for nonlinear costs or uncertain market impact. Under partial information, it would be interesting to test the performance of the model by introducing a filter into the actor, and practical applications for the partial information setting would include having a ξ+ηξ+η flow. References Andrychowicz et al. (2020) M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem What matters in on-policy reinforcement learning? a large-scale empirical study. External Links: 2006.05990, Document, Link Cited by: §2. Cartea et al. (2026) Á. Cartea, S. Jaimungal, and L. Sánchez-Betancourt Nash equilibrium between brokers and traders. Finance and Stochastics. Note: Published online 27 May 2026 External Links: Document, Link Cited by: §1, §2, §3.1, §3.1, §4.2, §5. Cartea and Jaimungal (2016) Á. Cartea and S. Jaimungal Incorporating order-flow into optimal execution. Mathematics and Financial Economics 10 (3), p. 339–364. External Links: Document, Link Cited by: §2. Casgrain and Jaimungal (2019) P. Casgrain and S. Jaimungal Trading algorithms with learning in latent alpha models. Mathematical Finance 29 (3), p. 735–772. External Links: Document, Link Cited by: §2. Engstrom et al. (2020) L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry Implementation matters in deep RL: a case study on PPO and TRPO. In International Conference on Learning Representations, External Links: Link Cited by: §2. Fazel et al. (2018) M. Fazel, R. Ge, S. M. Kakade, and M. Mesbahi Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1467–1476. External Links: Link Cited by: §2. Henderson et al. (2018) P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, Vol. 32, p. 3207–3214. External Links: Document, Link Cited by: §2. Hendricks and Wilcox (2014) D. Hendricks and D. Wilcox A reinforcement learning extension to the Almgren–Chriss framework for optimal trade execution. In 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr), p. 457–464. External Links: Document, Link Cited by: §1, §2. Higham (2001) D. J. Higham An algorithmic introduction to numerical simulation of stochastic differential equations. SIAM Review 43 (3), p. 525–546. External Links: Document, Link Cited by: §3.2. Igl et al. (2018) M. Igl, L. M. Zintgraf, T. A. Le, F. Wood, and S. Whiteson Deep variational reinforcement learning for POMDPs. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 2117–2126. External Links: Link Cited by: §2. Johannink et al. (2019) T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), p. 6023–6029. External Links: Document, Link Cited by: §2, §5. Koulouris and Campajola (2026) C. S. Koulouris and C. Campajola Memory-induced supra-competitive outcomes between deep reinforcement learning agents in optimal trade execution. External Links: 2605.20348, Document, Link Cited by: §2. Lambrechts et al. (2022) G. Lambrechts, A. Bolland, and D. Ernst Recurrent networks, hidden states and beliefs in partially observable environments. Transactions on Machine Learning Research. External Links: Link Cited by: §2. Lin and Beling (2020) S. Lin and P. A. Beling An end-to-end optimal trade execution framework based on proximal policy optimization. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, C. Bessiere (Ed.), p. 4548–4554. Note: Special Track on AI in FinTech External Links: Document, Link Cited by: §2. Macrì and Lillo (2024) A. Macrì and F. Lillo Reinforcement learning for optimal execution when liquidity is time-varying. External Links: 2402.12049, Document, Link Cited by: §1. Mania et al. (2019) H. Mania, S. Tu, and B. Recht Certainty equivalence is efficient for linear quadratic control. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: 2nd item. Moallemi and Wang (2022) C. C. Moallemi and M. Wang A reinforcement learning approach to optimal execution. Note: Working paper, current revision February 1, 2022 Cited by: §2. Mukherjee and Liu (2023) A. Mukherjee and J. Liu Bridging physics-informed neural networks with reinforcement learning: Hamilton–Jacobi–Bellman proximal policy optimization (HJBPPO). In ICML 2023 Workshop on New Frontiers in Learning, Control, and Dynamical Systems, External Links: 2302.00237, Link Cited by: §2. Nevmyvaka et al. (2006) Y. Nevmyvaka, Y. Feng, and M. Kearns Reinforcement learning for optimized trade execution. In Proceedings of the 23rd International Conference on Machine Learning, p. 673–680. External Links: Document, Link Cited by: §2. Ng et al. (1999) A. Y. Ng, D. Harada, and S. Russell Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning, p. 278–287. External Links: Link Cited by: §2, §4.3. Ning et al. (2021) B. Ning, F. H. T. Lin, and S. Jaimungal Double deep q-learning for optimal execution. Applied Mathematical Finance 28 (4), p. 361–380. External Links: Document Cited by: §1, §2, §5, §6. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §3.4. Sowerby et al. (2022) H. Sowerby, Z. Zhou, and M. L. Littman Designing rewards for fast learning. External Links: 2205.15400, Document, Link Cited by: §2. Wang et al. (2026) B. Wang, X. Gao, and L. Li Reinforcement learning for continuous-time optimal execution: actor-critic algorithm and error analysis. Finance and Stochastics 30, p. 597–655. External Links: Document, Link Cited by: §1, §2, §2, §5. Wu and Jaimungal (2024) X. Wu and S. Jaimungal Broker-trader partial information nash-equilibria. External Links: 2412.17712, Document, Link Cited by: §2.