Paper deep dive
SAiFE-gym: Model-based Environments for Automated Market Making with Concentrated Liquidity
Georgios Chionas, Charalampos Kleitsikas, Stefanos Leonardos, Leandro Sánchez-Betancourt, Carmine Ventre
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/17/2026, 3:56:16 AM
Summary
The paper introduces SAiFE_gym, a Python module providing vectorized Reinforcement Learning (RL) environments for simulating Automated Market Makers (AMMs) with Concentrated Liquidity (CL). It decomposes Constant Product Market (CPM) microstructure into modular components (price models, arrival processes, price impact) to facilitate high-dimensional RL research, demonstrating scalability and applicability in evaluating LP strategies under market uncertainty.
Entities (9)
Relation Signals (6)
SAiFE_gym → implements → Constant Product Markets
confidence 95% · We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs)
SAiFE_gym → supports → Concentrated Liquidity
confidence 95% · We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs) with Concentrated Liquidity (CL).
SAiFE_gym → uses → Reinforcement Learning
confidence 95% · We employ a vectorized approach to optimize our environments, making them scalable for high dimensional Reinforcement Learning (RL) workflows
Uniswap v3 → implements → Concentrated Liquidity
confidence 90% · Newer protocols such as Uniswap v3 (Adams et al., 2021) enable LPs to specify price ranges within which their liquidity is active. This mechanism, known as concentrated liquidity (CL)
SAiFE_gym → integrateswith → Stable Baselines 3
confidence 90% · We connect our environments to Stable Baselines 3 (SB3) (Raffin et al., 2021), which provides a rich suite of state-of-the-art RL algorithms.
Geometric Brownian motion → models → External Midprice
confidence 85% · There are various mid-price models implemented in our environments, such as the Geometric Brownian motion... We call this process, the external mid-price
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs) with Concentrated Liquidity (CL). These markets give Liquidity Providers (LPs) granular control over how their capital is allocated and enable them to adjust their range of liquidity provision dynamically based on market conditions, which in turn, dictates how they earn fees. We decompose the microstructure of CPMs with CL in interactive components that allow researchers and practitioners to capture various economic settings. We employ a vectorized approach to optimize our environments, making them scalable for high dimensional Reinforcement Learning (RL) workflows that best describe sequential decision problems. We demonstrate the benefits of our environments by evaluating the performance of RL agents in CPMs with CL under uncertainty in market parameters.
Tags
Links
- Source: https://arxiv.org/abs/2609.17788v1
- Canonical: https://arxiv.org/abs/2609.17788v1
Trouble viewing inline? Open PDF directly →
Full Text
49,136 characters extracted from source content.
Expand or collapse full text
SAiFE-gym: Model-based Environments for Automated Market Making with Concentrated Liquidity Georgios Chionas Affiliation: University of Liverpool, Liverpool, UK Note: Authors contributed equally to the paper. email: g.chionas@liverpool.ac.uk , Charalampos Kleitsikas Affiliation: King’s College London, London, UK email: charalampos.kleitsikas@kcl.ac.uk , Stefanos Leonardos Affiliation: King’s College London, London, UK email: stefanos.leonardos@kcl.ac.uk , Leandro Sánchez-Betancourt Affiliation: University of Oxford, Oxford, UK email: leandro.sanchezbetancourt@maths.ox.ac.uk and Carmine Ventre Affiliation: King’s College London, London, UK email: carmine.ventre@kcl.ac.uk Abstract. We present SAiFE_gym, a Python module that provides a collection of simulation environments for studying trading problems in Constant Product Markets (CPMs) with Concentrated Liquidity (CL). These markets give Liquidity Providers (LPs) granular control over how their capital is allocated and enable them to adjust their range of liquidity provision dynamically based on market conditions, which in turn, dictates how they earn fees. We decompose the microstructure of CPMs with CL in interactive components that allow researchers and practitioners to capture various economic settings. We employ a vectorized approach to optimize our environments, making them scalable for high dimensional Reinforcement Learning (RL) workflows that best describe sequential decision problems. We demonstrate the benefits of our environments by evaluating the performance of RL agents in CPMs with CL under uncertainty in market parameters. 1. Introduction Automated market makers (AMMs) are some of the most used applications built on top of blockchains for trading. These markets were proposed as an alternative to Limit Order Books, tailored to the computational and storage limitations of blockchains. AMMs, which mainly operate as constant function markets (CFMs), dictate the rules of clearing demand and supply through bonding curves, with the most dominant example being the constant product bonding curve. Despite their simple design and the inefficiencies in its first iterations, AMMs have facilitated $4 trillion in trading volume across Ethereum and other Ethereum-compatible networks (Uniswap Labs, 2026). Consequently, AMMs have attracted the interest of researchers within the mathematical finance community (Angeris and Chitra, 2020; Cartea et al., 2024a; Cartea et al., 2025). Most of these works study stylized models, either from the Liquidity Provider (LP) perspective (Cartea et al., 2024a) or from Liquidity Takers (LTs) (Cartea et al., 2025; Angeris et al., 2022), by capturing the AMM mechanics associated with their model specific assumptions. Despite the mathematical elegance of those methods, they are often low dimensional to accommodate analytical solutions and, thus, are unable to capture the high dimensionality observed in practice. Reinforcement Learning (RL), in turn, has emerged as a paradigm for tackling such high dimensional control problems. Nevertheless, we observe that there is fragmentation in model-based simulation environments tailored to the AMM microstructure, and to this day, there is not an open sourced unified and extensible collection of environments available to the research community. We introduce SAiFE_gym, a collection of Reinforcement Learning (RL) environments, that assembles components which govern each AMM case study. Environments have the property that once the skeleton is in place, the research community and practitioners can parallelize across different domains. For that reason, RL environments sit at the centre of current AI progress, where each new one expands what we can train, study, and evaluate. By lowering the barriers to conducting research on AMMs, our SAiFE_gym contributes directly to the intersection between Crypto and AI (Allen et al., 2026). The goal of this work is to assemble different aspects of the AMM microstructure in an extensible way while also demonstrating the potential of RL for solving this type of stochastic control problem. Broadly, the AMM microstructure is built from several interacting components, namely the evolution of market prices, the orderflow and the toxicity thereof, the price impact and the liquidity distribution, and the frictions imposed by blockchains, namely the gas costs for interacting with them and their inherent latency. Our main architecture design for the RL environments is a vectorized approach, whereby a batch of trajectories runs in parallel, aggregating the results. This design choice significantly improves computational efficiency (wall-clock efficiency) while also providing more stable estimates for policy optimization. The summary of our contributions is given below. Our Contributions. (1) We provide a unified set of RL environments that replicate the mechanics of AMMs under a wide range of combinations of market dynamics. This is a key strength of our approach as it supports picking and choosing different components of modelling that can simulate various economic settings. (2) We connect our environments to Stable Baselines 3 (SB3) (Raffin et al., 2021), which provides a rich suite of state-of-the-art RL algorithms. These algorithms allow a “plug-and-play” approach to train RL agents on previously unstudied problems, while also enabling users to develop their own customizable agents. (3) Following the design principles of (Jerome et al., 2023), we develop a high degree of parallelism in our collection of RL environments, which can work on top of common existing multiprocessing methods, achieving a level of efficiency for training AI agents that makes the task feasible even for conventional machines. (4) To demonstrate the applicability of our set of environments, we evaluate the performance of RL agents under uncertainty in market parameters embedded in the AMM microstructure. This approach, known as domain randomization , has found great applicability in simulation-to-real work, c.f., (Peng et al., 2018). The module is available on Github. 2. Related Work Automated Market Makers. The computational efficiency of using invariant functions (fixed trading functions) together with the computational constraints of the early blockchains led simple AMMs, in the form of Constant Product Markets, like Uniswap v2 (Adams et al., 2020), to become a cornerstone in decentralised finance. The work in (Angeris and Chitra, 2020) generalized the notion of the constant product invariant to the broader class of Constant Function Market Makers (CFMMs) and characterised conditions under which arbitrage aligns the price quoted by a CFMM with the (true) price of an asset on an external market, thus establishing the usefulness of CFMMs as price oracles. Despite their widespread adoption, simple AMMs such as Uniswap v2 suffer from capital inefficiency, since a substantial part of the provided liquidity is never used. Newer protocols such as Uniswap v3 (Adams et al., 2021) enable LPs to specify price ranges within which their liquidity is active. This mechanism, known as concentrated liquidity (CL), gives LPs more granular control over the allocation of their capital. Consequently, Uniswap v3 significantly expands the LPs’ action space, which was previously considered passive. This transition has sparked a significant line of research on profitable strategies for LPs in CPMs with CL. We outline notable works that inspired the collection of environments presented in this work. The authors in (Fan et al., 2023; Fan et al., 2022) study dynamic liquidity provision strategies in Uniswap v3 and provide insights on how aspects of a Uniswap v3 contract, notably the partition of price space, have implications for LPs’ profit. The authors in (Cartea et al., 2023) decompose the LP’s losses in CPMs with CL into two parts; the convexity cost, arising from the convexity of the pool’s invariant function, and the opportunity cost, arising from the LP’s locking assets in the pool instead of investing them in a risk-free account. The subsequent work in (Cartea et al., 2024a) formulates dynamic liquidity provision as a stochastic control problem and derives a closed form solution, under which the LP reallocates its accumulated wealth continuously across intervals as the price evolves. However, their model abstracts away from frictions inherent to blockchain, mainly the gas fees and latency. Reinforcement Learning for Market Making. In recent years, Reinforcement Learning has gained considerable traction in tackling model-based problems in high frequency trading. This is because market making in LOBs, as well as liquidity provision in CPMs with CL, can naturally be formulated as stochastic control problems. RL offers a complementary method to numerical solutions, which usually suffer from the curse of dimensionality and therefore struggle to capture realistic models. While there is now substantial literature and open-source infrastructures for model-based RL environments in LOB trading, similar tools remain limited in the DeFi space. In traditional finance, the work in (Spooner et al., 2018) introduces one of the earliest RL frameworks for market making in a LOB simulator and shows that inventory-aware RL agents can outperform simple known benchmarks. Closely related in spirit to our work, (Jerome et al., 2023) provides a modular suite of Gym-compatible environments for model-based LOB trading that supports picking different components of existing models in the mathematical finance literature. However, in that work, a reduced-form model of the LOB is implemented by abstracting away from the statefulness of the entire order book. In contrast, our implementation of the CPM with CL closely mirrors the respective pools that are implemented in practice. An alternative to model-based simulators has been the agent-based simulators. Rather than specifying the market dynamics directly, agent-based simulators generate these dynamics through the interaction of agents. Some notable works that implement agent-based simulators to train RL agents in financial markets include (Kumar, 2020), (Amrouni et al., 2022), and (Mascioli et al., 2024). These are some of the most important works that demonstrate the effectiveness of RL in traditional finance settings. Our work is one of the first to provide a modular infrastructure for testing RL agents in AMMs. 3. Design of the Module In the following sections, we give a detailed description of the implemented RL environments, including the mechanics of a CPM with CL and the microstructural dynamics that specify the environments. Formally, the interaction of the LP with the CPM with CL can be viewed as a stochastic control problem over a continuous time horizon ℐ=[0,T]I=[0,T], with terminal time T. In our RL environments, we approximate this problem on a discrete time grid with n steps. At each time step, every RL environment performs the observation phase, where the LP agent observes market information, action phase, where the agent takes a new action, market operation phase, where the market processes the new orders and time advancement phase. These phases are implemented in the class AMMEnvironment and it is inherited by all the RL environments. We then present experiments demonstrating the capabilities of our environments collection. Finally we discuss the implications of our work and outline directions for future research. 3.1. Constant product markets and concentrated liquidity Constant Product Markets. Let X denote a numéraire asset, such as USDC, and let Y denote a risky asset, such as ETH. A liquidity pool for the pair (X,Y)(X,Y) quotes a marginal exchange rate Z, measured in units of asset X per unit of asset Y. Let qX,qY>0q^X,q^Y>0 denote the pool reserves of assets X and Y, respectively. The reserves of a CPM satisfy the invariant (1) f(qX,qY)=qXqY=κ2,f(q^X,q^Y)=q^Xq^Y=κ^2, where κ>0κ>0 is the liquidity parameter, or depth, of the pool. In the absence of fees, LT trades move the reserves along the level set defined in (1). Differentiating the invariant shows that the marginal exchange rate (the spot price) is Z=−dqXdqY=qXqYZ=- dq^Xdq^Y= q^Xq^Y. Thus, Z is the pre-fee marginal price of asset Y in units of asset X for an infinitesimal trade. The CPM charges a proportional trading fee τ∈[0,1)τ∈[0,1) on the input amount. Consequently, only the fraction 1−τ1-τ of a trade’s gross input is used in the swap calculation. Consider now an LT who submits Δx>0 x>0 units of asset X and receives Δyout>0 y^out>0 units of asset Y. The output amount is determined by the invariant equation (qX+(1−τ)Δx)(qY−Δyout)=qXqY. (q^X+(1-τ) x ) (q^Y- y^out )=q^Xq^Y. Solving for Δyout y^out gives Δyout=qY−qXqYqX+(1−τ)Δx. y^out=q^Y- q^Xq^Yq^X+(1-τ) x. Conversely, suppose that the LT submits Δy>0 y>0 units of asset Y and receives Δxout>0 x^out>0 units of asset X. The invariant equation is (qX−Δxout)(qY+(1−τ)Δy)=qXqY, (q^X- x^out ) (q^Y+(1-τ) y )=q^Xq^Y, which implies that Δxout=qX−qXqYqY+(1−τ)Δy. x^out=q^X- q^Xq^Yq^Y+(1-τ) y. Concentrated Liquidity. In a CPM with CL, an LP can restrict a liquidity position to a specified interval of marginal exchange rates (Zℓ,Zu](Z ,Z^u]. The endpoints of the position are selected from a finite, ordered grid (2) =Z(−N)=Z¯,Z(−N+1),…,Z(0),…,Z(N−1),Z(N)=Z¯,G= \Z^(-N)= Z,\,Z^(-N+1),…,Z^(0),…,Z^(N-1),\,Z^(N)= Z \, for some N≥1N≥ 1. Each pair of neighbouring grid points (Z(i),Z(i+1)] (Z^(i),Z^(i+1) ] is called a tick range. Within a tick range, the pool satisfies a translated constant-product invariant. Equivalently, the CPM with CL is a segmented CPM tailored to the liquidity in each tick range When opening a position, the LP selects a range of rates (Zℓ,Zu](Z ,Z^u] and a liquidity amount κ~>0 κ>0. These variables combined with current marginal exchange rate Z, determine the corresponding quantities x and y of assets X and Y respectively, according to (3) (x,y)=(0,κ~((Zℓ)−1/2−(Zu)−1/2)),Z≤Zℓ,(κ~(Z1/2−(Zℓ)1/2),κ~(Z−1/2−(Zu)−1/2)),Zℓ<Z≤Zu,(κ~((Zu)1/2−(Zℓ)1/2), 0),Z>Zu. (x,y)= cases (0,\, κ ((Z )^-1/2-(Z^u)^-1/2 ) ),&Z≤ Z ,\\[5.55002pt] ( κ (Z^1/2-(Z )^1/2 ),\, κ (Z^-1/2-(Z^u)^-1/2 ) ),&Z <Z≤ Z^u,\\[5.55002pt] ( κ ((Z^u)^1/2-(Z )^1/2 ),\,0 ),&Z>Z^u. cases In words, when Z≤ZℓZ≤ Z , the position consists only of asset Y, while when Z>ZuZ>Z^u, it consists only of asset X. When Zℓ<Z≤ZuZ <Z≤ Z^u, the position is active and contains quantities of both assets. In a tick range i, the total liquidity depth κi _i is obtained by summing the liquidity amounts κ~i κ_i of each individual position in tick range i. Thus, the whole liquidity depth profile is captured by (4) =(κi)i=−N−1. κ= ( _i )_i=-N^N-1. The formulas in (3) and (4) give a reparameterization of the state of the CPM with CL from the quantities x,yx,y of assets X,YX,Y respectively, to the total liquidity depth in each tick range and the marginal exchange rate. The latter is more convenient, since, an LP action changes only the liquidity depth profile (4), while an LT action changes only the marginal exchange rate Z. Lastly, the trading fees are stored in each tick range, and are distributed pro-rata to each LP. For example, if the LP’s position with depth κ~ κ is in a tick range with total depth κ, then for every liquidity taking trade that pays in fees an amount of p, the LP earns the amount of p~=κ~κp 1Zℓ<Z≤Zu p= κp\, 1_Z <Z≤ Z^u. 3.2. Mid-price processes In our RL environments, we follow the standard assumption in the literature on AMMs, under which the price formation of the risky asset Y is exogenous to the AMM. Concurrently, the AMM quotes its own price for the risky asset Y, the marginal exchange rate Z. External Midprice (Exogenous) We call this process, the external mid-price and denote it by StS_t. There are various mid-price models implemented in our environments, such as the Geometric Brownian motion which is the solution of the stochastic differential equation (5) dSt=μStdt+σStdWt,S0>0,dS_t=μ S_tdt+σ S_tdW_t, S_0>0, with drift μ, volatility σ>0σ>0, and where WtW_t is a Brownian motion. Another implemented model is the Arithmetic Brownian motion, where the mid-price can be written as St=S0+μt+σWt,S0>0.S_t=S_0+μ t+σ W_t, S_0>0. We also provide an arithmetic midprice model with an Ornstein–Uhlenbeck alpha signal, from the algorithmic trading literature. dSt dS_t =αtdt+σdWt,S0>0, = _tdt+σ dW_t, S_0>0, dαt d _t =−kααtdt+σαdWtα,α0∈ℝ, =-k^α _tdt+σ^αdW^α_t, _0 , where kαk^α is the mean-reversion rate, σα>0σ^α>0 is the volatility of the signal, and WαW^α is a Brownian motion independent of W. Marginal Exchange Rate (Endogenous) The marginal exchange rate Z is modelled as a pure jump process. We assume that the marginal exchange rate takes values from the same discrete grid (2). The evolution of the marginal exchange rate is endogenous because it depends on the state of the environment as explained next. 3.3. Arrival processes Currently in AMMs, the formation of the marginal exchange rate is mainly due to LTs’ activity. In our RL environments, we decouple the LTs’ activity into an exogenous and an endogenous component. Noise Traders, Liquidity-attracted Traders and Informed Traders The entire LT action is captured by the next model for intensities. (6) λt±=maxα0,α1+α2Kt−±+α3(St−Zt−) _t^±= \ _0,\, _1+ _2K^±_t-+ _3 (S_t-Z_t- ) \ where λt+λ^+_t (resp. λt−λ^-_t) is the intensity for LT sells (resp. LT buys). The linear model (6), which was introduced in (Aqsha et al., 2026), captures three stylized facts. The parameter α1 _1 is the baseline intensity of order arrivals, which corresponds to noise traders, and this part is exogenous to the state of the AMM. The parameter α2 _2 models the relationship between arrivals and depth (which means that higher depth implies more arrivals) and α3 _3 captures the impact of LT action by informed traders who align the marginal exchange rate Z with the external mid-price S.11 1 Linear intensity models are widely used in the literature, see e.g., (Stoikov and Sağlam, 2009). Writing the dynamics of the AMMs à la Avellaneda-Stoikov was first proposed in (Cartea et al., 2024b). Lastly, α0>0 _0>0 is a safeguard parameter to prevent intensities from becoming negative. More specifically, when St−Zt−>0S_t-Z_t->0 (resp. St−Zt−<0S_t-Z_t-<0), all else being equal, we expect a higher (resp. lower) buying intensity and a lower (resp. higher) selling intensity, given that the marginal exchange rate is underpriced (resp. overpriced). The subscript t−t- represents the time just before t given that we are working with processes that jump. The liquidity statistic Kt−±K^±_t- in (6) is implemented through exponential kernels ±K^±, which are illustrated in Figure 1. These kernels encode how the liquidity standing ahead of the current tick i affects future arrivals. In particular, we implement (7) i+=∑h=0H−1wh+1κi+hand i−=∑h=0H−1wh+1κi−hK^+_i=Σ _h=0^H-1w_h+1 _i+h ^-_i=Σ _h=0^H-1w_h+1 _i-h where the weights w determine how liquidity in ticks away from the current tick i affects future arrivals. Through this channel, the LP’s action influences the future order flow.22 2 These liquidity kernels are similar in spirit to the queue-reactive model studied in LOBs (Huang et al., 2015), but, to the best of our knowledge, this is the first time considered in the context of CPMs with CL. Figure 1. Smoothed kernels with exponential weights capturing the impact of the neighbouring liquidity of the current marginal exchange rate to the arrivals intensity. In our RL environments, the parameters α1,α2,α3 _1, _2, _3 and the kernels ±K^± can be modulated to represent different AMM economic settings, or they may be estimated from empirical data, as in (Aqsha et al., 2026). 3.4. Price Impact The price impact refers to the function that connects the marginal exchange rate before and after the execution of an LT trade. We provide two such classes. We remind that the marginal exchange rate Z is implemented as a jump process and it lives in the grid (2). Depth-independent price impact. This is a reduced-form model whereby each LT trade pushes the marginal exchange rate to the neighbouring tick based on the direction of the trade. If the marginal exchange rate is at tick i, a buy LT trade (resp. a sell trade) pushes Z to tick i+1i+1 (resp. to tick i−1i-1). This reduced-form price impact model in CPMs with CL has been used in the works (Fan et al., 2022; Fan et al., 2023). Economic component Role in the environment Implemented models External midprice Generates the exogenous market reference price StS_t observed by the AMM and the LP. BrownianMotionMidpriceModel; GeometricBrownianMotionMidpriceModel; OrnsteinUhlenbeckMidpriceModel Order arrivals Generates LT action and updates its intensity from AMM state. LiquidityKernelArrivalModel Price impact Maps LT trades into jumps of the marginal exchange rate. OneTickUniswapV3PriceImpact; LiquidityDepthUniswapV3PriceImpact Reward function Defines the objective optimized by the agent. PnL; RunningInventoryPenalty Table 1. Components in SAiFE_gym. Each setting is exposed through a small interface, enabling plug-and-play of different components while keeping the remaining AMM engine fixed. Depth-dependent price impact. In practice, price impact depends on the curvature of the CPM and the concentration of liquidity around the current marginal exchange rate. In order to capture these factors, we compute a smoothed forward and backward depth of the CPM with CL, over a given horizon of neighbouring ticks. The smoothing approach is similar in spirit to the kernel implementation for the order arrivals (7). We choose a smoothing horizon H and non-negative weights w1,w2,…,wHw_1,w_2,…,w_H and ∑j=1Hwh=1Σ _j=1^Hw_h=1. The smoothed forward depth at tick i is (8) Di+=∑h=0H−1wh+1κi+h(Zi+h)1/2and Di−=∑h=0H−1wh+1κi−h(Zi−h)−1/2D^+_i=Σ _h=0^H-1w_h+1 _i+h(Z_i+h)^1/2\,and \,D^-_i=Σ _h=0^H-1w_h+1 _i-h(Z_i-h)^-1/2 where κjκ^j is the total liquidity at tick j. A buy trade (resp. a sell trade) causes a price impact tailored to the smoothed forward depth (resp. smoothed backward depth). In particular, the size Q of the LT trade (in units of the numeraire asset X) is drawn from a custom distribution (e.g. a lognormal distribution).33 3 For a sell LT trade, the size in units of asset Y is given by dividing the size Q with the current external price S. The price impact is the number of jumps of the marginal exchange rate in the grid (2) and it is calculated by m=⌊Q/D±i⌋m= Q/D^i_± . Therefore, when the marginal exchange rate is at tick i, a buy (resp. sell) trade of size Q moves it to tick i+mi+m (resp. to tick i−mi-m). 3.5. Blockchain Frictions Gas Costs Our RL environments capture the cost of rebalancing as a fixed transaction cost. The fixed transaction cost corresponds to the so-called gas fees paid by the LP to the network for interacting with the blockchain. Gas fees are mostly a flat fee paid to the blockchain and do not depend on LP’s specific position. Thus, gas fees scale with the frequency of rebalancing. In particular, one can write the cumulative cost up to time t as Ct=∑r≤tgr,C_t=Σ _r≤ tg_r, where r are the times at which a rebalancing took place and gr>0g_r>0 is the gas cost associated with the rebalancing. Latency In blockchain environments, there is inherent latency, which prevents the LP from reallocating her liquidity and rebalance her asset continuously. In our RL environments, we capture latency by specifying the number of decisions throughout training and the evaluation time. In other words, we allow deterministic latency. Overall, all the components described in this section, are summarized in Table 1. 3.6. Agents We have implemented custom agents by primarily focusing on liquidity provision in CPMs with CL. All the agents inherit from a base class Agent with the main function being get_action(state). Liquidity Providers We implement two ways in which the LP agent can interact with the CPM with CL that aligns with the admissible actions of an LP in CPMs with CL such as Uniswap v3 and v4. Firstly, the agent can open an active position, where she decides the lower and upper tick (Zℓ,Zu](Z ,Z^u] such that, the current marginal exchange rate Z∈(Zℓ,Zu]Z∈(Z ,Z^u]. Secondly, the LP agent can approximate a limit order by opening a position (Zℓ,Zu](Z ,Z^u] outside of the current marginal exchange rate, whereby Z≤ZℓZ≤ Z or Z>ZuZ>Z^u. This is essentially a single-sided asset provision and in particular, provision of the risky asset Y in the former case and provision of the numeraire asset X in the latter case as defined in (3). In both cases, every time the LP agent takes a new action, she first withdraws her holdings in assets X,YX,Y, collects the accumulated fees from her last position and then reinvests her whole wealth to the new position. Liquidity Takers Although our main focus on the presented RL environments is liquidity provision, we also provide an LT agent for statistical arbitrage against the CPM with CL.44 4 We refer to the work (Cartea et al., 2025) for a detailed treatment on liquidity taking in CPMs. The control of the LT agent is the signed trading speed v at which the agent is buying or selling the asset Y. In particular, if v>0v>0 (resp. v<0v<0), the agent sells (resp. buys) the risky asset Y. If v=0v=0, the LT agent does not trade. For each timestep dtdt, the trading speed v implies the size q of the LT’s trade is v⋅dtv· dt. The LT’s execution cost is specified through the price impact that depends on the LT trade size q, which is calculated by the Depth-dependent price impact given in Section 3.4. 3.7. Reward Functions Let T>0T>0 be the terminal time of the simulation and ℐ=[0,T]I=[0,T]. The reward PnL (the risk-neutral reward) consists of the mark-to-market value of the LP’s position. Let (xt)t∈ℐ (x_t )_t be the process of the agent’s holdings in the numeraire asset, (yt)t∈ℐ (y_t )_t be the process of the agent’s holding in the risky asset and (St)t∈ℐ (S_t )_t be the external midprice process. The final mark-to-market agent’s value is VT=xT+yTSTV_T=x_T+y_TS_T. For the LP agent, the gas fees are subtracted from xtx_t, at every rebalancing step. The PnL is the change in the agent’s value between steps and therefore it can be written as VT−V0V_T-V_0. Risk Aversion We implement a reward that captures risk aversion through an inventory penalty. We use the Cartea-Jaimungal criterion (Cartea et al., 2017). The reward RunningInventoryPenalty is calculated as (9) VT−V0−η(yT−y^)2−ϕ∫0T(yt−y^)2t,V_T-V_0-η (y_T- y )^2-φ _0^T(y_t- y)^2dt, where the constants η,ϕ∈ℝ+η,φ ^+ determine the terminal and running penalty respectively from deviating from some target level y y. The units of η,ϕη,φ are such that the penalty terms are in units of X, while the parameter ϕφ quantifies the urgency of the agent to liquidate the risky asset Y. 4. Reinforcement Learning In this Section, we describe how the AMM engine, presented in Section 3, works along with RL workflows. This is achieved through the wrappers that we outline in Section 4.1. Subsequently, in Section 4.2, we present the custom vectorized approach that we follow across all the environments, which enables noticeably faster convergence in the agents’ rewards. Lastly, in Section 4.3, we demonstrate a potential use case of our collection of environments in domain randomized reinforcement learning; specifically, the evaluation of RL agents when training is driven by uncertainty in the market parameters embedded in the stochastic processes presented in Section 3. Figure 2. Deep RL wrapper workflow 4.1. Wrappers SAiFE_gym provides Gymnasium APIs for single agent deep reinforcement learning in the simulated CPM with CL. Figure 2 illustrates how a learning agent interacts with the AMM engine through the SB3 wrapper. From the learning agent’s perspective, the AMM mechanics, the external midprice, and the arrivals are all part of the environment. At each decision time, the learning agent observes the current state, chooses an action, and passes this action to the environment through the step interface. The environment then updates the AMM state by applying the agent’s action together with the simulated market dynamics. The resulting transition follows the standard Gymnasium convention; the step(action) returns the next observation, reward, and episode termination information. In this way, the wrapper layer exposes transition tuples consisting of observation, action, and next observation, while remaining agnostic to the specific RL algorithm used. This wrapper is the main compatibility layer between the AMM engine and the SB3 training algorithms. Next, we briefly describe the functionality of the main agent-specific wrappers that are implemented. StructuredMultiDiscreteVecEnv This wrapper re-parameterizes the action space from [lower_offset, upper_offset, hold], which is given in (3), to [center, half_width, hold]. The former parameterization is Uniswap v3-native, while the latter one is more suitable for RL training since the decisions are separable. DecisionStrideEnv This wrapper separates the market decision frequency from the agent’s decision frequency. The role of this wrapper is to capture the latency inherent in blockchain environments. Intuitively, it transforms a high-frequency control problem into a lower-frequency decision problem.55 5 Dynamic liquidity provision strategies are often slower than the underlying market simulation. The AMM pool might update every time step, but an LP would not necessarily rebalance after every LT trade. Through this wrapper, one can customize the number of decisions per episode the agent will take in equally split time intervals. For example, by setting K_stride=0, we recover the continuous-time setting where the market decision frequency aligns with the agent’s decision frequency. ReduceStateSizeWrapper The implementation of the CPM with CL is stateful enough; hence, one could customize the observation space of the agent. We do offer several standard features that can be used as components of the observations, namely: • time remaining T−tT-t, • the mispricing St−ZtS_t-Z_t between the external midprice and the marginal exchange rate, • the LP’s current position, • the accrued fees. Figure 3. Speedup of vectorization using NumPy. 4.2. Vectorized approach RL agents require training policies across many market paths in order to obtain reliable gradient estimates and reduce sensitivity to any single price or order-flow realization. To support this at scale, our implementation adopts a vectorized simulation architecture in which all state variables; actions, rewards, and the associated stochastic processes presented in Section 3 carry a leading batch dimension of size num_trajectories, the number of rollouts simulated in parallel, using NumPy arrays (van der Walt et al., 2011). Consequently, each environment step produces num_trajectories transitions.66 6 This vectorization is native to the AMM structure, and it differs from Gymnasium generic VectorEnv , which is agnostic to the AMM simulation engine. Our design contrasts with the typical approach to parallelize RL “rollouts” at the process level, which instantiates many environments across multiple CPU cores and sequentially generates trajectories on each one. Our implementation, instead, amortizes memory allocation and mathematical operations over the full batch in a single pass. Figure 3 illustrates the speedup offered by our NumPy implementation over the multiprocessing approach that uses the concurrent.futures package for identical per-trajectory workload.77 7 The benchmark was run on a single machine with 8 CPU cores and 16 GB of RAM. For rolling out 1000 trajectories, the vectorized approach takes ≈0.05≈ 0.05 s versus ≈1.6≈ 1.6 s for the multiprocessing approach, achieving a speedup of roughly 30×. Typical RL training involves millions of rollouts, making the multiprocessing approach prohibitively slow. Additionally, in Figure 4, we plot the evolution of the mean reward per episode for different degrees of parallelization in terms of the number of parallel trajectories. By increasing the number of parallel trajectories from 10 to 10,000 the variance in the reported mean episode reward drops across training updates, though we observe little difference in the training performance for n=100 and n=1000. Figure 4. Impact of vectorization in the stability of rewards. 4.3. Reinforcement Learning under Domain Randomization To demonstrate the potential of SAiFE_gym, we consider the evaluation of liquidity provision policies under distributional shifts in market parameters. As a warmup, we provide an example of a nominal RL agent, under stationary transition dynamics, whereby the learning agent is trained and evaluated under the same market parameters. Subsequently, we proceed to domain randomization, whereby we sample market parameters during training, a process that we call non-stationary transition dynamics. Stationary Transition Dynamics As a first example of training an agent using SAiFE_gym, we use proximal policy optimization (PPO) (Schulman et al., 2017) to train an LP agent in a CPM with CL under fixed market parameters. In this example, the external midprice StS_t follows a geometric Brownian motion (5), with μ=0μ=0 and σ=0.03σ=0.03. The evolution of the marginal exchange rate ZtZ_t is formed by incoming LT trades, for which we use the intensity model in (6), with α1=450,α2=10 _1=450, _2=10 and α3=4000 _3=4000, where the latter captures the arbitrage pressure and is the main driver through which the marginal exchange rate ZtZ_t follows the external midprice StS_t throughout the simulation period. We use the depth-dependent price impact model described in Section 3.4. We set the initial price at Z0=S0=100Z_0=S_0=100 and the LP agent’s initial wealth at 10001000 in asset X. We initialize the liquidity depth profile (4) with 100,000100,000 liquidity units in each tick range. Near the tick ranges of the initial marginal exchange rate at 100100, these liquidity units correspond to roughly 5050 units in asset X. Moreover, we fix the size Q of each LT trade at 250250 units in asset X, which causes a price impact of around 5 ticks in the marginal exchange rate Z. We set a fixed gas cost at 22 units in asset X. We allow the agent to take one action every 100100 time steps through the wrapper DecisionStrideEnv, and we allow the agent to choose her new positions within an interval of 5050 tick ranges from the current marginal exchange rate. The restrictions on the agent’s action space are natural, but, importantly, they improve the agent’s exploration. As a reward class, we choose the running inventory penalty from (9), setting the per-step inventory penalty at ϕ=0.4φ=0.4, the terminal inventory penalty at η=0η=0, and the target level at y^=0 y=0. We compare the performance of the RL agent with that of a baseline agent that rebalances every 100100 time steps symmetrically around the current marginal exchange rate Z, with 5050 tick ranges on each side. We spawn 10001000 trajectories that run in parallel, and we sample three of those during evaluation, as illustrated in Figure 5. Figure 5. Dynamic liquidity provision policies. On the left column, we plot the strategy of the baseline agent (periodic rebalancing), that rebalances symmetrically around the marginal exchange rate. On the right column, we plot sampled trajectories from the learnt policy of the PPO agent. This is for illustrative purposes to visualize how the agent react to market conditions. Non-Stationary Transition Dynamics In order to expose the agent to market uncertainty, we implement episode-level domain randomization through a wrapper around the base environment class AMMEnvironment. We consider the market domain θ=(σ,α1)θ=(σ, _1), where σ is the volatility of the external midprice process StS_t, and α1 _1 is the baseline arrival intensity in (6), which captures the intensity of noise trading. During training, these market parameters are sampled from the joint distribution ρθ _θ for the independent random variables σ,α1σ, _1, according to σ∼(0.01,0.05)σ (0.01,0.05) and α1∼(300,600) _1 (300,600). At the start of each rollout, the wrapper samples D domains; that is, D tuples of (σ,α1)(σ, _1) and assigns them into balanced groups across the n trajectories that run in parallel. The wrapper creates D groups, each containing n/Dn/D trajectories, and uses the same domain within the group throughout the episode. The hyperparameter D offers the following simple tradeoff. Larger D gives each PPO update broader coverage of the domain distribution. Smaller D gives more repeated stochastic trajectories per sampled domain. The remaining market parameters are kept fixed across domains. Naturally, the market domain θ=(σ,α1)θ=(σ, _1) is not included in the RL agent’s observation space; thus, the agent faces a latent-parameter partially observable decision problem. Consequently, the domain randomized agent must learn behaviors that perform well across the domain distribution. We use the exact same market configuration as in the Stationary Transition Dynamics example in the previous paragraph. The reward is the running-inventory criterion defined in (9) with a per-step inventory penalty ϕ=0.4φ=0.4. The objective is now modified to maximize the expected reward across the distribution ρθ _θ (10) θ∼ρθ[ζ∼p(ζ|π,θ)[∑n=0N−1(Vn+1−Vn−ϕΔtyn+12)]],E_θ _θ [E_ζ p(ζ|π,θ) [ _n=0^N-1 (V_n+1-V_n-φ\, t\,y_n+1^2 ) ] ], the trajectory ζ is sampled from the trajectory distribution induced by policy π in the environment with the market domain θ. For evaluation, we compare the domain-randomized agent with the nominal PPO, trained under the fixed domain (σ,α1)=(0.03,450)(σ, _1)=(0.03,450). Policies are evaluated using Monte Carlo trajectories at each of the 9 parameter points 0.01,…,0.05×300,…,600\0.01,…,0.05\×\300,…,600\, which include in-distribution parameters as well as a few out of distribution market parameters for stress testing. We report the mean objective within each regime, which is illustrated in Figure 6. We note that the policy of domain-randomized agent uses a feedforward method. An alternative approach would be to implement a recurrent policy that learns to infer dynamics from observation and action history, as is done in (Peng et al., 2018). Figure 6. Evaluation of the agents under different market regimes. The left panel shows the periodic-rebalancing baseline agent, the middle panel shows the nominal PPO agent, trained in the market domain ((σ,α1)=(0.03,450))((σ, _1)=(0.03,450)) and the right panel shows the domain-randomized agent, trained on market domains sampled from the distribution (ρθ)( _θ). The evaluation covers both in-distribution market parameters (the dashed area) and out-of-distribution market parameters. Conclusion and Further Research In this paper, we introduce SAiFE_gym, a collection of RL environments for solving trading problems in constant product markets with concentrated liquidity. The goal was to assemble the components of the AMM microstructure in a modular way, whereby researchers and practitioners can create their desired economic settings. We demonstrate the versatility of SAiFE_gym through experiments under different economic settings. The vectorized design of the environments makes these experiments computationally tractable and allows policies to be evaluated across many trajectories under controlled variations of the underlying market model. While SAiFE_gym provides a solid foundation for research, there are several natural directions for future work. A first extension is to develop multi-agent RL environments. In practice, liquidity provision is strategic, and, unlike market making in limit order books, it is highly transparent. Hence, multiple LPs compete for order flow by adjusting to common market signals and to other LPs’ actions (Baggiani et al., 2026). A second direction is to further expand the action space of LPs in our gym environment. In particular, our implementation is forward-compatible, allowing LPs to set dynamic trading fees (Baggiani et al., 2025). Last but not least, we invite the community to implement additional agents from the LT perspective. Our SpeedControlArbitrageurAgent is a first building block towards this direction. Such extensions would make it possible to study settings that interpolate between the model-based environments developed in this work and more agent-based simulations, in which market dynamics emerge from the interaction of heterogeneous LPs, arbitrageurs, and noise traders rather than from exogenously specified stochastic processes. References Adams et al. (2020) H. Adams, N. Zinsmeister, and D. RobinsonUniswap v2 core(Website) External Links: Link Cited by: §2. Adams et al. (2021) H. Adams, N. Zinsmeister, M. Saleh, R. Keefer, and D. Robinson Uniswap v3 whitepaper. Cited by: §2. Allen et al. (2026) S. Allen, P. Anchuri, J. Austgen, M. Bahrani, S. Breckenridge, A. Buchwald, C. Cachin, A. Fábrega, J. Fernandez, J. H. Chiang, et al. Crypto x ai, ai x crypto: a survey. arXiv:2606.13892. Cited by: §1. Amrouni et al. (2022) S. Amrouni, A. Moulin, J. Vann, S. Vyetrenko, T. Balch, and M. Veloso ABIDES-gym: gym environments for multi-agent discrete event simulation and application to financial markets. In Proceedings of the Second ACM International Conference on AI in Finance, ICAIF ’21. External Links: ISBN 9781450391481, Document Cited by: §2. Angeris and Chitra (2020) G. Angeris and T. Chitra Improved price oracles: constant function market makers. In Proceedings of the 2nd ACM AFT Conference, AFT ’20, p. 80–91. External Links: ISBN 9781450381390, Document Cited by: §1, §2. Angeris et al. (2022) G. Angeris, A. Evans, T. Chitra, and S. Boyd Optimal routing for constant function market makers. In Proceedings of the 23rd ACM Conference on Economics and Computation, EC ’22, p. 115–128. External Links: ISBN 9781450391504, Document Cited by: §1. Aqsha et al. (2026) A. Aqsha, P. Bergault, and L. Sánchez-Betancourt Equilibrium reward for liquidity providers in automated market makers. Mathematical Finance, to appear, arXiv:2503.22502. Cited by: §3.3, §3.3. Baggiani et al. (2025) L. Baggiani, M. Herdegen, and L. Sánchez-Betancourt Optimal dynamic fees in automated market makers. arXiv:2506.02869. Cited by: Conclusion and Further Research. Baggiani et al. (2026) L. Baggiani, M. Herdegen, and L. Sanchez-Betancourt Competition between dexs through dynamic fees. arXiv:2603.09669. Cited by: Conclusion and Further Research. Cartea et al. (2023) Á. Cartea, F. Drissi, and M. Monga Predictable losses of liquidity provision in constant function markets and concentrated liquidity markets. Applied Mathematical Finance 30, p. 69–93. Cited by: §2. Cartea et al. (2024a) Á. Cartea, F. Drissi, and M. Monga Decentralized finance and automated market making: predictable loss and optimal liquidity provision. SIAM Journal on Financial Mathematics 15 (3), p. 931–959. Cited by: §1, §2. Cartea et al. (2017) Á. Cartea, R. Donnelly, and S. Jaimungal Algorithmic trading with model uncertainty. SIAM Journal on Financial Mathematics 8 (1), p. 635–671. External Links: Document Cited by: §3.7. Cartea et al. (2025) Á. Cartea, F. Drissi, and M. Monga Decentralised finance and automated market making: execution and speculation. Journal of Economic Dynamics and Control 177, p. 105134. External Links: ISSN 0165-1889 Cited by: §1, footnote 4. Cartea et al. (2024b) Á. Cartea, F. Drissi, L. Sánchez-Betancourt, D. Siska, and L. Szpruch Strategic bonding curves in automated market makers. Available at SSRN 5018420. Cited by: footnote 1. Fan et al. (2022) Z. Fan, F. J. Marmolejo-Cossío, B. Altschuler, H. Sun, X. Wang, and D. Parkes Differential liquidity provision in uniswap v3 and implications for contract design. In Proceedings of the Third ACM International Conference on AI in Finance, p. 9–17. Cited by: §2, §3.4. Fan et al. (2023) Z. Fan, F. Marmolejo-Cossio, D. Moroz, M. Neuder, R. Rao, and D. C. Parkes Strategic Liquidity Provision in Uniswap V3. In 5th ACM AFT Conference (AFT 2023), Vol. 282, p. 25:1–25:22. Note: Keywords: blockchain, decentralized finance, Uniswap v3, liquidity provision, stochastic gradient descent External Links: ISBN 978-3-95977-303-4, ISSN 1868-8969, Document Cited by: §2, §3.4. Huang et al. (2015) W. Huang, C. Lehalle, and M. Rosenbaum Simulating and analyzing order book data: the queue-reactive model. Journal of the American Statistical Association 110 (509), p. 107–122. External Links: Document Cited by: footnote 2. Jerome et al. (2023) J. Jerome, L. Sánchez-Betancourt, R. Savani, and M. Herdegen Mbt-gym: reinforcement learning for model-based limit order book trading. In 4th ACM International Conference on AI in Finance, ICAIF 2023, Brooklyn, NY, USA, p. 619–627. External Links: Document Cited by: item 3, §2. Kumar (2020) P. Kumar Deep reinforcement learning for market making. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), p. 1892–1894. Cited by: §2. Mascioli et al. (2024) C. Mascioli, A. Gu, Y. Wang, M. Chakraborty, and M. Wellman A financial market simulation environment for trading agents using deep reinforcement learning. In Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF ’24, p. 117–125. External Links: ISBN 9798400710810, Document Cited by: §2. Peng et al. (2018) X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), p. 3803–3810. Cited by: item 4, §4.3. Raffin et al. (2021) A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann Stable-baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22 (1). External Links: ISSN 1532-4435 Cited by: item 2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §4.3. Spooner et al. (2018) T. Spooner, J. Fearnley, R. Savani, and A. Koukorinis Market making via reinforcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’18, Richland, SC, p. 434–442. Cited by: §2. Stoikov and Sağlam (2009) S. Stoikov and M. Sağlam Option market making under inventory risk. Review of Derivatives Research 12, p. 55–79. External Links: Document Cited by: footnote 1. Uniswap Labs (2026) Uniswap LabsUniswap powers ~31% of MetaMask swaps(Website) External Links: Link Cited by: §1. van der Walt et al. (2011) S. van der Walt, S. C. Colbert, and G. Varoquaux The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering 13 (2), p. 22–30. External Links: ISSN 1558-366X, Link, Document Cited by: §4.2.