Paper deep dive

Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix

Shree Murthy, Rohan Pandey

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 26

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/9/2026, 1:25:38 AM

Summary

This paper investigates two reproducible failure modes in deep multi-agent reinforcement learning applied to continuous-time pricing markets: tacit cartel formation (FM1) among DDPG agents in synchronous settings, and critic instability (FM2) at high event rates. Using a CT-MARL benchmark, the authors quantify collusion via a scalar index and trajectory-level trace diagnostics. A partial microstructure fix leveraging Poisson-clocked asynchrony and observation latency reduces FM1 collusion by 48–59%, though the mitigation remains partial, non-monotonic in latency, and ineffective against FM2. The findings highlight implications for antitrust policy and algorithmic pricing regulation.

Entities (9)

Collusion Index (Δ) · metric · 95%DDPG Agents · agent-architecture · 95%Deep Multi-Agent Reinforcement Learning (MARL) · algorithm · 95%Failure Mode 1 (FM1) · phenomenon · 95%Failure Mode 2 (FM2) · phenomenon · 95%CT-MARL Benchmark · framework · 90%Event Rate (λ) · environment-parameter · 90%Observation Latency (δ) · environment-parameter · 90%Poisson-clocked Asynchrony · environment-parameter · 90%

Relation Signals (8)

DDPG Agents → exhibits → Failure Mode 1 (FM1)

confidence 95% · synchronous DDPG agents reliably trigger Failure Mode 1 with collusion index

High Event Rate (λ=5) → triggers → Failure Mode 2 (FM2)

confidence 95% · DDPG critic divergence at λ = 5

CT-MARL Benchmark → evaluates → Failure Mode 1 (FM1)

confidence 90% · We quantify both failure modes on a single benchmark

Poisson-clocked Asynchrony → mitigates → Failure Mode 1 (FM1)

confidence 90% · asynchrony alone cuts collusion by 48%

Microstructure Fix → partiallyaddresses → Failure Mode 1 (FM1)

confidence 90% · quantify a partial microstructure fix: asynchrony alone cuts collusion by 48%

Synchronous Environment → triggers → Failure Mode 1 (FM1)

confidence 90% · Calvano-style synchronous environment: Failure Mode 1 is reliably triggered

Failure Mode 2 (FM2) → corrupts → Phase-diagram cell (λ=5, δ=1)

confidence 85% · corrupts the phase-diagram cell at (λ=5, δ=1)

Observation Latency (δ) → modulates → Collusion Index (Δ)

confidence 85% · adding latency drives it to a minimum of ∆ = 0.28

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We study two reproducible failure modes of deep multi-agent reinforcement learning in continuous-time pricing markets: (i) tacit cartel formation between competing DDPG agents, and (ii) actor--critic instability at high event rates. We instantiate both inside a single CT-MARL benchmark (Poisson-clocked price updates, observation latency $\delta$, interior-optimum logit demand), show that synchronous DDPG agents reliably trigger Failure Mode 1 with collusion index $\Delta = 0.69 \pm 0.11$, and quantify a partial microstructure fix: asynchrony alone cuts collusion by 48\% and adding latency drives it to a minimum of $\Delta = 0.28$. The fix has clearly documented costs: it is partial ($\Delta$ remains supra-Bertrand), it is non-monotone in $\delta$, and it does not survive Failure Mode 2, which emerges as DDPG critic divergence at $\lambda = 5$ and corrupts the phase-diagram cell at $(\lambda{=}5, \delta{=}1)$. We accompany the scalar collusion index with trajectory-level trace diagnostics that expose the within-episode signalling collapse and the post-shock non-recovery.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

26,122 characters extracted from source content.

Expand or collapse full text

Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix Shree Murthy 1 * Rohan Pandey 1 * Abstract We study two reproducible failure modes of deep multi-agent reinforcement learning in continuous- time pricing markets: (i) tacit cartel formation between competing DDPG agents, and (i) actor– critic instability at high event rates.We in- stantiate both inside a single CT-MARL bench- mark (Poisson-clocked price updates, observa- tion latencyδ, interior-optimum logit demand), show that synchronous DDPG agents reliably trig- ger Failure Mode 1 with collusion index∆ = 0.69 ± 0.11, and quantify a partial microstruc- ture fix: asynchrony alone cuts collusion by 48% and adding latency drives it to a minimum of ∆ = 0.28. The fix has clearly documented costs: it is partial (∆remains supra-Bertrand), it is non- monotone inδ, and it does not survive Failure Mode 2, which emerges as DDPG critic diver- gence atλ = 5and corrupts the phase-diagram cell at(λ=5,δ=1). We accompany the scalar collusion index with trajectory-level trace diag- nostics that expose the within-episode signalling collapse and the post-shock non-recovery. 1. Introduction Multi-agent reinforcement learning (MARL) systems are increasingly being deployed in mixed-motive economic set- tings where the boundary between “competitive equilib- rium” and “coordinated harm” is not a property of the algo- rithm, but an emergent property of the environment that the agents learn in. Calvano et al. (2020) showed that tabular Q- learning agents charging prices in a synchronous Bertrand oligopoly converge to supra-competitive prices sustained by reward-and-punishment strategies without explicit com- munication. Subsequent work has extended this to deep RL methods (Schlechtinger et al., 2024; Deng et al., 2024), alternative agent architectures including LLMs (Fish et al., * Equal contribution 1 DigitalOcean, USA. Correspondence to: Shree Murthy<smurthy@digitalocean.com>, Rohan Pandey <rpandey@digitalocean.com>. Preprint. June 10, 2026. 2024), and macroeconomic shocks (Tinoco et al., 2025). Surveys identify “asynchronous timing, communication la- tency, and event-driven pricing” as understudied robustness conditions (Bichler et al., 2025; Deng et al., 2024). We adopt the failure-mode lens of recent agentic-AI eval- uation work: a failure mode is a reproducible behaviour, elicited by a small number of named environment and agent factors, with a verifiable trace and a partially understood cost-aware mitigation. The substantive contribution of this paper is to characterise two such failure modes in CT-MARL pricing: Failure Mode 1 (FM1): tacit cartel formation. Two DDPG agents trained in a continuous-time Bertrand duopoly converge to prices well above the Bertrand–Nash equilib- rium. The mode is robust across seeds and is detectable from a single scalar (the collusion index∆) plus a visible plateau in the per-episode price trace. Failure Mode 2 (FM2): critic instability at high event rate. Atλ = 5Poisson-clocked events per agent per unit time, the DDPG critic loses stability before the buffer accu- mulates enough trajectory diversity, and the policy diverges (mean prices spiralling above the monopoly level on a subset of seeds). The mode is reproducible on the corresponding (λ,δ) phase-diagram cell. We quantify both failure modes on a single benchmark, sup- ply trace-level diagnostics for each, and study a candidate mitigation for FM1 (continuous-time market microstruc- ture: Poisson asynchrony plus observation latency) under which collusion drops by 48–59%. The fix is partial:∆ remains substantially above Bertrand–Nash, the effect is non-monotone inδ, and it does not address FM2. We ar- gue this is the kind of result the failure-mode framing is designed to surface: a precisely characterised, reproducibly triggered behaviour; a verifiable trace; a fix with bounded but documented effect. Contributions. •A self-contained, reproducible CT-MARL pricing benchmark with interior-optimum logit demand, Poisson-clocked agents, observation latency, and an 1 arXiv:2606.09884v1 [cs.MA] 3 Jun 2026 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing optional transient demand shock. •A semi-MDP DDPG agent that uses the continuous- time discountγ τ = e −ρτ over the agent’s own sojourn timeτ, extending standard DDPG to the semi-MDP setting with a continuous pricing action space. • A 16-cell(λ,δ)phase diagram of∆, plus eight head- line conditions with five seeds each, isolating FM1 and FM2. • Trajectory-level trace diagnostics complementing the scalar collusion index, including a stress-condition tra- jectory that exposes post-shock non-recovery. • A partial mitigation of FM1 with cost-of-fix account- ing. 2. Related Work The foundational result is Calvano et al. (2020): tabular Q-learning agents in a synchronous Bertrand–Edgeworth oligopoly learn supra-competitive strategies. Klein (2021) shows the result is robust to sequential moves. Schlechtinger et al. (2024) and Deng et al. (2024) extend to deep RL methods (PPO, DQN, DDPG) and find generally weaker but still super-competitive collusion. Paudel & Das (2024) apply the same setup to EV-charging pricing. Tinoco et al. (2025) show that inflation shocks reshape but do not eliminate collusion; Fish et al. (2024) replicate the phenomenon with LLM agents. The methodological tools we use sit at the intersection of CT-MARL and asynchronous decision-making. Wang et al. (2026) introduce continuous-time value iteration in a general MARL setting. Du et al. (2020) formalise sojourn-time decisions as semi-MDPs solved with neural ODEs. Xiao et al. (2025) construct macro-action MARL that re-decides on event triggers in cooperative robotics. Sutton et al. (1999) provide the classical bridge between MDPs and semi-MDPs that we adopt. To our knowledge, none of these threads has been applied to the algorithmic-collusion question, and none has documented FM2-style critic instability in the high-event-rate regime. 3. Background Continuous-time duopoly. Two firmsi ∈ 1, 2post pricesp i ∈ [p,p]in real timet ∈ [0,T max ]. The instanta- neous logit demand for firm i at (p 1 ,p 2 ) is q i (p i ,p −i ) = e (1−p i )/μ 1 + e (1−p i )/μ + e (1−p −i )/μ ,(1) with marginal costc = 0. Profit per unit time isπ i = (p i − c)q i . We setμ = 0.25, which yields a symmetric Bertrand– Nash pricep BN ≈ 0.473and a symmetric joint-monopoly pricep M ≈ 0.925, both strictly interior in the action space [0, 2]. This rules out the “hit-the-ceiling” artefact in which collusion is bounded by an arbitrary action-space upper bound. Poisson clocks and latency. Each firmihas an indepen- dent Poisson event clock with rateλ i . Between events its price is held constant. Each firm observes the rival’s price with delayδ ≥ 0:ˆp −i (t) = p −i (t− δ). A demand shock multipliesq i byσ shock ∈ (0, 1)during an interval [t s ,t s + ∆ s ]. Semi-MDP formulation.The decision process for firmi is a semi-MDP (Sutton et al., 1999). At each eventk, the agent observess k i , chooses actiona k i = p i , and receives the integrated rewardR k i = R t k+1 i t k i π i (t)dt over its sojourn τ = t k+1 i −t k i . The continuous-time discount isγ τ = e −ρτ . Failure-mode metric: collusion index. For policies in- ducing average prices ̄p 1 , ̄p 2 over the evaluation window, with ̄p = ( ̄p 1 + ̄p 2 )/2, define ∆ = ̄p− p BN p M − p BN .(2) ∆ = 0 corresponds to the Bertrand–Nash equilibrium and ∆ = 1to perfect collusion at the joint monopoly price. 1 We treat any∆≳ 0.5in expectation across seeds as a positive trigger of Failure Mode 1. 4. Method Agent architecture.Each agent is a semi-MDP Deep De- terministic Policy Gradient (DDPG) agent (Lillicrap et al., 2016) with a continuous price actiona ∈ [0, 2]. The ac- torπ θ : S → [p min ,p max ]is a two-layer MLP (64 units, ReLU) with atanhoutput scaled to[p min ,p max ]. The critic Q ω (s,a)is a two-layer MLP (64 units, ReLU) that takes the concatenation(s,a). Both networks have separate target networks (θ ′ ,ω ′ ) soft-updated at rate τ soft = 0.005. Semi-MDP update.At each pricing event the agent stores the transition(s,a,R,s ′ ,τ )in a replay buffer of capacity 50,000. After each event, a mini-batch of 128 transitions is sampled every four steps and the critic is updated by minimising L(ω) =E (Q ω (s,a)− y) 2 ,(3) y = R + e −ρτ Q ω ′ (s ′ ,π θ ′ (s ′ )),(4) 1 We define∆in price space. Calvano et al. (2020) use a profit- space index∆ π = ( ̄π − π BN )/(π M − π BN ). Under symmetric play the two are ordinally equivalent (profit is monotone increasing in price on[p BN ,p M ]) but numerically distinct: for this demand specification,∆ π runs approximately0.15–0.20higher than∆ across all conditions. 2 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing where the discountγ τ = e −ρτ is sojourn-dependent, imple- menting the semi-MDP discounting from Section 3. The actor is updated by deterministic policy gradient: ∇ θ J ≈E −∇ a Q ω (s,a) a=π θ (s) ·∇ θ π θ (s) .(5) Exploration uses Gaussian noiseN (0,σ 2 )added to the deterministic action, withσdecaying from0.20to0.02 at rate0.9998per step. Gradient norms are clipped to1.0; we use Adam optimisers with learning rates10 −4 (actor) and10 −3 (critic), andρ = 0.05. The integrated rewardRis computed by 5-point trapezoidal integration ofπ i over the agent’s sojourn. Algorithm 1 CT-MARL-DDPG (one episode, one agent shown). Require: env(T max ,λ,δ); DDPG params(θ,ω,θ ′ ,ω ′ ); bufferB Reset env; s prev [i]← None while t < T max do (i,s,R,done)← env.step tonextevent() if s prev [i]̸= None then τ ← t− t prev [i] B.push(s prev [i],a prev [i],R,s,τ ) if|B|≥ B and step≡ 0 (mod 4) then Sample mini-batch(s j ,a j ,R j ,s ′ j ,τ j ) γ j ← e −ρτ j y j ← R j + γ j Q ω ′ (s ′ j ,π θ ′ (s ′ j )) Critic step: minimise P j (Q ω (s j ,a j )− y j ) 2 Actor step: minimise− P j Q ω (s j ,π θ (s j )) Soft-update θ ′ ,ω ′ end if end if if not done then a← π θ (s) +N (0,σ 2 ), clipped env.apply action(i,a) s prev [i]← s; a prev [i]← a; t prev [i]← t end if end while Trace diagnostic.We report per-episode price trajectories alongside the scalar∆. Trace plots reveal mechanism (when in the episode the collusive plateau forms, whether prices recover after a shock) that final-window averages necessarily collapse. We use them in Section 5 to characterise both FM1 and the stress-condition trajectory. 5. Experiments Conditions.Eight headline conditions plus a4× 4 (λ,δ) phase-diagram sweep: S0 (synchronous,∆t = 0.5); A0– A3 (async withδ ∈ 0, 0.5, 1.0, 2.0,λ = 1); AR (slow, λ = 0.5,δ = 1); AF (fast,λ = 5,δ = 1); ST (async, λ = 1,δ = 1, demand shock0.4×overt ∈ [25, 35]). All conditions:T max = 60, 300 episodes per seed, five seeds per main condition, three seeds per phase-diagram cell. We report ̄p 1 , ̄p 2 averaged over the last 40 episodes. Baselines.The static Bertrand–Nash and Monopoly price constants supply the∆scale; we additionally report the synchronous-only baseline S0 to anchor the discrete-time, Calvano-style result. 5.1. FM1 Trigger and a Partial Microstructure Fix Table 1 summarises the seven stable main conditions (AF discussed separately in Section 5.3). The synchronous base- line S0 achieves∆ = 0.69± 0.11, confirming that DDPG agents learn strong tacit collusion in the Calvano-style syn- chronous environment: Failure Mode 1 is reliably triggered, providing the upper anchor for the ∆ scale. Switching to Poisson-clocked asynchrony alone (A0,δ = 0) reduces collusion to∆ = 0.36 ± 0.08, a 48% reduc- tion. Paired across seeds,∆ S0 − ∆ A0 = 0.33(95% CI [0.14, 0.53],t(4) = 4.68,p < 0.01). Increasing observa- tion latency fromδ = 0toδ = 1.0pushes collusion to its minimum:∆ A2 = 0.28± 0.09. The cumulative S0 to A2 effect is a 59% reduction (∆ S0 − ∆ A2 = 0.41, 95% CI [0.19, 0.64],t(4) = 5.07,p < 0.01). Table 2 reports all six paired contrasts; every one is significant atp < 0.01. At very high latency (δ = 2.0, condition A3),∆rebounds slightly to0.30, suggesting non-monotonicity at extreme delay that warrants further study (Section 6). Table 1. Headline results across 5 seeds per condition (AF ex- cluded; see Section 5.3).∆ = 0denotes Bertrand–Nash,∆ = 1 denotes joint monopoly. The S0→A0 (48%) and S0→A2 (59%) drops are both significant at p < 0.01 on a paired t-test. Condition ̄p∆Reward S0 (sync)0.7850.69± 0.1118.87 A0 (λ=1, δ=0)0.6350.36± 0.0816.50 A1 (λ=1, δ=0.5)0.6190.32± 0.0915.97 A2 (λ=1, δ=1.0)0.5990.28± 0.0915.74 A3 (λ=1, δ=2.0)0.6090.30± 0.1015.93 AR (slow, λ=0.5)0.6250.34± 0.0916.25 ST (shock)0.6250.34± 0.0714.56 Table 2. Pairedt-tests, S0 vs. each CT condition (n = 5seeds). Positive values indicate that S0 achieves higher collusion. Comparison∆ S0 −∆ cond t(4)p S0 vs. A0 (δ=0)0.3324.680.009 S0 vs. A1 (δ=0.5)0.3696.150.004 S0 vs. A2 (δ=1.0)0.4125.070.007 S0 vs. A3 (δ=2.0)0.3905.070.007 S0 vs. AR (λ=0.5)0.3564.730.009 S0 vs. ST (shock)0.3544.770.009 3 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing 5.2. FM1 Trace Diagnostic The collusion index∆compresses an entire training trajec- tory into a single scalar; the underlying mechanism is more legible in the trace itself. Figure 1a plots mean episode price for S0, A2, A3, and ST. S0 climbs steadily and plateaus near the monopoly level by episode 200, the textbook signature of an emergent reward-and-punishment policy. The CT con- ditions plateau lower and flatter: the policy finds a stable interior price, but does not reach the monopoly attractor. The shock condition (Figure 1d) shows a different trace sig- nature: prices fall during the shock windowt∈ [25, 35]and do not recover by episode end, indicating that the implicit signalling mechanism that sustains the collusive plateau is disrupted faster than the agents can re-bootstrap it. We treat trace plots like these as part of the trigger characterisation: two failure modes can produce similar end-of-episode∆ but qualitatively different trajectories. 5.3. FM2: Critic Instability at λ = 5 The high event-rate condition AF (λ = 5,δ = 1) produces mean∆ = 1.03with SD= 0.95, driven by two of five seeds in which DDPG diverged: prices spiralled above the monopoly level (∆ > 1.9). The remaining three seeds yield ∆ ≈ 0.35, consistent with other async conditions. We at- tribute the divergence to the rapid reward feedback at high λ: each agent receives many transitions per unit time and the off-policy critic updates become unstable before suf- ficient buffer diversity is accumulated. We label this as a distinct failure mode (FM2) rather than as noise on FM1: it has its own reproducible trigger (λ ≥ 5at the chosen learning rates), its own trace signature (monotone price escalation past the action-space midpoint), and its own can- didate fixes (TD3-style critic regularisation, slower replay- to-environment ratio, or actor-critic methods with explicit entropy regularisation). We exclude AF from the compara- tive statistics in Tables 1 and 2 and flag the corresponding phase-diagram cell(λ=5,δ=1)likewise; characterising and fixing FM2 is left for a follow-up paper. 5.4. Phase Diagram and Stress Condition Figure 1c shows the∆surface across(λ,δ). Forλ∈1, 2, increasingδgenerally reduces∆, with theλ = 2row show- ing the clearest monotone decline:0.40→ 0.37→ 0.34→ 0.21. The row minimum∆ = 0.21at(λ=2,δ=2)is the lowest stable cell. Atλ = 0.5(slow clocks) the trend is flat and non-monotone (0.35 → 0.30 → 0.36 → 0.32). Theλ = 5row contains one unstable FM2 cell (δ = 1.0, excluded). Across the 15 stable cells,∆ranges from0.21 to0.45, all well below the synchronous benchmark of0.69. The stress condition (ST) repeats A2 with a transient de- mand shock (σ shock = 0.4ont∈ [25, 35]). Episode-average ∆ ST = 0.34± 0.07is similar to A2; the trace-level effect is more interesting (Figure 1d): prices trim during the shock and stay trimmed for the rest of the episode, with mean per- agent reward dropping to14.56versus15.74for A2 (7.5% reduction directly attributable to the shock). 6. Discussion Why FM1 is suppressed by CT frictions. In the syn- chronous baseline, both agents update simultaneously, so one agent’s price rise is immediately observable and recip- rocable: the tit-for-tat structure that sustains tacit collusion. Under Poisson clocks, updates interleave: by the time agent jreprices in response toi’s move,imay have already moved again on stale information aboutj. Withδ > 0, even the state representation that the DDPG critic conditions on is misaligned with current market conditions. DDPG’s con- tinuous action space prevents the policy from accidentally landing on a coarse-grid cooperative attractor; the policy must explicitly coordinate in real-valued price space, which is more sensitive to informational disruption. The non-monotone rebound atδ = 2.The slight rise in ∆from A2 (δ = 1.0) to A3 (δ = 2.0) deserves comment. At very high latency, the observed rival price is so stale as to be nearly uninformative about current market conditions. Agents may retreat to a near-myopic strategy that happens to price conservatively near the interior optimum, producing moderate∆through inaction rather than coordination. This is distinct from the signal-based collusion in S0 and warrants theoretical analysis in future work; we note it here as a caveat to the “larger δ is always better” reading of the fix. Cost of the fix.The continuous-time microstructure miti- gation reduces FM1 collusion from∆ = 0.69to∆≈ 0.28 at the optimum(λ,δ) = (1, 1), a 41-percentage-point abso- lute reduction. The cost ledger: • Coverage cost. The fix is partial:∆ ≈ 0.28is well above the Bertrand–Nash level (∆ = 0), so under any reasonable threshold the residual collusion remains a market-design concern. Mitigation̸= removal. •Brittleness inδ. Pushingδfrom 1 to 2 raises∆back to 0.30. The mitigation is non-monotone in the parameter that a regulator could plausibly tune. •Brittleness inλ. The fix interacts with FM2:λ = 5 destabilises the critic and the resulting∆becomes uninformative. •Generalisation cost.We test only a symmetric duopoly. Real markets haveN ≥ 3firms and asym- metric clock rates; the experiments in Section 6 are needed before claiming the fix transfers. 4 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing •Implementation cost. Realising the fix requires the market designer to impose minimum-decision-interval style microstructure (analogous to speed bumps in HFT exchanges), a real intervention with its own efficiency costs that this paper does not measure. Implications for antitrust policy. If our partial fix gen- eralises, the gap between collusion achievable in idealised models (∆ ≈ 0.7) and in markets with even modest mi- crostructure (∆≈ 0.3) is large enough that regulators may be overstating cartel risk when calibrating against discrete- time benchmarks. That said,∆≈ 0.3remains meaningfully above competitive pricing, so CT frictions should be treated as a partial empirical mitigation, not an absolution. Limitations. • Duopoly only. Whether FM1 and the partial fix gener- alise toN ≥ 3firms is open. Discrete-time literature suggests collusion typically weakens with more com- petitors, in which case the fix may compound; an N=3 experiment is the most informative single follow-up. •Symmetric clocks.λ 1 = λ 2 throughout. Asymmetric clocks, e.g. a slow incumbent vs. a fast HFT entrant, may create exploitable structure and are an important real-world scenario. • FM2 left open. We characterise but do not fix the λ = 5critic instability; TD3 (Fujimoto et al., 2018) or SAC are natural candidates. •Five seeds per condition. Borderline; our paired tests rely on within-condition variance being substantially smaller than between-condition variance, which holds in the present data but warrants more seeds. • 300-episode budget. With more training, agents may discover stable collusive strategies under latency; the reported∆should be read as characterising conver- gence within a practical compute budget. 7. Conclusion We documented two reproducible failure modes of deep multi-agent RL in continuous-time pricing: tacit cartel formation among DDPG agents in the synchronous base- line (FM1), and DDPG critic instability at high event rates (FM2). FM1 admits a partial microstructure fix (Poisson- clocked asynchrony plus observation latency drops the collu- sion index by 48–59% relative to the synchronous baseline) that we accompany with explicit cost-of-fix accounting: the fix is partial, non-monotone inδ, and orthogonal to FM2. The 16-cell(λ,δ)phase diagram, the per-episode trace diag- nostics, and the stress-condition trajectory together provide a reproducible workbench on which subsequent CT-MARL methods can be evaluated and on which the open follow-ups (N > 2firms, asymmetric clocks, and FM2 mitigation) can be addressed. Impact Statement This paper studies failure modes of multi-agent reinforcement-learning agents in pricing markets. Two dimensions of impact warrant comment. First, the substantive claim of the paper (that continuous- time microstructure is a partial but non-trivial mitigant of emergent algorithmic collusion in deep MARL) has direct relevance to ongoing antitrust policy debate over algorithmic pricing. Our result should not be read as evidence that algorithmic collusion is benign in real markets: even under the strongest mitigation we find, the post-fix collusion index remains substantially above the competitive benchmark, and the fix is brittle in both microstructure parameters. We have framed the fix’s costs explicitly in Section 6 for this reason. Second, the experimental benchmark we release is intended as a tool for evaluating mitigation strategies (microstructure rules, representation choices, critic regularisers). It is not a validated model of any specific real-world pricing market, and conclusions drawn from it about specific industries or regulatory interventions require domain-specific extension and grounding. References Bichler, M., Durmann, J., and Oberlechner, M. Algorith- mic pricing and algorithmic collusion. arXiv preprint arXiv:2504.16592, 2025. Calvano, E., Calzolari, G., Denicol ` o, V., and Pastorello, S. Artificial intelligence, algorithmic pricing, and collusion. American Economic Review, 110(10):3267–3297, 2020. Deng, S., Schiffer, M., and Bichler, M. Algorithmic collu- sion in dynamic pricing with deep reinforcement learning. arXiv preprint arXiv:2406.02437, 2024. Du, J., Futoma, J., and Doshi-Velez, F. Model-based rein- forcement learning for semi-Markov decision processes with neural ODEs. In Advances in Neural Information Processing Systems (NeurIPS), 2020. Fish, S., Gonczarowski, Y. A., and Shorrer, R. I. Algorith- mic collusion by large language models. arXiv preprint arXiv:2404.00806, 2024. Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018. 5 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing Klein, T. Autonomous algorithmic collusion: Q-learning un- der sequential pricing. The RAND Journal of Economics, 52(3):538–558, 2021. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous con- trol with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. Paudel, D. and Das, T. K. Tacit algorithmic collusion in deep reinforcement learning guided price competition: A study using EV charge pricing game. arXiv preprint arXiv:2401.15108, 2024. Schlechtinger, M., Kosack, D., Krause, F., and Paulheim, H. By fair means or foul: Quantifying collusion in a market simulation with deep reinforcement learning. In Proceed- ings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), 2024. Sutton, R. S., Precup, D., and Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2): 181–211, 1999. Tinoco, S., Abeliuk, A., and Ruiz del Solar, J. Impact of price inflation on algorithmic collusion through reinforce- ment learning agents. arXiv preprint arXiv:2504.05335, 2025. Wang, X., Zhang, L., Pu, H., Qureshi, A. H., and Li, H. Continuous-time value iteration for multi-agent reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2026. Xiao, Y., Tan, W., Hoffman, J., Xia, T., and Amato, C. Asyn- chronous multi-agent deep reinforcement learning under partial observability. International Journal of Robotics Research, 2025. 6 Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing A. Result Figures 050100150200250300 Episode 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 Mean of agents' average prices (per episode) Learning dynamics across pricing regimes Sync (baseline) Async, = 1.0 Async, = 2.0 Async, = 1.0 + shock BertrandNash (0.47) Monopoly (0.93) (a) Learning dynamics (FM1 trace). Mean price per episode (±1SD across seeds). S0 climbs to the monopoly attractor; CT conditions plateau lower. S0 (sync) A0 ( =0) A1 ( =0.5) A2 ( =1) A3 ( =2) AR ( =0.5) ST (shock) 0.0 0.2 0.4 0.6 0.8 1.0 Collusion index Per-seed collusion index by condition (dots = individual seeds, bar = mean) BertrandNash (=0, p=0.47) Monopoly (=1, p=0.93) (b) Per-seed∆by condition. Dots = seeds, bar = mean. S0 (∆ = 0.69) is highest; all CT conditions substantially lower. 0.00.51.02.0 Observation latency (time units) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Collusion index Effect of latency on collusion across Poisson rates S0 sync baseline (=0.69) = 0.5 = 1.0 = 2.0 = 5.0 (unstable cell excl.) (c) Phase diagram.∆generally falls with latency;λ = 5,δ = 1 is excluded as FM2. 0102030405060 Continuous time t (final episode) 0.5 0.6 0.7 0.8 0.9 Posted price (mean across seeds) Stress-condition response: averaged across 5 seeds Agent 1 Agent 2 Demand shock BN (0.47) M (0.93) (d) Stress trace. Prices fall during the shock windowt∈ [25, 35]and do not recover. Figure 1. Empirical results. Trace-level diagnostics (a, d) complement the scalar collusion index (b) and the phase diagram (c). See Section 5. 7

Instant research discovery