Paper deep dive

Scaling Curriculum Learning For Autonomous Driving

Cevahir Koprulu, David Paz, Feng Tao, Yuliang Guo, Xinyu Huang, Ufuk Topcu, Liu Ren

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 100

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/30/2026, 2:42:58 AM

Summary

The paper introduces CL4AD, a framework integrating curriculum learning into batched autonomous driving simulators like GPUDRIVE. It addresses the sample inefficiency of domain randomization by using unsupervised environment design (UED) to adaptively prioritize training scenarios. CL4AD employs novel utility functions based on success rates (Learn-Hard), realism (GC-ADE, Act-MAE), and regret (AMGAE, PVL, MaxMC). Experiments show CL4AD achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77% and improving sample efficiency by 67% under limited compute.

Entities (12)

CL4AD · method · 98%GPUDRIVE · softwaresimulator · 95%Curriculum Learning · concept · 92%Domain Randomization · method · 90%Reinforcement Learning · concept · 90%WOMD · dataset · 90%Unsupervised Environment Design · concept · 88%Act-MAE · utility-function · 85%GC-ADE · utility-function · 85%Learn-Hard · utility-function · 85%Self-play · method · 85%PLR · algorithm · 80%

Relation Signals (10)

CL4AD → outperforms → Domain Randomization

confidence 95% · curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization

CL4AD → runson → GPUDRIVE

confidence 95% · We release an implementation of CL4AD in GPUDRIVE.

CL4AD → uses → Unsupervised Environment Design

confidence 95% · We present CL4AD... by framing scenario selection as an unsupervised environment design problem.

CL4AD → implements → PLR

confidence 90% · CL4AD integrates variants of a UED method, prioritized level replay (PLR)

CL4AD → improves → Sample Efficiency

confidence 90% · curriculum learning improves sample efficiency by 67%.

CL4AD → reduces → Wall-clock Time

confidence 90% · reducing wall-clock time by 77%

GPUDRIVE → uses → WOMD

confidence 90% · Agents trained on GPUDRIVE using the Waymo Open Motion Dataset (WOMD)

CL4AD → utilizes → Act-MAE

confidence 90% · CL4AD introduces... action mean absolute error (Act-MAE) U Act-MAE

CL4AD → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Batched simulators for autonomous driving have recently enabled training reinforcement learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We present CL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent's behavior, in addition to existing regret-estimation functions. Large-scale experiments in GPUDRIVE demonstrate that curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by 67%. We also investigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CLForAD in GPUDRIVE.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

97,069 characters extracted from source content.

Expand or collapse full text

Scaling Curriculum Learning For Autonomous Driving Cevahir Koprulu ∗ 1 , David Paz 2 , Feng Tao 2 , Yuliang Guo 2 , Xinyu Huang 2 Ufuk Topcu 1 , Liu Ren 2 1 The University of Texas at Austin, 2 Bosch Center for AI, North America Abstract Batched simulators for autonomous driving have recently enabled training reinforce- ment learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We presentCL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent’s behavior, in addition to existing regret-estimation functions. Large-scale experiments inGPUDRIVEdemonstrate that curriculum learning achieves a99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by67%. We also inves- tigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CL4AD in GPUDRIVE. 1 Introduction Batched simulators for autonomous driving (AD) have recently empowered sample-inefficient but effective reinforcement learning (RL) algorithms by enabling training for billions of interactions within days [10,22]. These simulators achieve such scale by training RL agents on hundreds to thousands of scenarios in parallel through self-play [38], where a single policy controls all vehicles, taking millions of actions per second. Agents trained onGPUDRIVE[22] using the Waymo Open Motion Dataset (WOMD) [14] generalize to unseen test cases in less than a day.GIGAFLOW[10], further scales self-play to 1.6 billion kilometers of simulated driving within 10 days, producing generalist driving policies that outperform benchmark-specific agents on CARLA [12], nuPlan [7], and Waymax [16] without any training on these benchmarks. Despite advances in high simulation throughput, training RL agents in batched driving simulators remains sample-inefficient due to a standard training strategy: uniform scenario sampling, i.e., domain randomization (DR). This approach wastes interactions on scenarios that are either too easy to provide a sufficient learning signal or too difficult for the current policy to make progress on. Curriculum learning (CL) offers a remedy by adaptively prioritizing scenarios that contribute the most to policy improvement [27]. In particular, curriculum learning has successfully fulfilled that promise in multiple large-scale RL domains. For example, [5] demonstrate that scaling meta-RL ∗ Corresponding author: cevahir.koprulu@utexas.edu Preprint. arXiv:2608.22549v1 [cs.AI] 23 Aug 2026 CL4AD New scenarios Scenario sampler Regret: Difference between learned and optimal behaviors Utility function Scenario scores Success: Consistency of the learned policy in reaching the goal Realism: Difference between learned behavior and logged trajectory UED Method Batched AD Simulator Observations Self-play agent Actions RL Algorithm Update policy PolicyRollouts Figure 1:CL4ADintegrates a UED method into a batched AD simulator to adaptively prioritize traffic scenarios based on three types of utility functions: regret, success, and realism. (blue) A self-play agent interacts with sampled scenarios across concurrent worlds. (yellow) An RL algorithm of choice updates the policy using collected rollouts in said scenarios. (green)CL4ADscores scenarios based on a chosen utility function and updates its sampler via the integrated UED method. with automated curricula yields agents capable of human-timescale adaptation across thousands of procedurally generated environments. [42] introduce curricula for open-ended environments with infinitely many tasks, showing that curriculum learning enables faster and broader skill acquisition. Inspired by the success of CL in large-scale RL, we introduce Curriculum Learning for Autonomous Driving (CL4AD) (see Figure 1), the first integration of automated curricula into a batched AD simulator. We frame scenario selection as an unsupervised environment design problem (UED), and equip a UED method with utility functions that adaptively shape training. Therefore, the curricula prioritize scenarios at the edge of the agent’s capabilities. Across experiments, we show that CL accelerates RL training by hundreds of millions of steps compared to DR and heuristic curricula. Our key contributions are three-fold: •We presentCL4AD, the first integration of curriculum learning methods from unsupervised environment design to the scale of GPU-accelerated, self-play simulators for AD, and provide an implementation in an open-source batched AD simulator, GPUDRIVE. •We propose novel utility functions based on realism, and a safety-critical success criterion that balances robustness and safety, exposing the gap between optimal and realistic driving. •We conduct a large-scale empirical study 1) showing thatCL4ADaccelerates RL training by up to a billion interactions compared to DR and outperforms heuristic curricula; 2) investigating how prioritized scenarios evolve during training; 3) analyzing the correlation among utility functions and performance metrics; 4) describing how multi-agent self-play creates a persistent learning frontier; 5) demonstrating that reward optimality and behavioral realism are distinct axes; and 6) providing practitioner guidelines for curriculum learning in batched AD simulators. 2 Related Work Autonomous driving simulators have enabled RL to train self-driving agents in multiple ways: Waymax [16] and Nocturne [41] use traffic scenarios from open-source driving datasets such as WOMD [14], whereas CARLA [13] is not data-driven, and Metadrive enables procedural scenario generation as well as integration of real driving data. Attempts to scale RL for AD have led to batched simulators such as Waymax,GPUDRIVE[22], andGIGAFLOW[10], which significantly increase data throughput for RL algorithms but still rely on random scenario generation/sampling. Curriculum learning for RL accelerates learning optimal policies by sequencing different configu- rations of the environment [27]. Automated curriculum generation studies goal-conditioned domains [4,15,40], contextual settings [23,24,34], and more popularly UED [11]. Across many settings, a curriculum requires a signal that measures the contribution of a task to policy improvement, e.g., learning progress, i.e., the change in an agent’s competence, which [30] estimates over a continuous parameter space and [21] measures as the change in success probability. UED methods can generate levels through trained teachers or by sampling free parameters. PAIRED [11] trains a teacher that 2 emits level parameters maximizing regret against an antagonist, and RE-PAIRED [18] stabilizes this game by replaying high-regret levels. ACCEL [29] instead mutates the parameters of high-regret levels in a replay buffer and curates the mutants that score highly. PLR [19], as one of the earlier UED approaches, neither trains a teacher nor mutates parameters; instead, it scores and replays levels it has already encountered. PLR has also shown evidence of scalability in meta RL [5,17], and open-ended environments [42]. Beyond individual methods, libraries such as DCD [18], minimax [20], and Syllabus [39] implement curricula for batched or asynchronous training, though they target procedurally generated environments outside AD. Curriculum learning for AD aims to speed up training self-driving policies via RL, e.g., ScenarioNet [25], which unifies heterogenous data for simulation, showcases benefits of heuristic-based curricula. Similarly, [2,3] propose a multi-stage curriculum learning method for CARLA, making the number of agents, their initial positions, or weather conditions incrementally more difficult. In contrast to manual curricula, [31] develops an automated method for urban intersections. Recently, [6] and [1] have demonstrated that UED methods, RE-PAIRED and ACCEL, respectively, accelerate training AD agents in CARLA. However, there has been no investigation into whether CL can scale with the high throughput and scenario diversity enabled by batched simulators such asGPUDRIVE, which trains RL agents on real-world scenarios from the Waymo Open Motion Dataset (WOMD) [14]. Other curriculum strategies for AD include adversarial scenario generation [43], continual policy adaptation with individualized curricula [28], and VLM-guided safety-critical curriculum design [37]. These works train single-ego agents on a smaller scale, in contrast to CL4AD, which enables training with real-world datasets across thousands of concurrent self-play agents in hundreds of parallel worlds. 3 Background We model a traffic scenario as a partially observable stochastic game (POSG) [6], to accommodate the multi-agent nature of driving. Then, we frame curriculum learning for autonomous driving as unsupervised environment design, and describe how to measure the utility of traffic scenarios. 3.1 Traffic scenarios as partially observable stochastic games Definition 3.1. A POSG is a tupleG = ⟨N,S,A,O,T,Z,R,I,γ⟩, whereN = [N]is the set of agents withN ∈ Z + ,Sis the state space,A = Q N i=1 A i andO = Q N i=1 O i are the joint action and observation spaces.T : S ×A → ∆(S)is the stochastic dynamics, i.e, the probability of transitioning from states ∈ Sto states ′ ∈ Sgiven joint actiona ∈ A.Z : S ×A → ∆(Z) determines the probability of observingo = (o 1 , o 2 ,· , o N )∈Oin statestaking joint actiona. R :S×A→ R N is the reward function, i.e.,R(s, a) = (R i (s, a)) i∈[N] whereR i (s, a)∈ Ris the reward for agenti∈N.I ∈ ∆(S)is the initial state distribution, andγ ∈ [0, 1]is the discount factor. A policyπ i : O i → ∆(A i )describes the behavior of agentiin POSGG. The value function forπ i is the expected cumulative discounted rewards over a horizon ofHsteps, i.e.,V (π i ) = E T,Z h P H−1 t=0 γ t R i (s t , a t )|s o ∼ I, a t = (a j,t ) j∈N i wherea j,t ∼ π j (o j,t ). Agentiaims to find an optimal policy π ∗ i , which maximizes its value V (π i ) in POSGG. In a traffic scenario modeled asG, considerπ i as a policy that controls vehiclei. The road layout, traffic rules, and collision dynamics in a scenario specify the dynamicsT. Initial states 0 ∼ Iconsists of the initial positions of all vehicles, pedestrians, cyclists, etc. Observationo i,t of vehicleiat time t∈ [H]is what the controller perceives about the surroundings based on its sensors as well as specific attributes, e.g., the type of vehicle, its velocity, acceleration, etc. The rewardr i = R i (s t , a t )can incentivize the policy to reach a goal location, stay within lanes, and avoid collisions. To model multiple traffic scenarios, we formalize them as an underspecified POSG (UPOSG). Definition 3.2. An underspecified POSGG Θ =⟨Θ,N Θ ,S,A Θ ,O Θ ,T Θ ,Z Θ ,R Θ ,I Θ ,γ⟩models a set of POSGs through parametersθ ∈ Θthat determine all attributes of a POSGθ ∈ Θdepending on its agents, such as the dynamics T Θ :S×A Θ × Θ→ ∆(S). Consider scenariosΘ = θ m m∈[M] in WOMD, whereM ≈ 100, 000. A scenarioθ m may corre- spond to an urban intersection or a highway, with varying speed limits, number of vehicles, etc. In practice, θ m is merely an identification number, i.e., θ m ∈ [M], hence it is underspecified. 3 3.2 Unsupervised Environment Design UED [11] aims to generate a sequence of levels 2 , i.e., scenariosθ ∈ Θin the case of AD, to accelerate learning a policy that generalizes across all levels. One solution to UED is a level generator Λ : Π→ ∆(Θ)that produces a distribution over the set of levelsΘgiven a policyπ ∈ Π. A level generator Λ maximizes a utility function U (π,θ) measuring the contribution of θ for improving π. Domain randomization, i.e., uniformly sampling levels throughout the training, is the default way of training an RL agent where the utility is constant for each level, namely,U (π,θ) = C∈ R,∀θ ∈ Θ. UED methods primarily differ in their utility functions of choice. There are two common categories of utility functions: regret and success-based. Regret Realism Success Novel Existing Utility functions and their derivations Figure 2: Utility functions:CL4ADadapts existing (a-d) utility functions as well as proposes (e-g) novel ones. Regret, i.e., the difference between the expected discounted return of the current policy and the optimal one, enables prioritizing the easiest levels that the agent cannot currently solve [11]. More formally, a regret-based utility isU Regret (π,θ) = V θ (π ∗ θ ) − V θ (π),whereπ ∗ θ is an optimal pol- icy in levelθ, i.e., a policy collecting the maximum expected discounted re- turnV θ (π ∗ θ ). However, as the optimal expected discounted return or the op- timal policy for each level is rarely available, UED methods estimate re- gret in various ways. [19] propose learning potential, i.e., average mag- nitude of the generalized advantage estimate (AMGAE) [35], (a) in Fig- ure 2, as a utility function that es- timates regret over a single episode. Here,δ k.n = r k,n +γV θ,π (o k+1,n )− V θ,π (o k,n )is the temporal difference error at timestepkof agentn∈ [N θ ]. V θ,π (o k,n ) = E T Θ ,Z Θ h P H−k−1 t=k γ t−k R Θ (s t,n , a t,n ,θ)|a t,n ∼ π(o t,n ) i is the expected discounted return of agentnins k on levelθ, andλis the discount factor for GAE. Alternatively, [18] and [29] employ positive value loss (PVL), (b) in Figure 2. As PVL uses the bootstrapped value target to compute the temporal difference error, [18] also propose maximum Monte Carlo (MaxMC), (c) in Figure 2, which instead utilizes the highest return obtained by agentnon levelθto mitigate potential bias issues, where R θ,n max is the maximum discounted return achieved in level θ so far during training. Success-based utility functions address settings where a levelθis considered solved for agentn when a policyπreaches a goal states∈S θ,n Goal ⊂S. Such utility functions use the success ratep θ,π , i.e., the fraction of controlled agents that reach a goal state in an episode of levelθunder policyπ, p θ,π = 1 N P n∈N 1[∃t∈ [H] : s t ∈S θ,n Goal ].Inspired by [40], [33] propose Sampling for Learnability (SFL), along with learnability, (d) in Figure 2, a utility function corresponding to the variance of a Bernoulli distribution with parameterp θ,π , namely, how inconsistently policyπsolvesθacross all controlled agents. [33] argues that, in sparse-reward settings, regret-based utility functions exhibit low correlation with success rates, as regret-based utility functions become noisy in such settings, causing inaccurate identification of the learning frontier. In AD, rewards commonly occur after sparse events, such as goal completion or collisions; thus, learnability becomes a viable alternative. 4 Curriculum Learning for Autonomous Driving at Scale CL4AD scales UED to batched AD simulators.CL4ADintegrates variants of a UED method, prioritized level replay (PLR) [19], which lays the foundation for approaches such as Robust PLR 2 As level is the common term in the UED literature to describeθ, we use it interchangeably with scenario. 4 Algorithm 1 Curriculum Learning for Autonomous Driving (CL4AD) Input: Set of training scenarios Θ train Parameters: Replay rated, Staleness coefficientρ, temperatureβ, utility functionU, max buffer sizeB max , total number of iterationsT train , scenario sampling intervalT sce , policy update intervalT pol , number of worlds W Output: Final policy π φ 1: B ← (),D ← () t← 0,l← 0, π φ ← π φ 0 ▷ Reset scenario/experience buffers, iterators, and policy 2: while t < T train do 3:if 0≡ t mod T sce then 4:l← l + 1▷ Increment sampling iteration 5:(θ w ) W w=1 ,B ← SAMPLEFROMCURRICULUM(B, Θ train ,l)▷ Sample scenarios for worlds 6:end if 7: D t =o n,w , a n,w , o ′ n,w ,r n,w ,e n,w n∈[N θ w ] w∈[W] ▷ Record experiences over a single step 8: B ← UPDATECURRICULUM(D t ,U,B) ▷ Update curriculum with the scores of terminated scenarios 9: D ←D∪D t ▷ Update experience buffer with new interactions 10:if 0≡ t mod T pol then 11:π,D ← Φ(D) ▷ Update self-play policy via RL algorithm Φ, and reset the experience bufferD 12:end if 13: t← t +|D t |▷ Update training iteration 14: end while [18], REPAIRED, ACCEL, and SFL.CL4ADscales them up for a batched AD simulator on four axes: (1) concurrent simulation of hundreds of scenarios, (2) tracking tens of agents per scenario, (3) training in tens of thousands of scenarios, and (4) for billions of steps. CL4AD efficiently scales via asynchronous episode management.CL4ADtracks the behavior of all controlled agents in all concurrent scenarios to compute their utility, which captures the expected collective behavior of the self-play policy. InGPUDRIVE, where we implementCL4AD, simulated scenarios come from real-world datasets, and each scenario has a specific horizonHdue to the nature of the logged data. Since scenarios have different horizons and agents terminate at different times, CL4ADprocesses episode terminations asynchronously: it computes the utility of a scenario as soon as all its agents terminate, updates the score by averaging over the episodes of that scenario since the last sampling call, and frees the rollout buffer for the next episode. This keeps memory usage proportional to the number of active episodes rather than the total number of episodes between curriculum sampling steps. Wall-clock measurements show that curriculum updates account for approximately 1% of total training time (see Section B). CL4AD operates over large training sets by adapting PLR’s sampling mechanism, which has two parts: uniformly sampling scenarios from the training setΘ train , and replaying levels from a rolling bufferB. At the beginning of the training, PLR uniformly randomly samples scenarios, and scores them based on a utility functionU. Then, it adds scenarios with the highest scores to its buffer. Subsequently, PLR makes a random decision with probabilitydto sample unseen levels inΘ train or seen levels from the buffer via a distribution based on their scores and staleness, namely, P replay (θ i |B,U,l) = (1− ρ)· P utility (θ i |B,U ) + ρ· P staleness (θ i |B,l),(1) whereP utility (θ i |B,U )emphasizes levels with higher ranks with respect to their scores, and P staleness (θ i |B,l)assigns a higher likelihood for levels that have not been sampled for longer (see Section B). This distribution aims to prevent the scores of seen levels from becoming off-policy, as they may remain in the buffer for a while without being sampled. CL4ADintroduces three novel utility functions, (e-g) in Figure 2: learnability-hardU Learn-hard , goal-conditioned average distance error (GC-ADE)U GC-ADE , and action mean absolute error (Act- MAE)U Act-MAE .U Learn-hard is a success-based utility function that, in contrast toU Learn , utilizes the rate at which agents reach their goals without colliding or going off-road in scenarioθvia self-play policyπ. Since episodes do not terminate on collision or off-road events, an agent can complete its goal unsafely, whichU Learn counts as a success andU Learn-hard does not.U Learn-hard therefore applies learnability to a success criterion that AD works use, as it captures both robustness and safety [10]. U GC-ADE andU Act-MAE are realism-based utility functions that compute the distance between the positions and actions of RL agents and the logged trajectories, respectively, evaluating the plausibility of behavior [7, 16, 8]. 5 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.5 5.0 7.5 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.5 5.0 7.5 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (a) Evaluation in training scenarios 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.8 0.9 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (b) Evaluation in unseen scenarios 0.20.61.01.41.8 Number of policy updates (×10 3 ) 0 50 100 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (c) Realism progression in unseen scenarios 0.21.01.82.63.4 Number of policy updates (×10 3 ) 0.0 0.1 0.2 Learnability-Hard 0.21.01.82.63.4 Number of policy updates (×10 3 ) 0.0 0.1 0.2 Learnability Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (d) Learnability progression in training scenarios Figure 3: Case 1: Training with 1000 scenarios. (top) We demonstrate performance progression in (a) training, and (b) test partitions, where bold markers indicate the mean and the shaded area covers one standard deviation across three independent runs. (bottom) Box plots illustrate quartiles of two utility functions: (c) U GC-ADE and (d) U Learn progression in test/training splits. Algorithm 1 illustratesCL4AD. At the beginning of the training, we initialize self-play policyπ φ , and reset scenario and experience buffersBandD, as well as the training and scenario sampling iterations,tandl, respectively (Line 1). Until training iteration reachesT train ,CL4ADfirst checks if it is time to sample new scenarios based on its replay bufferB(Line 3-5). If so, PLR samples new scenarios, andCL4ADsets them to concurrently simulated worlds. Note that PLR only keepsB max highest ranking scenarios in the buffer for sampling. Then, the self-play policyπ φ takes a step in all scenarios, andD t records them (Line 7).CL4ADupdates the curriculum buffer using the utility of terminated scenarios (Line 8). Finally, an RL algorithm updates the policy using the experience bufferD (Line 9-12) every T pol steps. We refer the reader to Section B for more details. 5 Experimental Results We implementCL4ADinGPUDRIVE[22] and conduct experiments using traffic scenarios from WOMD [14]. We assess performance via return and success rates, safety via collision and off- road rates, and realism via GC-ADE and metrics from the Waymo Open Sim Agents Challenge (WOSAC)[26]. We train RL agents using self-play PPO following [22,9], with sparse rewards for goal completion, collisions, and going off-road. Note that an episode does not terminate upon collision/off- road. Qualitatively, we analyze curriculum evolution, the correlation among utility functions and performance metrics, and how multi-agent self-play shapes the learning frontier. We compare DR against four heuristic curricula and seven PLR variants combined with the utility functions in Figure 2. Heuristic-Dense/Sparse prioritize scenarios with high/low vehicle counts; Heuristic-Fast/Far prioritize by descending mean velocity and pairwise inter-agent distance, respectively. Note that we adopt U PVL andU MaxMC from Robust PLR andU Learn from SFL, rather than the algorithms themselves. See Section C for more details on experiments. 5.1 Can CL4AD accelerate learning? PLR variants significantly accelerate learning, outperforming DR and heuristic curricula in both sample-efficiency and realism. Figures 3a and 3b shows that PLR, with all utility functions exceptU AMGAE , achieves the highest returns and success rates in training scenarios. Figure 5b evidences that, in test scenarios, PLR achieves99%success rate a billion steps earlier than DR, reducing wall-clock time by77%. Compared with Heuristic-Sparse/Dense, PLR accelerates training to achieve the same success rate by 40% and 66%, respectively. Note that PLR withU AMGAE outperforms DR with a small margin in terms of return. PLR also yields realistic policies faster than 6 1.01.52.02.53.03.54.04.55.0 Lag (checkpoint intervals) 1.0 0.5 0.0 0.5 1.0 Spearman correlation Rank persistence vs lag (top-100 intersection) PLR+Act-MAE PLR+AM-GAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 500100015002000250030003500 Number of policy updates 10 20 30 40 50 60 70 Cumulative unique scenarios in top-k Cumulative unique top-10 PLR+Act-MAE PLR+AM-GAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 500100015002000250030003500 Number of policy updates 2 4 6 8 10 12 14 16 Traffic density in top-k Traffic density top-10 PLR+Act-MAE PLR+AM-GAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Training set density (a) Score ranking evolution. 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scene IDs Scene 58 Scene 264 Scene 569 seed: 42 200600 10001400180022002600300034003800 Number of policy updates Scene 58 Scene 264 Scene 569 seed: 12 200600 10001400180022002600300034003800 Number of policy updates Scene 58 Scene 264 Scene 569 seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents Scene 58 Scene 264 Scene 569 seed: 42 200600 10001400180022002600300034003800 Number of policy updates Scene 58 Scene 264 Scene 569 seed: 12 200600 10001400180022002600300034003800 Number of policy updates Scene 58 Scene 264 Scene 569 seed: 67 Number of controlled agents: 19 Scene 58 Number of controlled agents: 15 Scene 264 Number of controlled agents: 5 Scene 569 0.0 0.2 0.4 0.6 0.8 1.0 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 Replay Probability ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Replay Probability ×10 1 (b) Progression of P replay for U MaxMC Figure 4: Case 1: How CL4AD guides scenario selection. (a) We analyze score rankings: (top) Spearman rank correlation at increasing lags (∆ = 1, 2, 5) for the top-100scenarios, where lower correlation indicates faster reshuffling; (middle) cumulative number of unique scenarios appearing in the top-10over training; (bottom) mean number of controlled agents in the top-10scenarios, with the dashed line indicating the training set average. (b)P replay progression of PLR withU MaxMC : (top) the evolution ofP replay , darker segments are scenarios with higher likelihood, (middle) replay distribution with respect to the number of controlled agents, (bottom) three frequently replayed scenarios. DR (see Figure 3c). Lastly, Figure 3d illustrates that approaches that converge early obtain high learnability early on and achieve the lowest learnability fastest at the end. 5.2 How does CL4AD guide scenario selection? CL4AD actively reshuffles score rankings, though the rate varies across utility functions. The rank persistence plot (top row in Figure 4a) shows the Spearman rank correlation between score rankings.U GC-ADE andU MaxMC are the most rank-persistent, maintaining correlations of0.4− 0.55 at∆ = 5. Other utility functions, particularly success-based andU Act-MAE , decorrelate more rapidly, reaching0.1− 0.25at∆ = 5. Cumulative number of unique scenarios (middle row) corroborates this: PLR withU GC-ADE andU MaxMC see29and34, respectively, unique scenarios enter their top-10 over the full training run, while success-based and regret-based functions cycle through 48− 70. Different utility functions favor qualitatively distinct instances. The progression of traffic density across the top-10 ranked scenarios (bottom row of Figure 4a) reveals that utility functions guide the curriculum to qualitatively distinct scenarios. Regret-based functions increasingly prioritize scenarios with traffic density above the dataset average. For example, Figure 4b shows the replay distribution progression of PLR withU MaxMC : dense scenarios such as Scenes58and264maintain high probability across runs, consistent with this mechanism. Success-based functions shift toward scenarios with low density as training progresses: scenarios with few controlled agents have the highest Bernoulli variance, since a single failure has a larger impact on the success rate. 5.3 Is CL4AD effective under limited compute? CL4ADremains effective under strict computational constraints, outperforming DR in time to success. Figure 5a illustrates results from an ablation study using a GPU with significantly smaller memory, reducing the number of worldsWand the experience bufferD(see Section D ). Although a 7 0.0 0.5 1.0 1.5 2.0 Number of interactions 1e8 95%96%97%98%99% Success rate 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Number of policy updates 1e3 Method DRPLR+Act-MAEPLR+LearnPLR+MaxMC (a) Ablation: 1,000 scenarios 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Number of interactions 1e9 95%96%97%98%99% Success rate 0.0 0.5 1.0 1.5 2.0 2.5 Number of policy updates 1e3 Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (b) Case 1: 1,000 scenarios 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Number of interactions 1e9 95%96%97%98%99% Success rate 0.0 0.5 1.0 1.5 2.0 2.5 Number of policy updates 1e3 Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL (c) Case 2: 10,000 scenarios 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Number of interactions 1e9 95%96%97%98%99% Success rate 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Number of policy updates 1e3 Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast (d) Case 3: 80,000 scenarios Figure 5: Success progression in unseen test scenarios: (a) ablation study under compute constraints, (b-d) training in 1,000, 10,000, and 80,000 scenarios, respectively. Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE pearson Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE spearman 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 (a) Case 1: 1,000 scenarios Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE pearson Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE spearman 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 (b) Case 2: 10,000 scenarios Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE pearson Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE spearman 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 (c) Case 3: 80,000 scenarios Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE pearson Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE Return Success Collision Off-Road AMGAE PVL MaxMC Learn-Hard Learn GC-ADE Act-MAE spearman 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Figure 6: Pearson correlation between utility functions and performance metrics, i.e, success, collision, and off-road rates: Results from training in (a) 1,000 (b) 10,000, and (c) 80,000 scenarios. smaller buffer results in more frequent policy updates, this setup takes about four times as long in wall-clock time and limits the diversity of scenarios used for updates. DR needs fewer interactions than the regular set-up, yet PLR reaches a 99% success rate 67% faster than DR. 5.4 How does CL4AD scale up the dataset? CL4AD continues to deliver sample-efficiency gains as the number of training scenarios increases to tens of thousands. We train self-play agents in (case 2) 10,000 and (case 3) 80,000 scenarios from WOMD. Figure 5c demonstrates that PLR reduces the number of interactions needed to reach a99%success rate by over55%, when combined withU MaxMC andU Act-MAE in case 2. Similarly, Figure 5d shows that PLR improves sample-efficiency by72%when combined withU Learn in case 3. Here, only Heuristic-Dense matches PLR, whereas the other heuristics offer no advantage. 5.5 Do utility functions correlate with each other and performance metrics? Figure 6 presents correlations between utility functions and performance metrics across all training cases. We include policies from multiple checkpoints to capture agents at varying stages of learning. 8 Regret-based functions differ in how they handle sparse rewards, and their approximations improve with dataset scale.U AMGAE takes the absolute value of GAE, amplifying collision and off-road penalties into high-utility signals since episodes do not terminate on collision/off-road events. This explainsU AMGAE ’s positive correlation with collision/off-road rates in Figure 6.U PVL discards negative TD errors via a ramp function, avoiding crash amplification, but its bootstrapped value targets are noisy early in training, explaining its lack of correlation with performance metrics.U MaxMC uses empirical returns, avoiding both issues. The correlation between regret-based functions increases as the training set grows, because more diverse experience improves value estimation, consistent with U AMGAE and U PVL performing better in case 2 than in case 1. Success-based functions target the learning frontier by construction.U Learn measures Bernoulli variancep(1−p), peaking atp = 0.5and prioritizing scenarios where the policy succeeds half of the time. The negative correlation with return and success in Figure 6 makes sense as the dominant signal comes from later training phases, where high success (p→ 1) corresponds to low learnability. The positive correlation with collision/off-road rates follows the same logic: scenarios where collisions still occur have intermediate success rates and hence higher Bernoulli variance.U Learn-hard counts only collision-free, on-road goal completions, making it a stricter measure of frontier scenarios. Realism-based utilities capture information orthogonal to performance.U GC-ADE andU Act-MAE are standard AD evaluation metrics repurposed as utility functions. They affect only scenario prioritization, whereas the reward function captures only goal completion and collision/off-road avoidance. Therefore, realism-based functions cannot steer behavior toward realism. This explains their lack of correlation with performance metrics in Figure 6.U GC-ADE andU Act-MAE do not correlate with each other, either, as trajectory divergence compounds over time, while action divergence does not. However, they still accelerate training relative to DR, because early in training, high behavioral divergence from logged trajectories serves as an implicit proxy for scenario difficulty. Once the policy reaches goals reliably, divergence reflects only the gap between reward-optimal and logged driving, which explains the lack of correlation across checkpoints. Cross-category combinations are viable. The correlation analysis suggests that pairs drawn from different categories carry complementary information. We combine two utility functions by ranking scenarios independently under each and sampling from the averaged rank. Section E.5 reports three such pairs in case 1, which converge with their individual components and exceed DR throughout, andU MaxMC +U Learn-hard sits at or above both of its components early in training. Combining a success-based function with a regret-based one therefore does not average their behavior. 5.6 How does multi-agent self-play affect curricula at scale? Multi-agent self-play creates a persistent learning frontier that is exaggerated at scale. As discussed in Section 4, the utility of a scenario reflects the collective performance of the self-play policy. When a policy update improves some agents, it changes their trajectories, creating new conflicts for others in the same scenario. This coupling keeps scenario utility elevated, as improving some agents continuously introduces new difficulty for others. Dense scenarios amplify this as more agents means more coupled interactions. At the scale of batched simulators, each policy update consumes over half a million interactions across hundreds of scenarios, causing large policy shifts that further exaggerate the disruption and rapidly make buffer scores off-policy. We address this via higher score temperatures (see Table 4), which spread sampling probability across more scenarios. 5.7 Does reward-optimal driving imply realistic driving? Reward optimality and behavioral realism can be orthogonal objectives.GPUDRIVE’s reward function incentivizes goal completion and penalizes collisions/off-road events, but does not account for realism, e.g., comfort, speed limit compliance, or smooth lane changes. An RL agent can therefore behave optimally yet not so realistically. We run an evaluation using WOSAC metrics to assess realism (see Section E.1). DR achieves higher map-based scores than PLR variants because WOSAC measures distributional similarity to logged behavior, and uniform sampling stays closer to the data distribution. PLR variants achieve lower displacement error, consistent with more efficient goal-reaching, but diverge from human-like driving. Curricula cannot steer towards realism, as utility functions affect only scenario prioritization, not the reward. Improving both performance and realism requires reward functions that explicitly incentivize realistic or penalize unrealistic behavior. 9 5.8 What guidelines can we derive for practitioners? A full-size replay buffer and a score temperature well above prior work perform best at this scale. A replay buffer equal to the training set works consistently as the dataset grows, in contrast to prior UED work, which uses buffers much smaller than the level space. Score temperatures above the typical range ofβ ∈ [0.1, 1.0]perform better, and higher values become preferable as the dataset scales, since low temperatures concentrate sampling on top-ranked scenarios. The staleness coefficient remains in the range used in prior work [19], as higher temperature already spreads sampling across more scenarios. In addition, Section E.4 varies the sampling interval and the buffer size in case 1, where all configurations reach the same performance and exceed DR throughout, though a small buffer carries higher collision and off-road rates early in training. On utility functions, no single choice dominates across scales, though patterns emerge.U MaxMC is the most reliable regret-based function, as it does not depend on value function accuracy, whereas U AMGAE andU PVL underperform in case 1 and improve as the dataset grows. Success-based functions are effective at every scale and strongest in case 3, where success variance identifies the learning frontier. Realism-based functions accelerate training over DR but cannot improve realism, since they affect prioritization rather than the reward. Functions within a category prioritize similar scenarios, especially as the dataset scales, with realism as the exception. See Section B.3 for details. 6 Conclusion In this work, we introduceCL4AD, the first integration of CL into batched AD simulators.CL4AD frames scenario selection as a UED problem, enabling adaptive prioritization of traffic scenarios via PLR [19] combined with utility functions that measure regret, success, and realism. Large-scale experiments onGPUDRIVEshow that CL achieves99%success, up to77%faster than domain randomization, and outperforms heuristic curricula, with one exception at the largest scale, where the same heuristic offers no advantage at smaller ones. Our analysis reveals that different utility functions prioritize qualitatively different scenarios at varying rates; utility functions within the same category may correlate, but across categories capture distinct information; multi-agent self-play creates a persistent learning frontier that scale amplifies; and reward optimality and behavioral realism can be distinct axes. We derive practitioner guidelines for applying CL in batched AD simulators. Limitations and future work.CL4ADevidences that CL scales up to the high-throughput of batched AD simulators. However,CL4ADis currently limited to variants of PLR and thus requires access to a real self-driving dataset as a source of traffic scenarios for sampling, e.g., WOMD, since GPUDRIVEoperates on predefined scenarios. To address these limitations, future work will explore UED methods such as ACCEL [29], which randomly mutates prioritized scenarios, hence increasing scenario diversity for training robust policies. Applying ACCEL to GPUDrive requires a mutation operator over scenarios, e.g., one that perturbs the initial positions and velocities of agents, inserts or removes agents, or edits their goals, while keeping the scene drivable and consistent with the road graph. PAIRED and RE-PAIRED instead require a teacher policy that emits such scenario parameters directly, which entails a generative scenario representation rather than the identification numbers GPUDrive currently operates with. In addition, synthetic scenario generation tools, e.g., Scenario Dreamer [32], can enableCL4ADto further accelerate training and improve the robustness and generalization capabilities of trained agents by creating safety-critical or out-of-distribution scenarios that the agent struggles with.CL4ADalso trains on all sampled scenarios, whereas Robust PLR suppresses gradient updates on newly sampled levels and reports improved zero-shot transfer under sharper prioritization than our score temperatures induce. Whether that restriction transfers to batched self-play remains untested. 10 References [1]Ahmed Abouelazm, Tim Weinstein, Tim Joseph, Philip Schörner, and J Marius Zöllner. Auto- matic curriculum learning for driving scenarios: Towards robust and efficient reinforcement learning. arXiv preprint arXiv:2505.08264, 2025. [2]Luca Anzalone, Silvio Barra, and Michele Nappi. Reinforced curriculum learning for au- tonomous driving in carla. In 2021 IEEE International Conference on Image Processing (ICIP), pages 3318–3322. IEEE, 2021. [3]Luca Anzalone, Paola Barra, Silvio Barra, Aniello Castiglione, and Michele Nappi. An end-to- end curriculum learning approach for autonomous driving scenarios. IEEE Transactions on Intelligent Transportation Systems, 23(10):19817–19826, 2022. [4]Adrien Baranes and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1766–1773. IEEE, 2010. [5]Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, et al. Human- timescale adaptation in an open-ended task space. In International Conference on Machine Learning, pages 1887–1935. PMLR, 2023. [6] Axel Brunnbauer, Luigi Berducci, Peter Priller, Dejan Nickovic, and Radu Grosu. Scenario-based curriculum generation for multi-agent autonomous driving. arXiv preprint arXiv:2403.17805, 2024. [7]Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810, 2021. [8]Daphne Cornelisse and Eugene Vinitsky. Human-compatible driving agents through data- regularized self-play reinforcement learning. In Reinforcement Learning Conference, 2024. [9] Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, and Eugene Vinitsky. Building reliable sim driving agents by scaling self-play. arXiv preprint arXiv:2502.14706, 2025. [10]Marco Francis Cusumano-Towner, David Hafner, Alexander Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor W. Killian, Stuart Bowers, Ozan Sener, Philipp Kraehenbuehl, and Vladlen Koltun. Robust autonomy emerges from self-play. In Forty-second International Conference on Machine Learning, 2025. [11]Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020. [12] Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. arXiv preprint arXiv:1611.01779, 2016. [13]Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017. [14]Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9710–9719, 2021. [15]Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515–1528. PMLR, 2018. 11 [16]Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, et al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems, 36:7730–7742, 2023. [17] Matthew T Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Greg Farquhar, Shimon Whiteson, and Jakob Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. Advances in Neural Information Processing Systems, 36: 79980–79998, 2023. [18]Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884–1897, 2021. [19]Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Interna- tional Conference on Machine Learning, pages 4940–4950. PMLR, 2021. [20]Minqi Jiang, Michael D Dennis, Edward Grefenstette, and Tim Rocktäschel. minimax: Efficient baselines for autocurricula in JAX. In Second Agent Learning in Open-Endedness Workshop, 2023. [21]Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106.14876, 2021. [22]Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinit- sky. GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS. In The Thirteenth International Conference on Learning Representations, 2025. [23]Pascal Klink, Haoyi Yang, Carlo D’Eramo, Jan Peters, and Joni Pajarinen. Curriculum rein- forcement learning via constrained optimal transport. In International Conference on Machine Learning, pages 11341–11358. PMLR, 2022. [24] Cevahir Koprulu, Thiago D Simão, Nils Jansen, and Ufuk Topcu. Risk-aware curriculum generation for heavy-tailed task distributions. In Uncertainty in Artificial Intelligence, pages 1132–1142. PMLR, 2023. [25]Quanyi Li, Zhenghao Mark Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou. Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling. Advances in neural information processing systems, 36:3894–3920, 2023. [26]Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents challenge. Advances in Neural Information Processing Systems, 36:59151–59171, 2023. [27]Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1–50, 2020. [28]Haoyi Niu, Yizhou Xu, Xingjian Jiang, and Jianming Hu. Continual driving policy optimization with closed-loop individualized curricula. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6850–6857. IEEE, 2024. [29] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, pages 17473–17498. PMLR, 2022. [30]Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835–853. PMLR, 2020. 12 [31]Zhiqian Qiao, Katharina Muelling, John M Dolan, Praveen Palanisamy, and Priyantha Mudalige. Automatically generated curriculum based reinforcement learning for autonomous vehicles in urban environment. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1233–1238. IEEE, 2018. [32] Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, and Felix Heide. Scenario dreamer: Vectorized latent diffusion for generating driving simulation environments. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17207–17218, 2025. [33]Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. Advances in Neural Information Processing Systems, 37:16071–16101, 2024. [34]Erdi Sayar, Giovanni Iacca, Ozgur S Oguz, and Alois Knoll. Diffusion-based curriculum reinforcement learning. Advances in Neural Information Processing Systems, 37:97587–97617, 2024. [35]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [36]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [37] Zihao Sheng, Zilin Huang, Yansong Qu, Yue Leng, Sruthi Bhavanam, and Sikai Chen. Curricu- vlm: Towards safe autonomous driving via personalized safety-critical curriculum learning with vision-language models. Transportation Research Part C: Emerging Technologies, 185:105549, 2026. [38] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. [39]Ryan Sullivan, Ryan Pégoud, Ameen Ur Rehman, Xinchen Yang, Junyun Huang, Aayush Verma, Nistha Mitra, and John P Dickerson. Syllabus: Portable curricula for reinforcement learning agents. Reinforcement Learning Journal, 6:1816–1855, 2025. [40]Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal curriculum for reinforcement learning agents. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. [41]Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Noc- turne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems, 35:3962–3974, 2022. [42] Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via models of human notions of interestingness. In The Twelfth International Conference on Learning Representations, 2024. [43]Linrui Zhang, Zhenghao Peng, Quanyi Li, and Bolei Zhou. Cat: Closed-loop adversarial training for safe end-to-end driving. In Conference on Robot Learning, pages 2357–2372. PMLR, 2023. 13 A Nomenclature GPOSG N , NSet of agents, number of agents (|N| = N) in POSG S ,A,OState, action and observation spaces in POSG s, a, o, rState, action, observation, and reward in POSG T , Z, RTransition, observation, and reward functions in POSG IInitial state distribution in POSG γ, HDiscount factor and horizon in POSG G Θ , Θ, θUPOSG, its set of scenarios and a scenario, i.e., θ ∈ Θ N Θ , NSet of agents, number of agents (|N Θ | = N) in UPOSG S ,A Θ ,O Θ State, action and observation spaces in UPOSG s, a, o, rState, action, observation, and reward in UPOSG T Θ , Z Θ , R Θ , I Θ Transition, observation, reward functions and initial state distribution in UPOSG γ, HDiscount factor and horizon in UPOSG xPosition of an agent in a scenario in UPOSG MNumber of scenarios in a UPOSG, i.e.,|Θ| = M S θ Goal Goal states in a UPOSG ΛLevel generator for UED ΠPolicy space in UED ∆(Θ)Distribution over levels in UED U , CUtility function in UED, constant utility in UED B, P replay Replay buffer and distribution in PLR P utility , P staleness Score and staleness distribution in PLR lScenario sampling iteration in PLR ρ, β, d, B max Staleness coefficient, score temperature, replay rate, max replay buffer size π, VPolicy, value function λ, δGAE discount factor, TD error R θ,n max Maximum return of an agent pSuccess rate φTrainable policy parameter T train , T sce , T pol Number of interactions for training, sampling scenarios, and updating policy WNumber of concurrent worlds DExperience buffer Φ()RL algorithm of choice to update policy eEnd of episode flag τRollout 14 Algorithm 2 SAMPLEFROMCURRICULUM() Input: Replay bufferB, set of training scenarios Θ train , sampling iteration l Parameters: Replay rate d, staleness ρ, temperature β, max buffer size B max , number of worlds W Output: Sampled scenarios(θ w ) W w=1 , and bufferBwith updated staleness 1: B ← DISCARDLOWESTRANKINGSCENARIOS(B, B max ) 2: if|B|≡ 0 or (Bernoulli(d)≡ 0 and|Θ train −B scenario | > 0) then 3: P sample ← Uniform(Θ train −B scenario )▷ Uniformly randomly sample scenarios 4: else 5: P sample ← P replay ▷ Replay scenarios based on P replay 6: end if 7: (θ w ) W w=1 ← Sample(P sample , W)▷ Sample W-many scenarios based on P sample 8: B scenario ←B scenario ∪ (θ w ) W w=1 ▷ Update scenarios in the replay buffer 9: l θ w ← l,∀w ∈ [W ]▷ Update sampling iteration for staleness distribution 10: τ θ w ← (),∀w ∈ [W ]▷ Reset the rollout Algorithm 3 UPDATECURRICULUM() Input: Interaction setD t , utility function U , replay bufferB Output: Updated replay bufferB 1: for w ∈ [W] do 2:if e n,w is True∀n∈ [N θ w ] then 3:score θ w ,t ← U (τ θ w )▷ Compute utility score for terminated episode 4:score θ w ← MovingAverage(score θ w , score θ w ,t )▷ Update the score in the buffer 5:τ θ w ← ()▷ Reset the rollout 6:else 7: τ θ w ← τ θ w ∪o n,w , a n,w , o ′ n,w ,r n,w ,e n,w n∈[N θ w ] ▷ Update the rollout with new interactions 8:end if 9: end for B Details of CL4AD In this section, we provide a more detailed look into howCL4ADworks to support the material in Section 4 and share the wall-clock overhead of curriculum updates and sampling. B.1 Curriculum Sampling in CL4AD The score distribution P utility (θ i |B,U ) is based on the ranking of seen levels, P utility (θ i |B,U ) = rank(θ i |B) −1/β P j∈B scenario rank(θ j |B) −1/β ,(2) with a temperature parameterβtuning the impact of ranking. The staleness distribution assigns a higher likelihood for levels that has not been sampled for longer, namely, P staleness (θ i |B,l) = l− l θ i P j∈B scenario l− l θ j ,(3) wherelis the total number of sampling iterations so far, andl θ j is the iteration at which scenario θ j was last sampled. Algorithm 2 is a pseudocode for howCL4ADsamples new scenarios during training via PLR. First,CL4ADremoves scenarios with ranking lower thanB max in the buffer (Line 1), whereB max is the maximum size ofBfor sampling. If the buffer size is smaller than or equal to B max , then no scenario is removed. Then, it determines whether to sample traffic scenarios from the replay buffer. If the replay buffer is empty, or the random replay decision is False, conditioned on the fact that there are still unseen scenarios, thenCL4ADuniformly randomly samples unseen scenarios from the training dataset. Otherwise, it uses the replay distributionP replay to sample from the replay bufferB(lines 2-6). Then,CL4ADupdates the scenarios in the buffer with the newly sampled ones and sets their corresponding last sampling iteration to the current one for staleness computation later on (lines 7-9). 15 B.2 Curriculum Updates in CL4AD Algorithm 3 is a pseudocode for howCL4ADupdates the buffer.CL4ADgoes through every world and checks whether an episode has terminated. If so, it computes the utility of that episode based on the rollout thatCL4ADhas kept track of. Then it stores this score, and at the next sampling call, it averages the scores of all episodes in that scenario to update the buffer. Finally, the rollout is reset for a new episode to save memory. If the episode continues,CL4ADupdates the rollouts with the latest interactions. B.3 Practitioner Guidelines Based on our experimental findings, we offer the following recommendations to practitioners. PLR configuration at scale. 1)Set the replay buffer size equal to the training dataset size. Prior UED work uses buffers much smaller than the level space; our experiments show that a full-size buffer works consistently as the dataset scales. 2)Use a score temperature significantly higher than prior work. We useβ ∈2, 4, well above the typical range ofβ ∈ [0.1, 1.0]. Lower temperatures concentrate sampling on a few top-ranked scenarios, which becomes problematic as the buffer grows. Increasing βgenerally leads to better performance as the dataset scales up. Higher temperature also indirectly mitigates staleness by spreading sampling across more scenarios. 3)The staleness coefficient can remain in the same range as prior work,ρ ∈ 0.1, 0.3 [19], as higher score temperature already alleviates staleness. Utility function selection. 1)No single utility function dominates across all scales. However, several patterns emerge from our results (see Sections 5.2 and 5.5). 2)Among regret-based functions, MaxMC is the most reliable across scales, as it does not depend on value function accuracy. AMGAE and PVL rely on value estimation and can un- derperform at smaller scale (case 1). As the dataset scales, their performance improves (case 2), consistent with the increasing correlation between regret-based functions in Figure 6. 3)Success-based functions are effective across all scales and particularly strong at large scale (case 3), where identifying the learning frontier through success variance is effective. 4) Realism-based functions accelerate training over DR but cannot improve realism, as they only affect scenario prioritization, not the reward function. 5)Utility functions within the same category tend to prioritize similar scenarios, especially as the dataset scales, except the realism category (see Figure 6). B.4 Computational overhead of CL4AD To quantify the overhead of curriculum learning, we measured wall-clock times for allCL4AD components during training. Curriculum updates (utility computation and buffer score updates) and scenario sampling (PLR sampling and scenario assignment to worlds) occur everyT sce = 2, 000, 000 interactions. Per inter-sampling segment, curriculum update time totals∼ 4.3s, which is∼ 1%of the segment’s evaluation/rollout time (∼ 428.5s). The PLR sampling operation itself takes∼ 0.66msper call, the remaining∼ 6.15sof each sampling block is spent assigning scenarios to worlds and resetting the simulator, which is simulator overhead, not CL4AD overhead. Over a full training run of one billion steps (∼ 110hours on an A5000, Section D), there are approximately500curriculum sampling steps. The total curriculum update time is approximately36 minutes, and the total PLR sampling time is approximately0.3seconds. The overhead of assignment of scenarios (∼ 51minutes total) is attributed to the simulator, not toCL4AD. In total,CL4AD adds less than 1% to the training wall-clock time. 16 Table 1: Wall-clock time per inter-sampling segment ComponentTimePercentage Curriculum update (total)4.30± 0.21s1.04% Evaluation/rollout time428.50± 88.44s100% Table 2: CL4AD sampling vs. scenario assignment per curriculum sampling call ComponentTimePercentage Curriculum sampling0.66± 0.02ms0.01% Scenario assignment to worlds6.15± 0.40s99.99% C Experimental Details In this section, we describe the process of hyperparameter selection for our experiments. C.1 Simulation Set-up Our integration ofCL4ADintoGPUDRIVEfollows the simulation set-up in Kazemkhani et al. [22], where the simulator ignores collisions and going off-road, i.e., they do not lead to episode termination; the observation of a vehicle is its bird-eye-view of a radius of 50m; non-vehicle objects are omitted; a goal is considered to be achieved if an agent is in its proximity by 2m; the action consists of two discrete random variables for steering and acceleration inputs, divided into evenly spaced grids, 13 and 7, respectively; maximum number of controlled agents in a scenario is 64; the agents only observe the current time step; and the episode takes 91 timesteps, amounting to 9 seconds, with 1 second of history followed by 8 seconds of future. The public test partition withholds the future trajectories, as they are the prediction targets of the motion forecasting benchmark, leaving 11 logged steps per scenario. Since GPUDrive sets each agent’s goal to its final logged position, goals in test scenarios lie roughly one second of driving ahead, and agents are exposed to fewer timesteps in which collisions and off-road events can occur. For more details, we refer the reader to the default PufferLib configuration (seeenvironmentsection) in the repository published by Kazemkhani et al. [22]. C.2 Self-play PPO Training Self-play RL is an RL scheme for multi-agent settings where each agent samples their actions from a shared, decentralized policy. More formally, this scheme samples the actiona i,t ∼ π φ (o i,t )of agent i∈N Θ via a policyπ φ parameterized byφ, e.g., a neural network with learnable parametersφ, given the observationo i,t of said agent at timet. Batched AD simulatorsGIGAFLOWandGPUDRIVE use self-play RL as the strategy to train a single policy that controls all vehicles in a scenario in parallel. Their batched structure empowers parallelization further by concurrently simulating hundreds to thousands of traffic scenarios to accelerate experience collection. Both works employ an on-policy RL algorithm, proximal policy optimization (PPO) [36], where policy updates occur once the simultaneous data collection fills an experience buffer. As a result, batched simulation accelerates experience collection via parallelized scenarios, while self-play RL saves compute time and memory by training a single policy. We implementCL4ADonGPUDRIVE, which samples hundreds of traffic scenarios every couple of million interactions, with initial positions and goals from logged data in WOMD. The default scenario sampling is uniform, where each traffic scenario has equal likelihood. Table 3 lists the hyperparameters for self-play PPO training in cases 1, 2, and 3, as well as the ablation study. As the ablation study investigates limited compute resources, i.e., the use of fewer worlds and lower batch sizes, we essentially set them according to the hyperparameters in Kazemkhani et al.[22], where the number of worldsW = 50. In comparison, cases 1, 2, and 3 studies a larger scale in terms of throughput, hence utilize significantly more concurrent worlds and a larger experience buffer. As a result, their hyperparameters come from Cornelisse et al.[9], which focuses on a similar scale. The weights for collision/off-road penalties and goal completion rewards also come from Cornelisse et al. [9]. The experiments are over three independent runs, utilizing seeds42,12, and67. The network architecture also follows the settings in Cornelisse et al. [9]. 17 Table 3: Self-play PPO Hyperparameters ParameterCase 1,2,3Ablation total_timesteps T train 2, 000, 000, 0001, 000, 000, 000 num_worlds W800100 batch_size T pol 524, 288131, 072 minibatch_size16, 3848, 192 learning_rate0.00030.0003 anneal_lr false false gamma γ0.990.99 gae_gamma λ0.950.95 update_epochs24 norm_adv true true clip_coef0.20.2 clip_vloss false false vf_clip_coef0.20.2 ent_coef0.00010.0001 vf_coef0.50.3 max_grad_norm0.50.5 target_kl null null collision_weight−0.75−0.75 off_road_weight−0.75−0.75 goal_achieved_weight1.01.0 C.3 Scenario Sampling Details Table 4 demonstrates the hyperparameters used for the experiments we report in cases 1, 2, 3, and the ablation study. The search space for PLR hyperparameters is as follows: staleness coefficient ρ∈0.1, 0.3and score temperatureβ ∈2, 4, based mainly on Jiang et al.[19]. We first conduct a grid search in Case 1, where we train agents using all score functions on three independent runs for one billion interactions. Then we select the pair that yields the highest success rate, the fastest at test-time. Jiang et al.[18]suggests a lower temperature; however, our experiments indicate that a higher temperature, especially considering the size of the training dataset, is more performant in large-scale training. Case 3 and the ablation study also utilize these hyperparameters. In case 2, we find that a higher temperature yields better results. We set the replay buffer size to the size of the training dataset, and sample scenarios every2, 000, 000interactions. For heuristic curricula, once scenarios are ranked using the chosen heuristic, we sample from a score-rank distributionP utility with fixed utility and β ∈2, 4. D Computational Resources We run our experiments in cases 1, 2, and 3 on an NVIDIA H200, which has 141 GB of GPU memory. One training run, which amounts to 2 billion steps and approximately 3,800 policy updates, takes around 60 hours. For the ablation study, we train agents on NVIDIA RTX A5000, which has a GPU memory of 24GB, for a billion interactions, which takes over 110 hours. E Detailed Results E.1 WOSAC Evaluation We evaluate the final trained policies using the Waymo Open Sim Agents Challenge (WOSAC) metrics [26] in two settings: (1) all controlled agents via self-play, and (2) ego-only with other agents replaying logged trajectories. Table 5 and Table 6 report the results for case 1 (150 test scenarios, 3 seeds). Kinematics and interaction scores are comparable across all methods, indicating that the curriculum choice does not meaningfully affect these categories. The main difference is in map-based metrics (Dist Edge, Offroad), where DR scores higher because WOSAC measures distributional similarity to 18 Table 4: PLR Hyperparameters Utility Function d β ρ Case 1 U Act-MAE 0.520.3 U AMGAE 0.540.3 U GC-ADE 0.540.3 U Learn 0.540.1 U Learn-hard 0.520.1 U MaxMC 0.520.3 U PVL 0.540.3 Case 2 U Act-MAE 0.540.3 U Learn 0.540.1 U MaxMC 0.540.3 Case 3 U Act-MAE 0.540.3 U Learn 0.540.1 U MaxMC 0.520.3 Ablation U Act-MAE 0.520.3 U Learn 0.540.1 U MaxMC 0.520.3 Table 5: WOSAC metrics, self-play evaluation with final trained policies (case 1: 150 test scenarios). MethodRealism↑minADE↓Lin Spd↑Lin Acc↑Ang Spd↑Ang Acc↑Dist Obj↑Collis↑TTC↑Dist Edge↑Offroad↑ Oracle0.8320.000.4930.4480.5780.6940.4300.9990.9000.7720.999 DR0.68910.280.1610.2460.5100.6580.1630.8550.8560.7030.901 PLR+MaxMC0.6579.200.1650.2520.5140.6620.1580.8300.8570.7010.804 PLR+GC-ADE0.6529.580.1580.2290.5090.6600.1520.8400.8560.6910.783 PLR+Learn-Hard0.6499.760.1580.2300.5080.6600.1500.8370.8560.6940.778 PLR+AMGAE0.6449.940.1590.2440.5080.6590.1560.8490.8540.6740.747 PLR+PVL0.6409.710.1570.2310.5100.6600.1520.8580.8570.6700.724 PLR+Act-MAE0.6319.400.1580.2120.5120.6640.1470.8240.8560.6890.724 PLR+Learn0.62810.100.1530.2170.5070.6630.1460.8630.8560.6480.685 logged behavior, and uniform sampling stays closer to the data distribution. PLR variants achieve lower displacement error, consistent with more efficient goal reaching, but diverge from human driving patterns in map-based metrics. This gap widens in the ego-only setting. All methods remain far from the oracle, with kinematics representing the largest gap. These results support the analysis in Section 5.7: the curriculum improves task performance but cannot improve realism without changes to the reward function. E.2 Quantitative Results Figures 7, 8, 9, and 10 demonstrate the progression of trained agents in cases 1, 2, and 3, as well as the compute ablation, respectively. These figures provide details on the progression of performance, regret, realism, and learnability when agents are evaluated in the training and test partitions of their respective experiments. Regret, learnability, and realism in the training partition highlight how automated curricula impact training. In most cases, we observe that PLR variants are significantly faster than DR at achieving low utility scores in these metrics, indicating that they obtain more performant and realistic policies more quickly. The performance progression, when evaluated on the training partition, leads to a similar observation as well. Progression in test scenarios demonstrates the generalization capabilities of these trained agents, as these scenarios were not encountered during training. Overall, we observe that PLR variants are again quickly becoming more capable at generalization or becoming robust and reliable faster than agents trained via DR. 19 Table 6: WOSAC metrics, ego-only evaluation with final trained policies (case 1: 150 test scenarios). MethodRealism↑minADE↓Lin Spd↑Lin Acc↑Ang Spd↑Ang Acc↑Dist Obj↑Collis↑TTC↑Dist Edge↑Offroad↑ Oracle0.8710.000.5820.6940.7210.9170.4820.9990.9180.8540.999 DR0.6699.140.2790.4020.6170.8640.1440.7690.8430.7240.804 PLR+MaxMC0.6248.250.2770.4050.6200.8620.1360.7650.8420.6740.649 PLR+PVL0.6088.470.2740.3860.6150.8660.1250.7660.8440.6590.597 PLR+GC-ADE0.6038.580.2730.3850.6120.8630.1320.7530.8420.6610.591 PLR+AMGAE0.5958.720.2730.3880.6120.8640.1280.7490.8400.6370.572 PLR+Learn-Hard0.5868.450.2740.3850.6130.8680.1260.7570.8420.6360.529 PLR+Act-MAE0.5728.350.2710.3690.6150.8640.1240.7610.8430.6250.478 PLR+Learn0.5569.000.2670.3730.6110.8610.1200.7680.8410.5650.424 E.3 Qualitative Results Figures 11, 12, 13, 14, 15, and 16 illustrate theP replay progression of PLR in case 1. Here we omit U MaxMC , as we provide its illustration in the main document. The utility functions with a high score temperature, i.e.,β = 4, as opposed toβ = 2, lead to a more uniform replay distribution (see Figures 12, 13, 15, 16 forU AMGAE ,U GC-ADE ,U Learn , andU PVL , respectively). As the score temperature decreases, the impact of the ranking on the replay distribution also decreases. Furthermore, we observe that certain utility functions result in significant changes in the replay distribution throughout training, specifically when visualized with respect to the number of controlled agents in scenarios (see Figures 11 and 14 forU Act-MAE andU Learn-hard , respectively). This change may be due to a lower score temperature, which allows the ranking to impact the replay distribution more strongly. E.4 Sensitivity to sampling interval and buffer size We ablate the two PLR hyperparameters that govern how often scores are refreshed and how many scenarios the buffer retains. Our reported configuration uses a buffer equal to the training set and T sce = 2× 10 6 interactions, which is the scenario sampling interval GPUDrive uses under domain randomization. We compare against three variations, all withU MaxMC in case 1 across three seeds: (1) Small Buffer: a buffer of 100 scenarios, i.e.,|B = 100|, andT sce = 2× 10 6 , (2) Faster Sampling: full buffer, i.e.,|B| = 1, 000, andT sce = 1 × 10 6 , and (3) Slower Sampling: full buffer, i.e., |B| = 1, 000, andT sce = 4× 10 6 . Figure 17 reports the progression on the test split. All four configurations reach the same discounted return and success rate by 1,400 policy updates, with collision and off-road rates near zero from 1,000 updates onward, and all four exceed DR in return and success rate at every checkpoint, so CL4AD is insensitive to both hyperparameters over the ranges we test. The configurations differ only over the first few hundred updates, where faster sampling and the small buffer reach high success earliest and slower sampling trails the other three. This follows from the staleness mechanism in Section 5.6: shorter sampling intervals keep buffer scores closer to on-policy, and a small buffer has a similar effect, as evicting all but the top-ranked 10% of scenarios causes the retained ones to be revisited more often and prioritized more sharply. The small buffer carries elevated collision and off-road rates through 600 updates, since the curriculum spends less of its budget on the broader set of scenarios where the remaining safety failures occur. E.5 Combining utility functions across categories Section 5.5 shows that utility functions correlate within a category, except for realism, but not across categories, suggesting that cross-category pairs carry complementary information. We combine two utility functions by ranking the scenarios independently under each and sampling from the averaged rank, preserving the scale invariance ofP utility in Equation (2), since a weighted sum of raw scores does not consist of utilities expressed in different units. We evaluate three pairs in case 1 across three seeds, one from each pairwise combination of categories: (1)U MaxMC +U Learn-hard , (2)U MaxMC +U Act-MAE , and (3)U Learn-hard +U Act-MAE . Figure 18 reports the progression on the test split. All three combinations converge with their individual components in discounted return and success rate by 1,400 policy updates, with collision and off-road rates near zero, and all exceed DR at every checkpoint.U Learn-hard alone is the slowest PLR variant in this setting, whereasU MaxMC +U Learn-hard sits at or above both components at the early checkpoints, so combining a success-based function with a regret-based one does not average their behavior. One explanation, which we do not test here, is that the combined ranking draws from both density regimes, since Section 5.2 shows regret-based functions drifting above the dataset average traffic density while success-based functions shift below. 20 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.5 5.0 7.5 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.5 5.0 7.5 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 3 4 1e1 Average Magnitude of GAE 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 3 1e1 Positive Value Loss 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 5 0 5 1e1 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 3 1e2 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (a) Evaluation on training partition 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.8 0.9 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Average Magnitude of GAE 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Positive Value Loss 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability-Hard 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL Heuristic-Dense Heuristic-Sparse (b) Evaluation on test partition Figure 7: Case 1: Regret (U AMGAE ,U PVL ,U MaxMC ), realism (U GC-ADE ), and learnability (U Learn , U Learn-hard ), progression during training with 1000 scenarios from WOMD: We evaluate in (a) training partition, and (b) 150 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across three independent training runs. 21 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2 4 6 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 7.5 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2 4 6 1e2 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 2.5 5.0 7.5 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 3 1e1 Average Magnitude of GAE 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 1e1 Positive Value Loss 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.5 0.0 2.5 1e1 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 1 2 3 1e2 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.5 1.0 1.5 1e1 Learnability-Hard 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 1e1 Learnability Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL (a) Evaluation on training partition 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.9 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 2 1e3 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Average Magnitude of GAE 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 5 1e1 Positive Value Loss 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 4 6 8 1e1 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability-Hard 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability Method DR PLR+Act-MAE PLR+AMGAE PLR+GC-ADE PLR+Learn-Hard PLR+Learn PLR+MaxMC PLR+PVL (b) Evaluation on test partition Figure 8: Case 2: Performance, Regret (U AMGAE ,U PVL ,U MaxMC ), realism (U GC-ADE ), and learnability (U Learn ,U Learn-hard ), progression during training with 10,000 scenarios from WOMD: We evaluate in (a) training partition, and (b) 10,000 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across three training runs. 22 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.9 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 2 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.9 1.0 Success Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0 1 2 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Average Magnitude of GAE 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Positive Value Loss 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 2.8 3.0 3.2 3.4 1e1 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability-Hard 0.20.61.01.41.82.22.63.03.43.8 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC Heuristic-Dense Heuristic-Sparse Heuristic-Far Heuristic-Fast Figure 9: Case 3: Regret (U AMGAE ,U PVL ,U MaxMC ), realism (U GC-ADE ), and learnability (U Learn , U Learn-hard ), progression during training with 80,000 scenarios from WOMD: We evaluate in 10,000 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across two independent training runs. 23 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 1.0 0.5 0.0 0.5 Discounted Return 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.25 0.50 0.75 1.00 Success Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Collision Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0 1 2 1e1 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 1.0 0.5 0.0 0.5 Discounted Return 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.25 0.50 0.75 1.00 Success Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Collision Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0 1 2 1e1 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 2 4 6 1e1 Average Magnitude of GAE 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.5 1.0 1.5 2.0 1e1 Positive Value Loss 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 1.0 0.5 0.0 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 1 2 1e2 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.5 1.0 1.5 1e1 Learnability-Hard 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 0.5 1.0 1.5 1e1 Learnability Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC (a) Evaluation on training partition 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.9 1.0 Success Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0 2 1e3 Collision Rate 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0 1 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 2 4 6 1e1 Average Magnitude of GAE 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0 2 4 6 1e1 Positive Value Loss 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 2.5 5.0 7.5 1e1 Maximum Monte Carlo Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 2 3 4 5 1e1 Goal-Cond. Ave. Distance Err. Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability-Hard 0.21.01.82.63.44.25.05.86.67.4 Number of policy updates 1e3 0.0 0.5 1.0 1e1 Learnability Method DR PLR+Act-MAE PLR+Learn PLR+MaxMC (b) Evaluation on test partition Figure 10: Ablation: Performance, regret (U AMGAE ,U PVL ,U MaxMC ), realism (U GC-ADE ), and learn- ability (U Learn ,U Learn-hard ), progression during training for our ablation study on compute resources: We evaluate in (a) training partition, and (b) 150 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across three training runs. 24 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 21 Scenario 134 Number of controlled agents: 12 Scenario 469 Number of controlled agents: 16 Scenario 476 0.0 0.2 0.4 0.6 0.8 1.0 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 Replay Probability ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Replay Probability ×10 1 Figure 11:P replay progression of PLR combined withU Act-MAE in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 25 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 1 Scenario 249 Number of controlled agents: 1 Scenario 826 Number of controlled agents: 11 Scenario 854 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Replay Probability ×10 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Replay Probability ×10 1 Figure 12:P replay progression of PLR combined withU AMGAE in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 26 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 4 Scenario 393 Number of controlled agents: 10 Scenario 534 Number of controlled agents: 8 Scenario 539 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Replay Probability ×10 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Replay Probability ×10 1 Figure 13:P replay progression of PLR combined withU GC-ADE in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 27 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 2 Scenario 56 Number of controlled agents: 4 Scenario 306 Number of controlled agents: 2 Scenario 861 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Replay Probability ×10 2 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 ×10 1 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 ×10 1 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Replay Probability ×10 1 Figure 14:P replay progression of PLR combined withU Learn-hard in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 28 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 4 Scenario 306 Number of controlled agents: 4 Scenario 400 Number of controlled agents: 2 Scenario 818 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Replay Probability ×10 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Replay Probability ×10 1 Figure 15:P replay progression of PLR combined withU Learn in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 29 200600 10001400180022002600300034003800 Number of policy updates 0 200 400 600 800 Scenario IDs seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 200600 10001400180022002600300034003800 Number of policy updates 0 10 20 30 40 Number of controlled agents seed: 42 200600 10001400180022002600300034003800 Number of policy updates seed: 12 200600 10001400180022002600300034003800 Number of policy updates seed: 67 Number of controlled agents: 14 Scenario 49 Number of controlled agents: 19 Scenario 58 Number of controlled agents: 12 Scenario 439 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ×10 3 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Replay Probability ×10 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 ×10 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Replay Probability ×10 1 Figure 16:P replay progression of PLR combined withU PVL in mini WOMD: We illustrate (top) the evolution ofP replay , where darker line segments indicate scenarios with higher replay likelihood, (middle) a version of replay distribution under categorization with respect to the number of controlled agents in scenarios, and (bottom) we exemplify three scenarios that appear frequently. 0.20.61.01.4 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.4 Number of policy updates 1e3 0.9 1.0 Success Rate 0.20.61.01.4 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.4 Number of policy updates 1e3 0 1 1e2 Off-Road Rate Method DR PLR+MaxMC PLR+MaxMC (Faster Sampling) PLR+MaxMC (Slower Sampling) PLR+MaxMC (Small Buffer) Figure 17: Sensitivity to sampling interval and buffer size: Performance progression ofU MaxMC in case 1 under three variations of our reported configuration, i.e., a buffer of 100 scenarios and scenario sampling intervals of1× 10 6 and4× 10 6 interactions. We evaluate on 150 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across three independent training runs. 30 0.20.61.01.4 Number of policy updates 1e3 8 9 1e1 Discounted Return 0.20.61.01.4 Number of policy updates 1e3 0.9 1.0 Success Rate 0.20.61.01.4 Number of policy updates 1e3 0 2 4 1e3 Collision Rate 0.20.61.01.4 Number of policy updates 1e3 0 1 1e2 Off-Road Rate Method DR PLR+Act-MAE PLR+Learn-Hard PLR+MaxMC PLR+Learn-Hard,Act-MAE PLR+MaxMC,Act-MAE PLR+MaxMC,Learn-Hard Figure 18: Combining utility functions across categories: Performance progression of three cross- category combinations in case 1, reported alongside their individual components and DR. We evaluate in 150 test scenarios. Bold markers indicate the mean, whereas the shaded area covers one standard deviation around it across three independent training runs. 31

Instant research discovery