Paper deep dive
Collective Autonomous Vehicle Routing Needs Social Awareness for Coexistence with Human Drivers
Anastasia Psarou, Mikolaj Rams, Blazej Torbus, Rafal Kucharski
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/12/2026, 7:20:12 AM
Summary
This paper addresses the negative externalities caused by selfish Autonomous Vehicle (AV) routing in mixed-autonomy traffic, where AVs minimizing individual travel times can increase travel times for Human-Driven Vehicles (HDVs). The authors propose incorporating a 'marginal cost' component into the Multi-Agent Reinforcement Learning (MARL) reward function for AVs. This social component measures the AV's impact on the system by comparing travel times with and without the AV (counterfactual simulation). Experiments on Ingolstadt and Provins networks demonstrate that this approach limits HDV travel time increases and improves overall system performance, including AV travel times. To address the computational cost of counterfactual simulations, the authors introduce 'selective marginal cost reward shaping,' which applies the social reward only to the most impactful AVs, achieving similar performance with significantly reduced computational overhead.
Entities (10)
Relation Signals (7)
Selfish AV Routing → increases → HDV Travel Time
confidence 96% · selfish routing of 40% AVs increases the travel times of other vehicles by 1.7%
Marginal Cost Reward → improves → System Performance
confidence 95% · incorporating marginal cost into the AV reward not only limits the increase in HDV travel times and shortens AV travel times relative to selfish AV routing, but also improves overall system performance.
Marginal Cost → iscomponentof → AV Reward
confidence 95% · To address this, we introduce a social component, the marginal cost, to the selfish travel time AV reward.
Selective Marginal Cost Reward Shaping → reduces → Computational Cost
confidence 94% · This reduces the number of counterfactual simulations and achieves similar performance at a significantly lower computational cost.
Marginal Cost → measures → AV Impact
confidence 93% · Marginal cost captures an AV's impact on other road users by comparing their travel times with those in a counterfactual simulation in which the AV is removed.
Selective Marginal Cost Reward Shaping → appliesto → Impactful AVs
confidence 92% · we compute the marginal cost only for the AVs whose route choices most affect the traffic system.
SUMO → usedby → RouteRL
confidence 90% · we use the RouteRL [3] framework integrated with SUMO [22]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Collective Autonomous Vehicle (AV) route choices in real-world traffic networks can create externalities for Human-Driven Vehicles (HDVs), as routes that benefit AVs may increase travel times for other road users. To address this, we introduce a social component, the marginal cost, to the selfish travel time AV reward. Marginal cost captures an AV's impact on other road users by comparing their travel times with those in a counterfactual simulation in which the AV is removed. We study AV route choice in mixed-autonomy traffic settings using Multi-Agent Reinforcement Learning (MARL), where AVs learn routing policies through repeated interactions with the traffic environment. Experiments on the real-world networks of Ingolstadt and Provins show that incorporating marginal cost into the AV reward not only limits the increase in HDV travel times and shortens AV travel times relative to selfish AV routing, but also improves overall system performance. In Ingolstadt, selfish routing of 40\% AVs increases the travel times of other vehicles by 1.7\%, and this drops to 1.1\% when the social reward is included. Interestingly, not only does overall system performance improve, but the AVs are also better off when acting this http URL effect is consistent across tested cities, penetration rates, and replications, yet when the weight of the social component in the reward increases, the system fails to reach the desired performance. For a more scalable approach to social AV routing in mixed-autonomy traffic, we compute the marginal cost only for the AVs whose route choices most affect the traffic system. This reduces the number of counterfactual simulations and achieves similar performance at a significantly lower computational cost. Our results suggest that AV routing should go beyond selfish rewards and, for the benefit of all traffic participants, include a social component in its objective.
Tags
Links
- Source: https://arxiv.org/abs/2610.13439v1
- Canonical: https://arxiv.org/abs/2610.13439v1
Trouble viewing inline? Open PDF directly →
Full Text
51,357 characters extracted from source content.
Expand or collapse full text
Collective Autonomous Vehicle Routing Needs Social Awareness for Coexistence with Human Drivers Anastasia Psarou ∗ , Mikolaj Rams, Blazej Torbus, Rafal Kucharski Faculty of Mathematics and Computer Science, Jagiellonian University Krak ́ ow, Poland Abstract Collective Autonomous Vehicle (AV) route choices in real-world traffic networks can create externalities for Human-Driven Vehicles (HDVs), as routes that benefit AVs may increase travel times for other road users. To address this, we introduce a social component, the marginal cost, to the selfish travel time AV reward. Marginal cost captures an AV’s impact on other road users by comparing their travel times with those in a counterfactual simulation in which the AV is removed. We study AV route choice in mixed-autonomy traffic settings using Multi-Agent Reinforcement Learning (MARL), where AVs learn routing policies through repeated interactions with the traffic environment. Experiments on the real-world networks of Ingolstadt and Provins show that incorporating marginal cost into the AV reward not only limits the increase in HDV travel times and shortens AV travel times relative to selfish AV routing, but also improves overall system performance. In Ingolstadt, selfish routing of 40% AVs increases the travel times of other vehicles by 1.7%, and this drops to 1.1% when the social reward is included. Interestingly, not only does overall system performance improve, but the AVs are also better off when acting socially. This effect is consistent across tested cities, penetration rates, and replications, yet when the weight of the social component in the reward increases, the system fails to reach the desired performance. For a more scalable approach to social AV routing in mixed-autonomy traffic, we compute the marginal cost only for the AVs whose route choices most affect the traffic system. This reduces the number of counterfactual simulations and achieves similar performance at a significantly lower computational cost. Our results suggest that AV routing should go beyond selfish rewards and, for the benefit of all traffic participants, include a social component in its objective. Keywords: Autonomous vehicles, urban route choice, multi-agent reinforcement learning, social awareness, reward shaping. 1 Introduction Every day, human drivers commute through urban traffic net- works, choosing among several possible routes the ones with the minimal expected perceived travel time [4,38]. The grow- ing presence of Autonomous Vehicles (AVs) in cities may introduce new dynamics, as AVs may deviate from the shortest routes provided by real-time navigation systems and can use a variety of algorithms to learn routing strategies, potentially achieving shorter travel times than Human-Driven Vehicles (HDVs) or disadvantaging them by consistently increasing their travel times [16, 28]. We consider a city that initially consists entirely of HDVs who learn routing strategies by updating their utility through repeated days of commuting [8]. We assume this repeated learning process brings the traffic system close to an equilib- rium state [40]. Then, a subset of HDVs switches to traveling with AVs, which use Multi-Agent Reinforcement Learning (MARL) to learn routing policies. As learning progresses, AVs reduce their travel times and eventually achieve shorter average travel times than HDVs. However, we find that when AVs select their routes solely to minimize their individual travel times (selfish behavior), their routing policies can in- crease the travel times of HDVs relative to those experienced 0∗ Corresponding author: anastasia.psarou@uj.edu.pl before AVs were introduced in the system (see Figure 1A). We therefore ask: How can AV routing MARL algorithms be designed to reduce AV travel times while limiting the additional costs on other road users, given that urban road networks are shared public infrastructure? To solve this problem, we augment the selfish travel-time AV reward with a component that accounts for each AV’s impact on the sys- tem, the marginal cost, following the formulation introduced in [29]. We show that incorporating marginal cost into AV rewards can reduce the increase in HDV travel times and al- low AVs to achieve shorter travel times than under selfish AV routing (see Figure 1B). We further find that these travel-time improvements depend on the relative weighting of the two reward components: assigning a greater weight to marginal cost can further improve HDV travel times but may increase AV and overall system travel times. We further investigate how to make marginal cost compu- tation scalable, since it requires an additional counterfactual simulation per AV each day of commute. In each counter- factual, we remove an AV from the traffic system and com- pare the remaining vehicles’ travel times with those observed when the AV is present. For example, with 400 AVs and 10,000 training iterations (equivalent to commute days), com- puting marginal costs for all AVs would require four million 1 arXiv:2610.13439v1 [cs.MA] 9 Oct 2026 Shared mixed-autonomy traffic environment A) Selfish AV reward rᵢ = −eᵢ(퐮) Each AV minimizes its own travel time. B) Marginal Cost (MC) reward shaping Simulation: AV i presentCounterfactual: AV i removed AV i removed e j (u) e j (u −i ) remove AV i Identify high-impact AVs C) Selective MC reward shaping Compute mⱼ(퐮) only for selected high-impact AVs. AV AV i O1 O2 AV HDV HDV D1 D2 OD Origin pointDestination point AVAV AV m i (u) = [ e j (u) − e j (u −i ) ] rᵢ = −eᵢ(퐮) − m i (u) N Σ j=1 AV CDF Impact score H i 70% top 30% HDVAV HDVAV Counterfactual simulations HDVAV Counterfactual simulations Figure 1: We study AV route choice in traffic environments shared with human drivers. (A) When AVs selfishly minimize their own travel times, their routing decisions can increase HDV travel times. (B) Incorporating our marginal cost into the AV reward allows AVs to account for the impact of their route choices on other road users, mitigating the increase in HDV travel times observed under selfish AV routing while also improving AV travel times on average. However, computing marginal cost requires additional counterfactual simulations for each AV, adding computational cost. (C) To reduce this burden, we introduce selective marginal cost reward shaping, which computes marginal cost rewards only for a subset of the most impactful AVs. This preserves some of the benefits for AVs and HDVs’ travel times while reducing the number of counterfactual simulations. additional simulation runs. We therefore investigate whether marginal cost rewards can be computed and applied only to a fraction of the most impactful AVs on the traffic system (se- lective marginal cost reward shaping), rather than to all AVs (marginal cost reward shaping). We then evaluate whether selective marginal cost reward shaping can achieve system, AV, and HDV travel-time improvements comparable to those obtained with marginal cost reward shaping, relative to selfish AV routing (Figure 1C). The contributions of this work include: •We show that AV routing policies trained with MARL to min- imize individual AV travel times can increase HDV travel times relative to those experienced before the introduction of AVs in the system, revealing a negative externality of selfish AV routing (see Figure 3A, D). •We demonstrate that incorporating a social component, the marginal cost, into the AV reward can mitigate the increase in HDV travel times observed after AV introduction while also improving AV travel times compared to those achieved when AVs minimize their individual travel times. We further show that AV and HDV travel times depend on how much weight is assigned to the marginal cost. Higher weights can further improve HDV travel times but worsen AV and overall system performance (see Figures 3, 4). •We show that computing marginal costs only for a fraction of the most impactful AVs, selective marginal cost reward shaping, is sufficient to retain HDV, AV, and system travel times comparable to those achieved when all AVs receive the marginal cost reward. This approach also reduces the num- ber of counterfactual simulations (see Table 1). We identify impactful AVs from sampled joint actions that capture how total system travel time varies across alternative AV route choices. 1.1 Literature review Route choice can be formulated as a congestion game in which travelers repeatedly select among alternative routes in a traffic network while sharing limited road capacity [15]. Assuming homogeneous, fully rational users, this game converges to the Wardrop or User Equilibrium (UE), the Nash equilibrium in the route choice context, where no traveler can improve their travel time by unilaterally switching routes [6,39]. The System Optimum (SO) is the traffic allocation that minimizes total system travel time across the network [24], but achieving it may require some travelers to take routes that are less favorable for them to improve overall system performance. Human route choice is commonly modeled using the utility- maximization decision rule [12,16,20]. The main compo- nent of the utility is the expected travel time for each route [7], which is updated after each trip. In random utility the- ory, utility additionally includes a random stochastic variable that captures heterogeneity in travelers’ preferences, imperfect perception of travel time and congestion, and fluctuations in driving conditions [8,40]. This stochastic route choice behav- ior is assumed to converge to the Stochastic User Equilibrium (SUE), in which no driver believes they can improve their travel times by switching routes. In a mixed-traffic environment where HDVs and AVs coex- ist, AV route assignment can be solved in several ways. MARL is a promising method [1,2,19,36,37], where AVs can learn routing policies through repeated interactions with the traffic environment. These policies can be optimized using different objectives, such as minimizing AVs’ individual travel time, the system’s travel time, or even maximizing HDV travel times [1,2,16]. Among these objectives, if AVs optimize only their individual travel times, their routing choices may increase other road users’ travel times because of limited road capacity. This motivates the need to search for AV routing objectives that improve AV travel times while limiting the additional 2 costs imposed on other road users. In economics, marginal cost denotes the change in total cost resulting from an increase in the quantity produced [23]. In transportation research, marginal cost quantifies a vehicle’s impact on the system by measuring the additional travel time it imposes on other road users through its presence in the traffic system [25,32]. This idea forms the basis of marginal cost pricing, in which travelers are charged according to the increase in latency their action or presence causes to the system [10,26]. Marginal cost pricing has also been incorporated into reinforcement learning approaches for route choice. Ramos et al. [30], for example, incorporated marginal cost tolls into the reward signal for commuters using reinforcement learning for route choice, aiming to align individual routing decisions with system-level efficiency and minimize congestion. Inspired by the concept of marginal cost and following [29], we define the marginal cost of an AV as the difference in travel time for all vehicles in a system when the AV is present versus a counterfactual simulation in which that AV is removed. Marginal cost relates to difference rewards, introduced in cooperative multi-agent settings to address credit assignment when agents optimize a shared global objective [11,27,37]. Difference rewards compare the global reward with a coun- terfactual in which an agent’s contribution is removed or re- placed by a default action, providing a reward that reflects the agent’s individual contribution to the system. In traffic routing, Grunitzki et al. [13] studied route choice with reinforcement learning and compared agents trained with selfish travel time rewards and difference rewards, showing that the latter can achieve lower average system travel times. Prior work introduced the marginal cost reward to help MARL routing AVs reach SO solutions faster than selfish routing AVs in a toy network where the UE and SO solutions coincided [29]. In this work, we examine whether marginal cost reward shaping can prevent selfish AV routing from re- ducing AV travel times at the expense of HDVs in real-world traffic networks. We additionally propose a method to decrease the counterfactual simulations needed to compute the marginal cost reward by rewarding a fraction of the most impactful AVs in the system. In the literature, researchers have studied the identification of impactful agents to explain agents’ importance in multi- agent systems. For example, Heuillet et al., [14] used Shapley values to quantify each agent’s contribution to the global re- ward. Chen et al. [9] stated that an agent is impactful if the system total reward is largely changing when its action is ran- domized. Following a similar intuition, we consider an AV to be impactful when alternative route choices lead to substantial differences in total system travel time. Research gap: To the best of our knowledge, little attention has been given to designing AV routing objectives that min- imize AV travel times while mitigating travel time increases for other road users in mixed-autonomy traffic. Prior work has explored different AV routing objectives but has not explic- itly investigated how to mitigate the travel time externalities these objectives may impose on HDVs or the system [1,2,16]. Similarly, existing work on difference rewards has primarily focused on cooperative settings, whereas we use the coun- terfactual term to account for the externalities that one agent population can impose on another while coexisting in the same environment [11, 13, 27, 37]. We address this gap by incorporating marginal cost into the AV routing reward and examining how the trade-off between individual AV travel times and their system impact during MARL training affects AV, HDV, and system travel times after training. Since AV routing objectives can shape how congestion costs are distributed among agents participating in this complex system. We further investigate selective marginal cost reward shaping, in which we apply marginal cost only to a fraction of the most impactful AVs. We examine whether we can achieve similar AV, HDV, and system travel times with fewer counterfactual simulations, providing a more scalable approach to social AV routing. 2 Preliminaries 2.1 Urban routing problem We study the route choice problem in which, each day, a num- ber of vehicles make one trip from an origin to a destination point in an urban traffic network at a specific departure time by selecting a route from a set of discrete alternatives. To simulate this problem, we use the RouteRL [3] framework integrated with SUMO [22], a continuous microscopic traffic simulator that enables modeling complex urban traffic networks and agent interactions. Our urban environment initially consists only of HDVs, which choose routes using random utility theory [8]. After consecutive days of commuting, HDV route choices stabilize. Then, a number of HDVs switch to using AVs, which can use various MARL algorithms to learn routing strategies. AVs keep the departure time, origin, destination points, and action space (set of routes they can travel on) of their human prede- cessors. Human learning is thereafter disabled, and human route choices are sampled from the fixed learned distribution. This provides behaviorally plausible human route choices that AV routing policies can be trained on before deployment. AV observations include an estimate of each route’s travel time based on travel times of earlier-departing vehicles with the same origin and destination, together with the AV’s origin, des- tination, and departure time. The set of routes does not change during the simulation or from one day to the next, and both departure times and demand remain constant across simulation days. 2.2 Problem formulation We abstract the daily (repeated) route choices made by hu- mans (and AVs in the future) in capacitated networks (limited resources) as a repeated congestion game [15]. This can be formalized as a one-cycle Agent Environment Cycle (AEC) problem [35], defined in [2, 28] as a tuple: ⟨S,N,I,U i i∈N ,r i i∈I ,Ω i i∈I ,υ,⟩ whereN = I ∪Hdenotes the set of all agents, withI representing AV agents andHhuman agents. In each episode (day or iteration), each AV agent inIand each human agent inHselects a route from their respective action space, with agents acting sequentially according toυ. The AEC game tuple includes: • A finite set of statesS . • A finite set of AV agentsI, a finite set of human agentsH. • For each agent i∈N : – The agent selects an actionu i ∈ U i , forming a joint action u∈ U ≡ U N . 3 • For each agent i∈I: – r i is the immediate reward agentiaims to maximize, as defined in [2, 28]. – Ω i is the set of possible observations for agent i. • υis the agent-selection mechanism that determines which agent acts next by sorting agents by their departure time from their origin. In this problem, AVs learn routing policiesπ i (u i | ω i )over consecutive days (episodes) simulated in SUMO to maximize their immediate rewards. A wide range of MARL algorithms can be used to learn such policies, which select routes based on each AV’s current observation ω i . 3 Method 3.1 Reward with marginal cost We postulate that adding to the AV reward a component that captures the agent’s impact on all other agents in the system, in our case, the marginal cost, can guide AVs towards route choices that account for the travel time increases they may impose on HDVs. We formulate the reward of an AV agenti as: r i (u) =−e i (u)− βm i (u), i∈I(1) wheree i (u)denotes the travel time experienced by indi- vidual AVi, under the joint actionuas computed by SUMO, andm i (u)represents the marginal cost of AViin the system. The parameterβweights the marginal cost component in the agent’s reward. The marginal cost of an agentiis represented by: m i (u) = N X j=1 j̸=i e j (u)− e j (u −i ) , N =|H∪I|,(2) and corresponds to the travel time differences of all the agentsjin the system, when agentiis present, and in a coun- terfactual simulation where agentiis removed.u −i denotes the joint action of the system where agentiis not present in the simulation. Whenβ = 0, AVs behave selfishly by minimizing their individual (selfish) travel time. Positive values ofβintroduce a trade-off between AVs routing selfishly and accounting for the impact of their route choices on the system. 3.2 Marginal cost table To compute the marginal cost for each AV, we construct the marginal cost matrixM (u), which records the impact of each AV on the travel times of all other agents in the system. Each entryM i,j (u)quantifies the impact of agentjon the travel time of agent i. M i,j (u) = ( e i (u)− e i (u −j ), i̸= j, 0,i = j, j ∈I.(3) The marginal cost of agentj,m j (u), is obtained by sum- ming the entries in columniof the marginal cost matrix, cap- turing the total change in travel time experienced by all agents in the system due to the presence of AV i. We build the marginal cost matrixM (u)by rerunning the simulation for a specific joint actionu, once for each AV Algorithm 1 AV training with marginal cost reward Require:Human agentsH, AV agentsI, AV policies π i i∈I , HDV route-choice distributionsP h h∈H , set of AVs for which we compute the marginal costI MC ⊆ I, coefficient β 1: for each episode (day) do 2:u←∅ // Main simulation 3:for each agent k ∈H∪I by υ order do 4: Advance the simulation to agentk’s decision time 5:if k ∈H then ▷ HDVs 6:Sample route u k ∼ P k 7:else ▷ AVs 8:Observe ω k and select route u k ∼ π k (·| ω k ) 9:end if 10:Agent k takes route u k , u[k]← u k 11:end for 12:Record travel timese j (u) j∈H∪I // Counterfactual simulations 13:for each AV i∈I MC do 14:for each agent k ∈H∪I in υ order do 15: Advance the simulation to agentk’s departure time 16:if k ̸= i then 17:Apply the stored route u k 18:end if 19:end for 20:Record counterfactual travel timese j (u −i ) j̸=i 21:m i (u)← X j̸=i e j (u)− e j (u −i ) 22:r i (u)←−e i (u)− βm i (u) 23:end for // AV learning 24:for each AV i∈I do 25:Update policy π i using r i (u) 26:end for 27: end for agent. In each additional simulation, we remove a single AV j, while keeping the routes and departure times of all other agents fixed. As a result, constructing the matrix for one episode (joint action) requires generating|I|additional and separate environment instances. Since these counterfactual simulations are independent, they can be executed in parallel to reduce computation time. Algorithm 1 details the marginal cost reward computation during AV training. 3.3 Selective marginal cost To reduce the number of simulations needed to compute marginal cost counterfactuals for all AV agents in the sys- tem, we propose selective marginal cost reward shaping. In selective marginal cost reward shaping, we rank AVs by their impact on the system. We assess an AV’s impact by how much system travel times differ when it chooses different routes. Then, we compute the marginal costm i only for a fraction of the most impactful AVs, forming the subsetI MC ⊆ I. The remaining AVs aim to minimize their own travel time. In 4 this way, selective marginal cost reward shaping concentrates the additional counterfactual computation on the AVs with the largest estimated system impact, allowing us to investi- gate whether similar AV, HDV, and system travel times can be achieved with fewer counterfactual simulations. To identify impactful AV agents, we construct a dataset Dof sampled joint action realizations. Each sampled ∈ D represents a day of commute, whereu (d) = (u (d) 1 ,...,u (d) N ) is the vector of joint actions and the vector of realized travel times ise (d) = (e 1 (u (d) ),...,e N (u (d) )). For each sample, we compute the total system travel time as: T (u (d) ) = N X i=1 e i u (d) , N =|H∪I|.(4) For each AV agenti∈Iand each available actiona∈U i , we then extract the subset of samples in which agentiselects action a: D i,a = n d∈D u (d) i = a o .(5) Then, we computeT i,a as the average total system travel time over all such samples where agent i chooses action a: T i,a = 1 |D i,a | X d∈D i,a T u (d) (6) Then, for each actiona ∈ U i we collect the re- sulting action-conditioned system travel timesT i = h T i,a 1 , T i,a 2 ,..., T i,a |U i | i . We then quantify the historical impact of agenti, denoted asH i , as the standard deviation of the average system travel times across each possible action. H i = Std a∈U i T i,a .(7) A highH i value indicates that, among the samples in dataset D, the system travel times differ substantially when agenti chooses different routes, while a lowH i value indicates small system travel times changes when agentichooses different routes. By ranking AV agents by this historical impact score H i , we can reward a fraction of the most impactful AV agents with the marginal cost reward formulation. Selective marginal cost reward shaping therefore allows us to examine whether similar AV, HDV, and system travel times can be achieved with fewer counterfactual simulations. 4 Results We begin by introducing the experimental setting in Section 4.1. Next, in Section 4.2, we show that on two real-world traffic networks, selfish AV routing can disadvantage HDVs, whose travel times increase after AV introduction. We then demonstrate that rewarding all AVs with the marginal cost can mitigate the increase in HDV travel times and additionally improve AV travel times. Finally, in Section 4.3, we apply the marginal cost reward only to a fraction of the most impactful AVs, and show that rewarding just 40% of AVs can achieve AV, HDV, and system travel times comparable to those obtained when marginal cost is applied to all AVs, while reducing the number of additional simulation runs by 40%. 4.1 Experimental setting We assess the performance of our marginal cost reward for- mulation through experiments in the real-world Ingolstadt and a) Ingolstadtb) Provins Origin pointsDestination points Figure 2: Traffic networks of Ingolstadt with 1035 agents (a) [21], and Provins (b) with 523 agents [2]. Provins traffic networks, shown in Figure 2. The Ingolstadt traffic network includes 850 nodes, 1426 links, 306 unique Origin-Destination (OD) pairs, and a demand of 1035 agents sourced from the InTAS traffic scenario [21] and used in the RESCO benchmark [5]. The Provins traffic network includes 2173 nodes, 4216 links, 517 unique OD pairs, and 523 travel- ing vehicles, sourced from the URB benchmark [2]. At the beginning of each experiment, the system contains only HDVs, which repeatedly choose routes among four op- tions over 300 days of commute, as in [2,3,16]. On day 301, a specified fraction of HDVs is replaced by AVs that learn MARL routing policies using the REINFORCE algorithm [18,41], where each agent is an independent learner [34]. We use REINFORCE in our experiments, but the proposed reward formulation is not specific to this algorithm and can be incorpo- rated into any MARL algorithm. We train the AVs for 10,000 training iterations or days of commute in both the Ingolstadt and Provins traffic networks, followed by a 100-day testing phase in which the learned policies are no longer updated. All results reported in Sections 4.2 and 4.3 are computed over the 100-day testing phase. We report the REINFORCE algorithm hyperparameters in the Supplementary Material. For selective marginal cost reward shaping, we identified the most impactful vehicles from the 800k joint action dataset D. We sampled from enormous space of possible joint actions (for 1,000 vehicles where each vehicle can choose among four alternative routes, there are4 1000 possible joint actions) a diverse datasetDusing several sampling methods, including random, balanced, Dirichlet [33], Maximin distance sampling [17], and simplex-lattice design [31] (see Section 3.3). 4.2 Performance of the marginal cost We demonstrate in Figure 3(a, d) that in both the Ingolstadt and Provins real-world traffic networks, humans consistently experience longer travel times across all examined AV penetra- tion rates when AVs selfishly minimize their individual travel times (β = 0). The negative relative travel-time chage indicate that after AV introduction, HDV travel times increase. AVs, on the other hand, experience shorter travel times compared to those achieved by the human drivers they replaced at episode 300 (see Figure 3(b, e) forβ = 0). When marginal cost is incorporated into the AV reward (β = 0.1for Ingolstadt and 5 00.10.20.5 β 0.2 0.4 0.6 AV penetration rate -1.3%-1.1%-1.0%-0.9% -1.7%-1.1%-1.0%-0.7% -1.7%-1.2%-1.0%-0.8% a) HDVs 00.10.20.5 β +4.7%+4.7%+2.4%-0.3% +3.7%+3.8%+2.5%-1.0% +3.7%+4.0%+2.8%-1.8% b) AVs 00.10.20.5 β -0.1%+0.1%-0.3%-0.8% +0.5%+0.9%+0.4%-0.8% +1.5%+1.9%+1.3%-1.4% c) All vehicles 00.10.51 β 0.2 0.4 0.6 AV penetration rate -2.8%-2.5%-2.3%-1.9% -1.5%-1.2%-1.1%-0.9% -1.3%-1.0%-0.9%-0.7% d) HDVs 00.10.51 β +9.0%+9.2%+8.8%+5.3% +7.7%+8.0%+7.1%+2.6% +8.6%+8.8%+8.0%+1.2% e) AVs 00.10.51 β -0.4%-0.2%-0.1%-0.5% +2.1%+2.5%+2.2%+0.5% +4.6%+4.9%+4.4%+0.4% f) All vehicles −9.2% −4.6% 0.0% 4.6% 9.2% Relative travel-time improvement Ingolstadt Provins Figure 3: Average agent relative travel time change, computed as the difference between travel time before AV introduction and travel time during the testing phase, normalized by travel time before AV introduction. Negative values indicates the vehicle is disadvantaged after AV introduction, and a positive indicate the vehicle is better off after AV introduction. Values are averaged across three experimental seed replications, with standard deviations reported in the Supplementary Material. Provins), the HDV relative travel time change increases rela- tive toβ = 0, indicating that HDVs experience shorter travel times than under selfish AV routing. At the same time, AVs experience slightly shorter travel times on average, showing that accounting for the system does not necessarily come at the expense of AV travel times. β=0 β=0.1 β=0.2 β=0.5 Ingolstadt AVHDV AVHDV AVHDV AVHDV −3−2−10123 β=0 β=0.1 β=0.5 β=1 Provins AVHDV AVHDV AVHDV AVHDV Travel time relative to system average (%) Figure 4: Change in average AV, and HDV travel times at 40% AV penetration in Ingolstadt at various weights of social componentβin the reward. Negative values indicate shorter travel times compared to the system average. We then observe that asβincreases, e.g.,β = 0.2, 0.5in Ingolstadt, andβ = 0.5, 1in Provins, the relative travel-time change on HDVs increases, indicating that HDVs experience shorter travel times. However, this comes at the cost of longer AV travel times. Increasingβbeyond a certain value wors- ens both AV and overall system travel times (see Ingolstadt, β = 0.5, Provinsβ = 1). This indicates that placing too much weight on the social component can disadvantage 020406080100 β=0 β=0.1 β=0.2 β=0.5 a) Ingolstadt 9.4% 11.4% 14.0% 18.4% 62.7% 57.1% 52.6% 46.2% 26.8% 29.0% 29.3% 29.9% 1.1% 2.5% 4.1% 5.6% 020406080100 Mean percentage of AVs choosing each route option (%) β=0 β=0.1 β=0.5 β=1 b) Provins 2.6% 2.3% 3.7% 7.4% 67.4% 68.2% 63.9% 59.2% 25.4% 25.6% 30.1% 25.9% 4.6% 3.9% 2.3% 7.5% Option 1Option 2Option 3Option 4 Figure 5: Mean percentage of AVs choosing routes from respective groups averaged over the 100 days of the testing phase on the Ingol- stadt (a) and Provins (b) networks for a 40% AV penetration rate. the system and individual AVs. Figure 4 shows that forβ = 0.1, 0.2in Ingolstadt, the gap between HDV and AV average travel times relative to the sys- tem average narrows, indicating that AVs, by accounting for their impact on the system, can reduce the travel-time gap between the two groups. Atβ = 0.5, however, the HDV ratio decreases while the AV ratio increases, indicating that AVs placing greater weight on system performance can ben- efit HDVs relative to the system average while making AVs worse off. In Provins, increasingβfrom 0.1 to 1 progressively narrows the travel-time gap between AVs and HDVs, with the two groups approaching the system average at β = 1. Differences in HDV, AV, and system travel times across values ofβare accompanied by changes in AV route choices, as shown in Figure 5. As the weight on marginal cost increases, the mean percentage of AVs choosing routes from respective groups shifts from the one observed under selfish routing. In 6 Table 1: Selective marginal cost reward shaping, specifically when 40% or 50% of the most impactful AVs are rewarded with the marginal cost, can achieve relative travel-time changes (see Figure 3) close to those obtained when all AVs receive the marginal cost reward, while reducing the number of counterfactual simulations. The results are reported for the Ingolstadt traffic network with 40% AV penetration. The AV subgroups column reports the relative travel-time change of selfish AVs (Selfish) and AVs receiving the marginal cost reward (MC). MC denotes marginal cost. Values are reported as mean± standard deviation across experimental replications. BehaviorMC AVs Counterfactual simulations HDVsAVsAV subgroupsAll β = 0.0 Selfish–- −1.66± 0.19 +3.74± 0.75–+0.50± 0.41 β = 0.1 MC100%4,140k −1.06± 0.17 +3.79± 0.71 MC: +3.79± 0.71 +0.88± 0.39 Selective MC50%2,070k −1.17± 0.18 +3.78± 0.72 MC: +3.51± 0.74 Selfish: +4.05± 0.70 +0.81± 0.39 Selective MC40%1,660k −1.25± 0.17 +3.78± 0.73 MC: +3.30± 0.75 Selfish: +4.10± 0.72 +0.76± 0.38 Selective MC10%420k −1.50± 0.19 +3.78± 0.74 MC:−0.18± 0.70 Selfish: +4.21± 0.74 +0.61± 0.40 Selective MC (random)10%420k −1.60± 0.18 +3.75± 0.72 MC: +3.45± 0.70 Selfish: +3.78± 0.72 +0.54± 0.39 Ingolstadt, for example, the share of AVs choosing routes from the 2nd group decreases fromβ = 0toβ = 0.5, while the shares choosing Routes from groups 1, 3, and 4 increase. Here, Options 1-4 correspond to route options or groups rather than physical routes. In Ingolstadt, each of the 306 unique OD pairs has four candidate routes, which correspond to 1224 distinct route alternatives. 4.3 Performance of the selective marginal cost We evaluate selective marginal cost reward shaping on the Ingolstadt traffic network atβ = 0.1, where marginal cost improves travel times for AVs, HDVs, and the system com- pared with selfish AV routing. Our results show that applying this reward only to a subset of AVs yields relative travel-time changes (see Figure 3) for HDVs, AVs, and the system that are close to those obtained when all AVs receive the marginal cost reward, while requiring fewer counterfactual simulations. Figure 6 shows the cumulative distribution of the historical impact (H i see Equation 7) of each of the 1035 vehicles on the Ingolstadt traffic network. Most vehicles have impact scores concentrated around 30. A smaller fraction of vehicles show lower or higher impact, indicating that vehicles differ in how strongly their route choices are associated with total system travel time. Table 1 reports the relative travel-time change for HDVs, AVs, and all vehicles in the system, as shown in Figure 3, when a fraction of the most impactful AVs receive the marginal cost reward. Specifically, when marginal cost is applied to the 50% and 40% most impactful AVs, HDV travel times improve by 0.5% and 0.4% compared to selfish AV routing, respectively, around 0.1% and 0.2% below the improvement obtained when marginal cost is applied to all AVs. We then tested a smaller fraction, specifically the 10% most impactful AVs, and found that HDV travel times still improve by 0.16% compared to selfish AV routing, although this is about 0.45% below the case when marginal cost is applied to all AVs. Thus, applying selective marginal cost to either 40% or 50% of the most impactful AVs could approximate the results of applying marginal cost to all AVs, retaining approximately 68% and 82% of the improvement in HDV relative travel-time change over selfish AV routing, respectively, while reducing the number of counterfactual simulations. To see if rewarding a fraction of the most impactful AVs based onH i is better than choosing a random fraction of AVs in the system. In the Selective MC (random) row of Table 1, we compare relative travel-time changes for HDVs, AVs, and all vehicles in the system when we apply marginal cost to a randomly selected fraction of AVs instead of the most impactful ones. We find a0.1%improvement in relative travel- time change for HDVs when we apply marginal cost to the 10% most impactful AVs instead of a random 10% of AVs in the system. 010203040 Historical agent impact score H i 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative fraction of agents 50th: 29.7 80th: 30.7 90th: 31.6 95th: 32.3 Figure 6: Empirical cumulative distribution of historical agent impact scoresH i retrieved from datasetDof 800 thousand joint actions sampled in Ingolstadt. Dashed lines indicate the 50th, 80th, 90th, and 95th percentiles. For the Ingolstadt network at a 40% AV penetration rate (around 414 AVs), 10k training iterations require about 4,140k counterfactual simulations. When we compute the marginal cost reward for 40% of the AVs, this reduces to about 1,660k 7 counterfactual simulations. We then need to account for the additional 800k joint-action samples of the datasetDused to identify the AVs with the largest system impact (see Figure 6), for a total of approximately 2,456k additional simulation runs. Thus, selective marginal cost reward shaping can reduce the additional simulation runs from 4 to 2.4 million, about a 40% reduction, while achieving an average HDV relative travel-time change of about 0.2 percentage points below that obtained when marginal cost reward shaping is applied to all AVs. In the AV subgroups column of Table 1, we compare relative travel-time changes of AVs receiving the selfish reward (Self- ish) and those receiving the marginal cost reward (MC). When the 50% or 40% most impactful AVs receive the marginal cost, MC AVs experience smaller relative travel-time change than selfish AVs. This difference is more pronounced when marginal cost is applied to the 10% most impactful AVs, where MC AVs experience a relative travel-time change of−0.18%, while selfish AVs experience a+4.21%change. Thus, AVs that account for their impact on the system may be individually worse off, experiencing smaller relative travel time changes than selfish AVs. 5 Conclusion AVs will increasingly operate in complex, capacitated urban traffic networks, where road users compete for limited road capacity, and each user’s route choices can affect others’ travel times. AVs may use a range of algorithms, with MARL being a promising one, to learn routing strategies that may differ from, and be more efficient than, those typically made by HDVs, potentially resulting in shorter AV travel times. Understanding how MARL AV routing can influence traffic performance is crucial to understanding how this emerging technology, AVs, may shape the performance of future urban transportation systems. In this work, we study social AV route choice in mixed- autonomy traffic environments shared with HDVs, where AVs use MARL to learn routing policies. To reduce the travel time costs AV routing imposes on other road users, we incorporate marginal cost into the AV reward. Marginal cost is a counterfactual defined as the difference in travel times for all vehicles in the system when a given AV is present versus absent from the simulation. This formulation lets us include in MARL training the trade-off between minimizing an AV’s own travel time and reducing its impact on the system. Our results show that incorporating marginal cost into the AV reward can mitigate increases in HDV travel times while also reducing AV travel times relative to selfish AV routing, indicating that AVs that account for their impact on the system can improve travel times for both HDVs, AVs, and the system. However, placing greater weight on marginal cost can further improve HDV travel times at the expense of AVs and worsen total system performance. For example, at 40% AV penetration in Ingolstadt, increasingβfrom 0 to 0.1 improves HDV, AV, and system travel time relative to the travel times before AVs were introduced by 0.6, 0.1, and 0.4 percentage points, respectively. Increasingβto 0.2 yields an additional 0.1 improvement in travel time change for HDVs but reduces the relative travel time change for AVs and the system by 1.3 and 0.5 percentage points, respectively, relative to β = 0.1. Since computing the marginal cost requires an additional counterfactual simulation for each AV, we propose selective marginal cost reward shaping to reduce this computational burden. We find that assigning the marginal cost reward to a fraction of the most impactful AVs yields improvement in relative travel-time change for HDVs, AVs, and the system that are close to those obtained when all AVs receive the marginal cost reward. In the Ingolstadt traffic network, rewarding the 40% most impactful AVs reduces the number of additional simulations by around 40%, while the HDV relative travel time change is around 0.2 percentage points lower than when all AVs receive the marginal cost reward. Overall, our findings show that AV routing algorithms can influence not only AV travel times but also how travel time costs are distributed across road users. AV routing policies that optimize only individual AV travel times may systematically disadvantage human drivers. By showing that marginal cost re- ward shaping can reduce these effects, our results highlight the importance of designing AV routing algorithms that account for AVs’ impact on shared transportation systems. References [1] Ahmet Onur Akman, Anastasia Psarou, Zolt ́ an Gy ̈ orgy Varga, Grzegorz Jamr ́ oz, and Rafal Kucharski. Im- pact of collective behaviors of autonomous vehicles on urban traffic dynamics: A multi-agent reinforcement learning approach.In Seventeenth European Work- shop on Reinforcement Learning, 2024. URLhttps: //openreview.net/forum?id=88zP8xh5D2. [2] Ahmet Onur Akman, Anastasia Psarou, Michał Hoff- mann,Łukasz Gorczyca,Łukasz Kowalski, Paweł Gora, Grzegorz Jamr ́ oz, and Rafał Kucharski. Urb – urban rout- ing benchmark for rl-equipped connected autonomous vehicles. In Advances in Neural Information Processing Systems, 2025. [3] Ahmet Onur Akman, Anastasia Psarou,Łukasz Gorczyca, Zolt ́ an Gy ̈ orgy Varga, Grzegorz Jamr ́ oz, and Rafał Kucharski.Routerl:Multi-agent re- inforcement learning framework for urban route choice with autonomous vehicles.SoftwareX, 31:102279, 2025.ISSN 2352-7110.doi: https: //doi.org/10.1016/j.softx.2025.102279. URLhttps: //w.sciencedirect.com/science/ article/pii/S2352711025002468. [4] W. Brian Arthur. Inductive reasoning and bounded ratio- nality. The American Economic Review, 84(2):406–411, 1994. ISSN 00028282. URLhttp://w.jstor. org/stable/2117868. [5] James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In Proceedings of the Thirty-fifth Conference on Neural Information Pro- cessing Systems (NeurIPS 2021) Datasets and Bench- marks Track, December 2021. [6] Martin J. Beckmann, C. B. McGuire, C. B. Winsten, and T. C. Koopmans. Studies in the Economics of Trans- portation. Yale University Press, New Haven, 1956. doi: 10.2307/2227641. 8 [7]Enide A. I. Bogers, Michel Bierlaire, and Serge P. Hoogendoorn. Modeling learning in route choice. Trans- portation Research Record, 2014(1):1–8, 2007. doi: 10.3141/2014-01. [8]Ennio Cascetta. Transportation Systems Analysis: Mod- els and Applications, volume 29 of Springer Optimiza- tion and Its Applications.Springer, 2009.ISBN 978-0-387-75857-2. doi: 10.1007/978-0-387-75857-2. URLhttps://link.springer.com/book/10. 1007/978-0-387-75857-2. [9] Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Jun Hu, Qing Wang, and Fanjiang Xu. Under- standing individual agent importance in multi-agent sys- tem via counterfactual reasoning. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelli- gence and Thirty-Seventh Conference on Innovative Ap- plications of Artificial Intelligence and Fifteenth Sym- posium on Educational Advances in Artificial Intelli- gence, AAAI’25/IAAI’25/EAAI’25. AAAI Press, 2025. ISBN 978-1-57735-897-8. doi: 10.1609/aaai.v39i15. 33733. URLhttps://doi.org/10.1609/aaai. v39i15.33733. [10]Richard Cole, Yevgeniy Dodis, and Tim Rough- garden.How much can taxes help selfish rout- ing?Journal of Computer and System Sci- ences, 72(3):444–467, 2006.ISSN 0022-0000. doi:https://doi.org/10.1016/j.jcss.2005.09.010. URLhttps://w.sciencedirect.com/ science/article/pii/S0022000005001121. Network Algorithms 2005. [11] Sam Devlin, Logan Yliniemi, Daniel Kudenko, and Kagan Tumer. Potential-based difference rewards for multiagent reinforcement learning.In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS ’14, page 165–172, Richland, SC, 2014. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450327381. [12]Lawrence Christopher Duncan, Thomas Kjær Ras- mussen, David Paul Watling, Laurent Cazor, and Otto Anker Nielsen. Identification of local detoured- ness and correlation effects in a bounded route choice model: solution & estimation in large-scale networks.Transportmetrica A Transport Science, 2025.ISSN 2324-9935.doi:https://doi.org/ 10.1080/23249935.2025.2543467.URLhttps: //w.sciencedirect.com/science/ article/pii/S2324993525001022. [13] Rafael Grunitzki, Guilherme de Oliveira Ramos, and Ana L C Bazzan. Individual versus difference rewards on reinforcement learning for route choice. In 2014 Brazilian Conference on Intelligent Systems, pages 179– 184. IEEE, 2014. [14] Alexandre Heuillet, Fabien Couthouis, and Natalia D ́ ıaz- Rodr ́ ıguez. Collective explainable ai: Explaining coop- erative strategies and agent contribution in multiagent reinforcement learning with shapley values. Comp. In- tell. Mag., 17(1):59–71, February 2022. ISSN 1556- 603X. doi: 10.1109/MCI.2021.3129959. URLhttps: //doi.org/10.1109/MCI.2021.3129959. [15]Ron Holzman and Nissan Law-Yone. Strong equilibrium in congestion games. Games and economic behavior, 21 (1-2):85–101, 1997. [16]Grzegorz Jamr ́ oz, Ahmet Onur Akman, Anastasia Psarou, Zolt ́ an Gy ̈ orgy Varga, and Rafał Kucharski. Social impli- cations of coexistence of cavs and human drivers in the context of route choice. Scientific Reports, 15(1):6768, 2025. [17]M.E. Johnson, L.M. Moore, and D. Ylvisaker. Minimax and maximin distance designs.Jour- nal of Statistical Planning and Inference,26 (2):131–148, 1990.ISSN 0378-3758.doi: https://doi.org/10.1016/0378-3758(90)90122-B. URLhttps://w.sciencedirect.com/ science/article/pii/037837589090122B. [18]Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How.A policy gradient algorithm for learning to learn in multiagent reinforcement learning.In Ma- rina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5541–5550. PMLR, 18– 24 Jul 2021. URLhttps://proceedings.mlr. press/v139/kim21g.html. [19]Daniel A. Lazar, Erdem Bıyık, Dorsa Sadigh, and Ramtin Pedarsani.Learning how to dynamically route au- tonomous vehicles on shared roads, 2021. URLhttps: //arxiv.org/abs/1909.03664. [20] Dawei Li, Siqi Feng, Yuchen Song, Xinjun Lai, and Shlomo Bekhor.Asymmetric closed-form route choice models: Formulations and comparative appli- cations. Transportation Research Part A: Policy and Practice, 171:103627, 2023. ISSN 0965-8564. doi: https://doi.org/10.1016/j.tra.2023.103627. URLhttps: //w.sciencedirect.com/science/ article/pii/S0965856423000472. [21] Silas C. Lobo, Stefan Neumeier, Evelio M. G. Fernandez, and Christian Facchi. Intas – the ingolstadt traffic sce- nario for sumo, 2020. URLhttps://arxiv.org/ abs/2011.11995. [22] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Fl ̈ otter ̈ od, Robert Hilbrich, Leonhard L ̈ ucken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic sim- ulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL https://elib.dlr.de/124092/. [23]A. Mas-Colell, M.D. Whinston, and J.R. Green. Mi- croeconomic Theory.Oxford student edition. Ox- ford University Press, 1995. ISBN 9780195073409. 9 URLhttps://books.google.pl/books?id= KGtegVXqD8wC. [24]Deepak K Merchant and George L Nemhauser. Optimal- ity conditions for a dynamic traffic assignment model. Transportation Science, 12(3):200–207, 1978. [25]Srinivas Peeta and Hani S. Mahmassani. System optimal and user equilibrium time-dependent traffic assignment in congested networks. Annals of Operations Research, 60:81–113, 1995. doi: 10.1007/BF02031941. [26]A. C. Pigou. The economics of welfare. The Economic Journal, 43(170):329–330, 06 1933. ISSN 0013-0133. doi: 10.2307/2224491. [27]Scott Proper and Kagan Tumer. Modeling difference rewards for multiagent learning. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 3, AAMAS ’12, page 1397–1398, Richland, SC, 2012. International Founda- tion for Autonomous Agents and Multiagent Systems. ISBN 0981738133. [28] Anastasia Psarou, Ahmet Onur Akman,Łukasz Gor- czyca, Michał Hoffmann, Grzegorz Jamr ́ oz, and Rafał Kucharski. Position: Collaboration between the city and the machine learning community is crucial to efficient autonomous vehicles routing. In AI4GOOD Workshop at ICML 2026, 2026. URLhttps://openreview. net/forum?id=24C18nSM4d. [29]Anastasia Psarou,Łukasz Gorczyca, Dominik Gaweł, and Rafał Kucharski. Autonomous vehicles need so- cial awareness to find optima in multi-agent reinforce- ment learning routing games. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’26, page 3417–3419, Richland, SC, 2026. International Foundation for Au- tonomous Agents and Multiagent Systems.ISBN 9798400723179.doi: 10.65109/FTHN6981.URL https://doi.org/10.65109/FTHN6981. [30]Gabriel de O. Ramos, Roxana R ̆ adulescu, Ann Now ́ e, and Anderson R. Tavares. Toll-based learning for min- imising congestion under heterogeneous preferences. In Proceedings of the 19th International Conference on Au- tonomous Agents and MultiAgent Systems, AAMAS ’20, page 1098–1106, Richland, SC, 2020. International Foun- dation for Autonomous Agents and Multiagent Systems. ISBN 9781450375184. [31]Henry Scheff ́ e. Experiments with mixtures. Journal of the Royal Statistical Society: Series B (Method- ological), 20(2):344–360, 07 1958. ISSN 0035-9246. doi:10.1111/j.2517-6161.1958.tb00299.x.URL https://doi.org/10.1111/j.2517-6161. 1958.tb00299.x. [32]Yosef Sheffi. Urban Transportation Networks: Equilib- rium Analysis with Mathematical Programming Methods. Prentice-Hall, Englewood Cliffs, 1985. [33]Chenxiao Song and Reiichiro Kawai. Sampling and change of measure for monte carlo integration on sim- plices. Journal of Scientific Computing, 98:64, 2024. doi: 10.1007/s10915-024-02461-0. [34]Ming Tan. Multi-agent reinforcement learning: inde- pendent vs. cooperative agents, page 487–494. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1997. ISBN 1558604952. [35]J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learn- ing, 2021. URLhttps://arxiv.org/abs/2009. 14471. [36]Luiz A. Thomasini, Lucas N. Alegre, Gabriel O. Ramos, and Ana L. C. Bazzan. Routechoiceenv: a route choice li- brary for multiagent reinforcement learning. In Adaptive and Learning Agents Workshop at AAMAS, 2023. [37]Kagan Tumer and Adrian Agogino. Agent reward shap- ing for alleviating traffic congestion. 01 2006. [38]J G WARDROP. Road paper. some theoretical aspects of road traffic research. Proceedings of the Institution of Civil Engineers, 1(3):325–362, May 1952. ISSN 1753- 7789. doi: 10.1680/ipeds.1952.11259. URLhttp:// dx.doi.org/10.1680/ipeds.1952.11259. [39] J. G. Wardrop.Road paper. some theoretical as- pects of road traffic research. 1952. URLhttps: //api.semanticscholar.org/CorpusID: 131127018. [40]David Paul Watling,Thomas Kjær Rasmussen, Carlo Giacomo Prato, and Otto Anker Nielsen. Stochastic user equilibrium with a bounded choice model.Transportation Research Part B: Method- ological, 114:254–280, 2018. ISSN 0191-2615. doi: https://doi.org/10.1016/j.trb.2018.05.004. URLhttps: //w.sciencedirect.com/science/ article/pii/S0191261517304083. [41]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3–4):229–256, May 1992. ISSN 0885- 6125. doi: 10.1007/BF00992696. URLhttps://doi. org/10.1007/BF00992696. 10