Paper deep dive
Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback
Chenyang Zhao, Vinny Cahill, Ivana Dusparic
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 2:05:59 PM
Summary
This paper proposes a multi-objective Reinforcement Learning from AI Feedback (RLAIF) framework to address reward design challenges in self-adaptive systems, specifically using Urban Traffic Control (TSC) as a case study. The authors utilize Large Language Models (LLMs) as preference annotators to generate scalable, user-aligned preference labels for trajectory pairs, mitigating the need for hand-crafted reward functions. The framework employs a structured chain-of-thought prompting strategy to handle conflicting objectives (e.g., throughput vs. emissions) and demonstrates that RLAIF can produce balanced trade-off policies without extensive reward engineering.
Entities (9)
Relation Signals (7)
RLAIF → uses → LLM
confidence 95% · RLAIF leverages large AI systems (generally LLMs) to automate large scale preference data collection.
RLAIF → appliedto → Urban Traffic Control
confidence 93% · We illustrate the proposed approach in a traffic signal control (TSC) problem... TSC provides an excellent exemplar for applicability of RLAIF algorithms
LLM → actsas → Preference Annotator
confidence 92% · LLMs can serve as annotators to autonomously generate preference labels.
SUMO → usedforsimulation → Urban Traffic Control
confidence 90% · We evaluate RLAIF on a single four-way intersection simulated in Simulation of Urban MObility (SUMO)
DQN → usedforpolicyupdate → RLAIF
confidence 88% · The policy is updated with standard off-policy RL algorithms (DQN in our implementation)
RLAIF → optimizes → Throughput
confidence 85% · The objectives include: {maximizing traffic throughput and minimizing carbon emission}
RLAIF → optimizes → Carbon Emission
confidence 85% · The objectives include: {maximizing traffic throughput and minimizing carbon emission}
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.
Tags
Links
- Source: https://arxiv.org/abs/2602.20728v1
- Canonical: https://arxiv.org/abs/2602.20728v1
Trouble viewing inline? Open PDF directly →
Full Text
36,823 characters extracted from source content.
Expand or collapse full text
Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback Chenyang Zhao zhaoc4@tcd.ie Trinity College Dublin Dublin, Ireland Vinny Cahill vinny.cahill@tcd.ie Trinity College Dublin Dublin, Ireland Ivana Dusparic ivana.dusparic@tcd.ie Trinity College Dublin Dublin, Ireland Abstract Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alterna- tive by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demon- strated that large language models (LLMs) can generate prefer- ence labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single- objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives. ACM Reference Format: Chenyang Zhao, Vinny Cahill, and Ivana Dusparic. 2026. Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback. In 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’26), April 13–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/ 3788550.3794864 1 Introduction Self-adaptive systems (SAS) increasingly rely on learning-based controllers to operate under uncertainty and evolving requirements. Reinforcement learning (RL), in particular deep RL utilizing deep function approximation, has demonstrated impressive capabilities across a variety of self-adaptive application domains, including traf- fic signal control [22], autonomous driving [5], and balloon naviga- tion [2]. However, RL systems rely heavily on a hand-crafted reward functions that encode the designer’s intent as a scalar feedback sig- nal. In practice, specifying a reward function that incentivises the agent to explore the optimal policy while aligning with the system objective is a challenging problem and often requires extensive trial-and-error. Misspecification can induce reward hacking, unsafe This work is licensed under a Creative Commons Attribution 4.0 International License. SEAMS ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2445-9/2026/04 https://doi.org/10.1145/3788550.3794864 behaviours, or brittle policies that overfit to proxy reward signals rather than truly reflect desired behaviour. The challenge becomes more pronounced in multi-objective settings, where performance must be balanced across multiple, potentially conflicting, criteria. Aggregating multiple objectives into a single scalar can be challeng- ing: poorly designed reward functions can lead the learning agent to over-optimize a dominant objective while neglecting others, and may fail to reflect the intent of users. To address the challenges in designing reward functions, prefer- ence based RL (PbRL) has emerged as a compelling alternative by replacing hand-crafted reward signals with pairwise comparisons of behaviours [6]. Instead of scalar rewards, the agent queries an anno- tator to indicate which of two presented behaviour trajectories (or segments) is preferred, then learns a reward model that incentivizes the agent to align with those preferences. More recently, RL from AI feedback (RLAIF) leverages large AI systems (generally LLMs) to automate large scale preference data collection. Despite promis- ing results [8,9], most existing work targets single-objective tasks, where the main challenges are encouraging sufficient exploration or expressing high-level intent as a scalar function. The question of how these approaches extend to multi-objective scenarios, with competing objective priorities, remains largely unexplored. In this paper, we investigate using LLMs as preference annota- tors for multi-objective RL (MORL). To address the gap in multi- objective RLAIF, we introduce an annotation workflow specifically tailored to multi-objective scenarios that elicits cross-objective com- parisons and conditions labels on user specifications. We empir- ically show that RLAIF can learn user-aligned policies, without requiring intensive work on reward engineering. We also demon- strate, in a scenario with directly competing objectives, that the policy behaviour can be directed to match desired trade-offs via natural language prompts. We illustrate the proposed approach in a traffic signal control (TSC) problem, as a canonical example of a multi-objective self-adaptive application. Traffic signal control inherently constitutes a multi-objective optimisation problem, in- volving competing performance metrics such as traffic throughput and ecological impacts (e.g., emissions). Furthermore, the priorities of different lanes or approaches across the same intersection lay- out may vary due to road importance or the presence of buses or emergency vehicles. An optimal controller must therefore adapt its behaviour to such variations and deploy different control policies. The inherent trade-offs among these objectives render the problem particularly challenging. Given this complexity, TSC provides an excellent exemplar for applicability of RLAIF algorithms, suitable for testing their performances in managing multiple conflicting objectives under dynamic conditions. arXiv:2602.20728v1 [cs.AI] 24 Feb 2026 SEAMS ’26, April 13–14, 2026, Rio de Janeiro, BrazilChenyang Zhao, Vinny Cahill, and Ivana Dusparic In summary, this paper posits that the recent advancements in LLMs and RLAIF techniques pave a path to a self-managed, verbally- steered learning loop for SAS with multiple objectives. Specifically, stakeholders can express high-level goals and constraints in natural language, and an LLM can be prompted to turn these directives into trade-off-aware preference labels over trajectories. An RL agent can then optimize its behaviour against these generated labels to produce either a single policy aligned with the stakeholder prefer- ences, or a diverse set of policies spanning qualitatively different trade-offs between objectives. 2 Related Work In this section, we introduce background work in Preference-based RL (PbRL) and in utilizing LLMs to provide feedback for RL systems, followed by related work on learning multi-objective self-adaptive systems in traffic signal control. 2.1 Preference-based Reinforcement Learning PbRL has emerged as a principled way to mitigate the challenges of reward engineering in reinforcement learning [6,11,24]. Rather than hand-crafting a reward, PbRL infers a reward model from pair- wise preference feedback, typically human comparisons between short trajectory segments or outcomes [6]. This is appealing be- cause humans generally find it more natural to compare behaviours than to specify a precise reward function [24]. PbRL has recently been extended to multi-objective RL (MORL), where agents must balance multiple criteria. One line of work elicits hierarchical or importance-aware preferences, comparing segments by scanning objectives in a ranked order until a salient difference is found [4]. Another assumes linearly weighted objectives and trains a weight- conditioned policy by sampling trade-off weights during learning [17]. Despite these advances, most prior systems depend on either human annotators or a scripted teacher—i.e., synthetic labels gen- erated from a oracle reward function [11,12]. To the best of our knowledge, we are the first to explore using LLMs as multi-objective annotators, scaling PbRL to real-world multi-objective problems with far less human effort. 2.2 Reinforcement Learning From AI Feedback The advent of powerful foundation models (e.g., GPTs, Llama) has catalysed efforts to integrate them into RL pipelines to reduce hu- man labour, by directly evaluating trajectories/transitions [10,25], helping specify reward functions [16,26], or providing action ad- vice [31]. These studies show that LLMs can benefit RL without requiring expert demonstrations or extensive reward engineering. Moreover, when explicit reward design is difficult and direct evalua- tions are noisy, Reinforcement Learning from AI Feedback (RLAIF) leverages such AI systems to generate pairwise preferences at scale, shown success in domains such as NetHack [9] and robotic manipu- lation [21]. Follow-up studies tackle label quality and robustness via crowd-sourced LLM annotators [20], ensembles of reward models [8], and richer supervision such as rating-style annotations [15]. However, prior work largely addresses single-objective tasks. In multi-objective settings, the preferred trade-off among objectives can vary across annotators. This heterogeneity induces noisy and inconsistent preference labels, complicating aggregation and re- ward learning in RLAIF. In this paper, we focus on multi-objective scenarios and study how to obtain and use LLM-generated prefer- ences that faithfully reflect these trade-offs while enabling stable, scalable policy optimisation. 2.3 Multi Objective Traffic Signal Control Intelligent traffic signal control (TSC) aims to adaptively adjust signal timings based on real-time traffic demand [29]. Deep RL approaches are frequently used in TSC systems due to their adapt- ability [23]. Different approaches use different reward designs to achieve the expected traffic behaviours, e.g., vehicle waiting time [7], queue length [30], and junction pressure [22]. Recently, grow- ing sustainability concerns have driven growing interest in multi- objective TSC which jointly optimises traffic efficiency, environ- mental impact, and fairness across stakeholders. Most methods use scalarization, typically combining different objectives in a weighted- sum reward, e.g., minimizing delay while penalizing carbon emis- sion consumption [27,28]. However, such approaches often require extensive weight tuning to find the balance between the objectives. Other work explores Pareto-based techniques that learn sets of poli- cies spanning different trade-offs[19], which provides flexibility in policy choices, at the cost of high computational resources. In this work, we explore the use of PbRL and AI feedback in addressing the multi-objective problem in TSC, aiming to reduce weight-tuning effort without incurring the computational burden of training an entire set of policies. 3 LLMs for RLAIF in Multi-objective Tasks In this section, we present our proposed RLAIF for multi-objective tasks framework. We tailor the annotation prompt template, explic- itly stating the task objectives and desired trade-offs via natural language, such that annotations better reflects user expectation over multiple objectives. Prior work [9,21] targets a single objective, e.g., opening a drawer with a robot arm or advancing to the next level in a game. Annotators can judge “how well the task was achieved” along one scalar criterion. In multi-objective settings, especially when objec- tives conflict, there is usually no single globally preferred solution. Consequently, annotator preferences become conditional on im- plicit weights over objectives, which vary across annotators and over time, yielding more ties and noisier, inconsistent labels. This in turn can destabilize reward-model training. To mitigate this issue, we propose the use of more structured prompts for LLM annotators tailored for multi-objective tasks, instructing them to evaluate the performances along each of the objectives and the overall alignment with user’s expectation. Fig. 1 provides an overview of our multi-objective RLAIF frame- work. The system alternates between data collection, preference- based reward learning, and policy learning. Provided with proper prompts about different task objectives and the overall goal, LLMs can serve as annotators to autonomously generate preference labels. Specifically, the current policy interacts with the environment to produce transitions(푠 푡 , 푎 푡 ,푠 푡+1 ), which are stored in an ordered replay bufferB. Periodically, pairs of trajectory segments, i.e., a sequence of transitions, are sampled fromBand an LLM queried Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI FeedbackSEAMS ’26, April 13–14, 2026, Rio de Janeiro, Brazil Collect transition & rewards Environment Agent Replay Buffer Reward Model Preference Dataset ... The objectives include: maximizing traffic throughput and minimizing carbon emission And the overall goal is: improving environmental impact while preserving throughput Compare and , which do you think performs better? Sample pairs Reward Learning RL AgentReward Model Update rewards in buffer Sample data & learn policy Figure 1: Our framework of RLAIF for multi-objective tasks: the policy interacts with the environment, transitions enter the replay bufferB; segment pairs are annotated by an LLM and stored into a preference bufferD; the reward model푟 휓 updates fromD; re-scoresB, and the policy is trained with RL algorithms with the updated replay buffer. During annotation, we make the optimization criteria explicit, stating the task objectives and desired trade-offs, so the learned reward reflects user expectations and aligns behaviour across multiple objectives without bespoke reward engineering. with an objective prompt to obtain preference labels; uncertain judgments are filtered and the remaining labels are stored in a preference bufferD. A reward model푟 휓 is then trained on these pairwise labels. After each update of the reward model, the tran- sitions inBare relabelled with new rewards using푟 휓 . The policy 휋 휃 is updated with standard off-policy RL algorithms (DQN in our implementation) using rewards provided by 푟 휓 . 3.1 Preference Annotation We modify the prompting template used in previous work [21] for generating preference labels via LLMs, so as to explicitly address the multi-objective nature of tasks and the users’ expectations. Specifically, the multi-stage chain-of-thought annotation process comprises the following steps: Observation analysisGiven a verbal instruction that specifies the target objectives, the LLM is prompted to examine a pair of observation segments휎 1 and휎 2 independently. To address the multiple objective nature, the LLM is prompted to evaluate each segment’s performance with respect to each individual objective, as well as how well the segment performs in balancing across objectives and addressing the overall goal. Segment comparisonNext, the LLM compares the two seg- ments휎 1 and휎 2 , drawing on both the raw observation data and the per-segment analyses generated in the previous step, to judge which segment better satisfies the overall goal instructions. Preference labellingFinally, the LLM is prompted to assign a preference label푦 ∈ 0,1,2, where 1 indicates a clear preference for the first segment휎 1 , 2 indicates a clear preference for the sec- ond segment휎 2 , and 0 indicates that the LLM is not confident in assigning a preference. Pairs labelled with 0 are excluded from the preference dataset, to improve the robustness of reward learning. The full annotation process is illustrated in Figure 2. When query- ing for annotation, a pair of segments is sampled, the chain-of- thought annotation workflow above applied, and the resulting label stored into the preference bufferD. Note that the human only participates before the start of the training process, providing in- structions on task objectives. Thereafter, all prompts are generated by the proposed framework and annotations during training are generated autonomously by the LLM. 3.2 Reward and Policy Learning Given collected transitions and preferences, the reward model is trained to approximate the underlying LLM preference distribution and the policy model is trained to maximize accumulated return. Following Bradley-Terry model [3], the estimated preference dis- tribution over a pair휎 1 , 휎 2 given a reward model푟 휓 is described as푃 휓 (휎 1 ≻ 휎 2 )= softmax 푟 휓 (휎 1 )− 푟 휓 (휎 2 ) /훽 . The objective of reward learning is to minimise the cross-entropy loss between estimated preference distribution and true labels: max 휓 E 퐷 h 1푦 푖 = 1 log 푃 휓 (휎 1 푖 ≻ 휎 2 푖 )+ 1푦 푖 = 2 log 푃 휓 (휎 2 푖 ≻ 휎 1 푖 ) i . For policy learning, DQN serves as the underlying RL algorithm. To bootstrap, we first collect a small replay buffer by training the agent under an arbitrary proxy reward and query preferences on segment pairs from this buffer to fit an initial reward model. Dur- ing training, comparison pairs are continually sampled from the evolving replay buffer. To address non-stationarity in rewards, we relabel the rewards of all stored transitions in the replay buffer with the latest reward model whenever it is updated, to ensure that rewards always are specified with the same reward model across the entire replay buffer [11, 21]. 4 RLAIF for Multi-objective TSC To demonstrate the effectiveness of RLAIF in a self-managed system with multiple objectives, we test the framework in traffic signal control (TSC) scenarios 1 . TSC regulates phase timing at intersec- tions to manage vehicle flows. The problem is inherently involving multiple objectives, including traffic throughputs along different junction approaches and carbon emissions. 1 Our code is publicly available on https://github.com/cyzhao1991/RLAIF_MO_TSC. SEAMS ’26, April 13–14, 2026, Rio de Janeiro, BrazilChenyang Zhao, Vinny Cahill, and Ivana Dusparic User Instruction You’re a traffic-flow supervisor, providing feedback on traf- fic conditions. You need to consider multiple objectives: OBJECTIVE 1, OBJECTIVE 2, ... The overall goal is USER SPECIFICATION. Segment 1 Consider the following scene: DESCRIPTION(휎 1 ). Summarize your key observations, and how it performs with respect to the overall goal. Segment 2 ... ... DESCRIPTION(휎 2 ). ... ... Comparison: How would you compare the performance between seg- ment 1 and segment 2? Is there any difference in terms of achieving balanced performance? Annotation: Please output a preference label over the compared pair. Output 1 if the first scene is better addressing the overall goal; 2 if the second one is better; and 0 if you are not sure or there is no difference. Figure 2: An illustrated workflow of the preference annota- tion process. The annotator is given considered objectives, the overall goal, and descriptions of the sampled pair of seg- ments휎 1 , 휎 2 , and outputs the preference label 푦 ∈ 0, 1, 2. 4.1 Experiment Setup Environment formulationWe evaluate RLAIF on a single four-way intersection simulated in Simulation of Urban MObility (SUMO) [13]. Each incoming approach comprises two lanes: one dedicated to left-turn movements and another for through and right-turn movements. In the experiments, each episode simulates the traffic for 10,000 seconds and each environment step corre- sponds to 5 seconds of real-time traffic flow. We follow the same configuration for traffic patterns as [1], with 200 vehicles per hour on each lane in the north-south (NS) direction and 600 each in the east-west (EW) direction. Signal actuation is framed as a four-phase discrete action space, NS left-turn, NS through and right-turn, EW left-turn, EW through and right-turn. Each action sets the lights for its direction as green and others as red. Note that a yellow phase of 2 seconds is automatically inserted before a red phase. The ob- servation is defined as [signal phase, min green, each lane vehicle density, each lane queue], where signal phase is a one-hot encoded vector indicating the current active green phase and min green is a binary value indicating whether minimal time for a green phase has already passed. This observation encodes information about both light phases and lane-level traffic. Implementation detailsTo translate simulator observations into natural language, we use a rule-based template that maps each element of the observation vector to a human-readable description and adds auxiliary context, e.g., the number of vehicles that passed The current light phase is North and South direction straight and right-turn allowed. The incoming lane densities are 0.16 for incoming lane 0 from north (straight and right-turn); 0.32 for incoming lane 1 from north (left-turn); ...... The outgoing lane densities are 0.11 for outgoing lane 0 towards south; 0.00 for outgoing lane 1 towards south; ...... The queue density are 0.00 for incoming lane 0 from north (straight and right-turn); 0.26 for incoming lane 1 from north (left-turn); ...... During the last 5 seconds, 4 vehicles passed through the junc- tion, and 381.5g of carbon was emitted. Figure 3: An exemplar text description of a traffic segment with length 1 in the task of balancing traffic throughput and environmental impact. The text description includes the observations, explained with their semantic meanings, as well as information on throughput and carbon emissions. in the last time step. An example translation in the task of balancing throughput and environmental impact is included in Figure 3. This representation helps LLMs assess traffic conditions and pro- vide consistent, interpretable preferences annotations with brief justifications. As for the prompts, we use a universal prompt tem- plate as described in Fig. 2 and gpt-4.1-nano [18] as the under- lying language model, with only the OBJECTIVES, USER SPECI- FICATION, and the text description for each compared segment DESCRIPTION (휎 ) varying across queries. Both the reward model and the Q network (policy model) use a 2-layer MLP architecture with 256 hidden units per layer and LeakyRelu activation. The reward model uses an ensemble of three networks with linear output heads and no activations. We use AdamW [14] as the optimizer with a weight decay coefficient of 0.01, a learning rate of 3푒−4, and a batch size of 128. The frequency at which we collect feedback is 5000 environment steps. We follow the disagreement-based strategy when sampling pairs for compar- ison, where pairs that models in the ensemble produce the most different estimations are prioritized for annotation. Concretely, at each sampling step, 1024×5 pairs are initially sampled from the replay buffer at random. For each sampled pair, three probabilities of휎winning are estimated with different reward models in the en- semble using Bradley-Terry model, and the standard deviation over the probabilities is considered as a measure for model disagreement. The sampled batch is then ranked based on their disagreement and the top 1024 samples are proceeded further for annotation. 4.2 Experiment Scenarios In the experiments, we evaluate RLAIF with the following two scenarios: throughput-emission and lane priorities. Throughput-emissionIn this setup, we consider two objec- tives that are most commonly discussed in TSC: overall through- put, which is the total number of vehicles exiting the junction, and the total carbon emission, which is calculated based on the HBEFA3 model in SUMO. Specifically, in the prompt template, we specify the OBJECTIVES as maximizing the throughput and mini- mizing carbon emission and the USER SPECIFICATION as improve environmental impact while preserving or only marginally reduc- ing throughput. We benchmark against a multi-objective baseline Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI FeedbackSEAMS ’26, April 13–14, 2026, Rio de Janeiro, Brazil 020k40k60k80k100k Step 60 80 100 120 140 Average Emission (g/s) RLAIF Linear Emission Throughput EW Only EW Priority Equal Priority NS Priority NS Only 26%74% 46%54% 49%51% 58%42% 77%23% NS ThroughputEW Throughput Figure 4: Left and middle: Average traffic throughput and CO2 emission in throughput-emission scenario through learning process. RLAIF learns better performances compared with single objective baselines, and falls short only of the Linear baseline, which requires reward engineering. Right: Comparison between 5 policies trained with different user specifications in lane priorities scenario with instructions towards different lane priorities. with linear reward scalarisation (“Linear”), using a weighted sum of the two pre-defined reward functions푟= 훼푟 1 +(1− 훼)푟 2 , with no learned reward models.푟 1 is the number of vehicles driving through the junction and푟 2 the total amount of carbon emissions in grams/second. To find the best weights, we conduct a grid search for훼, exploring values within the set0.1,0.2, ...,0.9. We monitor the visualized simulated traffic patterns of different agents as well as their real-time quantitive metrics, including throughput, carbon emission, average vehicle speeds, and queue length, and report the best validation result (훼=0.7). We also include single-objective DQN variants (“Throughput” and “Emission”), which are learned with 푟 1 and 푟 2 , respectively. Lane prioritiesFurthermore, to assess how prompt specifica- tions influence the agent’s behaviour, we include a scenario with directly competing objectives: throughput along different directions. In this experiment, we deploy a symmetric traffic demand pattern: For all four direction, 400 vehicles go straight or take a right-turn and 300 vehicles take a left-turn every hour. We prompt the LLM annotators with different instructions on lane priorities and evalu- ate the impact on lane-specific throughput. Concretely, we specify in the prompt template that the OBJECTIVES are to maximize the throughput along the north-south (east-west) direction. And the USER SPECIFICATION are to ensure high throughput along the north-south (east-west) direction since there are possibly emergency vehicles (NS only and EW only); to prioritise high throughput along the north- south (east-west) direction since there are buses along the lane (NS priority and EW priority); and to ensure balanced throughput along both directions (Equal priority). 4.3 Results and Discussions Learning PerformancesFigure 4 shows the performance in the throughput-emission scenario for RLAIF and three DQN base- lines. In early stage of training (<20k steps), RLAIF learns slower compared with other approaches with stationary rewards. We hy- pothesize that this is because the reward model is evolving with the policy in RLAIF, leading to lower sample efficiency in early stage. However, after convergence (>40k steps), RLAIF achieves better performances against single-objective baselines. RLAIF only falls short of the Linear baseline, achieving competitive results in throughput but emitting more carbon by a small margin (∼65(푔/푠) in RLAIF and∼60(푔/푠)in Linear). However, note that Linear base- line is selected via exhaustive grid search over 9 different weight settings, incurring additional simulation steps and human oversight to identify the best scalarization. With RLAIF, the budget in anno- tating preferences also need to be considered. In our experiments, about 20k annotations are made each run, which costs around 1−2 million tokens ($0.1−0.2 with model gpt-4.1-nano). Out of all LLM annotations, an average of 44% of them are labelled with 0 and thus filtered out in the reward model learning step. Overall, the comparison suggests that RLAIF can converge to a competitive policy without extensive manual reward engineering or human labelling, leveraging AI-based preference signals instead. Sensitivity to instructionsThe comparisons of RLAIF-learned performances with different lane priorities instructions are included in Figure 4. When no lane priority is specified, the agent learns al- most equal throughput along two directions in Equal Priority. When we gradually increase the importance to a specific direction through the prompts, we observe clear changes in the traffic behaviour, with the NS direction throughput moving to 58% when instructed to prioritise north-south direction and 77% to ensure north-south high throughput. This suggests tuning the prompts as a promising way for steering the expected behaviours of outcome policies in RLAIF, when learning a diverse set of policies is of interest. 5 Conclusions In this paper, we explore the extension of RLAIF paradigm, which leverages the LLM models for preference annotations in PbRL, to the context of multi-objective self-adaptive systems. We introduce a tailored LLM-annotation workflow for multi-objective scenar- ios that elicits cross-objective comparisons and conditions labels on user specifications. Our early evaluation in two traffic signal control scenarios shows that RLAIF yields policies with balanced performance across objectives and enables prompt-based tuning of policy behaviour. Overall, we show that recent advances in LLMs and AI-feedback methods point to a promising path for future self-adaptive systems, especially in learning user-aligned policies, reducing reliance on reward engineering through automated pref- erence annotation. Acknowledgements This publication has in part emanated from research supported by Taighde Eireann – Research Ireland under Frontiers for the Future Grant No. 21/FFP-A/8957. For the purpose of Open Access, the author has applied a C BY public copyright license to any Author Accepted Manuscript version arising from this submission. SEAMS ’26, April 13–14, 2026, Rio de Janeiro, BrazilChenyang Zhao, Vinny Cahill, and Ivana Dusparic References [1] Lucas N. Alegre. 2019. SUMO-RL. https://github.com/LucasAlegre/sumo-rl. [2] Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Mar- los C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. 2020. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588, 7836 (2020), 77–82. [3] Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345. [4]Alexander Bukharin, Yixiao Li, Pengcheng He, and Tuo Zhao. 2023. Deep Reinforcement Learning from Hierarchical Preference Design. arXiv preprint arXiv:2309.02632 (2023). [5] Kenneth H Chan, Sol Zilberman, Nick Polanco, Joshua E Siegel, and Betty HC Cheng. 2024. SafeDriveRL: Combining non-cooperative game theory with re- inforcement learning to explore and mitigate human-based uncertainty for au- tonomous vehicles. In Proceedings of the 19th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. 214–220. [6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017). [7]Juntao Gao, Yulong Shen, Jia Liu, Minoru Ito, and Norio Shiratori. 2017. Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network. arXiv preprint arXiv:1705.02755 (2017). [8]Shuaiyi Huang, Mara Levy, Anubhav Gupta, Daniel Ekpo, Ruijie Zheng, and Abhinav Shrivastava. 2025. Trend: Tri-teaching for robust preference-based reinforcement learning with demonstrations. arXiv preprint arXiv:2505.06079 (2025). [9] Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. 2023. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166 (2023). [10]Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward design with language models. arXiv preprint arXiv:2303.00001 (2023). [11]Kimin Lee, Laura Smith, and Pieter Abbeel. 2021. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091 (2021). [12] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. 2021.B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026 (2021). [13] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun- Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. 2018. Microscopic Traffic Simulation using SUMO, In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE Intelligent Transportation Systems Conference (ITSC). https://elib. dlr.de/124092/ [14] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). [15]Tung Minh Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, and Chang D Yoo. 2025. Enhancing Rating-Based Reinforcement Learning to Effec- tively Leverage Feedback from Large Vision-Language Models. arXiv preprint arXiv:2506.12822 (2025). [16]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931 (2023). [17]Ni Mu, Yao Luan, and Qing-Shan Jia. 2025. Preference-based multi-objective reinforcement learning. IEEE Transactions on Automation Science and Engineering (2025). [18]OpenAI. 2025. GPT-4.1-nano. OpenAI. https://platform.openai.com/docs/models/ gpt-4.1-nano Large language model accessed via the OpenAI API. Model: gpt- 4.1-nano. Accessed: 2025-10-23. [19]Takumi Saiki and Sachiyo Arai. 2023. Flexible traffic signal control via multi- objective reinforcement learning. IEEE Access 11 (2023), 75875–75883. [20] Ruiqi Wang, Dezhong Zhao, Ziqin Yuan, Ike Obi, and Byung-Cheol Min. 2025. Prefclm: Enhancing preference-based reinforcement learning with crowdsourced large language models. IEEE Robotics and Automation Letters (2025). [21]Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. 2024. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03681 (2024). [22]Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. 2019. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 1290–1298. [23]Hua Wei, Guanjie Zheng, Vikash Gayah, and Zhenhui Li. 2021. Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation. ACM SIGKDD explorations newsletter 22, 2 (2021), 12–18. [24]Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. 2017. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research 18, 136 (2017), 1–46. [25]Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. 2023. Read and reap the rewards: Learning to play atari with the help of instruc- tion manuals. Advances in Neural Information Processing Systems 36 (2023), 1009–1023. [26] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. 2023. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprint arXiv:2309.11489 (2023). [27] Gongquan Zhang, Fangrong Chang, Jieling Jin, Fan Yang, and Helai Huang. 2024. Multi-objective deep reinforcement learning approach for adaptive traffic signal control system with concurrent optimization of safety, efficiency, and decarbonization at intersections. Accident Analysis & Prevention 199 (2024), 107451. [28]Yuqi Zhang, Yingying Zhou, Beilei Wang, and Jie Song. 2024. MMD-TSC: An adaptive multi-objective traffic signal control for energy saving with traffic efficiency. Energies 17, 19 (2024), 5015. [29]Haiyan Zhao, Chengcheng Dong, Jian Cao, and Qingkui Chen. 2024. A survey on deep reinforcement learning approaches for traffic signal control. Engineering Applications of Artificial Intelligence 133 (2024), 108100. [30]Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash Gayah, Kai Xu, and Zhenhui Li. 2019. Diagnosing reinforcement learning for traffic signal control. arXiv preprint arXiv:1905.04716 (2019). [31]Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. 2023. Large language model as a policy teacher for training reinforcement learning agents. arXiv preprint arXiv:2311.13373 (2023).