Paper deep dive
AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution
Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 4:26:08 AM
Summary
The paper introduces AdsWorldEngine, a self-evolving agentic framework for conversational advertising that uses an Opportunity Gate, Orchestrator, and Evaluator to manage multi-turn ad delivery. It employs iterative actor-tool coevolution, where the Orchestrator is trained via supervised fine-tuning and reinforcement learning, and tools are improved using preference data derived from Orchestrator rollouts. Additionally, it utilizes label-grounded judgment modeling with cost-sensitive GRPO to handle subjective production decisions and asymmetric error costs.
Entities (8)
Relation Signals (10)
AdsWorldEngine → containscomponent → Opportunity Gate
confidence 95% · AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown
AdsWorldEngine → containscomponent → Orchestrator
confidence 95% · an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate
AdsWorldEngine → containscomponent → Evaluator
confidence 95% · an Evaluator to score delivered ads for offline optimization
AdsWorldEngine → improvesmetric → RPM
confidence 95% · In an online A/B test, it increases RPM by 22%
AdsWorldEngine → improvesmetric → Diversity
confidence 95% · Offline, AdsWorldEngine improves diversity by 60%
AdsWorldEngine → improvesmetric → Relevance
confidence 95% · and relevance by 80% over the current production ad delivery system
Orchestrator → trainedwith → Supervised Fine-tuning
confidence 90% · we first train the Orchestrator with supervised fine-tuning
Orchestrator → trainedwith → Agentic Reinforcement Learning
confidence 90% · train the Orchestrator with supervised fine-tuning and agentic reinforcement learning
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.
Tags
Links
- Source: https://arxiv.org/abs/2608.13833v1
- Canonical: https://arxiv.org/abs/2608.13833v1
Trouble viewing inline? Open PDF directly →
Full Text
68,298 characters extracted from source content.
Expand or collapse full text
AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool CoevolutionDOI: X.XXXXXXXConference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN: 978-1-4503-X-X/2018/06CCS: Information systems Information retrieval Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles Affiliation: Microsoft , Redmond , WA , USA email: simiaozuo@microsoft.com 2018© , 2018; Abstract. Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user’s intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-33 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60%60\% and relevance by 80%80\% over the current production ad delivery system. In an online A/B test, it increases RPM by 22%22\% and ads coverage by 74%74\%. 1. Introduction Online advertising has traditionally matched an advertisement to a compact serving context. In sponsored search, the main signal is the user’s query. This query is a short expression of need that supports retrieval, relevance, ranking, and pricing. Conversational advertising changes this setting. Users now often express needs through interactions with assistants over multiple turns. Their intent may be accumulated, refined, or only implied. A turn such as “cheaper ones,” “anything near me,” or “not in the Caribbean” is not an ad query by itself. It must be interpreted using the earlier turns. The assistant response is also informative. It may surface products, places, or brands that make the user’s commercial need more concrete. As a result, ad serving must consider the current user utterance together with dialogue history, optional user context, and the assistant response. These requirements make conversational advertising a full ad serving problem, not only a retrieval and ranking problem. The system must decide whether the current turn is suitable for ads. It must also infer commercial intents from the conversation, call retrieval and ranking tools, and construct a useful slate. Recent work studies ads in conversational AI, including temporal placement decisions and ad generation that accounts for dialogue (4; 34). However, these directions usually address individual components rather than the full ad serving system. To optimize the advertising experience across the entire interactive process, we introduce AdsWorldEngine, an agentic framework for conversational advertising that is designed for production settings. For each turn, an Opportunity Gate decides whether an ad is suitable. Eligible turns pass to an Orchestrator. The Orchestrator resolves context, proposes commercial intents, calls advertising tools, and selects a slate with the best ads. An offline Evaluator scores the delivered slate from the perspectives of users, advertisers, and publishers. It then supplies rewards for training. Figure 1. Overview of AdsWorldEngine. At serving time, the system observes the user query, dialogue history, optional user context, and the assistant response. The Opportunity Gate first decides whether ads should be shown. If the turn is eligible, the Orchestrator generates ad-seeking intents, calls advertising tools, and selects a top-33 slate. Offline, the Evaluator scores delivered ads and provides rewards for improving both the Orchestrator and the tools. A central technical challenge is optimization. Prompting and supervised fine tuning can teach the Orchestrator to follow a format, generate plausible intents for ads, and call tools correctly. However, they do not fully optimize the interaction between intermediate decisions and downstream tools. For example, two generated ad queries may look diverse to humans but retrieve the same merchant or landing page. Another pair may look similar but retrieve complementary ads because of inventory and retrieval model behavior. We therefore propose iterative actor and tool optimization. The Orchestrator is first trained with supervised fine tuning and agentic reinforcement learning using Evaluator rewards. Then, rollouts with high and low rewards are converted into preference data for retrieval, relevance, and ranking tools. This creates a loop that improves over time. The Orchestrator learns how to use the tools, and its rewarded behavior produces data that improves the tools themselves. A second challenge is judgment modeling under production standards. Many reward modeling approaches rely on LLM judge prompts or rubrics fitted by prompts (21; 45). In conversational advertising, however, judgments are not purely verifiable. They depend on usefulness, relevance, and user experience. Their errors also have asymmetric costs across tasks. For example, an inappropriate ad exposure can harm user trust more than a missed opportunity. Selecting an irrelevant ad can also be worse than leaving a slot empty. We therefore propose label grounded judgment modeling, a data generation pipeline for training production judgment models. The pipeline starts with task specific production guidelines and human labels collected under those guidelines. For each labeled example, we generate a thinking trace conditioned on the label. The trace explains how the assigned label aligns with the guideline and describes which types of recommendations may be helpful without being intrusive to the user-assistant interaction. This differs from asking the model to invent its own judgment. A reflection step then checks whether the trace supports the label, follows the guideline, and avoids unsupported assumptions. Examples with inconsistent rationales are removed. The remaining pairs of labels and traces provide dense supervision for judges. To optimize decisions with asymmetric costs, we further introduce a GRPO variant with cost sensitive rewards (33). This variant uses asymmetric rewards and omits group standard deviation scaling, which preserves task specific penalty gaps. The main contributions of this paper are: • We formulate conversational ad serving as a gated agentic framework for production settings. The framework uses turn context and assistant responses to decide when to show ads and how to construct slates with the best ads. • We propose iterative actor and tool optimization. In this approach, rewarded Orchestrator rollouts train both the actor and the retrieval, relevance, and ranking tools. This enables the system to learn how to call tools and how to improve them from rewarded behavior. • We introduce label grounded judgment modeling with cost sensitive GRPO. For data, the method generates label and trace pairs from task guidelines and filters them through reflection. For training, it uses cost sensitive GRPO with asymmetric rewards for production judgment tasks. • AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad serving system. It also increases RPM (Revenue Per Mille) by 22% and ads coverage by 74% in an online A/B test. 2. AdsWorldEngine Overview AdsWorldEngine formulates conversational advertising as a gated agentic decision process. At each turn, it observes the user query, dialogue history, optional user context allowed by policy, and the assistant response, whose recommendations or context carried over from earlier turns can clarify intent. As Fig. 1 illustrates, the online pipeline separates whether to show ads from which ads to show. Our framework primarily consists of four components: (1) Opportunity Gate, (2) Orchestrator, (3) Tool Set, and (4) Evaluator. The Opportunity Gate combines production rules with a learned triggering model to abstain or pass a suitable turn. The Orchestrator then resolves the conversation state, generates commercial intents, calls advertising tools from Tool Set, and selects a slate with three ads. This adapts language model agents that use tools to ad serving, where dialogue reasoning must be combined with retrieval, ranking, relevance, pricing, and other advertising signals (41; 30). Finally, an Evaluator provide feedback signals to update the Orchestor and Tool Set. We introduce these compontents in details in Sec.3. For subjective decisions such as ad triggering and conversation-to-ad relevance, we introduce label grounded judgment modeling, as will be discussed in Sec. 4. The method first generates thinking traces conditioned on guidelines and human labels, then filters them through a reflection step to remove inconsistent reasoning. For asymmetric binary tasks, we use cost sensitive GRPO, which preserves raw reward gaps and allows false positive triggers to carry a larger penalty than false negatives. We also propose iterative actor and tool optimization. The Orchestrator is trained with supervised fine tuning (SFT) and agentic reinforcement learning from Evaluator rewards. Comparisons between rollouts with high and low rewards then provide preference data for retrieval, relevance, and ranking tools. Alternating these updates lets rewarded Orchestrator behavior improve the tools used in later rollouts. We introduce details in Sec.5. 3. AdsWorldEngine Core Components 3.1. Opportunity Gate The Opportunity Gate is the first online stage of AdsWorldEngine. Given the full turn context, it decides whether ads should be shown. If the gate rejects the turn, the system abstains. If it accepts the turn, the conversation is passed to the Orchestrator for intent generation, tool calling, and slate selection. The gate combines production rules with a learned triggering model. The rules enforce hard constraints such as policy and privacy requirements. The learned model decides whether an ad would help the current task. The gate triggers for purchase planning or concrete options surfaced by the assistant response. It abstains in sensitive, purely informational, or disruptive contexts. Overall, the gate favors user experience and treats false positives as especially costly. The triggering model is trained with the label grounded judgment modeling framework in Section 4. Human labels are collected under the triggering guideline. They are then converted into label grounded thinking traces, filtered by reflection, and used for SFT and cost sensitive GRPO. This produces a judgment model that separates the production decision of whether to show ads from the downstream decision of which ads to show. Input conversation. The user first asks for beach holidays above 28∘28 C at Christmas, with direct flights from Heathrow and flight time no longer than 12 hours. After seeing options that include Caribbean destinations, the user asks: “Anything not in the Caribbean.” State resolution. The current turn is a refinement, not a topic shift. The omitted target is the earlier request for warm Christmas beach holidays with direct Heathrow flights under 12 hours. The new constraint is to exclude Caribbean destinations. Constraint extraction. Hard constraints: beach destination; Christmas timing; typical temperature above 28∘28 C; direct flight from Heathrow; flight time ≤ 12 hours; not in the Caribbean. Assistant-suggested attributes such as luxury or family resorts are not user constraints. Intent and query generation. [Indian Ocean beach holidays Christmas direct from London, Middle East beach holidays Christmas direct from London, Africa beach holidays Christmas direct from London] Tool results. The retrieval/relevance/ranking tools return specific Egypt hotel pages, broad Africa beach-holiday pages, repeated Asilia Africa safari-and-beach pages, repeated Ras al Hikmah pages, and one UK inland accommodation listing. Reflection and final selected ads. Reflection selects a7 and a9 as strong specific matches, keeps a1 for broader Africa coverage, removes near-duplicates, and rejects the UK property as off-intent. The final selected ads are a7, a9, a1. Figure 2. Example Orchestrator trajectory. Intermediate traces teach the model to resolve a short follow-up, preserve inherited constraints, generate useful retrieval queries, and recover from noisy or duplicate tool results. 3.2. Orchestrator The Orchestrator is the actor in AdsWorldEngine. Once the Opportunity Gate accepts a turn, the Orchestrator constructs the final slate with three ads. This is not simply ranking a fixed candidate set. The model must resolve the user’s current intent and preserve constraints from earlier turns. It must also generate intents (ad queries) for retrieval, call tools, inspect retrieved candidates, and select a slate that is relevant and diverse. The Orchestrator has three components. The intent generator resolves dialogue state, extracts constraints and preferences, and produces one to three intents. The tool caller submits these intents to advertising tools. These tools include retrieval, relevance and ranking tools. They can also include pricing, and signal models. The reflection module reviews the returned candidates and selects exactly three ads for the final slate. 3.2.1. Data Generation We generate supervised trajectories with intermediate traces before the final ad selection. Each trajectory teaches a reusable decision process. It covers state resolution and constraint extraction. It also covers intent generation, tool calling, and reflection. This is especially important for conversational follow up turns, where the current user query may be underspecified without earlier turns. Figure 2 illustrates an example of the Orchestrator’s thinking trace. The model learns that the short current turn is a refinement of the earlier request, not a standalone query. It also learns to preserve user constraints and avoid adopting unsupported assistant suggestions as user requirements. The trace further teaches the model to generate grounded retrieval queries and handle imperfect tool outputs. 3.2.2. Model Training We train the Orchestrator in two stages. First, supervised fine tuning teaches the model the basic procedure for tool use. The model learns to resolve the conversation, extract constraints, generate intents, call tools, inspect candidates, and output three ads. SFT is useful for format control and valid tool calls. It also helps with constraint preservation and decomposition. Second, we apply agentic reinforcement learning as will discuss in Sec.5. The Orchestrator samples complete trajectories that include intents, tool calls, reflection, and final slate selection. The Evaluator (see Section 3.3) scores the final slate using system level objectives such as relevance and diversity. We then use GRPO to reinforce trajectories that produce better final slates (32; 33). The main advantage of agentic reinforcement learning is that it aligns intermediate tool use with the final serving objective. For example, diversity is not rewarded merely because the Orchestrator writes intents that look different. It is rewarded only when those intents retrieve complementary ads that improve the final slate. Thus, GRPO turns instructions such as “generate diverse intents” from prompt level preferences into behaviors grounded in outcomes. The Orchestrator learns which intent decompositions, tool calls, and reflection decisions produce better ads for the user. 3.3. Evaluator The Evaluator is an offline component for measuring ad quality and producing training signals. Its scores are not used directly in online serving. Instead, they provide rewards for Orchestrator training and preference data for improving downstream tools. We use the term broadly to cover user, advertiser, and publisher perspectives. The user perspective measures whether the slate is useful to the user. It focuses on relevance and diversity. The advertiser and publisher perspectives capture production requirements. In the current implementation, we focus on two user perspective signals: relevance for each ad and diversity for the slate. The main learned evaluator is a conversation-to-ad relevance model. This model judges whether a candidate ad helps the user’s current task given the conversation context. We train it with the label grounded judgment modeling framework in Section 4. Human labels are collected under a relevance guideline, enriched with thinking traces, filtered by reflection, and optimized with SFT and GRPO. Relevance is evaluated at the individual ad level, but a useful slate should also avoid redundancy. Because only three ads are shown, repeated products or pages can noticeably reduce utility. We therefore measure slate level diversity with pairwise cosine similarity among the selected ad embeddings. We penalize slates that contain near duplicate ads. 4. Label Grounded Judgment Modeling Many production decisions in conversational advertising are subjective rather than directly verifiable. For example, deciding whether to show ads depends on whether an ad would be commercially useful and whether it would fit the user’s experience. It also depends on sensitivity and product policy. Existing LLM judge methods often prompt a model to approximate human preferences (21; 45). RLHF methods instead train reward models from human feedback (26). In AdsWorldEngine, we take a label grounded approach. We first define explicit production guidelines and collect human labels under those guidelines. We then train judgment models to match the resulting preferences. This framework is designed for production settings where errors often have asymmetric costs. For example, in ad triggering, a false positive can be more costly than a false negative. Showing an ad in an inappropriate context may harm user trust, while missing one commercial opportunity is usually less visible. Similar asymmetry appears in relevance modeling. Selecting an irrelevant ad for the final slate can be worse than failing to retrieve one relevant ad. 4.1. Data: Label Grounded Thinking Traces The data pipeline has four steps. First, human annotators assign labels using a task specific guideline. Second, for each labeled example, we generate a thinking trace conditioned on the guideline, the input, and the human label. Third, we use a reflection prompt to check whether the trace supports the label and follows the guideline. Finally, only examples that pass reflection are used for supervised fine tuning. The trace generation prompt asks the model to explain why the human label is correct under the guideline. This converts a categorical label into denser supervision. The model learns the target judgment and the decision pattern expected by the production policy. However, generated traces may contradict the label, add unsupported assumptions, or fail to follow the required format. We therefore filter examples by checking whether input→thinking trace→labelinput trace is internally consistent. Reflection filtering is important because inconsistent rationales teach the model to produce plausible explanations that do not support the target decision. Keeping only traces that follow the guideline improves the reliability of judgments. See Table 1 for an example. Table 1. Example of data filtering. The trace is fluent but contradicts the human label, so it is removed before SFT. Field Content Conversation User asks for a kid-friendly camping resort near Newton, MA with cabins, AC, kitchen, and teen activities. The assistant recommends specific campgrounds and links. Human label No (information-seeking conversation) Generated trace “… directly about commercial entities … ads for camping resorts or booking platforms would be relevant and helpful … the human label of No is incorrect …” Reflection result REASONING_LABEL_MISMATCH. The trace supports the opposite label and questions the given label. 4.2. Training: SFT and Cost Sensitive GRPO After filtering, we train judgment models in two stages. Supervised fine tuning teaches the model to follow the task format and produce reasoning grounded in the guideline. We then apply GRPO (33) to optimize task rewards. These rewards capture label correctness, format validity, and guideline consistency. For asymmetric binary judgment tasks, we use cost sensitive rewards. For example, in ad triggering, let Yes mean that ads should be shown and No mean that ads should not be shown. We use R(y,y^)=y^=Yesy^=Noy=Yes1−1y=No−21R(y, y)= array[]c|c& y= Yes& y= No\\ y= Yes&1&-1\\ y= No&-2&1 array This reward directly encodes the production preference that false positive ad triggering should be penalized more strongly than false negatives. Standard GRPO computes group relative advantages by centering and scaling rewards within a group: Aiscaled=Ri−R¯sR+ϵ,A_i^scaled= R_i- Rs_R+ε, where R¯ R and sRs_R are the group mean and standard deviation. This scaling can be undesirable for binary tasks because the raw reward gap encodes the cost of different mistakes. Dividing by the group standard deviation can remove that information. We therefore use group centering without group standard deviation scaling: Ainone=Ri−R¯.A_i^none=R_i- R. This preserves the within group comparison while keeping the magnitude of the reward gap. The motivation is related to recent analyses of normalization in GRPO training (22). Our setting, however, is specifically cost sensitive binary judgment. The effect can be seen in a group with two possible rewards. Suppose completions receive either a better reward H or a worse reward L, and k out of G completions receive H. Then R¯=kH+(G−k)LG. R= kH+(G-k)LG. Without standard deviation scaling, AHnone=G−kG(H−L),ALnone=−kG(H−L).A_H^none= G-kG(H-L), A_L^none=- kG(H-L). Thus, the learning signal is proportional to the raw reward gap H−LH-L. With group scaling, sR=k(G−k)G|H−L|,s_R= k(G-k)G|H-L|, so AHscaled=G−k,ALscaled=−kG−k.A_H^scaled= G-kk, A_L^scaled=- kG-k. The gap H−LH-L cancels out. In other words, scaling preserves which completion is better, but loses how much better it is. We use this cost sensitive GRPO variant for binary judgment models such as the Opportunity Gate. The same label grounded framework is also used for conversation-to-ad relevance modeling. Each task uses its own labels, guidelines, traces, and rewards. We also include a disscusion of serving distribution shift in Appendix A. 5. Orchestrator and Tool Iterative Training AdsWorldEngine is optimized with an iterative training loop for the actor and tools. The Orchestrator is the actor. It resolves the conversation, generates intents, calls tools, and selects the final slate. The tools define the actor’s action space. They include retrieval and relevance tools, along with pricing, signal, and ranking models. The key contribution is that we do not treat these tools as fixed infrastructure. Instead, we use rewarded Orchestrator rollouts to improve both the actor and the tools. Algorithm 1 Iterative optimization for actor and tools. 0: Conversations D, initial tools 0T_0, Evaluator E, Orchestrator SFT data sftD_sft 1: Train initial Orchestrator πθ0 _ _0 on sftD_sft 2: for t=0,1,…,T−1t=0,1,…,T-1 do 3: Orchestrator update with fixed tools 4: Sample rollouts from πθt _ _t using tools tT_t 5: Score final slates with Evaluator E 6: Update πθt→πθt+1 _ _t→ _ _t+1 with GRPO 7: Tool preference construction 8: Generate rollouts from πθt+1 _ _t+1 on D 9: Compare high-reward and low-reward rollouts for the same conversation 10: Construct tool preference data t=(intent,a+,a−)P_t=\(intent,a^+,a^-)\ 11: Tool update 12: Optimize t→t+1T_t _t+1 using tP_t with DPO or ranking losses 13: end for 14: return Final Orchestrator πθT _ _T and tools TT_T Algorithm 1 summarizes the procedure. We first train a stable initial Orchestrator with SFT. Each iteration then alternates between two updates. First, with tools fixed, we optimize the Orchestrator using GRPO and Evaluator rewards. This improves intent decomposition, tool use strategy, and reflection behavior under the current tool environment. Second, we roll out the improved Orchestrator to construct preference data for the tools. For a conversation, an ad from a slate with a high reward becomes a positive example. An ad retrieved or selected in a slate with a low reward becomes a hard negative. To reduce noise, we construct such pairs only when the reward gap is sufficiently large. This loop converts final rewards at the slate level into supervision for intermediate tools. For retrieval, rollouts with high rewards identify ads that should be retrievable for an intent. Rollouts with low rewards expose candidates that were retrieved but should not be preferred. For relevance and ranking, the same rollouts provide realistic preference pairs discovered by the Orchestrator. The resulting system improves along two coupled dimensions. GRPO makes the Orchestrator better at using the current tools. At the same time, preference learning from rollouts makes the tools better for future Orchestrator decisions. Better tools expand the actor’s effective action space. A stronger actor then produces more informative rollouts for the next tool update. This feedback loop between the actor and tools is the central mechanism of AdsWorldEngine. The system learns how to call tools and how to improve them from its own rewarded behavior. This connects conversational ad serving to recent work on optimizing compound language model systems and systems augmented with retrieval (16; 19). Example: preference training for intent-to-ads relevance models. For each conversational intent q, we construct a preference triple (q,a+,a−)(q,a^+,a^-), where a+a^+ is an ad from a high-reward slate and a−a^- is a hard negative from a low-reward slate. Let sθ(q,a)s_θ(q,a) be the score from the trainable relevance model and sref(q,a)s_ref(q,a) the score from a frozen reference model. We optimize (1) ℒ= = −(q,a+,a−)logσ(β[sθ(q,a+)−sθ(q,a−) -E_(q,a^+,a^-) σ (β [s_θ(q,a^+)-s_θ(q,a^-) −sref(q,a+)+sref(q,a−)]). -s_ref(q,a^+)+s_ref(q,a^-) ] ). For an encoder, sθs_θ can be the relevance logit. For a one-token decoder, we define sθ(q,a)=logpθ(Yes∣q,a)−logpθ(No∣q,a).s_θ(q,a)= p_θ( Yes q,a)- p_θ( No q,a). The proposed algorithm is similar to DPO (28) because it directly increases the preference margin of a+a^+ over a−a^-, relative to a frozen reference model. In particular, the scores induce a policy over the candidate pair C=a+,a−C=\a^+,a^-\: πθ(a∣q,C)= _θ(a q,C)= expsθ(q,a)∑a′∈Cexpsθ(q,a′), s_θ(q,a) _a ∈ C s_θ(q,a ), logπθ(a+∣q,C)πθ(a−∣q,C)= _θ(a^+ q,C) _θ(a^- q,C)= sθ(q,a+)−sθ(q,a−). s_θ(q,a^+)-s_θ(q,a^-). Therefore, the objective applies the DPO preference principle to relevance scores rather than autoregressive sequence probabilities. Table 2. Example of Opportunity Gate. The prompted GPT-5 judge and AdsWorldEngine (SFT only) overreact to product language and produce a false positive. AdsWorldEngine recognizes informational intent and correctly suppresses the trigger. Field Content Conversation User: “Which shade fits me?” Bot: “Choosing the right concealer shade depends on your skin undertone, depth, and the purpose … Step 1: Identify Your Undertone … Step 2: Match Depth … Step 3: Purpose of Use … To narrow it down, could you tell me:…” Human label No. The user is seeking personalized shade-matching information, not a retailer or purchase option. The assistant can complete the task directly, so inserting an ad would distract from the current interaction. GPT-5 prompt “The user is seeking help choosing a concealer shade, which is a beauty product they would likely need to purchase. The bot is guiding them toward identifying the right shade, which could directly lead to a buying decision. Ads for concealers or beauty products would be relevant and helpful here …” Prediction: Yes (incorrect). AdsWorldEngine (SFT only) “The user’s message is clearly about finding the right concealer shade … the conversation is directly about selecting a product variant. The user’s goal is to identify which specific shade to choose, which is a step toward a purchase decision. An ad could support their journey rather than distract …” Prediction: Yes (incorrect). AdsWorldEngine “The assistant is giving general advice based on the user’s undertone, skin depth, and existing swatches. The conversation is about matching a shade to the user’s skin, not about finding where to buy concealer. The focus is on completing a task that the assistant can handle directly, so inserting ads would distract from the user’s intent.” Prediction: No (correct). 6. Experiments: Label Grounded Judgment 6.1. Opportunity Gate: Triggering Models Triggering is not merely detecting product mentions. It is a production judgment: would an ad advance the user’s current goal without harming the conversation? We use label grounded judgment modeling to tie predictions to production requirements. Because a bad trigger is visible to the user, the target operating point emphasizes fewer false positives while preserving valid opportunities. We wrote annotation guidelines for Opportunity Gate that specify when an ad is useful and when it should be suppressed because the assistant can answer directly. With these guidelines, we collected 30,00030,000 human labeled conversations. For each conversation, we prompted GPT-5 with the guideline and human label to produce a reasoning trace, then used a reflection prompt to remove traces that contradicted the guideline or label, leaving 27,00027,000 examples. Starting from Qwen3-30B-A3B-Thinking, we used 24,00024,000 examples for SFT and 3,0003,000 for GRPO. Results. Table 3 reports relative changes from the GPT-5 prompt baseline. AdsWorldEngine with SFT substantially reduces FPR, but moves the decision boundary too far: TPR decreases by 17.19%17.19\%, resulting in a 4.09%4.09\% reduction in balanced accuracy. Thus, SFT yields a conservative model that misses the desired production operating point. In contrast, AdsWorldEngine with SFT and cost-sensitive GRPO preserves baseline TPR while improving FPR by 39.07%39.07\%, yielding a 2.51%2.51\% balanced accuracy gain. The resulting 30B model outperforms the GPT-5 prompted judge while better avoiding unnecessary triggers, and runs much faster than the GPT-5 prompt for a latency sensitive online gate. Table 3. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy. Model Δ (↑ ) Δ (↓ ) Δ Acc. (↑ ) GPT-5 prompt – – – AdsWorldEngine (SFT) −17.19%-17.19\% −75.42%-75.42\% −4.09%-4.09\% AdsWorldEngine 0.00%0.00\% −39.07%-39.07\% +2.51%+2.51\% These results show why reinforcement learning is preferable to prompt engineering or SFT alone. A prompt can state the policy, but it cannot precisely set the model’s implicit preference for positive or negative predictions. SFT transfers labeled reasoning patterns, but its token level objective does not optimize the final tradeoff between false positives and false negatives. GRPO directly adjusts the decision boundary for the label grounded production objective, reducing false positives without the large recall loss seen with SFT. Example. Table 2 shows a representative case. The user asks which concealer shade fits them, and the assistant responds with guidance about undertone, skin depth, and existing swatches. The correct decision is No: the user wants information, not a retailer or purchase path, so an ad would distract. This example highlights a bias that prompts and SFT struggle to remove: product related words can produce a trigger even when the goal is noncommercial. Production aligned reinforcement learning teaches that mentioning or selecting a product is not enough evidence that an ad would improve the experience. 6.2. Evaluator: Conversation-Ads Relevance The conversation to ads relevance model follows the same motivation and training procedure as the triggering model. Its purpose is not only to identify ads that are related to the conversation, but also to match the production preference for preserving user experience. In particular, a false positive allows an irrelevant ad to enter the final slate and can be more disruptive than filtering out one potentially relevant ad. We therefore optimize the model toward fewer false positive predictions while retaining useful ads. We first defined a human annotation guideline for conversation to ads relevance in Evaluator and collected 32,00032,000 labeled examples. For each conversation and candidate ad, we generated a reasoning trace conditioned on the guideline and human label. A reflection prompt then removed traces that were inconsistent with the intended judgment, leaving 30,00030,000 examples. We used 27,00027,000 examples for SFT and 3,0003,000 examples for GRPO, starting from Qwen3-30B-A3B-Thinking. Table 4 reports changes relative to the GPT-5 prompt. SFT improves TPR, but it also produces more false positives, suggesting that SFT alone favors accepting ads too broadly. GRPO directly adjusts this preference toward the desired production operating point. It reduces FPR by 12.71%12.71\% with only a 0.78%0.78\% decrease in TPR, resulting in a 7.51%7.51\% improvement in balanced accuracy. This tradeoff is better aligned with the user experience objective than either prompt based judgment or SFT alone. Table 4. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy. Model Δ (↑ ) Δ (↓ ) Δ Acc. (↑ ) GPT-5 prompt – – – AdsWorldEngine (SFT) +10.04%+10.04\% +6.36%+6.36\% +3.39%+3.39\% AdsWorldEngine −0.78%-0.78\% −12.71%-12.71\% +7.51%+7.51\% 7. Experiments: Orchestrator and Tools 7.1. Training and Evaluation We evaluate the iterative procedure in Section 5, using Qwen3-30B-A3B-Thinking as the base model for the Orchestrator. The full pipeline begins with supervised warmup and then alternates Orchestrator training and tool training for three rounds. In each round, the Orchestrator is first updated while the tool set is fixed. The improved Orchestrator is then used to generate preference data for updating the intent-to-ads relevance model in the tool set, which is deployed in the next round. Evaluator and metrics. We use a fixed Evaluator throughout both training and final evaluation. Its conversation-to-ads relevance model observes the complete dialogue and judges whether each selected ad is relevant. The Evaluator also measures slate diversity from the three pairwise cosine similarities among ad embeddings, counting a pair as diverse when its similarity is below 0.30.3. During training, these relevance and diversity judgments generate reward signals for Orchestrator optimization, and the relevance judgments additionally provide supervision for tool training. For final evaluation, all systems are run on the same conversation set and return exactly three ads per conversation. We report the Evaluator’s relevance score, defined as the number of selected ads judged relevant, and its diversity score, defined as the number of diverse ad pairs. The current Production System is the baseline, and all results are reported as relative gains rather than absolute metric values. Orchestrator warmup. We first train the Orchestrator with supervised reasoning traces generated by prompting. Each trace covers state resolution, constraint extraction, intent and query generation, tool interaction, and reflection over retrieved candidates. This stage teaches the model a stable problem solving pattern. In particular, the Orchestrator learns to resolve short follow up turns, preserve constraints from earlier messages, form valid tool inputs, and select three ads from a noisy candidate pool. Orchestrator training. After supervised fine tuning, we optimize the Orchestrator with GRPO while keeping all downstream tools fixed. Each rollout ends with a slate of three ads and receives the relevance and diversity rewards produced by the Evaluator. Optimizing these final-slate rewards allows the Orchestrator to adapt its intent decomposition, query generation, tool use, and reflection to the behavior of the actual retrieval and ranking tools. Tool training. The tool update trains an intent-to-ads relevance model that ranks candidates online for each intent generated by the Orchestrator. This model is distinct from the conversation-to-ads relevance model in the Evaluator: the Evaluator model sees the complete dialogue and supplies offline relevance judgments, whereas the tool model sees only a generated intent and a candidate ad. After each GRPO update, we run the improved Orchestrator ten times for every training conversation and construct preference triples (q,a+,a−)(q,a^+,a^-), where q is a generated intent, a+a^+ is an ad the Evaluator judges relevant to the full conversation, and a−a^- is judged irrelevant. We train the intent-to-ads relevance model on these preference pairs and deploy the updated model in the tool set for the next Orchestrator round. 7.2. Experiment Results Table 5 reports relative gains over the current Production System. Round 00 denotes supervised warmup; later rows report performance after the GRPO Orchestrator update with fixed tools and the subsequent intent-to-ads tool update. Table 5. Relative diversity and relevance gains over the Production System of AdsWorldEngine (30B-A3B). Round System state Diversity Relevance – Production System – – 00 AdsWorldEngine (SFT) +15.31%+15.31\% +49.37%+49.37\% 11 Orchestrator update +29.24%+29.24\% +53.15%+53.15\% 11 Tool update +53.90%+53.90\% +70.07%+70.07\% 22 Orchestrator update +58.74%+58.74\% +75.60%+75.60\% 22 Tool update +59.47%+59.47\% +78.04%+78.04\% 33 Orchestrator update +62.16%+62.16\% +80.28%+80.28\% 33 Tool update +62.87%+62.87\% +82.26%+82.26\% Table 6. Case study comparing generated intents before and after reinforcement learning. After the generated intents are used to call the tools, the reflection step selects three ads, which the Evaluator scores for relevance. The model trained with SFT and GRPO achieves a stronger relevance outcome than the SFT-only model. Conversation Previous user: Compare the Honda Pilot vs the Toyota Grand Hylander both 2026 Previous bot: Here is a clean, decision ready comparison … I can also break this down by family use, towing, … Current user: Yes please do Current bot: Absolutely—let’s take this comparison deeper and make it genuinely useful for a real world decision … AdsWorldEngine (SFT) Conversation to ads relevance: lower <think> … I think the cleanest final intent is that the user is researching and comparing the 2026 Honda Pilot and 2026 Toyota Grand Highlander to help decide which is better for their needs … the strongest queries are ones that directly target side by side comparisons and reviews … </think> <tool_call> 2026 Honda Pilot vs Toyota Grand Highlander comparison, 2026 Honda Pilot vs Toyota Grand Highlander family review </tool_call> AdsWorldEngine Conversation to ads relevance: higher <think> I think the cleanest final intent is that the user wants detailed, scenario based information to help decide between the 2026 Honda Pilot and the 2026 Toyota Grand Highlander … The only hard constraints are the two named vehicles and model year; everything else is open for exploration … For ads queries, the strongest grounded targets are the two specific SUV models for 2026. Each is a distinct product family, so having one query for each covers the full scope without redundancy … </think> <tool_call> 2026 Honda Pilot, 2026 Toyota Grand Highlander </tool_call> Supervised warmup provides strong gains, and each Orchestrator update further improves both metrics with the tools fixed. By optimizing final-slate relevance and diversity through the retrieval and ranking pipeline, GRPO adapts intent generation and reflection to what the downstream tools can retrieve. Tool updates further improve relevance and, indirectly, diversity. Before training, generic ads may rank highly for multiple intents, causing their top-K candidate sets to overlap. Preference training makes ranking more intent-specific by promoting ads for the intents they satisfy and demoting hard negatives. Distinct intents therefore retrieve less-overlapping inventory, increasing diversity despite the relevance-only tool objective. Gains are largest in the first round and diminish thereafter, suggesting gradual co-adaptation between the Orchestrator and relevance tool. After three rounds, the final system improves diversity by 62.87%62.87\% and relevance by 82.26%82.26\% over the Production System. 7.3. Case Study Table 6 compares the intents generated by the supervised model and the model after GRPO for the same conversation. The table presents the dialogue context, excerpts from the Orchestrator reasoning, the resulting tool calls, and the conversation to ads relevance scores. This comparison makes it possible to examine how reinforcement learning changes the intermediate intent generation strategy rather than only the final score. The SFT only AdsWorldEngine model produces two overlapping comparison intents. Although both are semantically related to the conversation, they target nearly the same comparison content and are poorly aligned with a product centered retrieval tool, resulting in a weak relevance outcome. After GRPO, the Orchestrator generates one grounded intent for each named vehicle. These product specific intents are easier for the tools to match to inventory and expose candidate sets with less overlap, giving the reflection step better options for selecting relevant ads. This separation may also improve diversity because each intent retrieves inventory for a different product family instead of repeating the same comparison pages. The improved relevance outcome therefore illustrates that reinforcement learning improves the operational compatibility of intent generation with downstream tools, while also creating conditions for a more diverse final slate. 8. Real World Online Experiments We deploy the trained AdsWorldEngine in Microsoft Copilot, a consumer facing conversational assistant, across its surfaces worldwide. The Opportunity Gate first identifies commercial opportunities where ads are unlikely to harm the user experience. For eligible turns, the Orchestrator retrieves, ranks, and selects the final ad slate. During a 20-day online experiment, AdsWorldEngine increases revenue per mille (RPM) by 22%22\% and ad coverage by 74%74\%. 9. Related Work Recent advertising research uses dialogue context to generate and assess ad text, with emphasis on contextual fit, linguistic quality, and personalization (34; 24; 44; 37). Sponsored question answering and response insertion instead study where ads can enter assistant outputs and how generated responses should incorporate them (4; 25; 14; 39). A complementary line develops pricing, auction, and position allocation mechanisms for generated content and summaries (7; 12; 6; 36; 3). Other studies identify technical opportunities while examining ad detection, user trust, and persuasive effects (10; 31; 8; 29). AdsWorldEngine differs by treating conversational advertising as an end to end serving problem that joins opportunity gating, intent formation, retrieval, ranking, and slate selection. Our Orchestrator builds on agents that interleave language reasoning with web interaction, external actions, and learned tool selection (40; 41; 30; 13). Recent benchmarks and algorithms improve API selection and generalization across agent tasks (18; 27; 20; 43). Trajectory tuning, retrieval with self reflection, pipeline optimization, and verbal feedback further improve how agents plan and recover from errors (5; 2; 16; 35). These methods mainly optimize the agent around a given environment. In contrast, our iterative procedure converts rewarded rollouts into preference data that improves both the Orchestrator and its retrieval and relevance tools. Finally, rationale supervision and iterative feedback show how generated reasoning can provide richer training signals than categorical targets alone (38; 42; 23). Model based evaluators use prompting or dedicated evaluator models to estimate generation quality (21; 45; 17), and related methods produce detailed critiques or task specific scores (15; 11). Preference learning aligns model behavior through human feedback or direct optimization of preferred outputs (26; 28; 9; 33). Our label grounded judges additionally condition rationales on production labels, remove explanations that conflict with those labels, and preserve asymmetric error costs during optimization. 10. Conclusion We presented AdsWorldEngine, a production-oriented framework for conversational advertising that jointly learns when to show ads, how to use advertising tools, and how to improve those tools from rewarded outcomes. By combining iterative Orchestrator-tool optimization with label grounded judgment modeling, the system improves both offline ad quality and online serving performance. These results demonstrate the value of optimizing agent decisions and downstream tools as a coupled system for real-world conversational applications. More broadly, AdsWorldEngine provides a practical approach for building compound language-model systems in which agents and tools co-adapt under production constraints. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §A.1. Asai et al. (2024) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi Self-RAG: learning to retrieve, generate, and critique through self-reflection. In Proceedings of the 12th International Conference on Learning Representations, Cited by: §9. Balseiro et al. (2025) S. Balseiro, K. Bhawalkar, Y. Deng, Z. Feng, J. Mao, A. Mehta, V. Mirrokni, R. Paes Leme, D. Wang, and S. Zuo Position auctions in AI-generated content. arXiv preprint arXiv:2506.03309. Cited by: §9. Banchio et al. (2024) M. Banchio, A. Mehta, and A. Perlroth Ads in conversations. arXiv preprint arXiv:2403.11022. Cited by: §1, §9. Chen et al. (2023) B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao FireAct: toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. Cited by: §9. Dubey et al. (2024) K. A. Dubey, Z. Feng, R. Kidambi, A. Mehta, and D. Wang Auctions with LLM summaries. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 713–722. External Links: Document Cited by: §9. Dütting et al. (2024) P. Dütting, V. Mirrokni, R. Paes Leme, H. Xu, and S. Zuo Mechanism design for large language models. In Proceedings of the ACM Web Conference 2024, p. 144–155. External Links: Document Cited by: §9. Erickson (2025) J. Erickson Fake friends and sponsored ads: the risks of advertising in conversational search. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, p. 1–8. External Links: Document Cited by: §9. Ethayarajh et al. (2024) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela KTO: model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235. Cited by: §9. Feizi et al. (2023) S. Feizi, M. Hajiaghayi, K. Rezaei, and S. Shin Online advertisements with LLMs: opportunities and challenges. arXiv preprint arXiv:2311.07601. Cited by: §9. Fu et al. (2024) J. Fu, S. Ng, Z. Jiang, and P. Liu GPTScore: evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 6556–6576. External Links: Document Cited by: §9. Hajiaghayi et al. (2024) M. Hajiaghayi, S. Lahaie, K. Rezaei, and S. Shin Ad auctions for LLMs via retrieval augmented generation. In Advances in Neural Information Processing Systems, Vol. 37, p. 18445–18480. Cited by: §9. Hao et al. (2023) S. Hao, T. Liu, Z. Wang, and Z. Hu ToolkenGPT: augmenting frozen language models with massive tools via tool embeddings. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §9. Hu et al. (2025) S. Hu, S. Zhang, Y. Shi, and X. Xiao GEM-Bench: a benchmark for ad-injected response generation within generative engine marketing. arXiv preprint arXiv:2509.14221. Cited by: §9. Ke et al. (2024) P. Ke, B. Wen, A. Feng, X. Liu, X. Lei, J. Cheng, S. Wang, A. Zeng, Y. Dong, H. Wang, J. Tang, and M. Huang CritiqueLLM: towards an informative critique generation model for evaluation of large language model generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 13034–13054. External Links: Document Cited by: §9. Khattab et al. (2024) O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts DSPy: compiling declarative language model calls into state-of-the-art pipelines. In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Cited by: §5, §9. Kim et al. (2024) S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo Prometheus: inducing fine-grained evaluation capability in language models. In Proceedings of the 12th International Conference on Learning Representations, Cited by: §9. Li et al. (2023) M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li API-Bank: a comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 3102–3116. External Links: Document Cited by: §9. Li et al. (2025) X. Li, S. Mei, Z. Liu, Y. Yan, S. Wang, S. Yu, Z. Zeng, H. Chen, G. Yu, Z. Liu, M. Sun, and C. Xiong RAG-ddr: optimizing retrieval-augmented generation using differentiable data rewards. In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Cited by: §5. Liu et al. (2024) X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang AgentBench: evaluating LLMs as agents. In Proceedings of the 12th International Conference on Learning Representations, Cited by: §9. Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 2511–2522. External Links: Document Cited by: §1, §4, §9. Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective. In Conference on Language Modeling (COLM), Cited by: §4.2. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, Vol. 36, p. 46534–46594. Cited by: §9. Mita et al. (2024) M. Mita, S. Murakami, A. Kato, and P. Zhang Striking gold in advertising: standardization and exploration of ad text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 955–972. External Links: Document Cited by: §9. Mordo et al. (2024) T. Mordo, M. Tennenholtz, and O. Kurland Sponsored question answering. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, p. 167–173. External Links: Document Cited by: §9. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. Cited by: §4, §9. Qin et al. (2024) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world APIs. In Proceedings of the 12th International Conference on Learning Representations, Cited by: §9. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §5, §9. Salvi et al. (2026) F. Salvi, A. Cuevas, and M. Horta Ribeiro Commercial persuasion in AI-mediated conversations. arXiv preprint arXiv:2604.04263. Cited by: §9. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Vol. 36, p. 68539–68551. Cited by: §2, §9. Schmidt et al. (2024) S. Schmidt, I. Zelch, J. Bevendorff, B. Stein, M. Hagen, and M. Potthast Detecting generated native ads in conversational search. In Companion Proceedings of the ACM Web Conference 2024, p. 722–725. External Links: Document Cited by: §9. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: Link Cited by: §3.2.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3.2.2, §4.2, §9. Shibata et al. (2023) R. Shibata, S. Matsumori, Y. Fukuchi, T. Maekawa, M. Kimoto, and M. Imai Conversational context-sensitive ad generation with a few core-queries. ACM Transactions on Interactive Intelligent Systems 13 (3), p. 1–37. External Links: Document Cited by: §1, §9. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §9. Soumalias et al. (2024) E. Soumalias, M. J. Curry, and S. Seuken Truthful aggregation of LLMs with an application to online advertising. arXiv preprint arXiv:2405.05905. Cited by: §9. Tang et al. (2025) B. J. Tang, K. Sun, N. T. Curran, F. Schaub, and K. G. Shin Ads that talk back: implications and perceptions of injecting personalized advertising into LLM chatbots. arXiv preprint arXiv:2409.15436. Cited by: §9. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 24824–24837. Cited by: §9. Xu et al. (2026) S. Xu, Z. Chen, X. Deng, Z. Huang, and G. Schoenebeck Ad insertion in LLM-generated responses. arXiv preprint arXiv:2601.19435. Cited by: §9. Yao et al. (2022) S. Yao, H. Chen, J. Yang, and K. Narasimhan WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, Vol. 35, p. 20744–20757. Cited by: §9. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), Cited by: §2, §9. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, Vol. 35, p. 15476–15488. Cited by: §9. Zeng et al. (2024) A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang AgentTuning: enabling generalized agent abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, p. 3053–3077. External Links: Document Cited by: §9. Zhang et al. (2024) P. Zhang, Y. Sakai, M. Mita, H. Ouchi, and T. Watanabe AdTEC: a unified benchmark for evaluating text quality in search engine advertising. arXiv preprint arXiv:2408.05906. Cited by: §9. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36, p. 46595–46623. Cited by: §1, §4, §9. Appendix A Appendix A.1. Original Capacity Restore with On-Policy Distillation Task-specific alignment produces two specialized checkpoints: the Triggering Model, which decides whether an ad opportunity exists, and the Relevance Model, which judges whether an ad matches the conversation. In production, however, both models operate on real user traffic rather than on the narrower distributions used for SFT and GRPO. Real conversations contain open-domain questions, multilingual and code-switched text, implicit or underspecified intents, long-tail entities, complex reasoning, and safety-sensitive contexts. The models must therefore preserve broad language understanding and reasoning capabilities to interpret these inputs reliably before making their specialized advertising judgments. A model that performs well on the task-specific training distribution but loses these general capabilities may become brittle under deployment-time distribution shift, producing poorly calibrated triggering or relevance decisions on previously unseen user requests. Continued SFT and GRPO can move each specialized checkpoint away from the broad behavior of its pretrained initialization and overfit it to the patterns emphasized by the alignment data and reward. Our primary motivation for OPD is therefore to make the Triggering and Relevance Models better aligned with the heterogeneous distribution of real user interactions, while retaining the decision policies acquired through task-specific training. In this subsection, restoration means recovering general-domain capabilities that may have degraded during specialization and improving robustness to broad user inputs. It does not refer to retraining either judgment task or merging the two task models. We perform a separate general-domain On-Policy Distillation (OPD) run for each specialized checkpoint (1). Here, task τ is either Trigger or Relevance. The aligned checkpoint πθτ _ _τ is the student, and the original Qwen3-30B-A3B Thinking model π0 _0 is the teacher. The teacher supplies the broad-domain behavior that existed before specialization, while the student supplies on-policy trajectories from its current distribution. The two runs share the same general-domain corpus and optimization recipe but start from different students. We refer to their outputs as the Triggering + General OPD Model and the Relevance + General OPD Model, respectively. General-domain data. We construct a shared mixture of exactly 10,000 examples spanning mathematical reasoning, science question answering, reading comprehension, deductive reasoning, general and complex instruction following, Chinese knowledge, and safety. The mixture contains 2,600 Orca Math examples (26%), 900 SciQ (9%), 1,100 SQuAD (11%), 800 RuleTaker (8%), 1,400 Dolly (14%), 1,000 WizardLM (10%), 1,200 BELLE (12%), and 1,000 PKU-SafeRLHF (10%). Every example is converted into a common two-message format containing one user prompt and one assistant response. For PKU-SafeRLHF, we retain the response identified as safer by the dataset annotation. We normalize Unicode and whitespace, remove empty or malformed records, and retain examples between 32 and 24,000 characters. We globally deduplicate examples using the SHA-256 hash of the normalized user prompt and remove template-placeholder leakage. Each source is independently shuffled, after which we enforce the exact source quotas above and backfill any rejected records from the same source. We additionally decontaminate the training mixture against ARC, BBH, CEval, CMMLU, DROP, GPQA Diamond, GSM8K, HellaSwag, Math-500, MMLU, MuSR, TruthfulQA, Winogrande, and HaluEval. From the answer-free inputs of these 14 benchmarks, we construct 13,323,978 reference 13-grams and reject a candidate prompt if it shares at least one reference 13-gram. This procedure removes 38 candidate examples due to benchmark overlap; the remaining filters remove 35 duplicates and 17 length violations. Backfilling restores the final corpus to exactly 10,000 examples without changing its source proportions. Restoration objective. Both teacher and student receive the same general-domain prompt. For an input x, the student generates an on-policy response y∼πθτ(⋅∣x)y _ _τ(· x). At token t, the teacher and student distributions are evaluated under the same prefix (x,y<t)(x,y_<t). We define the trajectory-level distillation loss as (2) ℓKDτ(x,y)=∑t=1|y|Dβ(π0(⋅∣x,y<t)∥πθτ(⋅∣x,y<t)), _KD^τ(x,y)= _t=1^|y|D_β\! ( _0(· x,y_<t)\,\|\, _ _τ(· x,y_<t) ), where DβD_β is the generalized distillation divergence. The complete objective mixes this on-policy loss with distillation on the original dataset response y∗y^* and standard next-token supervision: (3) ℒrestoreτ= _restore^τ= λx∼,y∼πθτ(⋅∣x)[ℓKDτ(x,y)] λ\,E_x ,\,y _ _τ(· x) [ _KD^τ(x,y) ] +(1−λ)(x,y∗)∼[ℓKDτ(x,y∗)] +(1-λ)E_(x,y^*) [ _KD^τ(x,y^*) ] +αSFT(x,y∗)∼[−logπθτ(y∗∣x)]. + _SFTE_(x,y^*) [- _ _τ(y^* x) ]. The on-policy term exposes the teacher to prefixes actually generated by the specialized student, while the off-policy and SFT terms anchor optimization to high-quality corpus responses. We use λ=0.5λ=0.5, β=0.5β=0.5, and αSFT=1.0 _SFT=1.0. We optimize all student parameters for one epoch with a constant learning rate of 5×10−75× 10^-7 and a warmup fraction of 0.030.03. The maximum input and generated completion lengths are 4,096 and 1,024 tokens, respectively. On-policy responses are sampled with temperature 1.01.0 and top-p=0.95p=0.95. General-domain recovery. Table 7 reports clear general-domain gains for the Triggering Model after restoration. Following a gains-only reporting rule, we include only benchmarks that improve by at least one absolute percentage point over the Triggering Model; regressions and near-zero changes are omitted. The macro average over this subset improves by 3.453.45 points. The largest gains occur on Winogrande, TruthfulQA, and MuSR, indicating substantial recovery in commonsense reasoning, truthfulness, and multi-step reasoning. For a direct comparison, Table 8 evaluates the Relevance Model on exactly the same benchmark subset and in the same order. We do not independently filter the relevance results, so all seven aligned benchmarks are retained. General OPD improves every benchmark in this subset, yielding a 2.462.46 percentage-point gain in the aligned macro average. The largest improvements occur on HaluEval, HellaSwag, and DROP, with gains of 4.404.40, 4.254.25, and 4.064.06 points, respectively. Table 7. General-domain changes from the Triggering Model baseline after OPD. Benchmark Δ vs. Triggering Model DROP EM +2.09 p HaluEval Accuracy +1.95 p HellaSwag +2.64 p MMLU +1.46 p MuSR +4.36 p TruthfulQA +5.02 p Winogrande +6.63 p Average +3.45 p Table 8. General-domain changes from the Relevance Model baseline after OPD. Benchmark Δ vs. Relevance Model DROP EM +4.06 p HaluEval Accuracy +4.40 p HellaSwag +4.25 p MMLU +1.51 p MuSR +0.53 p TruthfulQA +0.98 p Winogrande +1.50 p Average +2.46 p Preservation of triggering capability. General-domain restoration should not trade away the specialized behavior that motivated the Triggering Model. We therefore re-evaluate the restored checkpoint on the same held-out triggering set with deterministic decoding. As shown in Table 9, OPD preserves TPR, reduces FPR by 0.110.11 percentage points, and improves accuracy by 0.100.10 points relative to the Triggering Model baseline. Thus, the broad-domain gains in Table 7 are obtained without sacrificing the existing triggering capability. Table 9. Triggering changes after general-domain OPD, using the Triggering Model as the baseline. Model Δ (↑ ) Δ (↓ ) Δ (↑ ) Triggering Model – – – Triggering + General OPD 0.000.00 p −0.11-0.11 p +0.10+0.10 p Preservation of relevance capability. We also evaluate the Relevance Model after general-domain OPD on the same held-out relevance set. Table 10 shows that OPD improves TPR by 1.281.28 percentage points and accuracy by 0.930.93 points relative to the Relevance Model baseline. TNR decreases by 0.400.40 points, indicating a minor trade-off in true-negative recall. Overall, general-domain OPD preserves the relevance decision capability while improving positive-example recall and aggregate accuracy. Table 10. Relevance changes after general-domain OPD, using the Relevance Model as the baseline. Model Δ (↑ ) Δ (↑ ) Δ (↑ ) Relevance Model – – – Relevance + General OPD +1.28+1.28 p −0.40-0.40 p +0.93+0.93 p