Paper deep dive
AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization
Yuejia Dou, Hesong Wang, Xinyu Zhang, Tianyu Wang, Zhilin Zhang, Chuan Yu, Jian Xu, Bo Zheng, Qi Qi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/21/2026, 4:50:28 AM
Summary
The paper introduces AIGB-R1, a hierarchical self-evolving auto-bidding framework that leverages Large Language Models (LLMs) for macro-level strategy planning and a Prompt Decision Transformer (PDT) for fine-grained decision-making. It addresses limitations of existing generative auto-bidding methods, such as limited dataset coverage and numerical precision issues, by employing an experience-driven self-evolving loop and a novel Decoupled Group Relative Policy Optimization (D-GRPO) algorithm for end-to-end optimization.
Entities (8)
Relation Signals (7)
AIGB-R1 → contains → Planner
confidence 95% · AIGB-R1 ... comprising a high-level Planner module for macro-level strategy planning
AIGB-R1 → contains → Executor
confidence 95% · AIGB-R1 ... and a low-level Executor module for fine-grained decision-making
AIGB-R1 → uses → D-GRPO
confidence 94% · we propose Decoupled Group Relative Policy Optimization (D-GRPO) to achieve end-to-end optimization
AIGB-R1 → solves → Auto-bidding
confidence 93% · AIGB-R1, a hierarchical self-evolving auto-bidding framework
Executor → implements → Prompt Decision Transformer
confidence 92% · The Executor module employs Prompt Decision Transformer (PDT) to generate bidding parameters
AIGB-R1 → belongsto → AI-Generated Bidding
confidence 91% · enhance AI-Generated Bidding via LLMs’ Reasoning capabilities
Planner → uses → LLM
confidence 90% · the Planner module employs a large LLM to reason about macro-level strategies
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Auto-bidding plays an essential role in online advertising, automatically adjusting bids for advertisers to optimize their commercial goals. The emerging AI-Generated Bidding (AIGB) paradigm widely adopts generative modeling to optimize bidding strategies, yet suffers from the limited mode coverage of offline datasets and inadequate task-state understanding, hindering effective exploration of optimal strategies. Large Language Models (LLMs), with prior world knowledge and reasoning capabilities, offer a promising approach to overcome these limitations. However, directly applying LLMs to auto-bidding tasks faces inherent challenges in limited numerical precision, hallucinations, and inference latency. To address these limitations, we propose AIGB-R1, a hierarchical self-evolving auto-bidding framework aiming to enhance AI-Generated Bidding via LLMs' Reasoning capabilities, comprising a high-level Planner module for macro-level strategy planning and a low-level Executor module for fine-grained decision-making. Building upon this, we design an experience-driven self-evolving loop, enabling autonomous strategy exploration and optimization from accumulated experience. We adopt a two-stage pipeline of offline pre-training and post-training alignment, and build an interactive bidding simulation environment for strategy rollout. Furthermore, we propose Decoupled Group Relative Policy Optimization (D-GRPO) to achieve end-to-end optimization via advantage decoupling. Experimental results on a large-scale public dataset demonstrate the effectiveness of AIGB-R1.
Tags
Links
- Source: https://arxiv.org/abs/2607.17281v1
- Canonical: https://arxiv.org/abs/2607.17281v1
Trouble viewing inline? Open PDF directly →
Full Text
78,034 characters extracted from source content.
Expand or collapse full text
AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization Yuejia Dou douyuejia@ruc.edu.cn Gaoling School of Artificial Intelligence, Renmin University of ChinaBeijingChina , Hesong Wang whs040114@gmail.com Gaoling School of Artificial Intelligence, Renmin University of ChinaBeijingChina , Xinyu Zhang zxy479640@alibaba-inc.com Alibaba GroupBeijingChina , Tianyu Wang yves.wty@alibaba-inc.com Alibaba GroupBeijingChina , Zhilin Zhang zhangzhilin.pt@alibaba-inc.com Alibaba GroupBeijingChina , Chuan Yu yuchuan.yc@alibaba-inc.com Alibaba GroupBeijingChina , Jian Xu xiyu.xj@alibaba-inc.com Alibaba GroupBeijingChina , Bo Zheng bozheng@alibaba-inc.com Alibaba GroupBeijingChina and Qi Qi qi.qi@ruc.edu.cn Gaoling School of Artificial Intelligence, Renmin University of ChinaBeijingChina (2026) Abstract. Auto-bidding plays an essential role in online advertising, automatically adjusting bids for advertisers to optimize their commercial goals. The emerging AI-Generated Bidding (AIGB) paradigm widely adopts generative modeling to optimize bidding strategies, yet suffers from the limited mode coverage of offline datasets and inadequate task-state understanding, hindering effective exploration of optimal strategies. Large Language Models (LLMs), with prior world knowledge and reasoning capabilities, offer a promising approach to overcome these limitations. However, directly applying LLMs to auto-bidding tasks faces inherent challenges in limited numerical precision, hallucinations, and inference latency. To address these limitations, we propose AIGB-R1, a hierarchical self-evolving auto-bidding framework aiming to enhance AI-Generated Bidding via LLMs’ Reasoning capabilities, comprising a high-level Planner module for macro-level strategy planning and a low-level Executor module for fine-grained decision-making. Building upon this, we design an experience-driven self-evolving loop, enabling autonomous strategy exploration and optimization from accumulated experience. We adopt a two-stage pipeline of offline pre-training and post-training alignment, and build an interactive bidding simulation environment for strategy rollout. Furthermore, we propose Decoupled Group Relative Policy Optimization (D-GRPO) to achieve end-to-end optimization via advantage decoupling. Experimental results on a large-scale public dataset demonstrate the effectiveness of AIGB-R1. Generative Model, Large Language Model, Auto-bidding, Preference Alignment †copyright: acmlicensed†journalyear: 2026†doi: X.X†conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY†isbn: 978-1-4503-X-X/18/06†ccs: Information systems Computational advertising 1. Introduction With the rapid digitalization of commerce, online advertising platforms have continually expanded their traffic coverage, becoming critical channels for businesses to attract target audiences and boost sales (Evans, 2009; Ha, 2008; Wang and Yuan, 2015). Given billions of impressions and intense competition, the traditional paradigm of manual bid adjustment has become impractical. To alleviate this burden, major advertising platforms provide auto-bidding services that automatically determine bids for each impression to optimize advertisers’ commercial goals (Balseiro et al., 2021; Deng et al., 2021; Ou et al., 2023). In the emerging AI-Generated Bidding (AIGB) paradigm, auto-bidding can be formulated as a sequential decision-making task, where the objective is to generate optimal bidding parameters that maximize cumulative conversions subject to constraints such as cost-per-action (CPA) and budget, based on historical bidding sequences and current market states (Guo et al., 2024; He et al., 2021). Within this paradigm, generative auto-bidding methods represented by Decision Transformer (DT) (Chen et al., 2021; Zheng et al., 2022) have attracted widespread attention (Gao et al., 2025; Li et al., 2025b; Dou et al., 2026). These methods employ generative models to learn optimal strategies from historical bidding trajectories collected from online advertising platforms, autoregressively generating bidding parameters conditioned on return-to-go, state, and action sequences. However, existing generative auto-bidding methods face critical challenges. First, their performance is restricted by the quality and coverage of the offline dataset, leading to degraded decision quality when encountering out-of-distribution states (Li et al., 2026, 2025b). Second, existing generative auto-bidding methods simply condition policies on return-to-go without fully understanding the true value of actions and specific task states, making it difficult to plan optimal strategies (Ajay et al., 2022; Gao et al., 2025). Third, existing generative methods are typically trained under fixed preference distributions and struggle to adapt to diverse advertiser preferences in real-world bidding services, while retraining for different preferences incurs substantial training costs (Huang et al., 2025b; Song et al., 2026; Dou et al., 2026). These challenges limit the further development and widespread adoption of generative auto-bidding services (Wang et al., 2017), but the rapid advancement of Large Language Models (LLMs) in recent years offers new insights into how to overcome these bottlenecks (Bai et al., 2023; Yang et al., 2025; Guo et al., 2025). With prior world knowledge and reasoning capabilities, LLMs are expected to transcend the limitations of offline dataset coverage and adapt to diverse bidding strategy preferences through semantic understanding and multi-task generalization. However, directly applying LLMs to auto-bidding tasks faces inherent limitations. First, auto-bidding requires precise control over continuous numerical action spaces when generating bidding parameters, yet LLMs suffer from inherent limitations in numerical decision-making, including insufficient precision and hallucinations (Forootani, 2025; Huang et al., 2025a; Lv et al., 2026). Moreover, online advertising platforms require auto-bidding agents to process numerous bid requests under tight latency constraints, while LLMs face a fundamental trade-off between decision-making quality and inference latency: larger LLMs possess strong decision-making and planning capabilities but struggle to meet the low-latency requirements, whereas significantly reducing model size alleviates latency issues but inevitably leads to substantial degradation in decision-making and planning capabilities (Cai et al., 2025; Huang et al., 2025a; Xiao et al., 2025). Several works have explored integrating LLM techniques into auto-bidding systems from different perspectives. One straightforward approach uses an LLM as the decision-maker, leveraging reasoning capabilities with memory and reflection mechanisms to output bidding decisions, but is directly constrained by the aforementioned numerical precision, hallucination, and latency issues (Cai et al., 2025). Other methods optimize generative policy models through semantic representation embedding, or through post-training policy search techniques, but fundamentally fail to fully exploit the decision-making and planning potential of LLMs (Zhu et al., 2026; Dou et al., 2026; Li et al., 2025b). Recently, LBM (Li et al., 2026) designed a hierarchical framework with separate modules dedicated to high-level reasoning and low-level decision-making respectively, effectively retaining LLM reasoning and planning capabilities while circumventing inherent bottlenecks, representing a promising architectural direction for LLM-empowered auto-bidding (Wang et al., 2023b; Song et al., 2026; Wan et al., 2025; Ahn et al., 2022). However, the above methods still face the following challenges when learning optimal auto-bidding strategies. First, existing methods typically rely on offline pre-trained value functions to evaluate strategy quality, but these value functions often suffer from extrapolation errors on out-of-distribution strategies, struggling to guide exploration toward potentially optimal regions (Fujimoto et al., 2019; Kumar et al., 2020, 2019). Second, existing methods’ post-training search and alignment typically rely on repeated random sampling, which has limited exploration efficiency in auto-bidding tasks with high-dimensional continuous strategy spaces, generating massive redundant trajectories while failing to effectively reuse historical rollout experience for self-improvement (Guo et al., 2024; Gao et al., 2025; Li et al., 2024). Finally, existing methods either perform training-free test-time search or only optimize the high-level reasoning module while freezing the low-level execution module, thereby limiting convergence to globally optimal strategies compared with end-to-end optimization. To overcome these challenges, we propose AIGB-R1, a hierarchical self-evolving auto-bidding framework designed to enhance AI-Generated Bidding via LLMs’ Reasoning capabilities, achieving autonomous exploration and evolution of bidding strategies through an experience-driven self-evolving loop. AIGB-R1 temporally decouples the auto-bidding task into two stages: macro-level strategy planning and fine-grained bidding decision-making, based on which we construct a high-level Planner module and a low-level Executor module, respectively. Specifically, the Planner module employs a large LLM to reason about macro-level strategies for each bidding period and generate structured semantic strategy prompts. The Executor module employs Prompt Decision Transformer (PDT) to generate bidding parameters at each timestep within the bidding period, conditioned on both observed numerical sequences and the Planner’s strategy prompts. Building upon this, we introduce an experience-driven self-evolving loop that enables AIGB-R1 to autonomously explore and optimize strategies from accumulated rollout experience, rather than blindly exploring through random sampling. We adopt a two-stage training pipeline: In the pre-training stage, we train the Executor to equip it with the capability to fuse observed sequences and strategy prompts for bidding decisions. In the post-training stage, we build an interactive bidding simulation environment for strategy rollout, mitigating extrapolation errors associated with offline value estimation. We design Decoupled Group Relative Policy Optimization (D-GRPO) to achieve inter-layer credit assignment, enabling end-to-end optimization of the Planner and Executor. In summary, our contributions are as follows: • We propose AIGB-R1, a hierarchical self-evolving auto-bidding framework comprising a Planner for macro-level strategy planning and an Executor for fine-grained bidding decision-making, fully leveraging LLMs’ reasoning and planning capabilities while effectively circumventing their inherent limitations in limited numerical precision, hallucinations and inference latency. • We design an experience-driven self-evolving loop that enables AIGB-R1 to autonomously explore and optimize strategies from accumulated rollout experience, continuously approaching optimal bidding strategies through sustained interaction with the simulation environment during post-training. • We build an interactive bidding simulation environment for strategy rollout and propose D-GRPO to achieve inter-layer credit assignment through advantage decoupling, enabling end-to-end optimization of the LLM Planner and PDT Executor. • Experimental results on a large-scale public dataset demonstrate that AIGB-R1 significantly outperforms various state-of-the-art baselines, validating the effectiveness of the proposed framework. 2. Related Work 2.1. Auto-bidding Methods Auto-bidding helps advertisers achieve their commercial goals by adaptively optimizing bids for each impression (He et al., 2021; Xu et al., 2024b). Early methods primarily relied on predefined rules or online learning to adjust strategies (Chen et al., 2011; Yu et al., 2017). As online bidding environments grew increasingly complex, reinforcement learning methods such as USCB (He et al., 2021), SORL (Mou et al., 2022), and MAAB (Wen et al., 2022) became essential for auto-bidding. Due to the risks of online interaction, offline RL methods including BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and IQL (Kostrikov et al., 2021) have gained widespread attention for their ability to learn effective strategies from offline datasets. However, these methods are subject to the Markov assumption, which limits their sequential modeling capability. Recently, generative models represented by Decision Transformer (DT) (Chen et al., 2021; Zheng et al., 2022) have opened a new paradigm for auto-bidding. DiffBid (Guo et al., 2024) uses diffusion models to generate bidding trajectories, whereas CBD (Li et al., 2025a) introduces a diffusion completer–aligner for trajectory completion. GAS (Li et al., 2025b) performs post-training search via MCTS, GAVE (Gao et al., 2025) employs value-guided exploration. PRO-Bid (Wu et al., 2026) introduces Pareto-prioritized regret optimization, while GRAD (Lei et al., 2026) develops a large-scale pre-trained DT-based bidding model. For the broader auto-marketing setting, GAM (Dou et al., 2026) employs DTs for joint bidding and coupon distribution with GRPO-based preference alignment. Several works have begun exploring LLM-empowered auto-bidding. RTBAgent (Cai et al., 2025) uses prompt engineering to directly output bidding decisions via LLMs, SemBid (Zhu et al., 2026) integrates LLM semantic representations to assist generative sequence modeling, LBM (Li et al., 2026) designs a hierarchical Think-Act architecture and employs GQPO for fine-tuning. 2.2. LLM for Decision-Making Recently, the autonomous decision-making potential of LLMs has been extensively explored. Early works such as ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023) established the foundational paradigm of reasoning, acting, and reflection. However, recent studies reveal inherent limitations of LLMs as direct decision-makers (Schmied et al., 2025), motivating researchers to leverage LLMs for macro-level planning rather than direct action execution. SayCan (Ahn et al., 2022), LgTS (Shukla et al., 2023), and DART-LLM (Wang et al., 2024) translate LLM-generated language plans into executable actions through feasibility assessment and effectiveness scoring, while PAR (He et al., 2024) and ACE (Wan et al., 2025) propose hierarchical collaboration frameworks between LLMs and RL agents. Meanwhile, reinforcement learning has been widely adopted to optimize the decision-making capabilities of LLMs. RLHF (Ouyang et al., 2022) optimizes models using human feedback to construct reward signals. DPO (Rafailov et al., 2023) and its extensions (Azar et al., 2024; Ethayarajh et al., 2024; Meng et al., 2024; Xu et al., 2024a) directly fine-tune models with human preference datasets. GRPO (Shao et al., 2024) improves the stability and efficiency of fine-tuning through group relative advantage estimation. GiGPO (Feng et al., 2026) introduces fine-grained credit assignment for multi-step decision-making tasks. 2.3. Self-Evolving Agents Self-evolving agents continuously improve their capabilities through accumulated experience. Early works achieve self-improvement via in-context learning with feedback from experience. Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024a), and Voyager (Wang et al., 2023a) retain interaction experience in the form of linguistic reflections or generalized rules to iteratively improve subsequent decision quality. These methods enable strategy evolution without parameter updates, but are often constrained by the inherent limits of the model’s capabilities. To further enhance LLMs’ performance through parameter updates, STaR (Zelikman et al., 2024) and ReST (Gulcehre et al., 2023) bootstrap LLM reasoning capabilities via iterative self-training. WebRL (Qi et al., 2025) and SCA (Zhou et al., 2026) drive online RL training through automatic curriculum generation. EvolveR (Wu et al., 2025) designs a collect-refine-guide self-evolving loop. 3. Preliminary 3.1. Problem Formulation We consider the auto-bidding problem with cost-related constraints. Within a bidding period, we assume that I impression opportunities arrive sequentially, indexed by i. Advertisers submit bids on the platform to compete for each impression. For each arriving impression i∈[I]i∈[I], an advertiser wins when its bid bib_i exceeds those of other advertisers, and pays the winning cost cic_i, which is typically the highest bid of the other advertisers in the second-price auction (SPA) setting. The advertiser’s objective is to maximize the cumulative value of won impressions ∑ioivi _io_iv_i over the entire bidding period, where viv_i denotes the value of the impression and oio_i indicates whether the advertiser wins it. Moreover, to control advertising delivery performance, advertisers typically need to satisfy the budget and multiple Key Performance Indicator (KPI) constraints (He et al., 2021). The budget constraint is expressed as ∑ioici≤B _io_ic_i≤ B, where cic_i is the cost of impression i and B is the budget. Other KPI constraints are more complex and can be uniformly expressed as: (1) ∑icijoi∑ipijoi≤Cj, _ic_ijo_i _ip_ijo_i≤ C_j, where CjC_j is the upper bound of the j-th constraint, pijp_ij can be any performance indicator such as return or constant, and cijc_ij is the cost of the j-th constraint. Therefore, given J constraints, the auto-bidding problem can be formulated as: max ∑ioivi _io_iv_i s.t. ∑ioici≤B, _io_ic_i≤ B, (2) ∑icijoi∑ipijoi≤Cj,∀j, _ic_ijo_i _ip_ijo_i≤ C_j, ∀\,j, oi∈0,1,∀i. o_i∈\0,1\, ∀\,i. A previous study (He et al., 2021) has shown that the optimal solution is: (3) bi∗=λ0vi+∑j=1JλjpijCj,b_i^*= _0v_i+ _j=1^J _jp_ijC_j, where bi∗b_i^* is the optimal bid for impression i, and λj,j∈0,…,J _j,j∈\0,…,J\ are the optimal bidding parameters. However, the uncertainty and dynamics of online advertising systems make it impractical for advertisers to directly calculate these optimal bidding parameters. Instead, these parameters should be regularly adjusted in response to the dynamic environment, rendering auto-bidding a sequential decision-making task of iteratively identifying the optimal bidding parameters. 3.2. Sequential Modeling of Auto-Bidding Under the sequential modeling formulation of auto-bidding, we first divide the bidding period into T discrete timesteps. At each timestep t, the auto-bidding agent receives a real-time advertising state st∈s_t and then outputs an action at∈a_t according to its policy π to adjust the bidding parameters. The state transition dynamics of the advertising environment are unknown, and the next state st+1∈s_t+1 depends not only on the current state sts_t and action ata_t, but also potentially on the historical bidding trajectory τ. After transitioning to the next state, the environment returns a reward rtr_t, representing the cumulative impression value obtained during timestep t. The auto-bidding agent’s objective is to maximize the cumulative impression value ∑trt _tr_t over the entire bidding period. The detailed description of our modeling is as follows: • State sts_t: The state sts_t describes the advertising status at timestep t, which can include the remaining time of the bidding period, remaining budget, budget consumption speed, and other historical statistics. • Action ata_t: The action ata_t represents the adjustment of bidding parameters λj _j (j=0,…,Jj=0,…,J) at timestep t, modeled as (atλ0,…,atλJ)(a_t _0,…,a_t _J). • Reward rtr_t: The reward rtr_t represents the value contributed to the objective during the period from timestep t to t+1t+1. • Return-To-Go (RTG) RtR_t: The RTG value RtR_t represents the sum of rewards to be obtained in the future timesteps: (4) Rt=∑t′=tTrt′,R_t= _t =t^Tr_t , where T is the final timestep. 4. Method 4.1. Framework Overview Figure 1. Overall framework of AIGB-R1. Diagram showing the overall framework of AIGB-R1. In the emerging AIGB paradigm, generative models exemplified by Decision Transformer (DT) have been widely adopted for their ability to learn from offline datasets. However, these methods are limited by the mode coverage of offline datasets and struggle to fully understand task states and strategy preferences, motivating us to explore a new approach that leverages the semantic understanding, reasoning, and planning capabilities of LLMs to empower auto-bidding. However, LLMs’ inherent limitations, including limited numerical precision, hallucinations, and inference latency, make it impractical to directly use a single LLM for end-to-end bidding. To address these limitations, a practical architectural design is to decouple strategy reasoning and real-time decision-making into two separate modules using a hierarchical framework, where a large LLM asynchronously performs macro-level strategy reasoning, while a lightweight decision model makes real-time decisions at each timestep based on the derived strategy and current states. Accordingly, we temporally decompose the auto-bidding task into macro-level strategy planning followed by fine-grained bidding decision-making, handled by a high-level Planner and a low-level Executor respectively. Specifically, the Planner module employs a large LLM to reason about macro-level strategies for each bidding period and generates structured strategy prompts. The Executor module employs Prompt Decision Transformer (PDT) as a lightweight decision-maker, generating bidding parameters at each timestep within the bidding period, conditioned on both the observed numerical sequences and the strategy prompts. Next, we formally describe the sequential decision-making process of AIGB-R1. Consider a bidding period for advertiser g. Before the bidding period begins, we obtain the advertiser’s contextual information gC_g (including budget, CPA constraints, and advertising category) and its historical bidding performance from previous periods ℋgH_g. The high-level LLM Planner πθ _ _P then plans the macro-level strategy (e.g., risk preference) for the current bidding period based on gC_g and ℋgH_g, generating a structured strategy prompt p that contains enumerated bidding strategy labels and macro-level strategy reasoning: (5) p∼πθ(⋅∣g,ℋg),p _ _P (· _g,H_g ), Notably, this planning step is completed before the bidding period begins, and the generated strategy prompt remains effective throughout the entire period. Subsequently, at each timestep t within the bidding period, the low-level PDT Executor πθℰ _ _E generates an action ata_t conditioned on the observed numerical sequences of RTG values, states, and actions (Rt−L:t,st−L:t,at−L:t−1)(R_t-L:t,s_t-L:t,a_t-L:t-1) along with the strategy prompt p, where L denotes the context length of PDT: (6) at∼πθℰ(⋅∣p,Rt−L:t,st−L:t,at−L:t−1)a_t _ _E (· p,R_t-L:t,s_t-L:t,a_t-L:t-1 ) PDT embeds the strategy prompt as a conditional prefix to the standard Decision Transformer’s input sequence, enabling the Planner’s macro-level strategy to naturally influence the conditional autoregressive generation of actions at each timestep. The overall framework of AIGB-R1 is illustrated in Figure 1. We adopt a two-stage training pipeline of pre-training and post-training alignment: In the pre-training stage, we pre-train the Executor offline to equip it with the capability to fuse observed sequences and strategy prompts for bidding decisions. In the post-training stage, we fine-tune the entire framework end-to-end via reinforcement learning to further discover optimal strategies. 4.2. PDT Executor Pre-training with Strategy Prompts Despite LLMs’ strong semantic understanding and reasoning capabilities, directly using them as the low-level Executor faces fundamental limitations. First, industrial bidding systems typically require millisecond-level response times, yet LLMs struggle to simultaneously achieve high performance and low latency. Second, LLMs are inherently unsuitable for tasks with continuous numerical action spaces such as auto-bidding, where the decision model must output precise bidding parameters at each timestep, while LLMs’ encoding of numerical inputs as discrete tokens inevitably introduces precision loss and hallucination risks. Furthermore, current public LLMs have not been specifically pre-trained for auto-bidding tasks, meaning their general prior knowledge can hardly provide effective inductive biases for predicting bidding parameters, making it more practical to directly train a dedicated lightweight decision model on offline bidding trajectories. These limitations motivate us to adopt PDT rather than an LLM as the core architecture of the low-level Executor. PDT extends the standard DT by introducing a textual strategy prompt as a conditional sequence prefix. Specifically, PDT embeds the strategy prompt p into vector representations and prepends them to the standard DT input sequence (Rt−L:t,st−L:t,at−L:t−1)(R_t-L:t,s_t-L:t,a_t-L:t-1), forming the extended input sequence (p,Rt−L:t,st−L:t,at−L:t−1)(p,R_t-L:t,s_t-L:t,a_t-L:t-1). At timestep t, PDT autoregressively generates action ata_t conditioned on this sequence. In this way, the strategy prompt influences the conditional autoregressive sampling of actions at each subsequent timestep through the self-attention mechanism, achieving strategy-conditioned decision-making. The strategy prompt p of the PDT Executor contains two forms of information: enumerated strategy labels and macro-level strategy reasoning. The strategy labels cover predefined discrete strategy dimensions such as risk preference and budget pacing, while the macro-level strategy reasoning provides a summary of historical bidding performance along with semantic reasoning and detailed explanations for the current period’s bidding strategy. This dual design balances structural and semantic expressiveness: enumerated strategy labels effectively reduce the variance of strategy learning, enabling the Executor to efficiently learn coarse-grained strategy behavior patterns, while the macro-level strategy reasoning provides fine-grained semantic strategy expressions, compensating for the strategy details and decision rationale that discrete labels cannot capture. We use pre-trained text embedding layers to process strategy labels and macro-level strategy reasoning into vector representations of equal size, and concatenate them to form the complete strategy prompt representation p. During the pre-training stage of the PDT Executor, we first use an advanced LLM to annotate the offline bidding trajectory dataset with corresponding strategy prompts, and then perform offline supervised training on PDT. Whereas previous DT methods learn deterministic policies πθ(at|Rt−L:t,st−L:t,at−L:t−1) _θ(a_t|R_t-L:t,s_t-L:t,a_t-L:t-1), we opt to learn a stochastic policy that maximizes the dataset likelihood, as stochastic policies enable efficient strategy exploration through repeated sampling during post-training and can be directly fine-tuned with policy-based reinforcement learning algorithms. Specifically, since the action ata_t is a non-negative continuous value, we adopt the log-normal distribution to model the stochastic policy: (7) πθℰ(at∣p,Rt−L:t,st−L:t,at−L:t−1)=ℒ(μθℰ(at∣p,Rt−L:t,st−L:t,at−L:t−1),σθℰ(at∣p,Rt−L:t,st−L:t,at−L:t−1)) split& _ _E(a_t p,R_t-L:t,s_t-L:t,a_t-L:t-1)=\\ &LN ( _ _E(a_t p,R_t-L:t,s_t-L:t,a_t-L:t-1), _ _E(a_t p,R_t-L:t,s_t-L:t,a_t-L:t-1) ) split During pre-training, we train the PDT Executor to fit the offline data distribution by maximizing the log-likelihood of the stochastic policy over the trajectories in the training dataset, equivalently minimizing the Negative Log-Likelihood (NLL) loss: (8) ℒNLL=1L+1(p,R,s,a)∼[−∑t′=t−Ltlogπθℰ(at′∣p,Rt′−L:t′,st′−L:t′,at′−L:t′−1)] _NLL= 1L+1E_(p,R,s,a) [- _t =t-L^t _ _E (a_t p,R_t -L:t ,s_t -L:t ,a_t -L:t -1 ) ] 4.3. Self-Evolving Post-training via D-GRPO Limited by the quality and coverage of offline training datasets, pre-trained generative models often converge to local optima. We therefore employ reinforcement learning in the post-training stage to fine-tune the Planner and Executor end-to-end for further performance improvement. However, conventional RL methods typically rely on repeated random trial-and-error, resulting in massive redundant trajectories and slow convergence. To this end, we build a self-evolving post-training framework consisting of the interactive bidding simulation environment AuctionNetEnv, an experience-guided hierarchical rollout mechanism, and the D-GRPO algorithm for inter-layer credit assignment, thereby enabling AIGB-R1 to internalize and leverage accumulated experience and transforming strategy optimization from undirected trial-and-error into a more effective self-improvement process. 4.3.1. Interactive Bidding Simulation Environment To enable the auto-bidding agent to obtain reward signals through rollouts for guiding strategy optimization, we build a flexible interactive bidding simulation environment, AuctionNetEnv, based on a public large-scale auto-bidding decision-making benchmark (Su et al., 2024). AuctionNetEnv enables the auto-bidding agent to compete with diverse opponent agents under a second-price auction, with these opponents trained using different decision-making algorithms. The detailed construction of AuctionNetEnv and its distributional consistency validation are provided in Appendix A.1. At each training step, the auto-bidding agent samples a bidding task g from AuctionNetEnv’s task pool G, described by the advertiser’s contextual information gC_g including budget, CPA constraints, and advertising category, and then executes a complete bidding period for this task. At each timestep t, the agent generates an action ata_t and computes the bid btb_t. The environment then conducts the second-price auction with the agent’s bid btb_t and the opponents’ bids, returns outcomes such as impressions, conversions dtd_t, and cost ctc_t for this timestep, and updates the next state st+1s_t+1. At the end of the bidding period, the reward model of AuctionNetEnv returns a trajectory reward r based on the agent’s cumulative performance under the CPA constraint C, incorporating the constraint satisfaction objective into the reward signal to encourage a balance between maximizing impression value and satisfying constraints: (9) CPA=∑tct∑tdtℙ(CPA;C)=min(CPA)η,1r=ℙ(CPA;C)⋅∑tdt casesCPA= _tc_t _td_t\\ P(CPA;C)= \ ( CCPA )^η,1 \\\ r=P(CPA;C)· _td_t cases where η is a hyperparameter that controls the penalty intensity for constraint violation. By conducting online agent-environment interaction through AuctionNetEnv, AIGB-R1 mitigates the out-of-distribution extrapolation errors associated with offline value estimation methods, thereby providing more reliable learning signals for strategy optimization. 4.3.2. Experience-Guided Hierarchical Rollout To leverage historical experience for transforming strategy optimization from random trial-and-error into a more effective self-improvement process, we design an experience-guided hierarchical rollout mechanism that enables the LLM Planner to explore optimal strategies based on accumulated experience during post-training. We observe that optimal strategies for different bidding tasks often differ significantly due to variations in budget scale and CPA constraint tightness, necessitating task-specific experience accumulation. Therefore, we maintain an experience buffer ℋgH_g for each bidding task g in AuctionNetEnv, consisting of two components: the best-strategy anchor and the sliding memory window. The best-strategy anchor records the strategy prompt and bidding trajectory summary associated with the highest average reward across all rollouts for task g, serving as a reference direction for subsequent exploration and enabling the Planner to refine its search within the neighborhood of the known best strategy. The sliding memory window retains detailed bidding performance from recent periods in a FIFO manner, including the strategy prompt, bidding trajectory, trajectory reward, and corresponding performance statistics of each bidding period, allowing the Planner to observe performance differences across strategies on task g for targeted adjustments. Credit assignment is a core challenge in end-to-end RL training of hierarchical frameworks like AIGB-R1, as it requires distinguishing the respective contributions of the Planner’s strategy selection and the Executor’s bidding execution to overall performance. We address this challenge through hierarchical rollout. Specifically, for each bidding period of task g, we first use the LLM Planner to reason about potentially optimal strategies by referencing the advertiser’s contextual information gC_g and historical bidding experience ℋgH_g, and output the M most promising and diverse strategy prompts p1,p2,…,pM\p_1,p_2,…,p_M\: (10) pm∼πθ(⋅∣g,ℋg),m=1,2,…,M,p_m _ _P (· _g,H_g ), m=1,2,…,M, Then, for each strategy prompt pmp_m, we invoke the PDT Executor to perform K rollouts in AuctionNetEnv, producing K bidding trajectories (11) τm,k∼πθℰ(⋅∣pm),k=1,2,…,K, _m,k _ _E (· p_m ), k=1,2,…,K, After all rollouts are completed, the strategy prompts and bidding trajectories from this period are appended to the experience buffer ℋgH_g. At this point, for each bidding period of task g, we have collected an M×KM× K nested reward matrix, where the element (m,k)(m,k) corresponds to the k-th rollout trajectory under strategy prompt pmp_m, with reward rm,kr_m,k. The nested structure of this reward matrix encapsulates the dual contributions of the Planner’s strategy selection and the Executor’s bidding execution, providing a natural basis for inter-layer credit assignment via D-GRPO. 4.3.3. Decoupled Group Relative Policy Optimization In recent years, Group Relative Policy Optimization (GRPO) (Shao et al., 2024) has been widely adopted for reinforcement fine-tuning of LLMs. However, directly applying GRPO to the hierarchical architecture of AIGB-R1 faces the aforementioned credit assignment problem, as the reward signal couples the dual contributions of the Planner and the Executor, and directly using GRPO for advantage estimation would lead to misattribution between the two layers, undermining training stability. To this end, we propose Decoupled Group Relative Policy Optimization (D-GRPO), which constructs decoupled advantage signals for the Planner and the Executor respectively, through an orthogonal decomposition of the reward signal, achieving inter-layer credit assignment. GRPO measures the relative superiority of trajectory k within a sampled group based on its reward deviation from the group mean, i.e., rk−1K∑ℓ=1Krℓr_k- 1K _ =1^Kr_ . For the reward matrix produced by hierarchical rollout, we introduce the mean reward r¯m r_m of strategy prompt pmp_m to further decompose the reward deviation of the nested reward matrix along the sampling hierarchy into two orthogonal components, namely the inter-group reward component and the intra-group reward component: (12) r¯m=1K∑krm,kr¯=1MK∑m∑krm,k,rm,k−r¯=(r¯m−r¯)+(rm,k−r¯m) cases r_m= 1K _kr_m,k\\ r= 1MK _m _kr_m,k,\\ r_m,k- r= ( r_m- r )+ (r_m,k- r_m ) cases The inter-group component r¯m−r¯ r_m- r averages over K rollouts for each strategy prompt pmp_m, reducing the randomness of individual Executor executions, so that this signal reflects the quality of the Planner’s strategy selection. The intra-group component rm,k−r¯mr_m,k- r_m subtracts the mean reward of the corresponding strategy prompt from each trajectory’s reward, mitigating the influence of the strategy prompt itself, so that this signal reflects the Executor’s decision-making performance under a given strategy prompt. The orthogonality of this decomposition is formally proved in Appendix A.2. We normalize the inter-group and intra-group reward components along the strategy prompt dimension and the trajectory dimension respectively, yielding the advantage A^m A_m^P for each strategy prompt pmp_m output by the Planner and the advantage A^m,kℰ A_m,k^E for each bidding trajectory τm,k _m,k output by the Executor: (13) A^m=r¯m−r¯σ,A^m,kℰ=rm,k−r¯mσℰ A_m^P= r_m- rσ^P, A_m,k^E= r_m,k- r_mσ^E where σ^P and σℰσ^E denote the standard deviations of the inter-group and intra-group reward components, respectively. At each training step, we jointly optimize the Planner and the Executor by maximizing the following objectives, achieving end-to-end training: (14) (θ)GRPO=(,ℋ)∼,pmm=1M∼πold(p|,ℋ)1M∑m=1M1|pm|∑t=1|pm|min(ρm,tA^m,clip(ρm,t, 1±ϵ)A^m)−βKL[πθ∥πref] splitJ&_GRPO( _P)=E_(C,H) ,\;\p_m\_m=1^M _old^P(p\,|\,C,H) 1M _m=1^M 1|p_m| _t=1^|p_m|\\ & \ ( _m,t^P\, A_m^P,\;clip ( _m,t^P,\,1\!±\!ε )\, A_m^P )- _P\,D_KL [ _ _P\,\|\, _ref^P ] \ split (15) (θℰ)GRPO=pm∼P,τm,kk=1K∼πoldℰ(τ|pm)1K∑k=1K1|τm,k|∑t=1|τm,k|min(ρm,k,tℰA^m,kℰ,clip(ρm,k,tℰ, 1±ϵ)A^m,kℰ)−βℰKL[πθℰ∥πrefℰ] splitJ&_GRPO( _E)=E_p_m P,\;\ _m,k\_k=1^K _old^E(τ\,|\,p_m) 1K _k=1^K 1| _m,k| _t=1^| _m,k|\\ & \ ( _m,k,t^E\, A_m,k^E,\;clip ( _m,k,t^E,\,1\!±\!ε )\, A_m,k^E )- _E\,D_KL [ _ _E\,\|\ _ref^E ] \ split where ρ denotes the importance sampling weight, and ϵε and β are training hyperparameters. The detailed post-training procedure is shown in Appendix A.3. Table 1. Performance comparison. The boldface denotes the best performance. Dataset USCB CQL IQL BCQ DT DT-score DiffBid CBD GAS GAVE PRO-Bid GRAD LBM AIGB-R1 AuctionNet 157 171 281 321 329 334 152 298 359 376 372 372 348 385±3.61 AuctionNet-sparse 17.5 22.2 30.0 31.1 29.6 33.2 19.5 37.0 36.1 37.2 38.1 37.4 33.4 39.0±0.37 5. Experiment 5.1. Experimental Setup 5.1.1. Dataset To comprehensively evaluate the performance of AIGB-R1 on large-scale advertising auctions, we utilize AuctionNet, a large-scale public real-world bidding dataset released by Alibaba, together with its more challenging sparse variant, AuctionNet-Sparse. Both datasets contain approximately 480K bidding trajectories, each consisting of 48 timesteps and millions of impression opportunities. Details are provided in Appendix A.4.1. 5.1.2. Evaluation Metrics We adopt the following metrics to evaluate the performance: • Conversions: The total conversions obtained during the bidding period, calculated as ∑ioivi _io_iv_i. • Score: A metric for jointly evaluating conversions and CPA constraint satisfaction, defined as score=(∑ioivi)×penaltyscore=( _io_iv_i)×penalty, where the penalty term penalty=min(CCreal)2,1penalty= \ ( CC_real )^2,1 \, and CrealC_real and C denote the strategy’s realized CPA and CPA constraint, respectively. 5.1.3. Baselines We compare AIGB-R1 against various state-of-the-art baselines. For RL-based methods, we compare with the online RL method USCB (He et al., 2021) and offline RL methods BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and IQL (Kostrikov et al., 2021). For generative methods, we compare with both diffusion-based and DT-based methods. Diffusion-based baselines include DiffBid (Guo et al., 2024), which generates bidding trajectories via conditional diffusion modeling, and CBD (Li et al., 2025a), which adopts a diffusion completer-aligner framework for large-scale competitive auctions. DT-based baselines include DT (Chen et al., 2021) and DT-score (Xu et al., 2024b), which leverages a reward function to jointly represent the winning value and KPI constraints. We further include GAS (Li et al., 2025b), which performs post-training search via MCTS, GAVE (Gao et al., 2025), which employs value-guided exploration, PRO-Bid (Wu et al., 2026), which introduces Pareto-prioritized regret optimization for constraint-aware bidding, and GRAD (Lei et al., 2026), which develops a large-scale pre-trained DT-based bidding model. For LLM-based methods, we compare with LBM (Li et al., 2026), which designs a hierarchical Think-Act architecture with GQPO-based fine-tuning. 5.1.4. Implementation Details We deploy the baseline models following their standard implementations with default hyperparameters from the original papers. The Planner module of AIGB-R1 uses the Qwen2.5-32B-Instruct model with parameter-efficient fine-tuning based on the ms-swift framework (Zhao et al., 2024b), with a batch size of 128 and a learning rate of 5e-6. The Executor module follows the official code of DT (Chen et al., 2021) and Prompt-DT (Xu et al., 2022) with necessary adaptations. The Executor adopts a causal transformer architecture with 6 attention layers and 8 attention heads, with a hidden size of 512, constituting a lightweight decision model. The Executor is optimized using AdamW with a learning rate of 1e-5 and a batch size of 512. Our training is conducted on NVIDIA A100 GPUs using the PyTorch framework. We implement the D-GRPO algorithm compatible with the PDT architecture following GAM (Dou et al., 2026), with training hyperparameters set to default values. More implementation details are provided in Appendix A.4. 5.2. Performance Comparison We present a comprehensive comparison between AIGB-R1 and various baseline approaches, using Score as the evaluation metric, with results on AuctionNet and AuctionNet-sparse summarized in Table 1. To further assess the generalization ability and robustness of our method, we evaluate AIGB-R1 against state-of-the-art generative baselines under different budget settings. Specifically, we conduct experiments on both datasets across five budget ratios: 50%, 75%, 100%, 125%, 150%, with results reported in Table 2. Our experimental results show that: • DT-based generative bidding methods significantly outperform RL methods such as IQL, CQL, and USCB, demonstrating the effectiveness of generative models in solving complex sequential decision-making tasks like advertising auctions. Notably, advanced generative baselines such as GAS, GAVE, PRO-Bid, and GRAD further outperform DT and its simple variants through Monte Carlo tree search, value-guided exploration, constraint-aware regret optimization, and large-scale pre-training, respectively, indicating the effectiveness of incorporating specialized strategy exploration and optimization mechanisms into generative auto-bidding. • Although the LLM-based method outperforms the standard DT, it still falls significantly behind advanced generative baselines, likely due to the inherent limitations of LLMs as decision models in handling complex numerical decision-making tasks. In contrast, AIGB-R1 leverages LLMs for high-level strategy planning while training a dedicated PDT as the decision model, fully exploiting LLMs’ reasoning and planning capabilities while circumventing this limitation. • AIGB-R1 demonstrates the best performance across all experimental settings, consistently outperforming existing state-of-the-art baselines. These results validate the effectiveness of our hierarchical Planner-Executor architecture and end-to-end self-evolving training paradigm, while demonstrating strong generalization and robustness. Table 2. Generalization to various budget settings. AuctionNet Method 50% 75% 100% 125% 150% DiffBid 54 100 152 193 234 GAS 193 287 359 409 461 GAVE 201 296 376 421 467 PRO-Bid 204 291 372 426 471 GRAD 204 293 372 432 476 AIGB-R1 231 313 385 437 489 AuctionNet-sparse Method 50% 75% 100% 125% 150% DiffBid 9.87 15.4 19.5 25.3 30.8 GAS 18.4 27.5 36.1 40.0 46.5 GAVE 19.6 28.3 37.2 42.7 47.4 PRO-Bid 21.0 28.8 38.1 43.7 49.9 GRAD 20.0 28.5 37.4 43.2 47.5 AIGB-R1 21.1 29.6 39.0 45.1 50.9 5.3. Ablation Study To evaluate the effectiveness of each component in AIGB-R1, we conduct an ablation study by evaluating the following modified variants of AIGB-R1: • AIGB-R1(E) w/o RL: Removes the Planner module and the post-training stage, using only the pre-trained Executor to predict bidding parameters. • AIGB-R1(E): Removes the Planner module and applies standard GRPO to post-train the pre-trained Executor. • AIGB-R1(SP) w/o RL: Removes the post-training stage and uses a frozen LLM as a static Planner to provide strategy prompts for the pre-trained Executor at test time. • AIGB-R1(SP): Uses a frozen static Planner during post-training, i.e., only the Executor’s parameters are updated. Table 3 presents the performance comparison of modified variants, showing that every core component of AIGB-R1 is indispensable. Specifically, AIGB-R1(E) w/o RL performs the worst, reflecting the inherent limitation that purely offline pre-trained generative models are constrained by the quality and coverage of training datasets. AIGB-R1(E) achieves a significant improvement over AIGB-R1(E) w/o RL by applying post-training. This trend is also observed in the comparison between AIGB-R1(SP) and AIGB-R1(SP) w/o RL, confirming the importance of post-training for improving generative models’ performance. Meanwhile, the AIGB-R1(SP) series consistently outperforms the AIGB-R1(E) series, indicating that LLMs’ reasoning and planning capabilities provide effective macro-level strategy priors for generative decision models, validating the rationale for our hierarchical architecture that decouples planning and execution. Furthermore, the significant improvement of AIGB-R1 over AIGB-R1(SP) demonstrates that the static Planner with frozen parameters in AIGB-R1(SP) has limited ability to improve strategies through in-context learning. AIGB-R1 achieves the best performance by using D-GRPO to jointly optimize the Planner and Executor, validating that end-to-end training is indispensable for unleashing the full potential of the hierarchical architecture. Table 3. Ablation Study Model Score Conversions AIGB-R1(E) w/o RL 343 362 AIGB-R1(E) 367 395 AIGB-R1(SP) w/o RL 351 388 AIGB-R1(SP) 371 396 AIGB-R1 385 404 6. Conclusion In this work, we propose a hierarchical self-evolving auto-bidding framework, termed AIGB-R1, which achieves autonomous exploration and continuous evolution of bidding strategies through an experience-driven self-evolving loop powered by LLMs’ reasoning, planning, and reflection capabilities. We design a hierarchical Planner-Executor architecture that employs LLMs for high-level bidding strategy planning and PDT for low-level execution, fully leveraging LLMs’ reasoning and planning capabilities while circumventing their inherent limitations in complex numerical decision-making tasks. We adopt a two-stage training pipeline comprising offline pre-training and post-training alignment, and propose an end-to-end self-evolving post-training paradigm for efficient strategy exploration and continuous self-improvement. We also design a hierarchical rollout mechanism and the corresponding Decoupled Group Relative Policy Optimization (D-GRPO) algorithm, achieving credit assignment through advantage decoupling and ensuring stable end-to-end optimization. Experimental results on a large-scale public dataset demonstrate the effectiveness of AIGB-R1, providing a promising solution for LLM-empowered auto-bidding. References M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. Cited by: §1, §2.2. A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal (2022) Is conditional generative modeling all you need for decision-making?. arXiv preprint arXiv:2211.15657. Cited by: §1. M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello (2024) A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. Cited by: §2.2. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §1. S. Balseiro, Y. Deng, J. Mao, V. Mirrokni, and S. Zuo (2021) Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems 34, p. 17777–17788. Cited by: §1. L. Cai, J. He, Y. Li, J. Liang, Y. Lin, Z. Quan, Y. Zeng, and J. Xu (2025) Rtbagent: a llm-based agent system for real-time bidding. In Companion Proceedings of the ACM on Web Conference 2025, p. 104–113. Cited by: §1, §1, §2.1. L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch (2021) Decision transformer: reinforcement learning via sequence modeling. Advances in neural information processing systems 34, p. 15084–15097. Cited by: §1, §2.1, §5.1.3, §5.1.4. Y. Chen, P. Berkhin, B. Anderson, and N. R. Devanur (2011) Real-time bidding algorithms for performance-based display ad allocation. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, p. 1307–1315. Cited by: §2.1. Y. Deng, J. Mao, V. Mirrokni, and S. Zuo (2021) Towards efficient auctions in an auto-bidding world. In Proceedings of the Web Conference 2021, p. 3965–3973. Cited by: §1. Y. Dou, S. Dou, Y. Ma, B. Wang, T. Wang, Z. Zhang, C. Yu, J. Xu, and Q. Qi (2026) GAM: a generative auto-marketing framework in online e-commerce platforms. In Proceedings of the ACM Web Conference 2026, p. 7979–7988. Cited by: §1, §1, §1, §2.1, §5.1.4. K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) Kto: model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. Cited by: §2.2. D. S. Evans (2009) The online advertising industry: economics, evolution, and privacy. Journal of economic perspectives 23 (3), p. 37–60. Cited by: §1. L. Feng, Z. Xue, T. Liu, and B. An (2026) Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems 38, p. 46375–46408. Cited by: §2.2. A. Forootani (2025) A survey on mathematical reasoning and optimization with large language models. arXiv preprint arXiv:2503.17726. Cited by: §1. S. Fujimoto, D. Meger, and D. Precup (2019) Off-policy deep reinforcement learning without exploration. In International conference on machine learning, p. 2052–2062. Cited by: §1, §2.1, §5.1.3. J. Gao, Y. Li, S. Mao, P. Jiang, N. Jiang, Y. Wang, Q. Cai, F. Pan, P. Jiang, K. Gai, et al. (2025) Generative auto-bidding with value-guided explorations. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 244–254. Cited by: §1, §1, §1, §2.1, §5.1.3. C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. (2023) Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998. Cited by: §2.3. D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. J. Guo, Y. Huo, Z. Zhang, T. Wang, C. Yu, J. Xu, B. Zheng, and Y. Zhang (2024) Generative auto-bidding via conditional diffusion modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 5038–5049. Cited by: §1, §1, §2.1, §5.1.3. L. Ha (2008) Online advertising research in advertising journals: a review. Journal of Current Issues & Research in Advertising 30 (1), p. 31–48. Cited by: §1. J. He, S. Chen, F. Zhang, and Z. Yang (2024) From words to actions: unveiling the theoretical underpinnings of llm-driven autonomous systems. arXiv preprint arXiv:2405.19883. Cited by: §2.2. Y. He, X. Chen, D. Wu, J. Pan, Q. Tan, C. Yu, J. Xu, and X. Zhu (2021) A unified solution to constrained bidding in online display advertising. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, p. 2993–3001. Cited by: §1, §2.1, §3.1, §3.1, §5.1.3. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2025a) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1. Y. Huang, H. Ma, W. Chen, Z. Wang, S. Wang, Y. Zhang, X. Wei, Y. Zhu, H. Wang, and X. Wang (2025b) Generative bid shading in real-time bidding advertising. arXiv preprint arXiv:2508.06550. Cited by: §1. I. Kostrikov, A. Nair, and S. Levine (2021) Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169. Cited by: §2.1, §5.1.3. A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine (2019) Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems 32. Cited by: §1. A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems 33, p. 1179–1191. Cited by: §1, §2.1, §5.1.3. Y. Lei, J. Zhao, Y. Zhao, Z. Zhang, L. Cai, Q. Xie, and X. Wang (2026) Generative large-scale pre-trained models for automated ad bidding optimization. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2266–2277. Cited by: §2.1, §5.1.3. H. Li, Y. Huo, S. Dou, Z. Zheng, Z. Zhang, C. Yu, J. Xu, and F. Wu (2024) Trajectory-wise iterative reinforcement learning framework for auto-bidding. In Proceedings of the ACM Web Conference 2024, p. 4193–4203. Cited by: §1. Y. Li, J. Gao, N. Jiang, S. Mao, R. An, F. Pan, X. Zhao, B. An, Q. Cai, and P. Jiang (2025a) Generative auto-bidding in large-scale competitive auctions via diffusion completer-aligner. arXiv preprint arXiv:2509.03348. Cited by: §2.1, §5.1.3. Y. Li, Z. Lyu, P. Jiang, Q. Cai, F. Pan, B. An, and P. Jiang (2026) LBM: hierarchical large auto-bidding model via reasoning and acting. In Proceedings of the ACM Web Conference 2026, p. 75–86. Cited by: §1, §1, §2.1, §5.1.3. Y. Li, S. Mao, J. Gao, N. Jiang, Y. Xu, Q. Cai, F. Pan, P. Jiang, and B. An (2025b) Gas: generative auto-bidding with post-training search. In Companion Proceedings of the ACM on Web Conference 2025, p. 315–324. Cited by: §1, §1, §1, §2.1, §5.1.3. X. Lv, Z. Zhang, Y. Li, Y. Huo, S. Ju, X. Li, C. Hong, T. Wang, Y. Wang, P. Sun, et al. (2026) DecisionLLM: large language models for long sequence decision exploration. arXiv preprint arXiv:2601.10148. Cited by: §1. Y. Meng, M. Xia, and D. Chen (2024) Simpo: simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37, p. 124198–124235. Cited by: §2.2. Z. Mou, Y. Huo, R. Bai, M. Xie, C. Yu, J. Xu, and B. Zheng (2022) Sustainable online reinforcement learning for auto-bidding. Advances in Neural Information Processing Systems 35, p. 2651–2663. Cited by: §2.1. W. Ou, B. Chen, Y. Yang, X. Dai, W. Liu, W. Zhang, R. Tang, and Y. Yu (2023) Deep landscape forecasting in multi-slot real-time bidding. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 4685–4695. Cited by: §1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2.2. Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, J. Sun, X. Yang, Y. Yang, S. Yao, W. Xu, et al. (2025) Webrl: training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, Vol. 2025, p. 79791–79821. Cited by: §2.3. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2.2. T. Schmied, J. Bornschein, J. Grau-Moya, M. Wulfmeier, and R. Pascanu (2025) Llms are greedy agents: effects of rl fine-tuning on decision-making abilities. arXiv preprint arXiv:2504.16078. Cited by: §2.2. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2.2, §4.3.3. N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, p. 8634–8652. Cited by: §2.2, §2.3. Y. Shukla, W. Gao, V. Sarathy, A. Velasquez, R. Wright, and J. Sinapov (2023) LgTS: dynamic task sampling using llm-generated sub-goals for reinforcement learning agents. arXiv preprint arXiv:2310.09454. Cited by: §2.2. M. Song, Y. Huo, B. Zhou, S. Yin, Z. Xiao, J. Long, Z. Zhang, and C. Yu (2026) DARA: few-shot budget allocation in online advertising via in-context decision making with rl-finetuned llms. In Proceedings of the ACM Web Conference 2026, p. 40–50. Cited by: §1, §1. K. Su, Y. Huo, Z. Zhang, S. Dou, C. Yu, J. Xu, Z. Lu, and B. Zheng (2024) Auctionnet: a novel benchmark for decision-making in large-scale games. Advances in Neural Information Processing Systems 37, p. 94428–94452. Cited by: §A.1.1, §4.3.1. X. Wan, W. Xu, C. Yang, and M. Sun (2025) Think twice, act once: a co-evolution framework of llm and rl for large-scale decision making. arXiv preprint arXiv:2506.02522. Cited by: §1, §2.2. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023a) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Cited by: §2.3. H. Wang, B. Tang, C. H. Liu, S. Mao, J. Zhou, Z. Dai, Y. Sun, Q. Xie, X. Wang, and D. Wang (2023b) Hibid: a cross-channel constrained bidding system with budget allocation by hierarchical offline deep reinforcement learning. IEEE Transactions on Computers 73 (3), p. 815–828. Cited by: §1. J. Wang and S. Yuan (2015) Real-time bidding: a new frontier of computational advertising research. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, p. 415–416. Cited by: §1. J. Wang, W. Zhang, and S. Yuan (2017) Display advertising with real-time bidding (rtb) and behavioural targeting. Foundations and Trends® in Information Retrieval 11 (4–5), p. 297–435. Cited by: §1. Y. Wang, R. Xiao, J. Y. L. Kasahara, R. Yajima, K. Nagatani, A. Yamashita, and H. Asama (2024) Dart-llm: dependency-aware multi-robot task decomposition and execution using large language models. arXiv preprint arXiv:2411.09022. Cited by: §2.2. C. Wen, M. Xu, Z. Zhang, Z. Zheng, Y. Wang, X. Liu, Y. Rong, D. Xie, X. Tan, C. Yu, et al. (2022) A cooperative-competitive multi-agent framework for auto-bidding in online advertising. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, p. 1129–1139. Cited by: §2.1. B. Wu, Y. Zhang, X. Li, R. Deng, C. Yue, W. Zhang, and X. Zeng (2026) Constraint-aware generative auto-bidding via pareto-prioritized regret optimization. arXiv preprint arXiv:2602.08261. Cited by: §2.1, §5.1.3. R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, et al. (2025) Evolver: self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079. Cited by: §2.3. C. Xiao, J. Cai, W. Zhao, B. Lin, G. Zeng, J. Zhou, Z. Zheng, X. Han, Z. Liu, and M. Sun (2025) Densing law of llms. Nature Machine Intelligence, p. 1–11. Cited by: §1. H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim (2024a) Contrastive preference optimization: pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417. Cited by: §2.2. J. Xu, Z. Zhang, Z. Lu, X. Deng, M. P. Wellman, C. Yu, S. Dou, Y. Huo, Z. Xu, Z. Duan, et al. (2024b) Auto-bidding in large-scale auctions: learning decision-making in uncertain and competitive games. In NeurIPS 2024 Competition Track, Cited by: §2.1, §5.1.3. M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan (2022) Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, p. 24631–24645. Cited by: §5.1.4. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §2.2. H. Yu, M. Neely, and X. Wei (2017) Online convex optimization with stochastic constraints. Advances in Neural Information Processing Systems 30. Cited by: §2.1. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2024) Star: self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, Vol. 1126, p. 0–55. Cited by: §2.3. A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024a) Expel: llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 19632–19642. Cited by: §2.3. Y. Zhao, J. Huang, J. Hu, X. Wang, Y. Mao, D. Zhang, Z. Jiang, Z. Wu, B. Ai, A. Wang, W. Zhou, and Y. Chen (2024b) SWIFT:a scalable lightweight infrastructure for fine-tuning. External Links: 2408.05517, Link Cited by: §5.1.4. Q. Zheng, A. Zhang, and A. Grover (2022) Online decision transformer. In international conference on machine learning, p. 27042–27059. Cited by: §1, §2.1. Y. Zhou, S. Levine, J. Weston, X. Li, and S. Sukhbaatar (2026) Self-challenging language model agents. Advances in Neural Information Processing Systems 38, p. 113959–113991. Cited by: §2.3. G. Zhu, J. Luan, H. Du, X. Fang, S. Xu, E. Ni, H. Li, J. Fang, R. Chen, H. Wang, et al. (2026) On the role of language representations in auto-bidding: findings and implications. arXiv preprint arXiv:2605.05833. Cited by: §1, §2.1. Appendix A Appendix A.1. AuctionNetEnv Details A.1.1. Construction Details Based on the public, large-scale auto-bidding decision benchmark AuctionNet (Su et al., 2024), we construct an interactive bidding simulation environment, AuctionNetEnv, to provide flexible agent-environment interaction during the post-training phase. AuctionNetEnv reuses the ad opportunity generation module, the ad auction module, and various auto-bidding opponent agents provided by AuctionNet, aligns them with the distribution characteristics of the training dataset, and encapsulates them into an OpenAI Gym-style interface to be invoked during the rollout process. In AuctionNetEnv, each rollout trajectory corresponds to a complete bidding period of T=48T=48 timesteps. Within each bidding period, the ad opportunity generator produces approximately 500K impression opportunities in total, distributed across the 48 timesteps following the temporal traffic patterns of a real-world advertising platform. The task pool G contains 48 bidding tasks, each associated with a distinct advertiser configuration including budget, CPA constraint, and advertising category. During post-training, an advertiser configuration is sampled from G to serve as the player agent, which is controlled by AIGB-R1 and engages in online bidding against opponent agents that are pre-trained on the training dataset using various algorithms (PID Controller, Online LP, IQL, etc.), thereby simulating the heterogeneous competitive dynamics of real-world advertising platforms. The auction module employs the second-price auction mechanism, with each impression opportunity corresponding to a single ad slot. At each timestep, the player agent submits a bid for each impression opportunity; if the bid exceeds the highest bid among all opponent agents, the player wins the impression and pays the winning cost. For each won impression, the conversion outcome is generated by Bernoulli sampling with the impression value as the success probability. A.1.2. Distributional Consistency of AuctionNetEnv To assess whether AuctionNetEnv preserves the key distributional properties of the historical dataset, we compare the simulated statistics with the corresponding data statistics. We report the mean, standard deviation, and median of representative traffic-side and cost-side metrics, together with the Sim/Data ratio. A ratio closer to 1.0 indicates better alignment with the historical dataset. Table 4. Distributional consistency of AuctionNetEnv against the historical dataset. Ratio denotes Sim/Data, and values closer to 1.0 indicate better alignment with the data distribution. Dataset Metric Mean Std Median Data Sim Ratio Data Sim Ratio Data Sim Ratio AuctionNet pValue 0.00479 0.00480 1.003 0.00377 0.00364 0.966 0.00385 0.00388 1.008 LWC 0.0892 0.0847 0.950 0.0293 0.0243 0.830 0.0810 0.0785 0.968 PV count/tick 10416 10416 1.000 6353 6743 1.061 10573 10483 0.991 AuctionNet -sparse pValue 0.000479 0.000480 1.002 0.000377 0.000364 0.966 0.000385 0.000388 1.008 Marketing 0.0924 0.0883 0.956 0.0273 0.0213 0.780 0.0851 0.0835 0.980 PV count/tick 10416 10416 1.000 6353 6743 1.061 10573 10483 0.991 A.2. Proof of Reward Decomposition Orthogonality Proposition 1. The centered reward deviation of the hierarchical rollout reward matrix can be decomposed into two orthogonal components. Proof. Consider the reward matrix obtained from hierarchical rollout: (16) R=rm,km=1,k=1M,K,R=\r_m,k\_m=1,\,k=1^M,\,K, where m indexes the m-th strategy prompt and k indexes the k-th rollout trajectory under this prompt. Define the mean reward under the m-th strategy prompt as (17) r¯m=1K∑k=1Krm,k, r_m= 1K _k=1^Kr_m,k, and the global mean reward as (18) r¯=1MK∑m=1M∑k=1Krm,k. r= 1MK _m=1^M _k=1^Kr_m,k. Then the centered reward deviation of each element can be decomposed as (19) rm,k−r¯=(r¯m−r¯)⏟Bm,k+(rm,k−r¯m)⏟Wm,k,r_m,k- r= ( r_m- r)_B_m,k+ (r_m,k- r_m)_W_m,k, where B denotes the inter-group component across strategy prompts, and W denotes the intra-group component within the same strategy prompt. In matrix form, (20) R−r¯ 11⊤=B+W.R- r\,11^\! =B+W. We now prove that B and W are orthogonal under the Frobenius inner product. Substituting the definitions of Bm,kB_m,k and Wm,kW_m,k, (21) ⟨B,W⟩F=∑m=1M∑k=1K(r¯m−r¯)(rm,k−r¯m). B,\,W _F= _m=1^M _k=1^K( r_m- r)\,(r_m,k- r_m). For a fixed m, the factor (r¯m−r¯)( r_m- r) is independent of k, so (22) ⟨B,W⟩F=∑m=1M(r¯m−r¯)∑k=1K(rm,k−r¯m). B,\,W _F= _m=1^M( r_m- r) _k=1^K(r_m,k- r_m). By the definition of r¯m r_m, (23) ∑k=1K(rm,k−r¯m)=∑k=1Krm,k−Kr¯m=Kr¯m−Kr¯m=0. _k=1^K(r_m,k- r_m)= _k=1^Kr_m,k-K r_m=K r_m-K r_m=0. Therefore, ⟨B,W⟩F=0 B,\,W _F=0, proving that the inter-group and intra-group components are orthogonal. ∎ A.3. Post-training Procedure Algorithm 1 describes the complete post-training procedure of AIGB-R1. At each training step, we first sample a bidding task and obtain its contextual information and historical bidding experience. We then perform the experience-guided hierarchical rollout. The LLM Planner then generates M diverse strategy prompts based on the context and historical experience, and the PDT Executor performs K rollouts in AuctionNetEnv for each strategy prompt. During advantage estimation, we decouple the reward signal into inter-group and intra-group components to construct separate advantage signals for the Planner and the Executor respectively. Finally, we update the parameters of the Planner and the Executor using their corresponding advantages, and append the rollout results to the experience buffer. 1Input: Planner πθ _ _P, pre-trained Executor πθℰ _ _E, task pool G of AuctionNetEnv, number of strategy prompts M, number of rollouts per prompt K; 2 Initialize experience buffer ℋg←∅H_g← for each task g∈g ; 3 Initialize reference models πref←πθ _ref^P← _ _P, πrefℰ←πθℰ _ref^E← _ _E; 4 for each training step do 5 Sample bidding task g from G, obtain contextual information gC_g and historical experience ℋgH_g; /* Experience-Guided Hierarchical Rollout */ 6 Planner generates M diverse strategy prompts p1,…,pM∼πθ(⋅∣g,ℋg)\p_1,…,p_M\ _ _P(· _g,H_g); 7 for m=1,…,Mm=1,…,M do 8 for k=1,…,Kk=1,…,K do 9 Execute rollout τm,k∼πθℰ(⋅∣pm) _m,k _ _E(· p_m) in AuctionNetEnv; 10 Compute trajectory reward rm,kr_m,k; 11 12 end for 13 14 end for /* Decoupled Advantage Estimation */ 15 r¯m←1K∑k=1Krm,k r_m← 1K _k=1^Kr_m,k, r¯←1MK∑m,krm,k r← 1MK _m,kr_m,k; A^m←r¯m−r¯σ A_m^P← r_m- rσ^P ; // Inter-group A^m,kℰ←rm,k−r¯mσℰ A_m,k^E← r_m,k- r_mσ^E ; // Intra-group /* Parameter Update */ 16 Update θ _P by maximizing GRPO(θ)J_GRPO( _P); 17 Update θℰ _E by maximizing GRPO(θℰ)J_GRPO( _E); /* Update Experience Buffer */ 18 Update ℋgH_g with rollout results; 19 Update best-strategy anchor if maxmr¯m _m r_m exceeds current best; 20 21 end for 22 Output: Optimized Planner πθ _ _P and Executor πθℰ _ _E; Algorithm 1 Post-training Procedure A.4. Implementation Details A.4.1. Dataset Details The AuctionNet benchmark comprises two datasets, AuctionNet and AuctionNet-sparse, where AuctionNet-sparse is a sparse variant of AuctionNet with fewer conversions. Each dataset contains 21 advertising delivery periods, each with approximately 500,000 impression opportunities, divided into 48 time intervals. Detailed parameters are summarized in Table 5. Table 5. Detailed parameters of AuctionNet and AuctionNet-sparse. Parameters AuctionNet AuctionNet-sparse Trajectories 479,376 479,376 Delivery Periods 9,987 9,987 Timesteps in a trajectory 48 48 State dimension 16 16 Action dimension 1 1 Action range [0, 493] [0, 589] Impression’s value range [0, 1] [0, 1] CPA range [6, 12] [60, 130] Total conversion range [0, 1512] [0, 57] A.4.2. Hyperparameter Settings We summarize the hyperparameter settings used in AIGB-R1 in Table 6. Table 6. Detailed hyperparameter settings of AIGB-R1. Module Hyperparameter Value LLM Planner Batch size 128 Number of steps 500 Max completion length 512 Learning rate 5e-6 Epsilon for PPO clipping 0.1 Number of strategy prompts (M) 3 Experience buffer depth 4 Number of rollouts per prompt (K) 8 PDT Executor Batch size 512 Number of steps 250000 Sequence length 20 Learning rate 1e-5 Number of attention layers 6 Number of heads 8 Scale 1000 Hidden size 512 Target return 2 Activation function ReLU Epsilon for PPO clipping 0.1 A.4.3. Prompt Templates To provide a detailed illustration of the prompt design in AIGB-R1, we present the core prompt template used to guide the Planner in generating strategy prompts. System Prompt You are a professional ad bidding expert, responsible for helping advertisers plan optimal bidding strategies. BACKGROUND: The advertiser participates in a one-day auction period consisting of 48 timesteps. At each step, the advertiser determines the optimal bid multiplier α≥0α≥ 0 based on market conditions. The advertiser competes in a second-price auction; the highest bidder wins the impression and pays the second-highest price. The objective is to maximize total conversions while keeping actual CPA within the CPA constraint. OBJECTIVE: Your goal is to maximize the advertiser’s score within the auction period, defined as: score=total_conversion×penaltyscore=total\_conversion×penalty, where penalty=min((cpa_target/cpa_actual)2, 1.0)penalty= ((cpa\_target/cpa\_actual)^2,\;1.0 ). To maximize this objective function, you should fully utilize the budget to acquire impressions while satisfying the CPA constraint. If the CPA constraint is severely violated, consider lowering the bid multiplier. TASK: Based on the target advertiser’s profile and historical bidding performance (if available), prescribe promising macro-level bidding strategies for the next auction period. Your task consists of the following steps: 1. First, thoroughly summarize and reflect on historical bidding performance to infer the potentially optimal strategy adjustment direction. 2. Then, based on your chosen adjustment direction, select the most appropriate enumerated strategy label for each of the following dimensions ⟨ labels⟩ : ⟨ ⟩ – e.g., Risk Awareness: AGGRESSIVE / MODERATE / CONSERVATIVE 3. Finally, briefly summarize historical strategy performance and explain the rationale for your chosen adjustment direction, producing ⟨ reasoning⟩ : ⟨ ⟩ . Provide ⟨ ⟩ distinct strategies ranked by potential value in descending order; they must differ in at least one strategy dimension. NOTES: – Each advertiser has unique budget, CPA, and advertising category, which implicitly reflect strategy preferences. Personalize strategies accordingly. – The BEST-STRATEGY ANCHOR records the best-performing strategy across all historical periods for this advertiser, but may not be the global optimum. – The SLIDING MEMORY WINDOW records the most recent N periods of bidding experience, including strategy labels, strategy reasoning, performance statistics, and rollout trajectories, enabling intuitive comparison of performance across different strategies. – Strategy reasoning should be forward-looking and not exceed 300 characters. User Prompt ADVERTISER PROFILE: Budget: ⟨ ⟩ , CPA: ⟨ ⟩ , Category: ⟨ ⟩ BEST-STRATEGY ANCHOR: Strategy Labels: ⟨ labels⟩ Strategy Reasoning: ⟨ reasoning⟩ Trajectory Summary: ⟨ summary⟩ SLIDING MEMORY WINDOW (recent N periods): Period ⟨ index⟩ : Strategy Labels: ⟨ labels⟩ Strategy Reasoning: ⟨ reasoning⟩ Performance Statistics: ⟨ statistics⟩ Rollout Trajectory: ⟨ trajectory⟩ – Timestep t: State: ⟨ ⟩ , Action: ⟨ ⟩ , Reward: ⟨ ⟩ , Cost: ⟨ ⟩ RESPONSE FORMAT (JSON): Return only a valid JSON array containing exactly ⟨M⟩ M strategy objects, ranked by potential value in descending order: ”strategy_labels”: ⟨ labels⟩ , ”reasoning”: ⟨ reasoning⟩