Paper deep dive
Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation
Yi Chen, Rufeng Cheng, Qiang Xie, Tao Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In industrial recommendation feeds, presenting a static headline for an item often fails to satisfy the diverse, multimodal interests of the user population, particularly suppressing the needs of long-tail audiences. While Large Language Models (LLMs) have been integrated into recommendation for content understanding or ranking, directly optimizing them to output a single best headline typically leads to mode collapse---converging to generic patterns that satisfy average tastes but miss specific latent intents. To bridge this gap, we introduce GESE (Generate to Explore, Select to Exploit), a framework operating at the system's presentation layer that decouples personalization into generative exploration and selective exploitation. First, we treat the LLM as a probabilistic explorer, utilizing Group Sequence Policy Optimization (GSPO) with a hierarchical reward mechanism to generate a candidate set that maximizes the semantic coverage of potential user interests. Subsequently, a lightweight, real-time feedback-aware selector acts as the exploiter, identifying the optimal realization from the candidate pool based on instant contextual signals. Extensive deployment on a commercial platform with over 100 million daily active users demonstrates that GESE significantly outperforms state-of-the-art baselines, achieving a 2.57% lift in CTR and 0.87% in dwell time. These results validate that decoupling diversity-oriented generation from precision-oriented selection offers a robust blueprint for aligning generative AI with dynamic user utility.
Tags
Links
- Source: https://arxiv.org/abs/2609.15094v1
- Canonical: https://arxiv.org/abs/2609.15094v1
Trouble viewing inline? Open PDF directly →
Full Text
43,770 characters extracted from source content.
Expand or collapse full text
Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation Yi Chen Affiliation: Baidu Inc., Beijing, China Email: chenyi32 Rufeng Cheng Email: chengrufeng Qiang Xie* Email: xieqiang Tao Li* Email: litao@baidu.com $ˆ*$Corresponding author Abstract In industrial recommendation feeds, presenting a static headline for an item often fails to satisfy the diverse, multimodal interests of the user population, particularly suppressing the needs of long-tail audiences. While Large Language Models (LLMs) have been integrated into recommendation for content understanding or ranking, directly optimizing them to output a single best headline typically leads to mode collapse—converging to generic patterns that satisfy average tastes but miss specific latent intents. To bridge this gap, we introduce GESE (Generate to Explore, Select to Exploit), a framework operating at the system’s presentation layer that decouples personalization into generative exploration and selective exploitation. First, we treat the LLM as a probabilistic explorer, utilizing Group Sequence Policy Optimization (GSPO) with a hierarchical reward mechanism to generate a candidate set that maximizes the semantic coverage of potential user interests. Subsequently, a lightweight, real-time feedback-aware selector acts as the exploiter, identifying the optimal realization from the candidate pool based on instant contextual signals. Extensive deployment on a commercial platform with over 100 million daily active users demonstrates that GESE significantly outperforms state-of-the-art baselines, achieving a 2.57% lift in CTR and 0.87% in dwell time. These results validate that decoupling diversity-oriented generation from precision-oriented selection offers a robust blueprint for aligning generative AI with dynamic user utility. 1 Introduction Figure 1: Static Point Estimation vs. Distributional Coverage. Traditional methods (gray line) suffer mode collapse on the multimodal interest landscape. In contrast, GESE decouples the task: Exploration (small stars) maximizes semantic coverage across diverse peaks, while Exploitation (gold star) leverages real-time signals to pinpoint the optimal realization. In the architecture of modern industrial recommendation systems, the integration of Large Language Models (LLMs) has predominantly focused on upstream tasks, such as enhancing item representations via embeddings or modeling sequential user behaviors for retrieval. However, a pivotal problem remains underexplored: the Presentation Layer. While a ranking model may successfully retrieve a relevant item, the static presentation—specifically the headline displayed—often fails to resonate with the heterogeneous nature of the user base. A single editorial headline typically cannot satisfy the diverse, multimodal interests of millions of users, particularly suppressing the needs of long-tail audiences. The industry standard frames personalized generation as a precise point-estimation problem, training LLMs to output a single optimal headline for a given user profile. However, this pursuit of precision rests on a fragile assumption: that user intent is unimodal and noise-free. In reality, user interests in open-domain feeds are inherently multimodal and dynamic. As visualized in Figure 1, a user’s latent preference for an item often splits into distinct intent modes (e.g., Factual vs. Emotional). Under the aleatoric uncertainty of sparse industrial data[20, 5], standard alignment methods (like SFT or DPO) function as a static point estimator (represented by the dashed gray line). They tend to regress to the mean of the distribution, leading to mode collapse—generating generic, safe patterns that cut through the semantic space but miss the specific high-utility peaks of individual intents. This results in the Echo Chamber effect, where long-term engagement is degraded by repetitive, average-quality content. Current approaches struggle to resolve this tension because they couple generation and ranking into a single black-box optimization process. Methods based on Supervised Fine-Tuning (SFT) tend to mimic the mode of the training distribution, lacking the incentive to explore the semantic space[7]. Meanwhile, standard Reinforcement Learning from Human Feedback (RLHF) approaches, such as DPO[14], often optimize for a scalar reward that inadvertently encourages homogeneity. By collapsing complex user preferences into a single point-wise score, these methods sacrifice the semantic richness necessary to capture latent user interests[9]. Consequently, end-to-end models have no mechanism to hedge their bets against profile noise. In this paper, we propose to decouple the recommendation problem into two distinct phases: Generation as Exploration and Selection as Exploitation. We introduce GESE (Generate to Explore, Select to Exploit), a framework designed for the fusion stage of recommendation pipelines. Instead of demanding the LLM to perform the final ranking, we position the LLM as a generator of a diverse hypothesis space. We employ Group Sequence Policy Optimization (GSPO)[24] to train the LLM, shifting its objective from minimizing token perplexity to maximizing the distributional coverage of potential user interests (the white stars in Figure 1). Subsequently, a real-time feedback-aware selector acts as the exploiter. Unlike ID-based methods that rely on static history, this lightweight module utilizes instant contextual signals—such as immediate session feedback and real-time user meta-tags—to identify the optimal realization (the gold star) from the diverse pool. This decoupled paradigm offers a theoretically grounded solution to the Personalization Paradox: by generating for the group (diversity) and selecting for the individual (precision), we utilize generative diversity as a mathematical hedge against profile uncertainty. Our contributions are as follows: • Framework Decoupling: We propose the GESE framework, which redefines personalized headline generation by separating the responsibility of semantic exploration (LLM) from preference exploitation (Ranking Model), specifically addressing the robustness issue in the presentation layer of noisy industrial environments. • Reward Function Optimization via GSPO: We introduce a novel alignment strategy utilizing a hierarchical reward mechanism that synthesizes generation diversity, predicted CTR (Click-Through Rate), and content faithfulness. This approach not only prevents mode collapse by ensuring the candidate set covers the diverse interest spectrum of the target audience, but also significantly enhances computational efficiency and reduces inference costs by guiding the model to generate the entire candidate set in a single inference pass. • Industrial Verification: We deploy the proposed method on a commercial recommendation platform with over 100 million DAUs. Extensive offline evaluations and online A/B testing demonstrate that our approach significantly outperforms strong baselines (including standard SFT and DPO), achieving a 2.57% lift in CTR and 0.87% in dwell time, validating that diversity is indeed a prerequisite for superior personalization. 2 Related Work 2.1 LLM Alignment in Recommendation The integration of LLMs into recommender systems has shifted the paradigm from ID-based matching to generative content understanding[23, 4]. Early approaches primarily utilized Supervised Fine-Tuning (SFT) to adapt LLMs for tasks like rating prediction or explanation generation[11]. While SFT effectively instills domain knowledge, it suffers from the fundamental limitation of behavior cloning: the model learns to mimic the average distribution of the training data, often leading to generic, safe, and undistinguished outputs—a phenomenon known as the regression to the mode[11]. To transcend simple imitation, recent works have adopted Reinforcement Learning from Human Feedback (RLHF), specifically Direct Preference Optimization (DPO) and its variants[14, 3]. These methods align LLMs with user preferences by optimizing a contrastive loss between chosen and rejected pairs. However, a critical disconnect remains in industrial applications: standard alignment methods treat user preference as a static, point-wise label. They optimize for the probability of a single best response, assuming a deterministic mapping from user profile to user intent[26]. As highlighted by[16], this point-wise maximization is brittle against profile noise; when the user representation is ambiguous, DPO-trained models tend to overcommit to a specific (often incorrect) interest cluster or collapse into low-diversity patterns to maximize the scalar reward. In contrast, our work posits that alignment in recommendation must be distributional rather than point-wise. By employing Group Sequence Policy Optimization (GSPO)[24], we move beyond optimizing the expected value of a single item to optimizing the coverage of the user’s potential interest spectrum. 2.2 Diverse Generation Diversity in recommendation has traditionally been treated as a post-hoc constraint, enforced via re-ranking algorithms like Maximal Marginal Relevance (MMR) or Determinantal Point Processes (DPP)[22, 1]. These methods operate on a fixed candidate pool retrieved by upstream models. However, in the era of Generative Recommendation, diversity must be intrinsic to the generation process itself. If the LLM collapses to a single mode during generation, no amount of post-hoc re-ranking can recover the lost semantic nuances[21]. Recent efforts in diversity-aware generation have explored entropy-regularized decoding or prompt-based diversification strategies[18, 17]. While these methods increase lexical variety, they often sacrifice semantic faithfulness and relevance, leading to the diversity-quality trade-off[25]. More importantly, existing generative frameworks typically couple the generation and ranking objectives end-to-end. They demand the generator to be simultaneously diverse (to explore) and precise (to exploit), objectives that are often mathematically conflicting within a single policy network[15]. Our GESE framework addresses this by structurally decoupling these responsibilities: the LLM is released from the burden of ranking to focus on exploration via diverse hypothesis generation, while a dedicated real-time feedback model handles the exploitation via context-aware selection. This separation allows us to achieve robustness against profile noise without compromising the precision required for industrial CTR optimization. 3 Proposed Method Figure 2: The GESE Framework: Decoupling Exploration from Exploitation for Robust Personalized Headline Generation. Phase 1 (Offline) trains an LLM to generate a diverse set of candidate headlines via Group Sequence Policy Optimization (GSPO), using SFT data and CTR signals to shape a hierarchical reward that values diversity, faithfulness, and engagement potential. Phase 2 (Online) deploys a lightweight, real-time feedback Selection Model to pick the optimal headline from the candidate pool based on real-time user feedback. A Fair Exposure Strategy ensures that the online model receives sufficient user impression–click data as cold-start training signals, allowing the online selection module to effectively fit user-level click behaviors. Formally, standard personalized generation is a point-wise estimation task: training a model πθ(y|,c) _θ(y|x,c) to predict a single optimal headline y given a user profile x. This approach fails when x serves as a noisy, sparse proxy for the user’s true, multimodal latent intent z∼P(z|)z P(z|x), inevitably causing the model to suffer from mode collapse. To address this, we reformulate the problem as set-wise optimization via our GESE (Generate to Explore, Select to Exploit) framework. We decouple the task into two phases. Generative exploration is optimized for distributional coverage. Its goal is to produce a candidate set =y1,…,yKY=\y_1,…,y_K\ that is highly likely to contain an optimal match for any potential user intent z. This transforms the objective to maximizing the expected utility of the best candidate within the set, where u(y,z)u(y,z) is the utility function: maxθz∼P(z|)[maxyk∈u(yk,z)] _θE_z P(z|x) [ _y_k u(y_k,z) ] (1) Selective exploitation, A separate real-time-aware ranker ϕS_φ, leverages real-time contextual signals to identify and serve the optimal realization y y from the generated pool: y^=argmaxy∈ϕ(y,,c). y= _y S_φ(y,x,c). (2) This framework relieves the LLM of noise-sensitive ranking duties, allowing it to focus on semantic exploration while the ranker ensures precision. 3.1 Generative Exploration To align the Large Language Model (LLM) with the objective of constructing a high-coverage candidate set Y, we employ a three-stage training pipeline. This pipeline progressively transitions the model from adhering to linguistic instructions to optimizing for a complex, multi-objective utility function. Supervised Instruction Tuning (SFT). The foundation of our explorer is a Qwen3-14B backbone. While the pre-trained model possesses general linguistic knowledge, it lacks two critical capabilities required for our industrial pipeline: (1) adhering to strict output formatting (e.g., JSON), and (2) Single-pass Multi-candidate Generation. Existing methods often rely on repeated sampling (calling the LLM K times) to obtain multiple candidates. This approach is not only computationally prohibitive for real-time serving but also suboptimal for diversity, as independent samples tend to cluster around the mode of the distribution. To address this, we construct a dataset SFTD_SFT comprising 30k samples, where the target y∗y^* is a structured list of distinct headlines. We fine-tune the model to generate the entire candidate set =y1,…,yKY=\y_1,…,y_K\ in a single inference pass. The optimization objective is: ℒSFT(θ)=−(c,,)∼SFT∑t=1||logπθ(t|c,,<t)L_SFT(θ)=-E_(c,x,Y) _SFT _t=1^|Y| _θ(Y_t|c,x,Y_<t) (3) This warm start stage ensures the policy πSFT _SFT learns to output diverse options within a single context window, significantly reducing inference costs while establishing the structural basis for subsequent exploration. Latent Interest Calibration via Reward Modeling. To guide the exploration process, we require a robust proxy for user preference. We train a Click-Through Rate (CTR) Reward Model (RM) that predicts the scalar utility of a generated headline. Unlike standard RMs that rely solely on generic semantic embeddings, we design a Hybrid-Feature Reward Architecture. To ensure semantic alignment with our generator, we utilize the Qwen-Embedding model as the backbone encoder Eϕ(⋅)E_φ(·). Specifically, we feed the concatenated context and headline sequence c⊕yc y into the encoder and extract the last token’s hidden state as the dense semantic representation, preserving the full contextual understanding of the Qwen architecture. Simultaneously, we explicitly model display-bias features dispf_disp (e.g., character length, punctuation density) which significantly impact industrial CTR. Let sem∈ℝdh_sem ^d denote the extracted semantic vector and ψ(⋅)ψ(·) be a feature projection network. The predicted reward rϕ(y,c)r_φ(y,c) is formulated as a non-linear fusion of semantic and display signals: rϕ(y,c)=MLPhead(Eϕ(c⊕y)[EOS]⏟sem∥ψ(disp)⏟feat)r_φ(y,c)=MLP_head ( E_φ(c y)[ EOS]_h_sem ψ(f_disp)_h_feat ) (4) where ∥ denotes vector concatenation, and [EOS][ EOS] signifies the extraction of the representation at the end-of-sequence token position. We utilize 8 million pairs of high-confidence interaction logs RM=(yw,yl)D_RM=\(y_w,y_l)\, where ywy_w (chosen) has a significantly higher CTR than yly_l (rejected). The model is optimized via the Bradley-Terry pairwise ranking loss: ℒRM(ϕ)=−(yw,yl)∼RM[logσ(rϕ(yw)−rϕ(yl))]L_RM(φ)=-E_(y_w,y_l) _RM [ σ(r_φ(y_w)-r_φ(y_l)) ] (5) This RM serves as the primary signal for Attractiveness in our hierarchical reward function. Exploration via GSPO. Standard Reinforcement Learning methods like PPO often suffer from high variance and require a separate Value Network, while DPO is limited to pairwise comparisons, restricting the model’s ability to explore the global semantic space. To address the Echo Chamber effect and enforce diversity, we adopt Group Sequence Policy Optimization (GSPO). GSPO operates by sampling a group of G outputs y1,…,yG\y_1,…,y_G\ for each prompt (c,)(c,x) from the current policy πθ _θ. Instead of relying on an absolute scalar reward, GSPO optimizes the policy based on the relative advantage of each candidate within its group. Formally, for each sampled output yiy_i in the group, we compute a total reward R(yi)R(y_i) (synthesizing CTR, diversity, and faithfulness). The advantage AiA_i is derived by normalizing the rewards within the group: Ai=R(yi)−μ(R(yj)j=1G)σ(R(yj)j=1G)+ϵA_i= R(y_i)-μ(\R(y_j)\_j=1^G)σ(\R(y_j)\_j=1^G)+ε (6) where μ and σ are the mean and standard deviation of rewards within the group. The policy πθ _θ is then updated to maximize the following objective: ℒGSPO(θ)=q,yi[1G∑i=1G(ℒiclip−βDKL[πθ∥πref])]L_GSPO(θ)=E_q,\y_i\ [ 1G _i=1^G (L_i^clip-β D_KL[ _θ\| _ref] ) ] (7) where ℒiclip=min(ρiAi,clip(ρi,1−ϵ,1+ϵ)Ai)L_i^clip= ( _iA_i,clip( _i,1-ε,1+ε)A_i ), ρi=πθ(yi|c)πθold(yi|c) _i= _θ(y_i|c) _ _old(y_i|c) is the importance sampling ratio. The rationale for employing GSPO lies in its intrinsic mechanism for diversity promotion. By utilizing group-wise normalization, GSPO creates a local competition environment within the generated batch. For the model to minimize the loss, it cannot simply generate G identical safe headlines, as this would result in a zero standard deviation and undefined advantages. Instead, the objective function mathematically compels the policy to differentiate the candidates, pushing the model to explore the boundaries of the semantic space to find distinct, high-reward strategies. 3.2 Hierarchical Reward Shaping Since our policy πθ _θ generates a candidate set =y1,…,yKY=\y_1,…,y_K\ in a single pass, we define a hierarchical reward R(,c,)R(Y,c,x) that synthesizes individual semantic utility with collective distributional diversity. Semantic Utility. For each headline yky_k, we compute a utility score U(yk)U(y_k) balancing three objectives: 1) Attractiveness rctr_ctr: The predicted CTR from our Reward Model, clipped for stability. 2) Faithfulness rqualr_qual: A hard constraint using a discrimination model ℳqualM_qual, returning 1.01.0 if yky_k is factually consistent and safe, else a penalty ηpenalty _penalty. 3) Novelty rnovr_nov: To discourage simply copying the editorial title yorgy_org, we utilize the Normalized Edit Distance, defined as: rnov=Levenshtein(yk,yorg)/max(|yk|,|yorg|).r_nov=Levenshtein(y_k,y_org)/ (|y_k|,|y_org|). (8) The aggregated utility is a weighted sum: U(yk)=rctr(yk)+λ1rlen(yk)+λ2rqual(yk)+λ3rnov(yk)U(y_k)=r_ctr(y_k)+ _1r_len(y_k)+ _2r_qual(y_k)+ _3r_nov(y_k) (9) Set-wise Diversity. To prevent mode collapse—where the model generates K identical high-CTR captions—we explicitly penalize redundancy. We define set-wise redundancy using the Intersection-over-Union (IoU) of character sets S(yk)S(y_k), the diversity bonus RdivR_div is scaled inversely to this redundancy: Red()=|⋂kS(yk)||⋃kS(yk)|+ϵ.Red(Y)= | _kS(y_k)|| _kS(y_k)|+ε. (10) Final Objective. The final reward RtotalR_total for GSPO optimization is the Z-score normalized synthesis of average precision and set-wise diversity: Rtotal()=1K∑k=1KU(yk)⏟Average Utility+α⋅max(0,(1−Red())⋅β)R_total(Y)= 1K _k=1^KU(y_k)_Average Utility+α· (0,(1-Red(Y))·β ) (11) This objective compels the policy to explore novel, high-CTR angles (Novelty + Driver) while ensuring candidates remain diverse (Expander) and faithful (Guardrail). 3.3 Online Exploitation While the LLM acts as a semantic explorer providing a diverse candidate set Y, identifying the optimal title for a specific user context requires real-time exploitation. This phase faces the classic Explore-Exploit dilemma: we must accurately estimate the CTR of new candidates (Exploration) while maximizing immediate user engagement (Exploitation). We address this via a two-stage mechanism: a Confidence-Aware Bandit for cold-start and a Deep Neural Ranker for mature serving. Cold Start. New generated headlines lack interaction history. To ensure fair exposure and rapid convergence, we propose a Confidence-Aware Thompson Sampling (CATS) strategy. We model the click probability θi _i of each candidate yiy_i (including the original title yorgy_org) as a Beta distribution: θi∼Beta(αi,βi) _i ( _i, _i), where αi=1+clicksi _i=1+clicks_i and βi=1+viewsi−clicksi _i=1+views_i-clicks_i. Standard Thompson Sampling selects yk=argmaxiθ~iy_k= _i θ_i where θ~i∼Beta(αi,βi) θ_i ( _i, _i). However, in industrial settings, unbounded exploration is risky. We impose a Statistical Termination Mechanism to switch from exploration to exploitation dynamically. Let P(wini)=P(θi>θorg|)P(win_i)=P( _i> _org|D) denote the Bayesian confidence that candidate i outperforms the original title. The active candidate set tS_t available for Thompson Sampling is defined by the following Rotation Criteria: i∈t⇔ i _t Ni≤τwarm N_i≤ _warm (12) ∨(P(wini)<δhigh∧Ni≤τmax∧θ¯i>θ¯org) (P(win_i)< _high N_i≤ _max θ_i> θ_org ) ∨(P(lossi)<δlow∧Ni≤τmax∧θ¯i<θ¯org) (P(loss_i)< _low\ N_i≤ _max θ_i< θ_org ) Where NiN_i is the impression count and θ¯i θ_i is the empirical CTR. Here, τwarm _warm represents the minimum sample size required for statistical validity, τmax _max acts as a safety budget to cap the exploration cost, and δhigh/δlow _high/ _low are high/low confidence thresholds for declaring a candidate as a winner or loser, respectively. If a candidate yiy_i satisfies Eq. (12), its display probability is determined by Thompson Sampling; otherwise, the exploration terminates: the candidate is either permanently discarded (if proven suboptimal) or fixed as the winner (if proven superior). This mechanism ensures that high-potential titles accumulate sufficient data while inferior ones are pruned early. Online Selection. Once candidates accumulate sufficient interaction data via the Bandit phase, we transition to a supervised Context-Aware Ranker for fine-grained exploitation. This transition allows us to move beyond simple ID-based statistics and capture high-order feature interactions between the user’s heterogeneous state and the semantic nuances of the generated headlines. Table 1: Feature Set for Online Selection Model Domain Feature Name Description User Profile uid, cuid Unique User/Device Identifiers (Embedding) gender, age Basic Demographic Attributes education User’s Educational Background Level consumption Estimated Purchasing Consumption Level group User Group Segment User Interest user_metags_20 Sequence of last 20 fine-grained interest tags user_metags_top1 Most frequent fine-grained interest tag user_mecate_v2 Sequence of coarse-grained category interests mecates_v2_top1 Top-1 coarse-grained category preference Candidate original_title Semantic Embedding of the editorial title generated_title Semantic Embedding of the LLM hypothesis We construct the input instance by aggregating features from three views: User Profile, Dynamic Interest, and Candidate Semantics (detailed in Table 1). Let ux_u and yx_y denote the sparse feature vectors for user u and candidate headline y, respectively. We first employ an Embedding Layer (⋅)E(·) to project sparse categorical features into dense low-dimensional spaces. Continuous features and semantic embeddings are concatenated directly. The probability of user u clicking on candidate y given context c is modeled as a non-linear interaction function: P(click|u,y,c)=σ(Φθ((u)∥(y)∥ℐ(u,y)))P(click|u,y,c)=σ ( _θ (E(x_u) (x_y) (x_u,x_y) ) ) (13) where ∥ represents vector concatenation, ℐ(⋅)I(·) denotes explicit feature interaction operations (e.g., cross products), Φθ _θ represents a parameterized Deep Neural Network (DNN) that learns high-order non-linear combinations, and σ(⋅)σ(·) is the sigmoid function. Finally, the optimal headline y y is selected by maximizing this predicted probability: y^=argmaxy∈P(click|u,y,c) y= _y P(click|u,y,c) (14) By explicitly modeling features like education, this ranker captures personalized nuances—such as a specific segment preferring curiosity-driven titles over informative ones—that pure Bandit algorithms or simple semantic matching cannot resolve. 4 Experiments and Results 4.1 Experimental Setup To evaluate robustness across heterogeneous scenarios, we curated MSR-50k, a benchmark dataset comprising 50,000 high-quality resources spanning graphics, short videos, and micro-videos across over 40 vertical categories (e.g., Drama, History). For online verification, models were deployed on a commercial platform with over 100 million DAUs. Baselines. We adopt Qwen3-14B [13] as the foundational backbone for our GESE framework. To rigorously evaluate the efficacy of our approach, we benchmark it against two distinct categories of baselines. Firstly, we employ massive-scale, state-of-the-art proprietary and open-weights models, including DeepSeek V3.1[2], Qwen3-235B-A22B[13], and GPT-OSS-120B[10]. Secondly, We include an SFT-Only baseline, fine-tuned exclusively on high-CTR online interaction logs. This represents the standard industrial practice of behavior cloning, serving as a direct reference to quantify the marginal gains achieved by our diversity-aware exploration mechanisms over simple supervised imitation. Evaluation Metrics. To rigorously assess the trade-off between generative exploration and semantic precision, we employ a comprehensive evaluation protocol that scrutinizes the generated candidates from two complementary dimensions. We first quantify the breadth of the semantic space using Distinct-N (N=1,2N=1,2)[6] to measure lexical uniqueness, while scrutinizing intra-list redundancy via Self-BLEU[27] and Pairwise-BLEU[12]; lower scores in these metrics indicate a more heterogeneous candidate set capable of covering diverse latent user interests. Concurrently, to ensure that high diversity does not compromise factual consistency, we evaluate content fidelity using ROUGE-1/2/L[8] for surface-level alignment with editorial ground truth, and employ a robust NLI-based entailment metric [19] to detect and penalize hallucinations, thereby guaranteeing that the generated headlines remain faithful to the source content. 4.2 Online A/B Test We evaluated the framework on a information feed platform serving approximately 100 million Daily Active Users (DAU). Results are summarized in Table 2. The standard SFT-Only baseline underperformed the production baseline (-1.5% CTR), confirming that greedy decoding leads to mode collapse and safe but unengaging headlines. Coupling the SFT generator with a bandit selector (SFT & UCB) reversed the trend (+1.37%), proving that selection is essential for filtering suboptimal candidates. Crucially, replacing the SFT generator with our GSPO explorer (GSPO & UCB) yielded a further jump to +2.28%, validating our core hypothesis that diversity is a prerequisite for effective personalization. Finally, the full GESE framework, which integrates the diversity-aware generator with the real-time feedback deep selection model, achieved the highest performance across all metrics, yielding a 2.57% increase in CTR and a 0.79% increase in effective impressions. Crucially, this gain in click probability is accompanied by a 0.87% increase in User Dwell Time. The simultaneous improvement in CTR and Dwell Time confirms that our hierarchical reward shaping successfully aligns the model with genuine user utility. It indicates that the generated headlines are not merely attractive ”clickbait” but are faithfully aligned with the content, leading to sustained user engagement and satisfaction. Table 2: Online A/B testing results relative to Raw Headline. GESE achieves the best performance across all metrics. Methods CTR Lift Impressions Dwell Time Raw Headline – – – SFT-Only -1.50% N/A N/A SFT & UCB +1.37% +0.18% +0.33% GSPO & UCB +2.28% +0.53% +0.60% GESE (Ours) +2.57% +0.79% +0.87% 4.3 Offline Evaluation Table 3: Offline performance comparison on the MSR-50k dataset. ↓ implies lower is better, while ↑ implies higher is better. Bold denotes the best performance within the Qwen3-14B group, and underline denotes the best overall performance across all parameter scales. Model Diversity Metrics Quality & Fidelity Appeal Pairwise BLEU ↓ Self BLEU ↓ Distinct N-Gram ↑ NLI Score ↑ ROUGE-1 ↑ ROUGE-2 ↑ ROUGE-L ↑ CTR Score ↑ Base: Large Scale LLMs (Closed/Open) GPT-OSS-120B 3.92 7.96 52.89 70.48 34.91 13.27 30.38 10.57 Qwen3-235B 8.52 18.76 55.97 62.31 50.24 26.68 44.77 7.21 DeepSeek-V3.1 4.63 12.80 51.37 82.93 37.65 16.42 33.45 9.88 Base: Qwen3-14B (Ours) SFT-Only 8.98 18.02 49.40 75.66 48.04 26.34 44.09 9.16 GESE (Ours) 4.47 9.93 56.20 80.17 34.82 15.62 30.92 11.38 Table 4: Ablation studies on reward functions and training stages. w/o Quality: Removes the faithfulness constraints. w/o Diversity: Removes the set-wise IoU reward. w/o RL: Corresponds to the SFT-Only baseline. w/o SFT: Applies GSPO directly to the pre-trained base model. Method Diversity Metrics Quality & Fidelity Appeal Pairwise BLEU ↓ Self BLEU ↓ Distinct N-Gram ↑ NLI Score ↑ ROUGE-1 ↑ ROUGE-2 ↑ ROUGE-L ↑ CTR Score ↑ GESE (Ours) 4.47 9.93 56.20 80.17 34.82 15.62 30.92 11.38 Ablation: Hierarchical Reward Functions w/o Quality Constraint 7.35 12.68 53.48 47.32 17.64 7.32 11.65 12.79 w/o Diversity Bonus 9.36 19.73 55.84 79.32 34.93 15.72 31.06 11.29 w/o Both 9.27 18.68 55.32 44.32 16.25 7.17 10.86 12.86 Ablation: Training Components w/o RL (SFT-Only) 8.98 18.02 49.40 75.66 48.04 26.34 44.09 9.16 w/o SFT (Cold RL) 4.51 10.62 37.83 74.39 35.49 14.37 29.85 10.19 Table 3 presents the comparative performance across three critical dimensions: Diversity, Quality, and Appeal. As hypothesized, the SFT-Only baseline exhibits severe mode collapse, evidenced by high Pair-BLEU (8.98) and Self-BLEU (18.02) scores, comparable to the significantly larger Qwen3-235B. This confirms that supervised likelihood maximization tends to converge on repetitive patterns. In contrast, GESE reduces Self-BLEU by nearly 45% (18.02 → 9.93) and achieves the highest Distinct-N-Gram score (56.20) among all models. Notably, our 14B model rivals the diversity metrics of the 120B-scale GPT model, validating that the GSPO mechanism effectively forces the explorer to traverse the semantic latent space rather than memorizing training modes. An intriguing observation is the divergence between ROUGE and NLI scores. SFT-Only achieves high ROUGE scores (e.g., R-L 44.09) but a lower NLI score (75.66). This indicates a ”Parroting Effect,” where the model mechanically copies the original title without understanding. conversely, GESE shows lower ROUGE scores yet maintains a high NLI (80.17). This trade-off is desirable: it demonstrates that GESE generates semantically novel headlines (low lexical overlap with ground truth) that remain factually consistent (high entailment) with the content, successfully avoiding the hallucination trap while providing fresh perspectives. Most critically, GESE achieves the highest predicted CTR Score (11.38), outperforming both the SFT baseline (9.16) and even the 10x larger teacher models. This substantial margin underscores the efficacy of our hierarchical reward shaping, which aligns the generation directly with industrial engagement signals rather than mere linguistic plausibility. 4.4 Ablation Study To disentangle the contribution of each component in the GESE framework, we conducted a series of ablation studies on the MSR-50k dataset. The results are summarized in Table 4. A striking observation is that removing the Quality constraint (w/o Quality) results in the highest CTR Score (12.79), surpassing even our full model. However, this comes at a catastrophic cost: the NLI score plummets to 47.32, and ROUGE-L drops to 11.65. This phenomenon illustrates Reward Hacking: without semantic guardrails, the RL algorithm exploits the CTR model’s biases, generating ”clickbait” or hallucinations that differ wildly from the content but maximize predicted clicks. The Quality Model is thus proven essential not for engagement, but for maintaining the system’s industrial viability and trustworthiness. Ablating the diversity bonus (w/o Diversity) leads to a significant regression in exploration metrics, with Self-BLEU doubling from 9.93 to 19.73. This confirms that even with Group Sequence sampling, the model tends to collapse into a single high-reward mode without explicit set-wise penalties. Interestingly, the CTR score slightly decreases (11.29 vs. 11.38), suggesting that lack of diversity hinders the model from finding the global optimum in the reward landscape. Comparing the component ablations reveals the necessity of the two-stage pipeline. w/o RL (SFT-Only) suffers from low appeal (CTR 9.16) and high redundancy. Conversely, w/o SFT (Cold RL) exhibits degradation in linguistic quality, evidenced by the lowest Distinct-N-Gram (37.83) and ROUGE scores. This indicates that SFT is crucial for establishing the structural grammar of the task, while RL is required to align this capability with user preference. 5 Conclusion In this work, we challenge the conventional point-estimation paradigm in personalized generation, demonstrating that forcing LLMs to converge on a single optimal output under profile uncertainty inevitably leads to mode collapse. We introduce GESE, a decoupled framework that redefines the task as generative exploration via Group Sequence Policy Optimization (GSPO) followed by selective exploitation. By optimizing for the coverage of latent interest modes rather than mere token likelihood, our approach effectively functions as a mathematical hedge against the aleatoric uncertainty inherent in industrial user profiles. Extensive deployment on a commercial platform with 100 million DAUs confirms that this paradigm significantly boosts both engagement (CTR) and user satisfaction (Dwell Time), providing a generalized blueprint for robustly aligning generative AI with dynamic user utility in stochastic environments. References [1] L. Chen, G. Zhang, and H. Zhou (2018) Fast greedy map inference for determinantal point process to improve recommendation diversity. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, Red Hook, NY, USA, p. 5627–5638. Cited by: §2.2. [2] DeepSeek-AI (2025) DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: §4.1. [3] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 12634–12651. External Links: Link Cited by: §2.1. [4] S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang (2023) Recommendation as language processing (rlp): a unified pretrain, personalized prompt & predict paradigm (p5). External Links: 2203.13366, Link Cited by: §2.1. [5] Y. Jiang, Y. Yang, L. Xia, D. Luo, K. Lin, and C. Huang (2025) RecLM: recommendation instruction tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 15443–15459. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. [6] J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan (2016) A diversity-promoting objective function for neural conversation models. External Links: 1510.03055, Link Cited by: §4.1. [7] Z. Li, C. Chen, T. Xu, Z. Qin, J. Xiao, R. Sun, and Z. Luo (2024) Preserving diversity in supervised fine-tuning of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1. [8] C. Lin (2004) ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, p. 74–81. External Links: Link Cited by: §4.1. [9] Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao (2024) InfoRM: mitigating reward hacking in rlhf via information-theoretic reward modeling. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 134387–134429. External Links: Document, Link Cited by: §1. [10] OpenAI (2025) Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925, Link Cited by: §4.1. [11] L. O’Mahony, L. Grinsztajn, H. Schoelkopf, and S. Biderman (2024) Attributing mode collapse in the fine-tuning of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, External Links: Link Cited by: §2.1. [12] K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, USA, p. 311–318. External Links: Link, Document Cited by: §4.1. [13] Qwen (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1. [14] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §1, §2.1. [15] H. Tang, K. Hu, J. P. Zhou, S. Zhong, W. Zheng, X. Si, and K. Ellis (2024) Code repair with llms gives an exploration-exploitation tradeoff. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §2.2. [16] S. Wang, Z. Zheng, Y. Sui, and H. Xiong (2025) Unleashing the power of large language model for denoising recommendation. In Proceedings of the ACM on Web Conference 2025, W ’25, p. 252–263. External Links: Link, Document Cited by: §2.1. [17] R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen (2024) Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §2.2. [18] Z. Ye, X. Li, Q. Li, Q. Ai, Y. Zhou, W. Shen, D. Yan, and Y. Liu (2024) Beyond scalar reward model: learning generative judge from preference data. External Links: 2410.03742, Link Cited by: §2.2. [19] O. Yoran, T. Wolfson, O. Ram, and J. Berant (2024) Making retrieval-augmented language models robust to irrelevant context. External Links: 2310.01558, Link Cited by: §4.1. [20] K. Zhang, Q. Cao, Y. Wu, F. Sun, H. Shen, and X. Cheng (2024) LoRec: combating poisons with large language model for robust sequential recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, New York, NY, USA, p. 1733–1742. External Links: ISBN 9798400704314, Link, Document Cited by: §1. [21] Y. Zhang, W. Jiang, and Z. Yang (2025) MOSLIM:align with diverse preferences in prompts through reward classification. External Links: 2505.20336, Link Cited by: §2.2. [22] Y. Zhao, Y. Wang, Y. Liu, X. Cheng, C. Aggarwal, and T. Derr (2024) Fairness and diversity in recommender systems: a survey. ACM Transactions on Intelligent Systems and Technology 16, p. . External Links: Document Cited by: §2.2. [23] Z. Zhao, W. Fan, J. Li, Y. Liu, X. Mei, Y. Wang, Z. Wen, F. Wang, X. Zhao, J. Tang, and Q. Li (2024) Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering 36 (11), p. 6889–6907. External Links: Document Cited by: §2.1. [24] C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin (2025) Group sequence policy optimization. External Links: 2507.18071, Link Cited by: §1, §2.1. [25] Y. Zhou, M. Keuper, and M. Fritz (2025) Balancing diversity and risk in LLM sampling: how to select your method and parameter for open-ended text generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 26352–26365. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.2. [26] Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y. Qiao (2024) Beyond one-preference-fits-all alignment: multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 10586–10613. External Links: Link, Document Cited by: §2.1. [27] Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu (2018) Texygen: a benchmarking platform for text generation models. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. External Links: Link Cited by: §4.1.