Paper deep dive
LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising
Wen Zhang, Xiao Bai, Hsiang-wei Chao, Mengfan Zhu, Tong Zhao, Mingxuan Ju, Xiaodong Liu, Congfei Zhang, Yunzhi Zhou, Yajun Wang, Jinchao Li, Yu Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging. Conventional offline engagement metrics are limited by sparse, biased feedback, while online evaluation is costly and time-consuming. In this work, we propose a user--product relevance metric for offline evaluation of DPA retrieval systems. The metric first compresses the engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation, then assesses the relevance of a recommended product to each individual intent and aggregates them as a user-level relevance score, yielding a stable and interpretable measure of retrieval quality. To compute this metric at scale, we leverage large language models (LLMs) as semantic relevance judges, capturing nuanced functional and semantic relationships that heuristic methods often miss. We propose an agentic, human-aligned prompting framework that generates multi-category relevance guidelines and distills teacher-model judgments into an efficient student model for scalable evaluation. The resulting metric strongly correlates with offline and online business metrics, enabling faster and more reliable DPA retrieval iteration before A/B testing.
Tags
Links
- Source: https://arxiv.org/abs/2610.14550v1
- Canonical: https://arxiv.org/abs/2610.14550v1
Trouble viewing inline? Open PDF directly â
Full Text
47,779 characters extracted from source content.
Expand or collapse full text
LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising Conference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 07â11, 2026; Rome, ItalyProceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM â26), November 07â11, 2026, Rome, ItalyDOI: 10.1145/3799682.3840117ISBN: 979-8-4007-2539-5/2026/11CCS: Information systems Relevance assessment Wen Zhang Affiliation: Snap Inc., Bellevue, WA, USA email: wzhang7@snapchat.com , Xiao Bai Affiliation: Snap Inc., Palo Alto, CA, USA email: xbai@snapchat.com , Hsiang-wei Chao Affiliation: Snap Inc., Seattle, WA, USA email: hchao@snapchat.com , Mengfan Zhu Affiliation: Snap Inc., Santa Monica, CA, USA email: mzhu2@snapchat.com , Tong Zhao Affiliation: Snap Inc., Bellevue, WA, USA email: tzhao@snapchat.com , Mingxuan Ju Affiliation: Snap Inc., Bellevue, WA, USA email: mju@snapchat.com , Xiaodong Liu Affiliation: Snap Inc., Bellevue, WA, USA email: xliu9@snapchat.com , Congfei Zhang Affiliation: Snap Inc., Bellevue, WA, USA email: czhang3@snapchat.com , Yunzhi Zhou Affiliation: Snap Inc., Palo Alto, CA, USA email: yzhou10@snapchat.com , Yajun Wang Affiliation: Snap Inc., Palo Alto, CA, USA email: ywang30@snapchat.com , Jinchao Li Affiliation: Snap Inc., Bellevue, WA, USA email: jli18@snapchat.com and Yu Zhang Affiliation: Snap Inc., Palo Alto, CA, USA email: yzhang3@snapchat.com Š c Abstract. Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging. Conventional offline engagement metrics are limited by sparse, biased feedback, while online evaluation is costly and time-consuming. In this work, we propose a userâproduct relevance metric for offline evaluation of DPA retrieval systems. The metric first compresses the engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation, then assesses the relevance of a recommended product to each individual intent and aggregates them as a user-level relevance score, yielding a stable and interpretable measure of retrieval quality. To compute this metric at scale, we leverage large language models (LLMs) as semantic relevance judges, capturing nuanced functional and semantic relationships that heuristic methods often miss. We propose an agentic, human-aligned prompting framework that generates multi-category relevance guidelines and distills teacher-model judgments into an efficient student model for scalable evaluation. The resulting metric strongly correlates with offline and online business metrics, enabling faster and more reliable DPA retrieval iteration before A/B testing. Keywords: LLM Judge, DPA, Semantic ID, Agentic, Offline Evaluation, Product Relevance, E-Commerce â c-license: by 1. Introduction Modern E-commerce advertising platforms increasingly rely on high-quality Dynamic Product Ads (DPA) to deliver personalized user experiences at scale. Within these systems, the major task of retrieval stage is efficiently filtering a massive product catalog into a small subset of candidates based on diverse user interests and engagement histories. DPA retrieval systems typically rely on multiple algorithms to retrieve candidate products to ensure diversity. Although the top-line business objective is to maximize user engagement and revenue, the industry lacks a universal and reliable metric to compare the effectiveness of different retrieval algorithms. Common offline evaluation metrics such as the recall of future engagements (Zhang et al., 2025) are limited by selection bias and data sparsity. Unlike in the Search domain, where explicit queries provide a clear benchmark for success, the advertising domains lack a well-defined framework for user-product relevance (Bauer et al., 2024). Consequently, machine learning practitioners often struggle to quantify whether a retrieval model is successfully surfacing relevant products or merely exploiting noisy, short-term behavioral correlations that may not lead to sustainable long-term growth. Figure 1. LLM-based User-Product Relevance Evaluation framework. (1) User Intent Representation: a multimodal encoder plus residual vector quantization assigns each engaged product a hierarchical semantic ID (SID); events sharing (SID,e)(SID,e) collapse into a single weighted intent representative, yielding the compact intent set drd_r. (2) Intent-Product Relevance Judge: an agentic guideline aligned with human reviewers anchors a teacher, PEFT-distilled into a student model. (3) Metric Computation: MicroRel/MacroRel under threshold-k aggregation, validated against offline retrieval-source rankings and online CTR/CVR.Diagram illustrating the stages of the LLM-based offline evaluation pipeline, including multi-agent guideline generation and student model distillation. Defining and measuring relevance is the core obstacle. Prior work either uses explicit user studies that rate usefulness, trust, or satisfaction (Pu et al., 2012; Shani and Gunawardana, 2015), or implicit engagement logs of clicks, likes, shares, and reports (Zhang et al., 2025). User studies are expensive and slow (Shani and Gunawardana, 2015). Engagement is abundant but noisy: clicks are biased by presentation and trust effects (Joachims et al., 2017), and engagement-optimized ranking amplifies attention-grabbing content users do not actually prefer when asked (Milli et al., 2025). We instead use a content-based definition of relevance: an LLM judges each retrieved product against intents inferred from the userâs engagement history, and intent-level scores are aggregated into a user-level metric. This reduces dependency on exposure artifacts, covers users with sparse-feedback, and makes the relevance score interpretable at the attribute level. Recent work uses LLMs as semantic relevance judges for search and recommendation, scoring content alignment directly rather than relying on engagement signals (Sun et al., 2023; Tang et al., 2025; Choi et al., 2025; Rahmani et al., 2025). Sun et al. (Sun et al., 2023) show that prompted LLMs are effective zero-shot re-rankers and can outperform strong supervised baselines; Choi et al. (Choi et al., 2025) report substantial agreement between LLM evaluators and human raters; and large-scale benchmarks find that LLM-generated relevance labels broadly track human judgment at scale (Rahmani et al., 2025). These works target search-style relevance with an explicit query. In contrast, DPA retrieval system do not have search queries and typically require inferring user-level relevance signals from heterogeneous engagement histories. We propose an offline evaluation framework that uses LLM judges to score userâproduct relevance for DPA retrieval. Two operational problems shape the design: the engagement history is heterogeneous and long, and frontier LLMs are too expensive to call on every eventâcandidate pair at production volume. We address both with (i) an SID (Semantic ID (Singh et al., 2024))-grouped, event-stratified compression of engagement history into a small set of weighted intent representatives, and (i) an agentic guideline-discovery loop together with instruction-guided teacher-student finetune for production serving. Our contributions are 3-folds as follows: ⢠Query-free relevance metric. We define a user-level offline evaluation metric for DPA retrieval that aggregates LLM-judged intentâcandidate relevance under a threshold-k aggregation rule, without requiring an explicit query. ⢠A Scalable LLM-judge Pipeline. We propose an efficient LLM judge pipeline with 1) SID-grouped intent decomposition that reduces LLM calls by 30Ă30Ă, and 2) teacher-student distillation from an agentic, human-vetted guideline that replaces the frontier teacher with an efficient student at production cost. ⢠Online validation. The metric ranks six retrieval sources consistently with conversion precision/recall - how well the retrieved candidates aligns with userâs subsequence purchases - and is directionally aligned with online CTR/CVR deltas across four A/B tests; It thus provides a reliable view of retrieval system health and a low-cost signal for guiding offline model iteration. 2. LLM-Based User-Product Relevance Metric The proposed relevance evaluation framework has three blocks (Figure 1): a compact user intent representation that segments engagement history into SID-grouped representatives (§2.1); an intent-product relevance judge built from an agentically-discovered guideline (§2.2.1) and a teacher-student distillation (§2.2.2); and a metric-computation layer (§2.3) that aggregates pairwise judgments into request- and user-level scores. 2.1. User Intent Representation via SID-based Segmentation A user is represented by her engagement history Hu=(pt,et)t=1TH_u=\(p_t,e_t)\_t=1^T, where ptp_t is an engaged product and ete_t is the event type (view, add-to-cart, purchase, etc.). Treating every event as a distinct intent is the most expressive representation but costly in production: a typical 3030-day window holds hundreds of events, most of which are semantically redundant, inflating LLM cost and biasing the metric toward retrieval algorithms that re-surface near-duplicates of frequently-viewed items. SID grouping. To select a compact, signal-rich representative set, we run a lightweight grouping step before the judge. Each product in the catalog is mapped to a hierarchical Semantic ID (SID) via k-means residual vector quantization (RQ-kmeans) over an in-house content-based multimodal product embedding (Deng et al., 2025), using L stages with 512512-entry codebooks to provide sufficient granularity. Concretely, with r0=embedâĄ(p)r_0\!=\!embed(p) and Clââ512ĂdC_l\!â\!R^512Ă d the level-l codebook, (1) SIDâĄ(p)=(c1,âŚ,cL),cl=argâĄminkâĄârlâ1âCl(k)â2,rl=rlâ1âCl(cl), (p)=(c_1,âŚ,c_L),\;\;c_l= _k\|r_l-1-C_l^(k)\|_2,\;\;r_l=r_l-1-C_l^(c_l), so each stage encodes the residual left by the previous one and longer prefixes refine coarser ones. The length of the leading prefix is used as the grouping key to control granularity: a length-11 prefix groups products at roughly category-prototype resolution (e.g., all running shoes), while the full length-L prefix distinguishes near product-type-plus-attribute granularity (e.g., a specific brand-model-color combination). Two items sharing the chosen-length prefix are treated as content-equivalent; The SID lookup is precomputed and cached, so its runtime cost is negligible during LLM judge inference. With L=3L\!=\!3 in production â a balance between cost (fewer LLM calls at shallow L) and granularity (longer prefixes preserve product-level distinctions) â we partition HuH_u by (SIDâĄ(pt),et)(SID(p_t),e_t) and elect one representative igi_g per group â the most recent product, with ties broken by event-type strength (e.g., purchase >> add-to-cart >> others). The resulting weighted intent set is (2) du=(ig,wg):gâGroupsâĄ(Hu),wg=logâĄ(1+|g|)â βâĄ(eg), d_u=\(i_g,w_g):g (H_u)\,\;\;w_g= (1+|g|)¡β(e_g), with βâĄ(purchase)=3β(purchase)\!=\!3, βâĄ(add-to-cart)=2β(add-to-cart)\!=\!2, βâĄ(others)=1β(others)\!=\!1. The logâĄ(1+|g|) (1+|g|) factor lets a heavily-engaged content-equivalence group dominate over a one-off view of an unrelated product without scaling linearly with view count; event-type stratification keeps a low-intent signal from overruling a same-SID high-intent signal. 2.2. Intent-Product Relevance Judge Given the intent set drd_r from §2.1, we need a judge âĄ(i,c)J(i,c) that assigns a relevance label to each (intent representative i, candidate product c). Two issues shape the design. The taxonomy must capture DPA-specific product semantics without becoming so detailed that a frontier LLM applies it inconsistently; and running a frontier LLM as J at production volume is too expensive. We address the first with an agentic guideline-discovery loop followed by a human-alignment pass (§2.2.1), and the second with knowledge distillation into a small student (§2.2.2). 2.2.1. Agentic Guideline Generation We build the evaluation guideline in two phases. Phase A is an agentic loop where an LLM judge iteratively refines its own decision rules from labeled examples; Phase B is a human alignment pass where editors steer the resulting rule set toward DPA business priorities. Phase A: agentic rule discovery. We start from a minimal criteria-only seed prompt â a single instruction, âLabel the candidate as Exact, Similar, Substitute, Complementary, or Irrelevantâ â with no per-class definitions, examples, or attribute hints. A frontier LLM (e.g., Gemini-2.5-Pro) labels a stratified calibration split using this seed prompt, scored by per-class F1 against the human labels. The refinement loop iterates (i) clustering mis-classified pairs by the (predicted, label) confusion and the judgeâs free-form rationale; (i) prompting a separate reasoner to propose up to three new sub-rules targeting the true-label class for the largest cluster; and (i) provisionally appending each sub-rule, re-labeling the calibration set, and accepting iff the paired-bootstrap 95% CI on Î excludes zero with no per-class F1 regression beyond 0.50.5 percentage points. Rejected sub-rules are logged as âpreviously triedâ. A round ends on the first accepted sub-rule or after all proposals are rejected; the loop stops after N rounds or two consecutive empty rounds. We run multiple trials and retain only sub-rules that recur across trials, stabilizing the discovered set against proposer stochasticity. The candidate-rule pool surfaces seven of the twelve editorial-guideline reasons (attribute-level: type, brand, model/style, size, color, material, component-vs-whole); the five not surfaced â demographic gender/age, broad-Complementary/Irrelevant, asymmetric Complementary â are deferred to Phase B. Phase B: human business-alignment. The agentic phase optimizes calibration-set agreement but is silent on downstream business goals. Human reviewers audit the Phase-A output against three criteria: coverage of business-critical mismatches (demographic gender/age, regulated categories, advertiser exclusions), editorial-standard compatibility, and reproducibility on a 200200-pair business-stratified sample. Vetoed rules return to Phase A with an explanatory note. The most consequential intervention in our deployment promoted demographic mismatch from a Substitute-leaning signal to a hard Irrelevant override â a rule the loop repeatedly deprioritized because demographic-mismatch pairs are statistically rare in the calibration set. Phase B is load-bearing: demographic-override rules emerge from editorial review rather than from the agentic loop, because demographic-mismatch pairs are too rare in the calibration set to drive the loopâs statistical gate. Final taxonomy and rationale format. The resulting guideline â the seed taxonomy refined by Phase-A sub-rules under the Phase-B business overrides â is what the rest of the paper refers to as âthe guidelineâ. It instructs the LLM judge to assign each (intent, candidate) pair to one of five tiers and to emit a structured rationale by selecting from a controlled attribute vocabulary (Product Type, Brand, Demographics, Component-vs-Whole, etc.). The final taxonomy summarization: ⢠Exact: matches all specific attributes of the intent (type, brand, model, size, color). ⢠Similar: same product category, differs in brand, model, or color. ⢠Substitute: different type but functionally interchangeable. ⢠Complementary: different type, typically used together with the intent (Jeon et al., 2025). ⢠Irrelevant: no functional or categorical link, or a critical demographic mismatch. 2.2.2. Teacher-Student Knowledge Distillation The finalized guideline from §2.2.1 could in principle be deployed as an instruction prompt to a frontier LLM at serving time, but at production volume this is prohibitively expensive (Table 5). We instead use the guideline-prompted frontier LLM as a teacher that emits high-quality labels and rationales on a curated training corpus, then fine-tune a much smaller student LLM on those labels. The student is what we deploy as J. Distillation dataset construction. We construct a large distillation corpus that reflects real-world retrieval scenarios via a four-stage pipeline. Stratified seed sampling: intent representatives igi_g are sampled stratified across user engagements and demographics (age, gender, engagement type, engagement frequency), ensuring broader and more balanced pattern coverage than a uniform random distribution. Multi-source candidate generation: for each seed, candidates are aggregated from three complementary streams â production retrieval logs, historical conversion-driven engagement signals, and curated high-value advertiser groups. Hard-case mining: we downsample to N=10N\!=\!10 candidates per seed using a multimodal embedding filter to intentionally surface semantically-similar-but-irrelevant negatives. Labeling: pairs are labeled by the teacher protocol above; rationale traces are retained as auxiliary supervision for the student. Teacher labeling. The finalized guideline is loaded as the instruction prompt to a frontier LLM (Gemini-2.5-Pro), which annotates each seedâcandidate pair in the distillation datasets with a relation label and an attribute-level rationale. GPT-5 runs in parallel under the same guideline as a verification agent; pairs on which the two agents agree at high self-reported confidence are accepted directly, and the disagreed cases are sent to human auditors for relabeling. Student fine-tuning. We test both Gemini-2.5-Flash and Gemini-2.5-Flash-Lite as student backbones and apply parameter-efficient instruction fine-tuning (LoRA with adapter rank 1616, learning-rate multiplier 22, 55 epochs). The student is trained for multi-label classification over the five-tier taxonomy and jointly emits an attribute-level rationale conditioned on the label, matching the teacherâs output format. At serving time, the student replaces both teacher LLMs in the inference path; the teacher is invoked only on guideline-update or distillation-refresh cycles. 2.3. Metric Computation Given the intent set dud_u from §2.1 and the pairwise judge J from §2.2, we compute a per-user relevance score sâĄ(u)s(u) and then aggregate across users into a single retrieval-quality metric. The aggregation layer is decoupled from the upstream blocks so the same judgments can be re-aggregated under different rules without re-invoking the LLM. Algorithm 1 summarizes the procedure. Algorithm 1 Offline UserâProduct Relevance Metric Input: A group of users G (e.g., users routed to a specific retrieval source); per-user engagement history Hu=(pt,et)H_u=\(p_t,e_t)\ and candidate set CuC_u for each uâGuâ G; SID truncation L; event-type weights βâĄ(â )β(¡); threshold k; judge âĄ(i,c)J(i,c). Output: Per-user scores sâĄ(u)uâG\s(u)\_uâ G; group-level summaries MicroRelâ(G)MicroRel(G), MacroRelâ(G)MacroRel(G). // Step 1: per-user intent set + relevance score foreach uâGuâ G do uâPartitionâ(Hu,key=(SIDLâ(pt),et))P_uâ Partition (H_u,\ key=(SID_L(p_t),\,e_t) ) ; duâ(ig,wg):gâud_uâ \\,(i_g,\,w_g)\,:\,g _u\, \, where ig=argâĄmax(p,e)âgâĄrecencyâĄ(p)i_g\!=\! _(p,e)â grecency(p), wg=logâĄ(1+|g|)â βâĄ(eg)w_g\!=\! (1+|g|)\!¡\!β(e_g) ; foreach câCucâ C_u do muâ(c)ââ(ig,wg)âduwgâ âĄ(ig,c)m_u(c)â _(i_g,w_g)â d_uw_g¡J(i_g,c) ; // weighted intent-coverage vu(c)â[mu(c)âĽk]v_u(c) [\,m_u(c)⼠k\,] ; // candidate-level binary relevance sâĄ(u)â1|Cu|ââcâCuvuâ(c)s(u)â 1|C_u| _câ C_uv_u(c) ; // per-user score // Step 2: group-level aggregation MicroRelâ(G)ââuâGâcâCuvuâ(c)âuâG|Cu|MicroRel(G)â _uâ G _câ C_uv_u(c) _uâ G|C_u| ; // (u,c)(u,c)-equal weighting MacroRelâ(G)â1|G|ââuâGsâĄ(u)MacroRel(G)â 1|G| _uâ Gs(u) ; // user-equal weighting return sâĄ(u)uâG,MicroRelâ(G),MacroRelâ(G)\s(u)\_uâ G,\,MicroRel(G),MacroRel(G); Per-user score. For each user u and candidate câCucâ C_u, the judge J emits a categorical label that we map to a per-pair score (binary 0,1\0,1\ by default; alternatively a 5-grade gain gâ0,1,2,3,4gâ\0,1,2,3,4\ following (Reddy et al., 2022)). The weighted intent-coverage muâ(c)=â(ig,wg)âduwgâ âĄ(ig,c)m_u(c)= _(i_g,w_g)â d_uw_g¡J(i_g,c) aggregates the per-pair signal across the intent representatives, and the threshold k converts it to a binary candidate-level relevance vu(c)=[mu(c)âĽk]v_u(c)=I[m_u(c)⼠k]. The per-user score sâĄ(u)=1|Cu|ââcâCuvuâ(c)s(u)= 1|C_u| _câ C_uv_u(c) is the primary output of the metric layer. User vs. group aggregation. The score sâĄ(u)s(u) can be reported per user, or aggregated to any group G of users (retrieval source, demographic bucket, etc.). We use two complementary summaries: MicroRel(G)=âuâGâcvuâ(c)/âuâG|Cu|(G)= _uâ G _cv_u(c)\,/\, _uâ G|C_u| weights each (u,c)(u,c) pair equally; MacroRel(G)=1|G|ââuâGsâĄ(u)(G)= 1|G| _uâ Gs(u) weights each user equally and is therefore less sensitive to skewed traffic. In the rest of the paper we group by retrieval source; the same machinery applies to any other user grouping. Threshold k as a strictness knob. We operate at k=7k\!=\!7 in production (justified in §4.4); the simpler k=1k\!=\!1 at-least-one rule is included as a permissive baseline since it equates a single weak intent match with broad intent coverage. 3. Online-Offline Retrieval Metrics Alignment In this section, we benchmark the proposed LLM-based user-product relevance metric against conventional DPA retrieval offline metrics (e.g., Conversion Precision/Recall (Zhang et al., 2025)) and evaluate their consistency across various retrieval sources. A key advantage of the LLM-based approach is its independence from online traffic data. While conventional metrics often rely on future engagement signals to accurately measure precision/recall, our metric operates strictly offline, providing a high-fidelity relevance signal without the need for real-time interaction logs. 3.1. Online Conversion Precision/Recall Utilizing online traffic, we employ a look-back evaluation strategy that links observed conversions to their contributing ad requests (Zhang et al., 2025). We define relevance using a strict exact-match criterion, where a retrieved product must be identical to the converted item within a one-day post-impression attribution window. This focus assesses the systemâs ability to surface the precise product driving a transaction. Accordingly, Conversion Precision (CP) is the proportion of retrieved products that match the conversion, while Conversion Recall (CR) measures the fraction of ultimately converted products successfully surfaced by the system. Metrics are limited to users with both an eligible ad request and a subsequent confirmed conversion. 3.2. Retrieval Source Baselines A production DPA retrieval system blends several heterogeneous sources, each with distinct algorithmic and signal-coverage profiles, so a single offline metric that ranks them consistently must generalize across signal types. To make our evaluation reproducible and to expose where each source is expected to win or lose, we describe the six representative retrieval sources used throughout this study. Matched Features (MF) is a deterministic retargeting source: candidates are products whose canonical IDs appear in the userâs recent engagement log, optionally expanded by the sellerâs product-graph siblings. Coverage is narrow but precision is high by construction. Similar User (SU) is a two-tower collaborative model trained on (u,p)(u,p) engagement pairs with sampled softmax; at serving time the user towerâs vector is used to retrieve the top items from an ANN index over the global active catalog. Content-Based Retrieval (CBR) encodes each catalog item with an in-house multimodal encoder (image + title + structured attributes) and serves products that are nearest neighbors of the userâs recently engaged items in this content embedding space. Engagement-Based Retrieval (EBR) is a sequence-aware retrieval tower fine-tuned on impression-attributed engagement; it shares the same ANN backend as CBR but its encoder is trained on behavioral co-occurrence rather than content similarity, so it captures complementary patterns such as cross-category purchase pairs. Popularity-Based Retrieval (PBR) ranks the global catalog by a smoothed conversion rate over the trailing NNd, segmented by coarse demographic buckets; it has no per-user signal beyond the segment. The Random Baseline (Rand) samples products uniformly from a coarse category bucket aligned with the userâs last engaged category. All sources retrieve a top-K=15K=15 candidate set per request; the same downstream ranker and budget policy is applied so observed differences are attributable to the retrieval stage. Table 1. Retrieval source performance, reported as log-rel against the Random Baseline, i.e., log10âĄ(method/Rand) _10(method/Rand) measures relative improvement (higher is better). And () indicates the ranking across all retrieval sources. Src CP CR NDCG@15 MAP@15 MicroRel MacroRel ImpRel MF +3.13 (1) +0.91 (1) +0.50 (1) +0.29 (1) +1.21 (1) +1.02 (1) +0.60 (1) SU +2.04 (2) +0.37 (2) +0.16 (4) +0.19 (2) +0.75 (2) +0.30 (2) +0.39 (2) PBR +0.45 (3) +0.59 (3) -0.03 (6) +0.03 (4) +0.12 (5) -0.08 (6) +0.18 (4) CBR +0.15 (4) +0.30 (4) +0.35 (2) +0.11 (3) +0.32 (3) +0.07 (3) +0.23 (3) EBR +0.15 (5) +0.29 (5) +0.28 (3) +0.01 (5) +0.24 (4) +0.05 (4) +0.12 (5) Rand 0.00 (6) 0.00 (6) 0.00 (5) 0.00 (6) 0.00 (6) 0.00 (5) 0.00 (6) 3.3. Alignment Between User-Product Relevance and Conversion-Based Metrics Table 1 reports the LLM-based offline metrics alongside conversion precision/recall (CP, CR) over a 3030-day production window. NDCG@15 and MAP@15 are rank-aware retrieval metrics computed on the same LLM-judge labels, while the remaining columns follow the convention in §2.3. The two sets of rankings agree closely, indicating that the LLM judge tracks conversion-based outcomes. We also report IâmâpâRâeâlImpRel, a metric derived from impressions selected by the downstream click prediction model from the initial retrieval pool, which captures end-to-end performance. The rankings mostly agree: CP and MicroRel produce the same ordering except for Popularity-Based Retrieval, which carries global engagement trends rather than semantic content correlation. Pearson correlations between MicroRel and CP/CR/CVR under the k=7k\!=\!7 aggregation default are reported in Figure 2. Across the six retrieval sources, MicroRel achieves the highest cross-source Pearson with CP (r=0.97r\!=\!0.97), with ImpRel (r=0.95r\!=\!0.95) and MacroRel (r=0.93r\!=\!0.93) close behind. Standard rank-aware retrieval metrics computed on the same LLM-judge labels are weaker proxies for the production signal: MAP@15 reaches r=0.94r\!=\!0.94 and NDCG@15 only r=0.55r\!=\!0.55. Intent-aware aggregation therefore tracks online conversion better than off-the-shelf rank-aware metrics. Table 2. Offline vs. online metrics for two retrieval families (content-based, engagement-based) across four online experiments (two for each family). log-rel reports each version against its controls, so positive values across both rows reflect successive improvements. All four online CTR/CVR deltas are statistically significant under our internal A/B framework within 14 days of windows with 10% of DPA traffics. Retrieval Source Offline LLM Judge Online A/B Metrics MicroRel MacroRel CTR CVR Content-based Retrieval Model 1 +0.138 +0.122 +0.149% +1.472% Content-based Retrieval Model 2 +0.176 +0.169 +0.211% +4.98% Engagement-based Retrieval Model 1 +0.068 +0.019 +0.350% +2.604% Engagement-based Retrieval Model 2 +0.062 +0.095 +0.210% +2.194% 3.4. Alignment Between User-Product Relevance and Online Metrics A natural follow-up: do offline gains in the relevance metric translate into online wins? Table 2 shows directional agreement between offline relevance improvements and online CTR/CVR gains, so the offline metric is usable for guiding DPA retrieval model development before paying for an A/B test. In this experiment, we analyze two primary retrieval categories within our DPA system: content-based and engagement-based retrieval. We evaluate the LLM-based relevance scores across several iterations of these models to assess performance improvements. We then further validate them through online A/B experiments in the DPA online system. In Table 2, Model 1 and Model 2 are two successive iterations of each retrieval family; each row reports its version against its prior baseline (Model 1 vs. the preceding production model; Model 2 vs. Model 1) using log-rel=log10âĄ(NewVersionRelevance/BaseVersionRelevance)log-rel= _10(NewVersionRelevance/BaseVersionRelevance), so positive values consistently indicate improvement. Interestingly, engagement-based retrieval exhibits relatively weak correlations, likely because the enhanced capability reflects a blend of engagement signals and content features. What changed across iterations. The two retrieval families were iterated along orthogonal axes; within each family, the same axes apply to both Model 1 (over the preceding production model) and Model 2 (over Model 1). Each content-based iteration enlarges the training corpus to broaden catalog coverage and refines the seed-product selection strategy used to construct (seed, candidate) training pairs, together improving content alignment for under-represented product types. Each engagement-based iteration strengthens alignment with the downstream product ranker by jointly tuning their objectives and incorporates more engagement signals directly correlated with CVR (e.g., upweighting conversion-attributed events). Because the engagement-based improvement mixes ranker-alignment with the underlying engagement signal, its offline-online delta is smaller and noisier, which explains the weaker correlation in Table 2. Table 3. Intent-set construction ablation at k=7k\!=\!7. Column definitions: see §4.1. Intent-set Variant Cost Spear. vs. Full P(CVR) P(CR) P(CP) Full sequence (baseline) 100%100\% 1.0001.000 0.240.24 0.190.19 0.320.32 Random subsample (25%25\% L_seq) 25%25\% 0.9050.905 0.260.26 0.230.23 0.190.19 Last 1-day window 11%11\% 0.5280.528 0.210.21 0.250.25 0.310.31 SID only, no event split 3.4%3.4\% 0.9900.990 0.300.30 0.170.17 0.240.24 SID + event split, uniform 3.4%3.4\% 0.9910.991 0.290.29 0.180.18 0.260.26 SID + event split, weighted 3.4%3.4\% 0.9890.989 0.500.50 0.290.29 0.350.35 4. Ablation Study We construct a 10K dataset from retrieval logs, audited through multiple rounds of human review to reach consensus. To ensure reliable labels, we adopt a multi-agent framework in which GPT-5 and Gemini-2.5-Pro independently generate judgments that are aggregated via confidence scores; human annotators resolve disagreements until convergence, and the final judgments are treated as ground truth. Results of the main ablation are shown in Table 4. In addition, we ablate two design choices that are central to the framework but whose empirical justification would have interrupted the methodology in §2.1 and §2.2.1: the SID-based intent-set construction (§4.1), and the two-phase agentic-then-human guideline-generation procedure (§4.2). 4.1. Intent-set construction We ablate the intent-set decomposition along three axes â (i) random vs. recency-based vs. SID-based selection, (i) presence or absence of event-type stratification, and (i) uniform vs. size-and-event-weighted aggregation â to isolate the contribution of each design choice rather than of subsampling alone. To put all variants on the same label matrix, we exhaustively judge every eventâcandidate pair in the 3030-day window with the teacher LLM, then for each variant compute its offline MicroRel and report the Pearson correlation against per-(date, source) live CVR, CR, and CP. In Table 3, Cost is pairwise calls as a percentage of the raw 3030-day history (several hundred events per user after product-id deduplication), placing all variants on a common denominator; Spear. vs. Full is the per-user MicroRel rank correlation against the full sequence; and PâĄ(â )P(¡) is the Pearson between offline MicroRel and the named online metric. Random baselines average over multiple seeds. Table 4. Impact of Guidelines and Student Models Role Model Config Precision Recall F1 Teacher Gemini-2.5-Pro (no guidel.) 0.824 0.825 0.816 Gemini-2.5-Pro (with guidel.) 0.861 0.866 0.860 Student SigLIP-2-m (Tschannen et al., 2025) 0.780 0.751 0.746 Gemini-2.5-Flash (with guidel.) 0.840 0.843 0.832 Gemini-2.5-Flash-ft 0.869 0.873 0.869 Gemini-2.5-Flash-lite-ft 0.845 0.849 0.846 Two findings stand out. First, recency truncation halves intent-set fidelity (Last 1-day Spearman 0.530.53 vs. Fullâs 1.001.00) and yet barely moves online Pearson â a Last 1-day intent set tracks CP/CR/CVR within Âą0.03Âą 0.03â0.060.06 of the Full sequence. The signal that drives online business metrics is concentrated in a small, recent slice of engagement, even though the intent-set structure looks very different. This is what makes aggressive compression viable in the first place. Second, SID grouping with event-type stratification keeps that compression while also restoring the intent-set fidelity: it matches the full sequence at 0.990.99 Spearman at only 3.4%3.4\% of its pairwise cost â a âź 30Ă30Ă reduction in LLM calls â and the explicit wg=logâĄ(1+|g|)â βâĄ(eg)w_g= (1+|g|)¡β(e_g) weighting lifts the variant to P(CVR)= 0.50\,=\,0.50, P(CR)= 0.29\,=\,0.29, P(CP)= 0.35\,=\,0.35, beating Full and every other variant on all three online metrics. The margin over the Random subsample baseline (which itself costs 7Ă7Ă more) is +0.24+0.24 on CVR and +0.16+0.16 on CP. We attribute this to a regularization effect: by deduplicating near-identical events and explicitly downweighting low-intent views, the metric stops rewarding retrieval algorithms that re-surface repeated views of the same item â a failure mode the full-sequence MicroRel cannot distinguish from genuine relevance. 4.2. Impact of Guidelines Using Gemini-2.5-Pro as the backbone, a judge given only brief relevance definitions (without the guideline) is 4% worse on both precision and recall than the guideline-prompted version (Table 4) and even underperforms the smaller student model with guideline, showing that the guideline carries most of the lift. The gain concentrates on rule classes that the loose prompt systematically misjudges, e.g., demographic mismatches and brand-vs.-category confusions, which the agentic discovery loop in §2.2.1 surfaces. 4.3. Choice of Student Judge We adopt Gemini-2.5-Flash as the student backbone. Instruction fine-tuning improves precision by 2.9% and recall by 3.0%, outperforming detailed prompting alone and fine-tuned Gemini-2.5-Flash-Lite. The fine-tuned Flash student in fact matches the Pro teacher on F1F_1 (0.8690.869 vs. 0.8600.860) while running at a fraction of the cost (Table 5), so we deploy it as the production judge J. It surpasses SigLIP2-Mâwhich uses 10Ă more training data and a two-layer classifier over fused intentâproduct embeddingsâby 9% in precision and 12% in recall. The lightweight classifier struggles with complex relevance patterns learning from limited training data, highlighting the advantage of LLM-based judges with few-shot learning. Figure 2. MicroRelâs Pearson correlation with live CP, CR, and CVR as a function of the intent-aggregation threshold k. Stars mark per-metric peaks; all three peaks fall in kâ[5,7]k\!â\![5,7]. At the k=7k\!=\!7 default, every CI is strictly positive. 4.4. Intent-Aggregation Threshold k We compare the production default k=7k\!=\!7 against the at-least-one permissive baseline (k=1k\!=\!1), which equates a single weak intent match with broad intent coverage. We sweep kâ[1,15]kâ[1,15] on the same exhaustive eventâcandidate judgment matrix used in §4.1 so the only varying quantity is the aggregation rule. Figure 2 reports the Pearson correlation between the resulting MicroRel and CP, CR, and CVR measured per-(date, source) over a 3030-day production window across 22 retrieval sources. All three correlations reach their maxima within kâ[5,7]k\!â\![5,7]: P(CP) peaks at 0.430.43 (k=5k\!=\!5), and P(CR) and P(CVR) both peak at k=7k\!=\!7. Beyond kâĽ8k\!âĽ\!8, P(CP) and P(CR) decay as the candidate pass-rate drops below âź 5%5\%, while CVR retains most of its signal through k=13k\!=\!13. At the k=7k\!=\!7, all three 95%95\% bootstrap CIs are strictly positive: P(CP) [+0.04,+0.63][+0.04,+0.63], P(CR) [+0.02,+0.55][+0.02,+0.55], P(CVR) [+0.29,+0.68][+0.29,+0.68]. The intuition is that high-k aggregation filters out single weak intent matches that are merely category-coincidental, requiring multi-intent support before a candidate counts as relevant, which aligns better with multi-touch purchase behavior. 5. System Efficiency We benchmark four judge configurations on Vertex AI at 16-thread concurrency over 100100 production retrieval pairs (Table 5). The fine-tuned Gemini-2.5-Flash we deploy reaches 751751 ms p50, 1,1171,117 ms p95, and 1818 QPS â roughly 9Ă9Ă faster than the teacher on p50 latency and on throughput. The fine-tuned Gemini-2.5-Flash-Lite variant offers a marginal further gain at 647647 ms p50 and 2020 QPS. Most of the latency improvement therefore comes from fine-tuning rather than from picking a smaller backbone: prompt-only Flash still inherits the teacherâs long-tail p95 (8,8078,807 ms), whereas the fine-tuned student compresses both p50 and p95 by an order of magnitude. 6. Related Work Offline evaluation for recommendation and ads retrieval. Classical offline metrics for recommendation retrieval â Recall@K, NDCG@K over future-engagement labels, and conversion-attributed precision/recall (Zhang et al., 2025) â have well-known weaknesses in ads: severe selection bias from the exposure policy, heavy tail of low-engagement requests, and a lookback window that conflates retrieval quality with downstream ranker and budget effects. The offline-online evaluation gap in e-commerce recommendation has been documented at multiple scales (Peska and Vojtas, 2020). Counterfactual estimators (IPS, doubly robust) reduce some of this bias (Joachims et al., 2017) but assume access to logged propensities that are rarely available at the retrieval stage of a multi-tower ad system. Our work is complementary: rather than debiasing engagement labels, we replace them with a content-grounded relevance label produced by an LLM judge, and report alignment with engagement-based offline metrics. LLM judges for relevance. LLM-as-judge has moved from open-domain summarization (Sun et al., 2023) to e-commerce and recommender relevance labeling: LREF (Tang et al., 2025) fine-tunes an LLM as a query-product relevance scorer with chain-of-thought, BloomIntent (Choi et al., 2025) performs intent-conditioned evaluation of SERP satisfaction, Wang et al. (Wang et al., 2025) progressively train an LLM-based semantic-ads retrieval encoder at Walmart, Penha et al. (Penha et al., 2025) examine LLM-judge ranking agreement with humans in Cranfield-style movie and podcast recommendation, and large-scale studies show LLM labels broadly track human judgment in IR settings (Rahmani et al., 2025; Upadhyay et al., 2024). Two gaps remain. First, these works rely on either an explicit query (search) or a Cranfield-style topic per request (movie/podcast collections); DPA retrieval has neither â intent must be inferred from heterogeneous engagement signals, so we propose SID-grouped, event-stratified intent decomposition as the compression rule over the engagement history. Second, prior LLM-judge studies validate against human assessors at the pair or ranking level; we instead validate against online business KPIs (CTR, CVR, conversion P/R) on the launch-decision critical path. Table 5. System efficiency of teacher and student judges. Latency is per-pair; QPS is aggregate throughput at the listed concurrency. Judge p50 (ms) p95 (ms) QPS Gemini-2.5-Pro (teacher, with guidel.) 6,963 10,840 2 Gemini-2.5-Flash (prompt, with guidel.) 2,189 8,807 4 Gemini-2.5-Flash-ft (ours) 751 1,117 18 Gemini-2.5-Flash-Lite-ft 647 1,342 20 7. Conclusion and Limitation We present a query-free userâproduct relevance metric for offline evaluation of DPA retrieval. A userâs engagement history is compressed into a small SID-grouped, event-stratified intent set, and an LLM judge scores each retrieved candidate against this set; an agentically-discovered guideline anchors a Gemini-2.5-Pro teacher, distilled into a Gemini-2.5-Flash student that matches the teacher at âź 9Ă9Ă lower latency. On 30 days of production DPA traffic, the metric ranks six retrieval sources consistently with conversion precision/recall and is directionally aligned with online CTR/CVR across four A/B tests; it serves as the offline gate for DPA model evaluation before online deployment. 8. GenAI Usage Disclosure In accordance with the ACM Policy on the Use of Generative AI Tools and Technologies, we disclose that GenAI assistants (including ChatGPT and Claude) were used during the preparation of the manuscript for language editing and writing assistance, including improving grammar, clarity, readability, and consistency of terminology. We also used AI-assisted tools to help draft preliminary versions of certain figures and diagrams, which we subsequently reviewed, revised, and finalized. All research ideas, methodology, system design, experiments, analyses, results, and conclusions were developed and validated by the authors. All cited references were manually identified by the authors and verified against their original sources with the help of AI tools. Because this paper studies LLM-based relevance judge, the use of LLMs as components of the studied systems and experimental methodology is described separately in the relevant sections of the paper. References Bauer et al. (2024) C. Bauer, E. Zangerle, and A. Said Exploring the landscape of recommender systems evaluation: practices and perspectives. ACM Transactions on Recommender Systems 2 (1), p. 1â31. Cited by: §1. Choi et al. (2025) Y. Choi, E. Kim, H. Kim, D. Park, H. Lee, J. Y. Kim, and J. Kim BloomIntent: automating search evaluation with llm-generated fine-grained user intents. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, p. 1â34. Cited by: §1, §6. Deng et al. (2025) J. Deng, S. Wang, K. Cai, L. Ren, Q. Hu, W. Ding, Q. Luo, and G. Zhou Onerec: unifying retrieve and rank with generative recommender and iterative preference alignment. arXiv preprint arXiv:2502.18965. Cited by: §2.1. Jeon et al. (2025) B. E. Jeon, R. Bae, and X. Bai Leveraging large language models for complementary product ads recommendation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 4837â4841. Cited by: 4th item. Joachims et al. (2017) T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay Accurately interpreting clickthrough data as implicit feedback. In Acm Sigir Forum, Vol. 51, p. 4â11. Cited by: §1, §6. Milli et al. (2025) S. Milli, M. Carroll, Y. Wang, S. Pandey, S. Zhao, and A. D. Dragan Engagement, user satisfaction, and the amplification of divisive content on social media. PNAS nexus 4 (3), p. pgaf062. Cited by: §1. Penha et al. (2025) G. Penha, A. V. Petrov, C. Hauff, E. Palumbo, A. Vardasbi, E. DâAmico, F. Fabbri, A. Wang, P. Chandar, H. Lindstrom, H. Bouchard, and M. Lalmas Do LLM-judges align with human relevance in cranfield-style recommender evaluation?. arXiv preprint arXiv:2511.23312. Cited by: §6. Peska and Vojtas (2020) L. Peska and P. Vojtas Off-line vs. on-line evaluation of recommender systems in small e-commerce. In Proceedings of the 31st ACM Conference on Hypertext and Social Media, p. 291â300. Cited by: §6. Pu et al. (2012) P. Pu, L. Chen, and R. Hu Evaluating recommender systems from the userâs perspective: survey of the state of the art. User Modeling and User-Adapted Interaction 22 (4), p. 317â355. Cited by: §1. Rahmani et al. (2025) H. A. Rahmani, C. Siro, M. Aliannejadi, N. Craswell, C. L. Clarke, G. Faggioli, B. Mitra, P. Thomas, and E. Yilmaz Judging the judges: a collection of llm-generated relevance judgements. arXiv preprint arXiv:2502.13908. Cited by: §1, §6. Reddy et al. (2022) C. K. Reddy, L. MĂ rquez, F. Valero, N. Rao, H. Zaragoza, S. Bandyopadhyay, A. Biswas, A. Xing, and K. Subbian Shopping queries dataset: a large-scale esci benchmark for improving product search. arXiv preprint arXiv:2206.06588. Cited by: §2.3. Shani and Gunawardana (2015) G. Shani and A. Gunawardana Evaluating recommender systems. Recommender Systems Handbook, p. 257â297. Cited by: §1. Singh et al. (2024) A. Singh, T. Vu, N. Mehta, R. Keshavan, M. Sathiamoorthy, Y. Zheng, L. Hong, L. Heldt, L. Wei, D. Tandon, et al. Better generalization with semantic ids: a case study in ranking for recommendations. In Proceedings of the 18th ACM Conference on Recommender Systems, p. 1039â1044. Cited by: §1. Sun et al. (2023) W. Sun, L. Yan, X. Ma, S. Wang, P. Ren, Z. Chen, D. Yin, and Z. Ren Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542. Cited by: §1, §6. Tang et al. (2025) T. Tang, Z. Tian, Z. Zhu, C. Wang, H. Hu, G. Tang, L. Liu, and S. Xu LREF: a novel llm-based relevance framework for e-commerce search. In Companion Proceedings of the ACM on Web Conference 2025, p. 468â475. Cited by: §1, §6. Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: Table 4. Upadhyay et al. (2024) S. Upadhyay, R. Pradeep, N. Thakur, D. Campos, N. Craswell, I. Soboroff, H. T. Dang, and J. Lin A large-scale study of relevance assessments with large language models: an initial look. arXiv preprint arXiv:2411.08275. Cited by: §6. Wang et al. (2025) Z. Wang, W. Du, M. O. F. Rokon, P. Adhikary, Y. Xue, J. Xu, J. Zhou, K. Lee, and M. Wen Semantic ads retrieval at walmart ecommerce with language models progressively trained on multiple knowledge domains. arXiv preprint arXiv:2502.09089. Cited by: §6. Zhang et al. (2025) J. Zhang, S. Kumar, W. Chang, Y. Wang, F. Zhang, W. Mao, H. Yu, A. Singh, M. Li, and Q. Wang Optimizing recall or relevance? a multi-task multi-head approach for item-to-item retrieval in recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, p. 5194â5204. Cited by: §1, §1, §3.1, §3, §6.