Paper deep dive
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian Anderson
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing benchmarks often rely on manual curation, predefined taxonomies, or synthetic prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly released upon acceptance.
Tags
Links
- Source: https://arxiv.org/abs/2610.17841v1
- Canonical: https://arxiv.org/abs/2610.17841v1
Trouble viewing inline? Open PDF directly â
Full Text
100,312 characters extracted from source content.
Expand or collapse full text
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval Tripti Shukla Glance Bengaluru, India tripti.shukla@glance.com Shwetha Somasundaram Glance Bengaluru, India shwetha.s@glance.com Akshat Gupta Glance Bengaluru, India akshat.gupta@glance.com Rohit Garg Glance Bengaluru, India rohit.garg@glance.com Paul Duff Glance Vilnius, Lithuania paul.duff@glance.com Ian Anderson Glance London, United Kingdom ian.anderson@glance.com Abstract Evaluating fashion retrieval systems remains challeng- ing due to the limited availability of benchmarks that cap- ture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing bench- marks often rely on manual curation, predefined tax- onomies, or synthetic prompts, limiting scalability and se- mantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evalua- tion of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical dis- crete representations learned by Residual Quantized Vari- ational Autoencoders (RQ-VAE) organize products into se- mantically coherent clusters at progressively finer granu- larities. We leverage this structure to construct FASHION- RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four com- plementary tasks for evaluating ranking, relevance classifi- cation, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demon- strate that FASHION-RQBENCH differentiates model ca- pabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly re- leased upon acceptance. 1. Introduction Fashion retrieval is inherently set-valued and graded. A query such as âlightweight cotton kurtas for summerâ con- strains multiple attributes, including product type, material, and season, and many catalog items may satisfy these con- straints to different degrees. Relevance is therefore compo- sitional: an item may match some attributes while violating others. A single aggregate ranking score can hide these dif- ferences. For example, two systems may achieve similar nDCG while one retrieves the correct garment in the wrong color and the other retrieves the correct color for the wrong garment. Evaluating such systems requires relevance labels that are set-valued, graded, and decomposable by attribute. Existing fashion benchmarks do not provide these prop- erties. Manual fashion retrieval datasets typically associate a query with a single target item [22, 51], penalizing other valid results. Graded set-valued annotations are possible, but expensive: ESCI contains 2.6M judgments for 130K queries [34]. Moreover, manually collected labels are tied to a particular catalog snapshot and query distribution. Au- tomatic assessment using language models [44] removes annotation cost, but typically produces an independent rel- evance score for each queryâitem pair and does not iden- tify which query attributes are satisfied or violated. These approaches also treat items independently, overlooking the relationships among products in the catalog. We instead observe that the catalog itself contains the structure needed to define relevance sets.Recent re- trieval and recommendation systems such as DSI [40], RI- POR [55], and TIGER [32] represent items using sequences of discrete codes learned with residual quantization [19]. Because these codes are organized from coarse to fine, they induce a prefix tree over the catalog: items sharing a pre- arXiv:2610.17841v1 [cs.IR] 13 Oct 2026 > Sibling Clusters (Should rank lower) Exact (E) Target cluster items Product Catalog Multi modal fashion items RQ-VAE Hierarchical Space Learned semantic hierarchy Topwear T-shirt Shirt Blue Red Parent, sibling and related clusters provide contextual alternatives. Automatic Query Generation Context-aware natural queries Cluster attributes Parent & sibling context Exclusivity constraints Coverage of sub-types âblue striped t-shirtâ Hierarchical Relevance Construction Graded relevance from hierarchy Sibling (S) Itemsfromsibling clusters Related (R) Items from parent clusters Irrelevant (I) Items from unrelated branch FASHION-RQ: AUTOMATED BENCHMARK GENERATION FASHION-RQBENCH: FOUR EVALUATION TASKS T1. QUERY-PRODUCT RANKING Rank products so that E > S > R > I Query: âblue striped t-shirtsâ EESSRI T2. RELEVANCE CLASSIFICATION Classify each product into E, S, R, I Query: âblue striped t-shirtsâ E SRI T3.SIBLINGDISCRIMINATION Prefertargetoversiblingclusters Query: âblue striped t-shirtsâ Target Cluster (Exact) T4. QUERY ROBUSTNESS Consistent retrieval under query variation âblue striped t-shirtsâ âstriped t- shirts in blueâ âmenâs blue striped t- shirtsâ EVALUATION METRICS A.QUERY RELEVANCE METRIC (Weighted) Attribute importance í€ ! from cross-encoder relevance experiment B. ATTRIBUTE-LEVEL METRICS (Diagnostic) Score each attribute separately to reveal fine-grained strengths / failure modes Ă Color Material Category Price ... 0.95 0.80 1.00 0.60 Per-attribute match í ! í,íâ[0,1] 0.86 Weighted Query Relevance í í,íâ[0,1] Color 0.95 Material 0.80 Category 1.00 Price 0.60 Gender 0.78 Brand 0.85 Style 0.67 Metric: AttrScore(a) = mean match on queries that specify attribute a Figure 1. Overview of FASHION-RQ. The framework leverages an RQ-VAE hierarchy to automatically generate natural-language queries and graded relevance sets, with Exact, Sibling, Related, and Irrelevant relationships. The resulting benchmark supports four complementary retrieval tasks, while weighted query relevance and attribute-level metrics evaluate overall intent satisfaction and fine-grained attribute matching. fix form increasingly specific groups. We repurpose this structure not as an index for retrieval, but as an automatic annotation mechanism. However, vanilla RQ-VAE does not guarantee that a code corresponds to a single seman- tic attribute, which can produce heterogeneous clusters. We therefore introduce an attribute exclusivity loss that encour- ages each quantization level to specialize to a coherent at- tribute, making the resulting hierarchy suitable for defin- ing relevance sets. A prefix then defines a relevance set, its length controls granularity, and the shared-prefix length between an item and the target prefix provides graded rele- vance without a hand-authored taxonomy. Building on this hierarchy, we filter clusters that are too small or heterogeneous to serve as relevance sets and com- pute each retained clusterâs relational context: the attributes that distinguish it from its parent and sibling clusters. A language model then verbalizes this structured evidence into shopper-style queries. Importantly, the language model generates the query but does not determine relevance; rel- evance is inherited deterministically from the semantic ID (SID) [32] structure. We instantiate this framework on the public Myntra Fashion Product Images catalog [2], containing 44,446 items, and construct FASHION-RQBENCH: 588 clus- ters across three granularity levels and 5,380 attribute- enriched queries. The benchmark provides four comple- mentary tasks: queryâproduct ranking, hierarchical rele- vance classification, sibling discrimination, and paraphrase robustness. We additionally introduce a weighted query- relevance score and per-attribute scores to distinguish over- all retrieval quality from fine-grained attribute satisfaction. Figure 1 provides an overview of the complete FASHION- RQBENCH pipeline, from hierarchical benchmark con- struction to query generation, evaluation tasks, and metrics. Evaluation of ten off-the-shelf sparse and dense retrieval models, without fine-tuning, shows that aggregate ranking performance alone can mask important weaknesses. Dense encoders achieve substantially higher ranking quality than BM25 [36] (nDCG@10 = 0.87 vs. 0.77), yet the best model achieves only 0.51 accuracy on four-way hierarchical relevance classification. Model scale is also not consistently predictive: the 4B Qwen3 [42] encoder slightly outperforms its 8B counterpart on sibling discrimination. Attribute-level evaluation further localizes these differences: models per- form strongly on product type and gender (> 0.93) but sub- stantially worse on material (â 0.52) and style (â 0.43). Finally, a human study comparing benchmark queries with real user queries shows comparable perceived naturalness and strong intent clarity, supporting Fashion-RQBENCH as a proxy for real-world fashion search. Our contributions are: âą We repurpose RQ-VAE semantic identifiers as an auto- matic annotation mechanism and introduce an attribute exclusivity loss that encourages quantization levels to encode coherent attributes.Prefixes define relevance sets, prefix length controls granularity, and shared-prefix length defines graded relevance. âą We introduce a cluster-conditioned query generation pro- cedure that captures each clusterâs attribute distribution and its relational differences from parent and sibling clusters, enabling automatic generation of shopper-style queries. âą We introduce FASHION-RQBENCH, comprising 588 clusters, 5,380 attribute-enriched queries, and 44,446 products, with deterministic graded relevance and four complementary evaluation tasks. âą We propose weighted query-relevance and per-attribute metrics that distinguish overall intent satisfaction from fine-grained attribute satisfaction. 2. Related Works Fashion Retrieval Datasets and Benchmarks.Fash- ion retrieval involves fine-grained attributes such as product type, color, pattern, material, occasion, season, and price. Existing resources address different aspects of this prob- lem. DeepFashion [22] and DeepFashion2 [11] provide large-scale benchmarks for fashion recognition, detection, segmentation, and retrieval; Fashion200K [12] and Shop- ping100k [3] support attribute-based search; FACAD [53] focuses on fashion captioning; and Fashion Product Im- ages [2] provides the catalog schema used in our work. Language-conditioned retrieval is studied primarily through composed image retrieval, including FashionIQ [51], and more general benchmarks such as CIRR [23], CIRCO [4], and LaSCo [20], with fashion-oriented encoders includ- ing FashionCLIP [6], Marqo-FashionSigLIP[16], and GCL [57]. Most existing fashion benchmarks use single-target rele- vance, whereas open-ended product queries can correspond to a set of valid items; CIRCO [4] is a notable exception, but at modest scale. Graded set-valued relevance is avail- able in product search: ESCI [33], for example, provides exact, substitute, complement, and irrelevant labels, but re- quires 2.6M manual judgments for 130K queries. Manual construction also fixes the query distribution and catalog snapshot, making large-scale regeneration costly. Outside fashion, QUEST [25] automatically constructs set-valued queries from Wikipedia categories, but relies on an exist- ing taxonomy. In contrast, FASHION-RQ derives both rel- evance sets and queries from the catalogâs learned semantic structure, avoiding manual relevance annotation and prede- fined taxonomies. Generative Retrieval and Semantic Identifiers. Genera- tive retrieval directly decodes item identifiers from queries. DSI [40] and NCI [49] established this paradigm, with hier- archical identifier construction shown to be effective. Sub- sequent approaches learn identifiers through discrete auto- encoding [39] or residual quantization for relevance-aware, prefix-scorable representations [55]. In recommendation, TIGER [32] represents items using SIDs produced by an RQ-VAE [19, 45]; its coarse-to-fine codebooks induce a prefix hierarchy, while Singh et al. [38] demonstrate bene- fits for cold-start and long-tail recommendation. Recent work explicitly shapes this hierarchy.HiD- VAE [9] supervises quantization levels with hierarchical item tags and introduces a uniqueness objective to reduce code collisions. These approaches, however, use seman- tic identifiers primarily as representations for downstream retrieval or recommendation. FASHION-RQ instead repur- poses the hierarchy as an annotation structure: prefixes de- fine product groups at different granularities and shared- prefix length provides graded relevance. We further intro- duce level-wise attribute supervision with an attribute ex- clusivity loss to encourage semantically coherent clusters, using the resulting tree to construct the benchmark rather than the retrieval identifiers themselves. Automated Query and Relevance Construction.Syn- thetic query generation reduces the cost of manual anno- tation. Doc2Query [30] generates queries for documents, while InPars [5] and Promptagator [8] use LLM-generated queryâdocument pairs for retrieval training.These ap- proaches are primarily document-conditioned: a query is generated from an individual document and typically inher- its a single positive target. Rahmani et al. [31] extend synthetic generation to evaluation, using LLM-generated queries and judgments to reproduce retrieval rankings, but the resulting relevance labels remain pairwise LLM assess- ments. BEIR [43], in contrast, aggregates existing man- ually annotated datasets, while ExcluIR [56] focuses on phenomenon-specific retrieval with manually constructed queries. FASHION-RQ uses cluster-conditioned generation: queries are generated from a purified clusterâs attribute dis- tribution together with its relational differences from par- ent and sibling clusters. The cluster itself therefore de- fines a multi-item relevance set, while the relational con- text encourages queries that distinguish the target group from nearby alternatives. The LLM only verbalizes catalog- derived evidence; it does not assign relevance, keeping the resulting labels independent of the generator. Evaluating Retrieval Beyond Binary Relevance. Graded metrics such as nDCG [15] require relevance judgments, motivating LLM-based assessors such as UMBRELA [44]. Such assessors produce holistic queryâitem scores and therefore do not reveal which attributes are satisfied or vio- lated. Attribute-level evaluation has been explored through attribute-specific retrieval metrics [24], conditional simi- larity [47], compositionality benchmarks [13, 54], and constraint-satisfaction measures [1]. However, these set- tings either evaluate individual attributes, use curated dis- tractors, or operate over restricted constraint types rather than decomposing relevance over a ranked product catalog. FASHION-RQ combines set-valued graded relevance with joint attribute-level evaluation. Our weighted query- relevance score and per-attribute metrics are computed from the structured query representation Ï(q), separating overall intent satisfaction from fine-grained attribute satisfaction. Because relevance is inherited deterministically from the semantic-ID hierarchy and attribute scores use a fixed cata- log schema, the evaluation is reproducible and inspectable without per-pair human or LLM judgments. 3. FASHION-RQ: Automated Benchmark Generation We introduce FASHION-RQ, an automated framework for constructing retrieval benchmarks from product catalogs without manual query or relevance annotation. It exploits the hierarchical structure induced by residual quantization: SID prefixes define relevance sets at multiple granularities, while catalog statistics generate natural-language queries that represent the target cluster and discriminate it from nearby alternatives. Given a catalog I = i 1 ,...,i N , each item i is rep- resented by a dense embeddinge i âR 1024 and catalog metadata including product type, category, color, gender, usage, price, and textual information. A hierarchical to- kenizer maps each item to a SID s i = (s (1) i ,s (2) i ,s (3) i ), with L = 3 quantization levels and K = 64 codes per level. A prefix p (â) = (s (1) ,...,s (â) ) defines the cluster C p (â) = n iâI : s (1:â) i = p (â) o , where â = 1 and â = 3 correspond to the coarsest and finest granularities, respec- tively. Our benchmark construction consists of four stages: (1) hierarchical semantic representation, which induces a coarse-to-fine catalog hierarchy; (2) cluster purification, which removes unreliable clusters; (3) relational context construction, which identifies attributes distinguishing each cluster from its parent and siblings; and (4) automatic query generation, which produces queries satisfying coverage and discriminability requirements. The resulting benchmark contains natural-language queries paired with automatically derived relevance sets at multiple hierarchy levels. Problem Formulation. Fashion retrieval should be eval- uated across multiple levels of semantic granularity, as queries may target broad product groups or fine-grained subsets. FASHION-RQ uses semantic IDs (SIDs) to con- struct such an evaluation hierarchy: each SID prefix p (â) defines a catalog cluster C(p (â) ), from which we generate natural-language queries Q(p (â) ) and derive their ground- truth relevance sets. We use an RQ-VAE to induce this hierarchy, as residual quantization captures entangled attribute interactions and organizes products into semantically coherent clusters at multiple granularities. This structure provides a catalog- grounded basis for realistic query generation and repro- ducible evaluation of fashion retrieval systems. With K = 64 codes and L = 3 levels, the hierarchy admits P L â=1 K â = 64 + 64 2 + 64 3 â 2.7 Ă 10 5 pre- fixes, making manual benchmark construction impractical. FASHION-RQ therefore automatically selects prefixes, de- rives their relevance sets, and generates queries: B = n (q j ,p (â) j ,C(p (â) j ),Ï(q j )) o N j=1 ,(1) where q j is the natural-language query, p (â) j its semantic granularity, C(p (â) j ) its relevance set, and Ï(q j ) its struc- tured attributes. Proposed Attribute Exclusivity Loss íł íííí Level0 (masterCategory) Level1 (articleType) Level2 (joint attributes) RQ-VAE (3 Levels) Per-level Supervised Contrastive (SupCon) Loss on quantized vectors Pullsame category together, push others apart Pull same article type together Pull same attribute combination together Same label (positives) Different labels (negatives) Codebook space (level 2) Without íł íííí Same attribute combination scattered across codes. Codebook space (level 2) With íł íííí (ours) Same attribute combination forms a tighter cluster (owned by one code region) Figure 2. Attribute Exclusivity Loss. Per-level supervised con- trastive supervision encourages progressively finer attribute se- mantics to form compact, distinct regions in the RQ-VAE code- books, reducing the scattering of identical attribute combinations across codes. Hierarchical Semantic Representation.We first rep- resent each product using a multimodal embedding from Marqo-FashionSigLIP [16], trained with Generalized Con- trastive Learning (GCL) on rich product signals, including product images, descriptions, categories, style, color, ma- terial, keywords, and fine-grained details, across over one million fashion products. This provides a semantically rich representation beyond text-only descriptions. Givene i âR 1024 , an MLP encoder f Ξ maps each prod- uct to z i = f Ξ (e i ) âR 32 with architecture 1024â 512â 256 â 128 â 32. We apply residual quantization [19] with L = 3 codebooks of K = 64 codes. Starting from r (0) i = z i , residual quantization selects the code and up- dates the residual at each level: s (â) i = argmin kâ1,...,K r (ââ1) i âc (â) k 2 2 , r (â) i = r (ââ1) i âc (â) s (â) i . (2) The quantized representation is Ëz i = P L â=1 c (â) s (â) i . Thus, the hierarchy captures coarse-to-fine product structure. For benchmark construction, the induced clusters should be both exhaustive and exclusive: items sharing an at- tribute combination should be grouped together, while dis- tinct combinations should map to distinct codes. Recon- struction and VQ losses promote exhaustivity by preserv- ing the input embedding, but do not explicitly prevent se- mantically different combinations from sharing a code. We therefore augment the objective with an attribute-based ex- clusivity loss: L =L recon + L X â=1 L (â) codebook + ÎČL (â) commit + λ excl L excl . (3) The exclusivity loss applies supervised contrastive learn- ing [18] directly to the quantized code vectors q (â) i =c (â) s (â) i , keeping the supervision local to each codebook. Figure 2 illustrates this level-wise exclusivity supervision. We use masterCategory at level 1, articleType at level 2, and all articleType-anchored joint attributes (color, gender, usage, season, age group, price tier, and fashion type) at level 3. For attributes with at least one valid positive pair in a mini- batch, the corresponding losses are averaged: L excl = L X â=1 1 |A (â) valid | X aâA (â) valid SupCon(q (â) ,y a ;Ï ).(4) This prevents sparse attributes from diluting the available supervision while enforcing exclusivity directly at each dis- crete level. We set Ï = 0.07, following supervised con- trastive learning [18], and λ excl = 0.05 to keep the exclu- sivity term subordinate to the primary reconstruction and quantization losses. At inference, SIDs are obtained deter- ministically using the same residual quantization procedure, yielding s i â0,..., 63 3 . Cluster Purification. The induced hierarchy may contain small or heterogeneous clusters. We therefore purify it post- hoc, independent of RQ-VAE training. Using articleType as the anchor attribute, cluster purity is the fraction of labeled items belonging to the dominant product type: Purity(p (â) ) = max a |iâ C(p (â) ) : aT i = a| |iâ C(p (â) ) : aT i Ìž= null| .(5) We retain prefixes with |C(p (â) )| â„ 5 and Purity(p (â) ) â„ 0.80; at the finest level, we additionally require at least two items. All qualifying prefixes are retained to preserve hier- archy coverage. Relational Context. Purity does not capture how a cluster differs from nearby alternatives. For each surviving prefix p (â) , we compare it with its parent and up to three siblings. For categorical attribute value (a,v), we compute â a=v (p (â) ,R) = pct C(p (â) ) (a = v)â pct R (a = v), (6) where R denotes the parent or a sibling. For numeric at- tributes, we compare mean, median, minimum, maximum, and quartiles. Differences of at least 20 percentage points are retained as exclusivity constraints, identifying attributes that distinguish the target from nearby clusters. Automatic Query Generation.For each purified clus- ter, we generate Q(p (â) ) from its attribute distribution and relational context, targeting exhaustiveness and exclusivity. Each sub-type occupying â„ 5% of the cluster receives at least one query; a catch-all query covers the remainder. At- tribute combinations coveringâ„ 90% &â„ 50% of a cluster define discriminative and coverage signatures, respectively. We use an instruction-following LLM (Gemini [41]; temperature 0.2, top-p 0.8) whose prompt includes these signatures, attribute distributions, parent/sibling context, and exclusivity constraints. It is instructed to produce com- plete shopper-style queries using discriminative attributes, verify separation from siblings, and avoid unsupported uni- versal claims. The LLM returns 6â12 queries plus cluster insights, validation checks, and a coverage map linking ma- jor product types to queries. Outputs are parsed leniently; the full prompt and schema are provided in the supplemen- tary material. Query Attribute Enrichment. For attribute-aware evalu- ation, each unique generated query is converted by the same LLM intoproduct_type,gender,age_group,color, pattern,material,occasion,brand,price_hint,style_ tags. Unexpressed attributes are null, multi-valued at- tributes may contain multiple values, and brand is popu- lated only when explicitly mentioned. Duplicate queries are removed globally before enrichment. 4. Benchmark Definition & Tasks. Source Dataset. We propose FASHION-RQBENCH, a benchmark for evaluating fine-grained fashion retrieval and ranking. It is constructed from the public Myntra Fashion Product Images [2] catalog, comprising 44,446 products with structured metadata, free-text descriptions, and prod- uct images. Each item is represented by a 1024-dimensional embedding generated using Marqo-FashionSigLIP [16], and assigned a 3-level SID using the RQ-VAE model de- scribed in Section 3. Table 1 provides an overview of Fashion-RQBENCH. More details are provided in the sup- plementary. Table 1. Overview of the Fashion-RQBENCH evaluation set. StatisticValueDetails Master categories5Apparel, Accessories, Footwear, Personal Care, Free Items Subcategories45â Article types143â Base colors47â Genders5Men, Women, Unisex, Boys, Girls Benchmark clusters58817 / 135 / 436 at levels 1 / 2 / 3 Enriched queries5,3802â12 per cluster Unique query strings5,30598.7% of queries Mean query length10.03â26 words Level-1 cluster size823.9Range: 6â5,349 Level-2 cluster size219.2Range: 5â4,896 Level-3 cluster size79.5Range: 5â4,888 Generated Benchmark. We exploit the hierarchical RQ- VAE SID structure to construct benchmark instances with- out manual taxonomy design. Each qualifying 1-, 2-, or 3-level SID prefix that satisfies the purity and minimum- size criteria is materialized as a cluster record. This yields 588 clusters, comprising 17, 135, and 436 clusters at levels 1, 2, and 3, respectively (Table 1). Attribute values exhibit substantial lexical diversity, supporting evaluation beyond a fixed vocabulary. Benchmark Tasks. We define four complementary tasks over the same SID hierarchy, with candidate construction and relevance labels derived deterministically, eliminating the need for an independently authored product taxonomy. For a level-3 anchor cluster, candidates receive four graded relevance levels: Exact (same level-3 prefix), Sibling (same level-2 prefix), Related (same level-1 prefix), and Irrele- vant (different level-1 prefix), with scores 3, 2, 1, and 0, respectively. Thus, a single representation defines semantic neighborhoods, evaluation instances, and graded relevance, yielding a consistent and reproducible protocol. âą QueryâProduct Ranking.Given a natural-language query and a fixed candidate pool containing Exact, Sib- ling, Related, and Irrelevant items, rank the candidates according to relevance.We report nDCG@10 and Recall@10, both overall and for explicitly constrained at- tributes. âą Hierarchical Relevance Classification. Given a queryâ product pair, predict one of the four relevance classes. We report accuracy and macro-F1. âą Sibling Discrimination.Given one target item and closely related sibling distractors, identify the target leaf cluster. The task emphasizes fine-grained distinc- tions between semantically similar products. We report nDCG@3 and Recall@3 for ranked predictions, or binary accuracy when a single item is selected. âą Query Paraphrase Robustness.We use the query groups generated by our method in Section 3 as para- phrases: since all queries from a cluster describe the same target intent by construction, we evaluate them against a shared candidate pool. This isolates sensitivity to query wording from candidate-pool variation. We measure ro- bustness using Consistency@K, the mean pairwise Jac- card overlap of top-K retrieved item sets. 5. Evaluation Framework FASHION-RQ evaluates retrieval quality at three com- plementary levels: (i) task-level performance using the hierarchy-derived relevance labels, (i) overall query rel- evance based on attribute satisfaction, and (i) attribute- level diagnostics that identify specific strengths and fail- ure modes. All metrics are computed from the automati- cally constructed queries, query metadata, and SID hierar- chy, without requiring additional human annotation. The four tasks use the common relevance structure defined in Section 4. Task-Level Metrics.We evaluate each task using met- rics suited to its corresponding objective as defined in Sec- tion 4.For QueryâProduct Ranking (T1), we report nDCG@5/10 and Recall@5/10, both overall and for ex- plicitly constrained attributes. For Hierarchical Relevance Classification (T2), we report accuracy and macro-F1, with per-class precision, recall, and F1 for additional analysis. ForSiblingDiscrimination(T3),wereport nDCG@1/3 and Recall@1/3 to measure fine-grained target-cluster identification.For Query Paraphrase Robustness (T4), we measure retrieval stability using Consistency@5/10, defined as the mean pairwise Jaccard overlap between the top-K result sets across paraphrases, and report the mean nDCG@10 across paraphrases. Weighted Query Relevance. The hierarchy-based metrics assess whether retrieved products belong to the intended se- mantic neighborhood, but do not quantify satisfaction of the individual attributes expressed in a query. Moreover, at- tributes are not equally important in shopping intent: a user may accept a different brand, but not a different product type. We therefore introduce a weighted query relevance score that measures query-level intent satisfaction, with at- tribute weights derived from a cross-encoder trained to re- flect relevance patterns in real user data. Let A(q) denote the attributes explicitly expressed in query q, as specified by its structured representation Ï(q). Table 2. Performance on the four FASHION-RQ tasks. Best results are shown in bold and second-best results are underlined. T1: RankingT2: Relevance ClassificationT3: Sibling DiscriminationT4: Paraphrase Robustness ModelnDCG@10Recall@10AccuracyMacro-F1nDCG@3Recall@3Consistency@5Consistency@10 BM25 [36]0.77360.73760.40230.39190.61420.77230.50670.7419 RoBERTa-large [21]0.73910.72520.37490.34810.54910.72600.44620.7135 SimCSE-RoBERTa-large [52]0.86550.82420.50460.50470.64210.81450.59140.8197 Sentence-T5-large [29]0.82350.7845 0.43280.42270.66100.83180.58790.8115 Qwen3-Embedding-4B [42]0.87190.82790.51450.51460.66410.82320.60410.8373 Qwen3-Embedding-8B [42]0.86860.8241 0.50360.50370.65500.82110.60220.8295 E5-Mistral-7B-Instruct [48]0.86990.82580.51340.51350.64620.81450.58930.8233 ColBERTv2 [37]0.85970.81030.48270.48270.66240.82420.56080.7889 SPLADE++ [10]0.86280.81410.49360.49370.66620.83490.58550.8089 Contriever-MSMARCO [14]0.86280.81640.47960.47970.65530.82770.56210.8073 For each aâA(q), an attribute-match function m a (q,p)â [0, 1] measures agreement between the query and product. The comparator depends on attribute type: exact or syn- onym matching for categorical attributes, distance-based matching for ordinal attributes, and semantic similarity for perceptual attributes. For color, perceptual CIE76 [35] dis- tance is additionally used when both values are recognized colors. Missing product attributes receive a neutral score to avoid penalizing incomplete catalog metadata. Details are provided in the supplementary material. We aggregate attribute matches using importance weights w a obtained from the cross-encoder relevance ex- periment in Section 6: R(q,p) = P aâA(q) w a ,m a (q,p) P aâA(q) w a .(7) Only explicitly specified attributes contribute, ensuring that R(q,p) measures expressed user intent rather than generic product similarity. The score lies in [0, 1], with higher val- ues indicating stronger overall query satisfaction. Attribute-Level Scores. While R(q,p) measures overall query satisfaction, we retain individual attribute matches to diagnose fine-grained retrieval behavior. For each attribute a, we report the mean match score over queries that ex- plicitly specify a: AttrScore(a) = 1 |Q a | P qâQ a m a (q,p q ), where Q a denotes the queries specifying a. We report scores for product type, gender, age group, color, material, occasion, brand, pattern, price, and style, enabling attribute- specific strengths and failure modes that may be obscured by aggregate metrics. 6. Experiments Cross-encoder derived attribute importance.The attribute weights in R(q,p) are distilled from a cross- encoder trained on real product-search data, including the ESCI [34] filtered to fashion relevant queries and inter- nal production-scale proprietary data. Since its proprietary training data precludes direct use as an evaluation model, we use the cross-encoder as a knowledge source and esti- mate attribute importance through matched twin-pair abla- tion and regression-based analysis. We normalize the re- sulting estimates to obtain non-negative weights w a that sum to one, which are used to aggregate per-attribute match scores into the model-agnostic query-relevance score. The resulting weights are shown in Table 3; full estimation pro- cedures, per-attribute confidence diagnostics, and sample sizes are provided in supplementary. Baselines. We evaluate FASHION-RQ using ten off-the- shelf retrieval baselines spanning sparse lexical and dense semantic retrieval. The sparse family includes BM25 [36] on settings. The dense family comprises nine pretrained encoders covering 355Mâ8B parameter scales, different pooling strategies, and both non-instruction-tuned and instruction-tuned query encoders. These include general- purpose sentence encoders as well as recent large embed- ding models. All baselines are evaluated without train- ing or fine-tuning on FASHION-RQ; queries and catalog products are encoded using their native published config- urations, and candidates are ranked by lexical matching or embedding similarity, respectively. This setup evaluates the extent to which existing retrieval representations generalize to the fine-grained, hierarchically structured fashion queries in FASHION-RQ. Results.Table 2 reports the primary results across the four FASHION-RQ tasks, using two complementary met- rics for each task. Overall, dense and learned sparse re- trievers consistently outperform the lexical BM25 baseline, demonstrating the benefit of semantic representations for fine-grained fashion retrieval. For T1, Qwen3-Embedding-4B achieves the best per- formance on both nDCG@10 (0.8719) and Recall@10 (0.8279), followed closely by E5-Mistral-7B-Instruct and SimCSE-RoBERTa-large. The dense and learned sparse models form a relatively tight performance range, while RoBERTa-large, which lacks retrieval-specific training, per- forms worst. For T2, Qwen3-Embedding-4B again achieves the best performance, with 0.5145 accuracy and 0.5146 macro-F1, narrowly followed by E5-Mistral-7B-Instruct. The close agreement between accuracy and macro-F1 suggests that the relative ordering is not driven by a single relevance Table 3. Weighted query-relevance and attribute-level matching scores across retrieval models. Attribute weights w a are derived from the cross-encoder and used to compute the overall weighted relevance score, Rel. ModelRel.AgeBrandColorGenderMat.Occ.PatternPriceProd.Style Qwen3-Emb.-8B [42]0.92740.99070.85600.91250.97500.53670.96320.85210.64840.94090.4291 E5-Mistral-7B [48]0.91580.99000.89140.87820.96760.52320.96180.84650.65430.93950.4265 Qwen3-Emb.-4B [42]0.92540.98950.82210.89650.97210.52170.96240.83270.64260.94380.4204 Sentence-T5-L [29]0.90180.99180.75870.91590.99110.53100.96280.83950.65620.93920.4229 Sup-SimCSE-RoBERTa-L [52]0.89230.99220.66130.90080.98730.50030.96160.82960.62470.93680.4292 BM25 [36]0.87530.98260.79270.84510.88890.49870.95850.79340.61690.89290.4211 RoBERTa-L [21]0.83400.96710.67760.73880.87930.33610.93380.63730.51930.85280.4222 ColBERTv2 [37]0.91980.98850.85710.86150.96720.5307 0.96380.85600.62280.93500.4271 SPLADE++ [10]0.92090.98770.8597 0.85940.97120.53430.96390.85260.63250.93610.4278 Contriever-MSMARCO [14]0.91550.98500.86150.84790.95600.52680.96000.83320.62940.93200.4247 w a â0.0660.2110.1380.1270.0630.0080.0600.0240.2900.012 class. RoBERTa-large obtains the lowest accuracy (0.3749), highlighting the importance of retrieval-oriented represen- tation learning for hierarchical relevance discrimination. T3 is more challenging, requiring discrimination among products from closely related sibling clusters. SPLADE++ achieves the best nDCG@3 (0.6662) and Recall@3 (0.8349), followed by Qwen3-Embedding-4B on nDCG@3 and Sentence-T5-large on Recall@3. This shows that fine- grained sibling discrimination does not necessarily improve with model size or dense semantic representations. For T4, Qwen3-Embedding-4B achieves the highest Consistency@5 (0.6041) and Consistency@10 (0.8373), indicating the most stable retrieval across paraphrases. Dense and learned retrievers generally show higher consis- tency than BM25, particularly at the shallow cutoff, where lexical matching is more sensitive to wording changes. The weaker ranking but comparatively stable behavior of RoBERTa-large further demonstrates that paraphrase con- sistency captures a distinct retrieval property. Overall, Qwen3-Embedding-4B achieves the strongest results on three of the four tasks, while SPLADE++ per- forms best on sibling discrimination. The complementary results demonstrate that FASHION-RQ assesses not only overall retrieval quality, but also hierarchical relevance dis- crimination and robustness to query variation. Table 4. Mean ratings from the user study on a 5-point Likert scale. Higher scores indicate greater perceived naturalness and intent clarity. SourceNaturalnessIntent Clarity CHAT3.522.83 LLM2.843.76 RQBENCH (Ours)3.623.91 Query-Relevance Evaluation. Table 3 shows that Qwen3- Embedding-8B achieves the highest weighted relevance score (0.9274), followed closely by SPLADE++ (0.9209) and ColBERTv2 (0.9198), demonstrating that both dense and learned sparse retrievers can capture fine-grained query intent. BM25 scores lower (0.8753), while RoBERTa-Large ranks last (0.8340). Attribute-level results reveal strong alignment for age, occasion, gender, and product type across models, whereas material and style remain consis- tently weaker. Notably, the weighted score distinguishes models with similar aggregate performance through their attribute-level behavior, demonstrating its ability to expose fine-grained differences in query satisfaction. User Study. We conducted a user study to assess the nat- uralness and intent clarity of queries from three sources: our Fashion-RQBENCH (RQBENCH), real user queries (CHAT), and LLM-generated queries (LLM). 32 partici- pants rated each query on a 5-point Likert scale for both dimensions. Inter-rater reliability was high, with Cron- bachâs [7] α = 0.977 for naturalness and α = 0.969 for intent clarity, indicating strong agreement among raters. Table 4 summarizes the mean ratings. Naturalness rat- ings are comparable between RQBENCH and CHAT, with mean scores of 3.62 and 3.52, respectively. In contrast, in- tent clarity shows a strong separation: RQBENCH achieves the highest mean score (3.91), followed by LLM (3.76), while CHAT is substantially lower (2.83), with all pair- wise differences statistically significant. These results in- dicate that RQBENCH queries preserve the naturalness of real user language while providing substantially clearer and more actionable intent. We provide more details on the study in supplementary. 7. Conclusion We introduced FASHION-RQ, an automated benchmark construction framework that leverages hierarchical seman- tic structure to generate realistic queries and graded rel- evance sets without manual annotation.Using an RQ- VAE with attribute exclusivity supervision, FASHION- RQ produces coherent fine-grained clusters and exploits parentâsibling relations to construct diverse retrieval sce- narios.We instantiate this framework as FASHION- RQBENCH, a benchmark for evaluating fashion retrieval models through complementary retrieval tasks, weighted query-relevance evaluation, and attribute-level metrics. To- gether, FASHION-RQ and FASHION-RQBENCH provide a scalable and semantically grounded alternative to manu- ally curated fashion retrieval benchmarks. FASHION-RQ depends on catalog quality, the underlying multimodal rep- resentation, and LLM-generated queries, which may intro- duce coverage and linguistic biases. The automatically de- rived relevance structure also approximates rather than re- places human judgments. Future work will explore human validation, broader domains and attributes, richer semantic relations, and adaptive benchmark generation. 8. Appendix 8.0.1. Codebook Size Ablation Study. We study the effect of RQ-VAE [19] codebook size on the semantic-ID hierarchy by comparing codebook sizes of 64, 128, and 256 with three quantization levels, while keeping all other settings fixed. The results characterize the trade-off between cluster purity and cluster fragmentation. As the codebook size increases, the available semantic-ID space grows from 64 3 = 262,144 to 256 3 = 16,777,216, substan- tially exceeding the 44,446 catalog items. Consequently, larger codebooks produce increasingly fragmented clusters, reducing the number of items sharing the same semantic-ID prefixes. Table 5. Effect of codebook size on weighted attribute purity for 64, 128 and 256 size. Attribute64128256 Master Category99.8%50.4%1.0% Subcategory99.4%50.2%1.0% Article Type98.3%49.7%1.0% Gender98.1%49.6%1.0% Base Color87.9%44.5%1.0% Usage97.7%49.4%1.0% Interestingly, weighted global purity and mean per- cluster purity exhibit different trends. Weighted purity em- phasizes the large clusters and therefore decreases sharply with increasing codebook size. In contrast, mean per-cluster purity can increase because fragmentation creates many small, trivially pure clusters. Thus, high unweighted clus- ter purity alone does not imply a useful semantic hierarchy: excessively large codebooks can produce highly pure but near-singleton clusters that provide little shared semantic structure. The fragmentation trend is evident across all three levels (Table 6). At the coarsest level, the average cluster size decreases from 824 items for the 64-codebook to 166.5 for the 256-codebook. At full depth, the corresponding cluster size decreases from 1.86 to 1.23 items, indicating that many semantic IDs become effectively item-specific. The 128- codebook configuration provides an intermediate operating point, with an estimated 285.0 items per D1 cluster and 1.48 items per full-depth cluster. Overall, these results motivate using a moderate code- book size when constructing a semantic-ID benchmark: the codebook must provide sufficient capacity to separate fine- grained products while retaining meaningful item sharing within semantic-ID prefixes. In our benchmark construc- tion, we therefore use the codebook configuration size of 64 which provides the desired balance between semantic purity and cluster density. 8.0.2. Query Generation Methodology. Signature construction. For each semantic-ID prefix, we derive attribute signatures using combinations of masterCategory, subCategory, articleType, brandName, baseColor, gender, usage, price, and productDisplayName. We construct two comple- mentary signature sets. Discriminating signatures identify highly specific cluster characteristics, requiring at least 10% cluster coverage and 90% precision against the full cata- log; the top eight are retained based on exclusivity, pre- cision, and coverage. These signatures guide query gen- eration toward attributes that distinguish the target cluster from its siblings. Coverage signatures require at least 5% coverage and 50% precision, with the top 20 retained, and are used to ensure that queries collectively cover the ma- jor product types within the cluster. The primary product- type axis is selected as the most specific attribute whose dominant value provides the highest cluster purity, with masterCategory as the fallback. Query generation. For each qualified prefix, the Gemini prompt combines fixed generation instructions with cluster- specific evidence.The instructions require natural and complete-sentence queries, coverage of all product types accounting for at least 5% of the cluster plus a catch-all for minority types, and discrimination from sibling clusters. Queries are grounded in productDisplayName and de- scription language while avoiding queries based solely on an attribute value unless that value is shared by all clus- ter items. The prompt further provides the major product types and their frequencies, sibling-level attribute contrasts, and a structured DATA CONTEXT containing the prefix metadata, attribute signatures, attribute distributions, parent statistics, and sibling differences. Query Generation Prompt. You are an expert analyst of hierarchical fashion semantic Table 6. Cluster fragmentation as a function of codebook size. 64128256 LevelClustersItems/Cl.ClustersItems/Cl.ClustersItems/Cl. D144823.9156285.0267166.5 D23,378219.210,37848.2817,3783.56 D323,95679.530,04513.4836,1221.23 ID clusters and a specialist in e-commerce search behavior. Your task is to understand what a given semantic prefix represents by analyzing: - High-precision attribute signatures (combinations of attributes strongly associated with the prefix) - Differences from its parent cluster (what makes it more specific) - Differences from sibling clusters (what makes it unique and non-overlapping) Based on this, infer the underlying product concept that this prefix encodes. Then generate natural-language user queries that a real shopper would type into a fashion recommendation system (e.g., Myntra, Amazon, Flipkart). These queries must: 1. Sound natural and human-like (NOT attribute lists like âcolor=black AND material=leatherâ). 2. Reflect real shopping intent (style, occasion, product type, etc.). 3. Be phrased properly. 4. Be EXHAUSTIVE: the set of queries must collectively retrieve EVERY item in the prefix group. - Generate at least one distinct query for each major product type (>5% share) listed in the coverage section. - Add a broad catch-all query to cover minor types (<5% share) that are not individually addressed. 5. Be EXCLUSIVE: no query should retrieve items from sibling prefixes or from outside this cluster. - For every query, verify it cannot be satisfied by the dominant attributes of each sibling listed in the exclusivity section. - Use target-only discriminating attributes as anchors wherever possible. 6. Prefer commonly used fashion terminology over internal attribute names. 7. Ground each query in the dominant phrases from description/productDisplayName to make the intent explicit. 8. Never use a single attribute value unless 100% of items share it; describe multi-value ranges collectively. Additional guidelines: - Use attribute signatures to ground the meaning. - Use parent deltas to explain what makes this cluster more specific than its parent. - Use sibling contrasts and the exclusivity constraints below to sharpen boundaries. Return ONLY a valid JSON object (no markdown fences, no extra text) with these keys: "insights": "<concise explanation of what this prefix represents>", "user_queries": ["<query 1>", "<query 2>", ...], // 6-12 queries, covering all major AND minor product types "delta_vs_parent": "<key ways this prefix is more specific than its parent>", "delta_vs_siblings": "<key distinguishing factors vs sibling prefixes>", "validation_checks": ["<rule 1>", ...], // one rule per major product type confirming coverage; one rule per sibling confirming exclusivity "coverage_map": "<product_type>": "<which query covers it>", ... // explicit mapping from each major product type to the query index that covers it EXHAUSTIVE COVERAGE REQUIREMENT -- this prefix contains N distinct product types: âą value: pct% (count items)â MUST be reachable by at least one query ... âą Minority types (<5% each): names â cover collectively with a broad query EXCLUSIVITY CONSTRAINTS -- your queries must NOT retrieve items from these sibling prefixes: Sibling prefix (size items) Contains: top_categories Sibling-only attributes (avoid these in target queries): attr=val (+Xpp in sibling); ... Target-only attributes (safe anchors for exclusivity): attr=val (+Xpp in target); ... ... For EVERY query you write, verify it would NOT match the dominant attributes of the sibling prefixes above. DATA CONTEXT: json: target_prefix.metadata, target_prefix.blueprint, target_prefix.discriminating_signatures, target_prefix.coverage_signatures, target_prefix.common_attributes, target_prefix.full_attribute_distribution, parent.record, parent.category_deltas_vs_parent, parent.numeric_deltas_vs_parent, sibling_comparisons Generation and validation. Queries are generated us- ing gemini-2.5-flash with temperature 0.2 and top- p = 0.8. The requested output contains an interpreta- tion of the cluster, 6â12 user queries, parent/sibling differ- ences, validation checks, and an explicit product-type-to- query coverage_map. Responses are parsed using struc- tured JSON extraction with a free-text fallback to preserve valid queries from partially malformed outputs. The re- turned coverage map is subsequently compared against the required product-type set, and uncovered types are recorded as coverage gaps. Prefix qualification. To ensure that query generation op- erates on semantically coherent groups, a prefix is pro- cessed only when its articleType purity is at least 0.80 and its cluster contains at least two items. Prefixes failing either criterion are excluded from generation. 8.0.3. Per-task Benchmark Statistics Table 1 in main paper reports aggregate statistics for Fashion-RQBENCH; here we break these down per evalu- ation task and report the relevance-label balance, attribute- tag taxonomy, and catalog coverage underlying each task. Table 7. Per-task composition of Fashion-RQBENCH. All queries are anchored at level-3 (leaf) clusters; Task 1 and Task 4 share the same 2,477 anchor queries (Task 4 paraphrases each one 2â10Ă). Task# Queries# Pairs/CandsCands per QueryLeaf ClustersQuery Length T1: Ranking2,47736,78212â15 (mean 14.8)4163â26 words (mean 10.1) T2: Relevance Classification2,47719,7317â8 (mean 8.0)4163â26 words (mean 10.1) T3: Sibling Discrimination1,9689,8014â5 (mean 5.0)3313â26 words (mean 10.1) T4: Query Robustness (Paraphrase)416 groups2,477 queries12â15 (mean 14.9)4162â10 paraphrases/group (mean 6.0) Relevance-label balance. Across the ranking and classifi- cation tasks, candidates are drawn with near-uniform cov- erage of the four relevance tiers (Exact / Sibling / Related / Irrelevant), avoiding trivial class-imbalance shortcuts: Table 8. Relevance-label distribution over (query, candidate) pairs. TaskExactSiblingRelatedIrrelevant T1: Ranking (36,782 pairs)20.2%25.9%26.9%26.9% T2: Classification (19,731 pairs)25.1%24.7%25.1%25.1% Attribute-tag taxonomy. Each ranking query is anno- tated with 0â8 attribute tags (mean 3.8) drawn from a 10-way taxonomy, with the following coverage over the 2,477 queries: product_type (2,475), gender (1,892), oc- casion (1,349), brand (945), style_tags (896), color (845), age_group (627), pattern (156), material (153), price_hint (80). Sibling discrimination. Each of the 1,968 queries pairs ex- actly one correct item against 3â4 distractor clusters (mean 3.89) sampled from neighboring leaf clusters, yielding 4â5 candidates per query â a harder, lower-cardinality setting than Task 1/2 designed to isolate fine-grained (sibling-level) confusability. Catalog coverage. The benchmark draws from the full 44,446-item catalog with broad metadata diversity: sam- pled items span masterCategory Apparel: 7,622, Acces- sories: 6,760, Footwear: 3,699, Personal Care: 1,570, Free Items: 73, Sporting Goods: 7 (Task 2 pairs) and gen- der Men: 8,996, Women: 8,913, Unisex: 1,279, Boys: 284, Girls: 259, with price ranging $90â$17,995 (median $1,199, mean $1,744). Across all four tasks, the benchmark touches 8,841 unique items (19.9% of the catalog) as either queriesâ anchor items or candidates/distractors. 8.0.4. Attribute-Level Relevance Scoring This section details the comparator m a (q,p) used for each attribute a â A(q) in weighted query relevance section of the main paper. All soft-text comparators use cosine sim- ilarity between Marqo-FashionCLIP [16] text embeddings of the compared strings, denoted sim(·,·). When a product lacks a structured value for an expressed query attribute, the comparator returns a neutral score of 0.5 rather than 0, so that missing catalog metadata is not conflated with an actual mismatch. Producttype. Ahierarchicalcomparisonagainst the productâs full category path masterCategory > subCategory >articleType.If the queryâs product-type string occurs as a substring anywhere in the lower-cased path,the score is floored at 0.85 (refined upward by sim against the leaf cat- egory when available).Otherwise, m a (q,p)= max max ââlevels sim(q a ,â),sim(q a , full path) ,i.e. the best similarity across the three individual hierarchy levels and the concatenated path. Gender. Categorical, after synonym normalization (e.g. "men", "man", "male", "boys"â male). Exact match after normalization scores 1.0; a query or product value of uni- sex against the other gender scores 0.7; any other mismatch scores 0.1. Agegroup. Ordinal,overtheorderedscale baby, toddler, kids, adult (Myntraâs ageGroup field, e.g. "Adults-Men", is reduced to this scale by keeping only the age segment).Exact match scores 1.0; oth- erwise the score decays linearly with ordinal distance, max(0, 1â 0.35| idx(q a )â idx(p a )|); a value outside the ordered scale scores 0.3. Color. Perceptual CIE76 âE distance in CIELAB [35] space between a query color and a product color, computed via a curated dictionary of âŒ90 named colors mapped to representative sRGB values. Identical color strings score 1.0 directly. When both values resolve to a known color, m a (q,p) = max(0, 1â âE/120), taking the maximum over all queryĂproduct color pairs. When either color is not in the dictionary, the comparator falls back to sim(q a ,p a ). If the product has no structured baseColor, the query color list is instead compared against the product description text. Material, occasion, pattern, style. These four attributes share one comparator shape: the productâs value list is parsed from the semi-structured articleAttributes field (fab- ric/pattern/fit/occasion keys) and, when that yields noth- ing, backfilled by matching a curated domain vocabu- lary (e.g. pattern terms like "floral print", "polka dot"; style/fit terms like "slim fit", "off shoulder"; occasion terms like "business casual", "loungewear") against the product title and description. Given the resulting product value list, m a (q,p) = max vâq a , uâp a sim(v,u) over all queryĂproduct value pairs. If the product has no values for the attribute at all, q a is instead compared against the raw description (and, for occasion/pattern/style, the full cat- egory path) text as a single string. Brand. Case-insensitive exact-or-substring match against the catalogâs brand field: m a (q,p) = 1.0 if either string contains the other, else 0.0. Price hint. Both query and product price are bucketed into three tiers (budget / mid-range / premium) using INR thresholds (< 700, 700â2000, > 2000, from the catalogâs list price). Matching tiers score 1.0; any mismatch scores 0.3 (rather than 0, since adjacent price tiers are a much smaller miss than an unrelated product). Optional visual grounding. When enabled,prod- uct_type, color, pattern, material, occasion, and style additionally probe a FashionCLIP image embedding of the product photo with a short attribute-specific text prompt (the attribute value plus the queryâs product type as context, e.g. "striped shirt"), and the attributeâs final score is max(m text a (q,p), m image a (q,p)) â letting either signal recover from a missing or noisy metadata field. Aggregation. For a given query, only the subset of at- tributes A(q) actually expressed in that queryâs structured representation contribute to its relevance score; the cor- responding cross-encoder weights w a are renormalized to sum to 1 over A(q) before the weighted sum is taken. A query with a multi-intent structured representation (e.g. a gift-set query implying several distinct product asks) is scored against every intent variant independently, and the best-matching variantâs score is kept â a retrieved product is credited if it satisfies at least one of the queryâs intents well. 8.0.5. Cross-Encoder-Based Attribute Importance Esti- mation We estimate attribute-level importance using a cross- encoder trained on real fashion search interactions. The cross-encoder serves as an independent relevance oracle, al- lowing us to derive attribute weights without relying on the retrieval systems or evaluation metrics being compared in Fashion-RQBENCH. We use two complementary estima- tion procedures: a controlled matched-pair ablation and an independent regression-based cross-check. Cross-encoder oracle. We train a cross-encoder using the ModernBERT-large [50] architecture on a combination of proprietary internal relevance dataset (prop-rel) and a fashion-filtered subset of ESCI [34]. The model predicts binary queryâproduct relevance, and we use its raw out- put logit as the relevance score.As ESCIâs US-locale subset spans fashion and non-fashion domains, we filter it to (query, product) pairs whose title or description con- tains fashion-specific keywords, treating products labeled exact for a given query as relevant and those labeled sub- stitute, complement, or irrelevant as irrelevant; a query may be paired with multiple products, yielding multiple query- product pairs. Combining this ESCI-filtered subset with prop-rel gives 539,849 training samples over 44,670 unique queries and 58,792 validation samples over 4,985 unique queries. Of the training queries, 14,467 (32.4%) come from the ESCI-filtered set and 30,203 (67.6%) from prop-rel; the validation split has an analogous 1,611 (32.3%) / 3,374 (67.7%) split. At the (query, product) sample level, ESCI accounts for 251,100 (46.5%) of training data and 27,876 (47.4%) of validation data, with the remainder â 288,749 (53.5%) train, 30,916 (52.6%) val â from prop-rel. We initialize the model from a checkpoint [28] pre- trained on sentence similarity, and represent product text as "name |brand |gender |color |materials |description". The model is trained with binary cross-entropy loss, mean pool- ing, label smoothing (Δ = 0.1), and batch size 512. On our internal test set of 2,353 unique queries and 12,619 query- product pairs (positive rate 0.632), with threshold=0.5, the cross-encoder achieves 0.818 ROC-AUC and a Brier score of 0.162, versus 0.233 for a baseline predicting the marginal positive rate for every pair. We validate the oracle on a manually constructed san- ity set of 100 queryâproduct examples covering multiple attributes. The model achieves a relevantâirrelevant prob- ability margin of 0.616 and pairwise ranking accuracy of 97.0%, exceeding our validation criteria of 0.15 and 75%, respectively. Attribute annotations. Item-level attributes are obtained independently from the public Myntra fashion catalog [2]. Product type, gender, age group, color, and usage/occasion are obtained from structured metadata, while material and pattern are extracted from the attribute text. Price is dis- cretized into budget, mid-range, and premium categories. Query-side attributes are obtained from naturally occurring enriched queries. Query and item attributes are canonical- ized using the same normalization procedure before com- puting attribute matches. Matched-pair ablation. For each attribute a, we con- struct matched âtwinâ item pairs that agree on relevant con- trol attributes but differ in a. Given a query specifying value v 1 of attribute a, the two items are scored by the cross- encoder and their logit difference is computed as â i = s CE (q i ,x 1 )â s CE (q i ,x 2 ).(8) We additionally score each twin pair against randomly sam- pled queries to obtain a null distribution. For each at- tribute, we report the mean logit difference, a twin-pair- clustered 95% bootstrap confidence interval using 2,000 re- samples, sign consistency, and a distinguishability-from- null diagnostic. Confidence tiers reflect both sample size and whether the confidence interval excludes zero. Results are shown in Table 9. Regression cross-check. As an independent validation, we construct queryâproduct pairs from real queries and en- code each attribute as f a â 1, 0, 0.5 for match, mis- match, and unspecified, respectively. After standardizing the ten attribute features, we fit a Ridge regression [46] to the cross-encoder logits using N = 6,205 pairs. The regression achieves R 2 = 0.490. Variance inflation fac- tors (VIFs) [26] between 1.00 and 1.10 indicate negligible multicollinearity among the attribute features. The regres- sion coefficients, VIFs, and resulting blended weights are reported in Table 10. Weight construction. For each method, negative esti- mates are clipped to zero and the resulting values are nor- malized to sum to one. We compute the Spearman rank cor- relation between the two sets of raw estimates as an agree- ment measure. The final attribute weight w a is obtained by averaging the two normalized weight vectors and renor- malizing the result to sum to one. These weights are sub- sequently used by the attribute-aware relevance metric in Section 3 of main paper. 8.0.6. Baseline Models and Implementation Details 8.1. Baseline Model Configurations We evaluate 12 retrieval baselines spanning four archi- tecture familiesâlexical, learned-sparse, multi-vector late- interaction, and dense single-vector modelsâfollowing the taxonomy of Kamalloo et al. [17]. All models are evaluated as frozen, off-the-shelf checkpoints without task-specific fine-tuning. The exact configurations used for each base- line are provided in Tables 11â13. â The base T5-large hidden size is 1024; the modelâs built-in projection head produces 768-dimensional embeddings. ⥠auto EOS handling was empirically verified against the tokenizer for Qwen3-Embedding-0.6B and the same policy was applied to the 4B and 8B checkpoints without indepen- dent re-verification. § For E5-Mistral-7B-Instruct, EOS is appended once manually after truncation. . 9. User Study Additional Details Query sources. We evaluated three blinded query sources, with 20 queries per source (N = 60): RQBENCH sampled our generated queries; CHAT, real user shop- ping queries filtered by majority-vote intent labels (â„ 2/3 LLM judges), scope, length, deduplication, and manual NSFW/PII screening; and LLM, unconditioned, persona- free shopping-query generations without dataset condition- ing or few-shot examples. For RQBENCH, 20 queries were randomly sampled (seed 42). CHAT and LLM were simi- larly sampled from curated pools of 50 using seed 42. All 60 queries were then randomly interleaved (seed 2026), as- signed hidden IDs, and presented without source labels. Rating protocol. Each query was rated independently on two 5-point Likert scales: naturalness (1 = very unnatu- ral/scripted, 5 = very natural) and intent clarity (1 = very Table 9. Controlled matched-twin-pair ablation using the cross-encoder relevance oracle. N denotes the number of queryâtwin-pair obser- vations.â is the mean cross-encoder logit difference, and the confidence interval is obtained by resampling twin pairs with replacement. Sign consistency denotes the fraction of observations having the same sign as â. AttributeNâ95% CISign Consistency Product Type5610.579[0.440, 0.713]0.78 Gender7300.730[0.625, 0.838]0.80 Age Group1290.737[0.537, 0.951]0.83 Color7860.926[0.828, 1.025]0.86 Material2590.589[0.384, 0.796]0.75 Occasion9520.072[â0.010, 0.153]0.53 Pattern2820.581[0.342, 0.830]0.71 Style Tags60.002[â0.274, 0.513]0.50 Brand6331.409[1.282, 1.537]0.90 Price Hint1720.242[0.046, 0.433]0.60 Table 10. Regression-based attribute importance and final blended weights. Ridge coefficients are estimated from cross-encoder log- its. VIF measures multicollinearity among attribute features. AttributeRidge Coef.VIFFinal w a Product Type0.4671.100.290 Gender0.1251.060.127 Age Group0.0061.000.066 Color0.1141.010.138 Material0.0261.020.063 Occasion0.0041.020.008 Pattern0.0211.000.060 Style Tags0.0241.020.012 Brand0.1771.020.211 Price Hint0.0061.000.024 Table 11. Lexical, learned-sparse, and multi-vector retrieval base- lines. ModelScoringKey Hyperparameters BM25 [36]Sparse lex- ical k 1 = 1.2, b = 0.75; English stopwords; Porter stemmer SPLADE++ EnsembleDis- til [10] Learned sparse max_seq_len=512; batchsize=64; max(log(1 + ReLU(·))) over MLM logits; vocabu- lary size=30,522 ColBERTv2 [37]Multi- vector MaxSim 128dimensions/token; query_maxlen=32; doc_maxlen=180; FP16 with FP32 MaxSim accu- mulation; batch size=128 vague, 5 = very clear/specific). Raters were explicitly in- structed to treat the dimensions independently. Table 12. Dense single-vector retrieval baselines with non-LLM backbones. ModelDim.PoolingMax Seq. Len Contriever-MSMARCO [14]768Mean512 RoBERTa-large [21]1024Mean(first + last layer)512 SimCSE-sup (RoBERTa-large) [52]1024 pooler_output (CLS)512 Sentence-T5-large [29]768 â Mean + Dense(1024â 768, tanh)256 Table 13. LLM-backbone embedding baselines. All models use last-token pooling, bfloat16, and â 2 normalization. Queries are prefixed with âGiven a fashion product search query, retrieve relevant product listings that match the queryâ, while documents are unprefixed. ModelParams.Dim.Max Seq. LenEOS Policy Qwen3-Embedding-0.6B [42]0.6B10248192auto ⥠Qwen3-Embedding-4B [42]4B25608192auto ⥠Qwen3-Embedding-8B [42]8B40968192auto ⥠E5-Mistral-7B-Instruct [48]7B40964096manual § SFR-Embedding-Mistral [27]7B40964096auto Participants and reliability. Thirty-two raters evaluated all 60 queries on both dimensions, yielding valid ratings after removing unparseable responses. Cronbachâs α was 0.977 for naturalness and 0.969 for intent clarity. Statistical analysis. For each dimension, we performed a KruskalâWallis omnibus test followed by pairwise Mannâ Whitney U tests for all three source pairs. Pairwise p-values were Holm-corrected, with α = 0.05. Qualitative examples. Representative naturalness scores (lowest/highest) were: RQBENCH, âCasual black or brown sunglasses.â(3.10) / âCeline Dion brown sunglasses.â (4.03); CHAT, âPut me in a dusty pink lilac saree with like 50% sheer flowy fabric and a light golden blouse with straps sweetheart neckline.â (2.52) / âSilk gown.â (4.23); and LLM, âCan you show me some elegant evening gowns for a black-tie event?â (2.77) / âShow me some stylish hand- bags for everyday use.â (4.33). Representative intent-clarity scores (lowest/highest) were: RQBENCH, I want to buy a perfume gift set.â (3.37) / Show me Celine Dion brown sunglasses.â (4.31); CHAT, Dark green pieces.â (1.66) / Put me in a dusty pink lilac saree with like 50% sheer flowy fabric and a light golden blouse with straps sweetheart neckline.â (4.00); and LLM, I need a dress that transitions from day to night.â (2.33) / I want a raincoat thatâs both stylish and waterproof.â (4.29). References [1] Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Peshawaria, Ranjita Naik, and Besmira Nushi.Kitab: Evaluating llms on constraint satisfaction for information retrieval. In Inter- national Conference on Learning Representations, pages 30664â30686, 2024. 4 [2] Param Aggarwal. Fashion product images dataset, 2019. 2, 3, 6, 13 [3] Kenan E. Ak, Joo Hwee Lim, Jo Yew Tham, and Ashraf A. Kassim. Efficient multi-attribute similarity learning towards attribute-based fashion search. In 2018 IEEE Winter Con- ference on Applications of Computer Vision (WACV), pages 1671â1679, 2018. 3 [4] Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Al- berto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 15292â15301. IEEE, 2023. 3 [5] Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Ro- drigo Nogueira. Inpars: Data augmentation for informa- tion retrieval using large language models. arXiv preprint arXiv:2202.05144, 2022. 3 [6] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhaes, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. Contrastive language and vi- sion learning of general fashion concepts. Scientific Reports, 12(1):18958, 2022. 3 [7] Lee J. Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297â334, 1951. 8 [8] Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming- Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. ArXiv, abs/2209.11755, 2022. 3 [9] Dengzhao Fang, Jingtong Gao, Chengcheng Zhu, Yu Li, Xi- angyu Zhao, and Yi Chang. Hid-vae: Interpretable gener- ative recommendation via hierarchical and disentangled se- mantic ids. arXiv preprint arXiv:2508.04618, 2025. 3 [10] Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and StĂ©phane Clinchant. Splade v2: Sparse lexical and ex- pansion model for information retrieval, 2021. 7, 8, 14 [11] Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for de- tection, pose estimation, segmentation and re-identification of clothing images. In 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 5332â 5340. IEEE, 2019. 3 [12] Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Au- tomatic spatially-aware fashion concept discovery. In 2017 IEEE international conference on computer vision (ICCV), pages 1472â1480. IEEE, 2017. 3 [13] Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36:31096â31116, 2023. 4 [14] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning, 2022. 7, 8, 14 [15] Kalervo JĂ€rvelin and Jaana KekĂ€lĂ€inen. Cumulated gain- based evaluation of ir techniques. ACM Trans. Inf. Syst., 20 (4):422â446, 2002. 4 [16] Myong Chol Jung and Jesse Clark. Marqo-FashionCLIP and Marqo-FashionSigLIP, 2024. 3, 5, 6, 11 [17] Ehsan Kamalloo,Nandan Thakur,Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin.Re- sources for brewing beir: Reproducible reference models and statistical analyses. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pages 1431â1440, New York, NY, USA, 2024. Association for Computing Machinery. 13 [18] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2021. 5 [19] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.Autoregressive image generation using residual quantization.In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11513â11522. IEEE, 2022. 1, 3, 5, 9 [20] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischin- ski. Data roaming and quality assessment for composed im- age retrieval. In Proceedings of the AAAI conference on ar- tificial intelligence, pages 2991â2999, 2024. 3 [21] Yinhan Liu, Myle Ott, Naman Goyal, Du Jingfei, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 7, 8, 14 [22] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xi- aoou Tang. Deepfashion: Powering robust clothes recog- nition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2016. 1, 3 [23] Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre- trained vision-and-language models. In 2021 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 2105â2114. IEEE, 2021. 3 [24] Zhe Ma, Jianfeng Dong, Zhongzi Long, Yao Zhang, Yuan He, Hui Xue, and Shouling Ji. Fine-grained fashion similar- ity learning by attribute-specific embedding network. In Pro- ceedings of the AAAI Conference on artificial intelligence, pages 11741â11748, 2020. 4 [25] Chaitanya Malaviya, Peter Shaw, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Quest: A retrieval dataset of entity-seeking queries with implicit set operations. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14032â14047, 2023. 3 [26] Donald W. Marquardt. Generalized inverses, ridge regres- sion, biased linear estimation, and nonlinear estimation. Technometrics, 12(3):591â612, 1970. 13 [27] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-mistral: Enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. 14 [28] D. Lee Miller. Modernce sts: An sts cross encoder model, 2025. 13 [29] Jianmo Ni, Gustavo HernĂĄndez Ăbrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence- T5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864â1874, Dublin, Ireland, 2022. Association for Computational Linguistics. 7, 8, 14 [30] Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho.Document expansion by query prediction.arXiv preprint arXiv:1904.08375, 2019. 3 [31] Hossein A Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, and Daniel Campos. Synthetic test collections for re- trieval evaluation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2647â2651, 2024. 3 [32] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36:10299â10315, 2023. 1, 2, 3 [33] Chandan K Reddy, LluĂs MĂ rquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A large-scale esci benchmark for improving product search. arXiv preprint arXiv:2206.06588, 2022. 3 [34] Chandan K. Reddy, LluĂs MĂ rquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A large-scale ESCI benchmark for improving prod- uct search. 2022. 1, 7, 12 [35] Alan R. Robertson. The cie 1976 color-difference formulae. Color Research and Application, 2:7â11, 1977. 7, 12 [36] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc, 2009. 3, 7, 8, 14 [37] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022. 7, 8, 14 [38] Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Ke- shavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, et al. Bet- ter generalization with semantic ids: A case study in ranking for recommendations. In Proceedings of the 18th ACM Con- ference on Recommender Systems, pages 1039â1044, 2024. 3 [39] Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval. Advances in Neural Information Pro- cessing Systems, 36:46345â46361, 2023. 3 [40] Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. Advances in neural information processing systems, 35:21831â21843, 2022. 1, 3 [41] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrit- twieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Er- ica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Sev- eryn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Itty- cheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, An- dras Orban, Fabian GĂŒra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ăgoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jar- rod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Lau- rent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Niko- lay Savinov, Ivo Danihelka, Becca Roelofs, AnaĂŻs White, Anders Andreassen, Tamara von Glehn, Lakshman Ya- gati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Na- talie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Al- cober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Ja- cob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gau- rav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Ko- rnakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunx- uan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob De- vlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, AdriĂ PuigdomĂšnech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, An- nie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, An- ton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sang- havi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Ke- hang Han, Peter Humphreys, Thibault Sellam, James Brad- bury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, SĂ©bastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hod- kinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexan- der Neitz, Zaheer Abbas, Sarah York, Machel Reid, Eliza- beth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozi Ì nska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedek- erke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swan- son, Dessie Petrova, Shashi Narayan, Arthur Guez, Sid- dhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai GimĂ©nez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Ab- hanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross McIlroy, Mario Lu Ë ci Ì c, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tac- chetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Se- bastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, So- phie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, RaphaĂ«l Lopez Kauf- man, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Ne- nad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Ange- los Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih- Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, SĂ©bastien Cevey, Zach Gle- icher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, LĂ©onard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, AdriĂ Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, VĂctor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thorn- ton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Ăa Ì glar ĂnlĂŒ, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGer- ald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mand- hane, Pratik Joshi, Anirudh Goyal, Matthew Tung, An- drew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Ne- manja Raki Ì cevi Ì c, Mostafa Dehghani, Fangyu Liu, Sid Mit- tal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Co- hen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz K Ìšepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Putta- gunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Sicil- iano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Mi- lad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mer- cado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Ălgmyr, TimothĂ©e Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Sel- van, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Hi- manshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Mari- nescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styrc, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih- Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ă Donnaile, SĂ©bastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wo- jciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen OâNeill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikala- pudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brov- elli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghor- bani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre- Louis Cedoz, Chenmei Li, Shiyuan Chen, NiccolĂČ Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappagan- thu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Ju- rdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ăhdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski, Fiona Macintosh, Roopali Vij, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur BraĆŸinskas, Andrei Sozan- schi, Matthew Hayes, HĂ©ctor FernĂĄndez Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante KĂ€rrman, PaweĆ Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jes- sica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irv- ing, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Pi- otr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung- Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhi- jit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Lu- owei Zhou, Jonathan Evens, William Isaac, Geoffrey Irv- ing, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Cas- taño, Irene Giannoumis, Wooyeol Kim, MikoĆaj Rybi Ì nski, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Al- lica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kush- man, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Padu- raru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKin- non, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, RĂ©mi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHos- sein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Ko- rchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Lint- ing Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont- Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, AmĂ©lie HĂ©liou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria- Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim PĂ”der, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mo- jtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gu- lati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christo- pher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane RiviĂšre, Alanna Walton, ClĂ©ment Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre- Chimoto, Hanna Klimczak-Pluci Ì nska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushk- ina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agar- wal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Ram- mohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Ju- lia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Miku- lik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Var- gas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moran- ski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan- Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, VĂt ListĂk, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Za- her, Paul MĂŒller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Fed- erico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Da- nenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pela- gatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subra- manya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2025. 5 [42] Qwen Team. Qwen3 embedding: Advancing text embedding and reranking. arXiv preprint arXiv:2506.05176, 2025. 3, 7, 8, 14 [43] Nandan Thakur, Nils Reimers, Andreas RĂŒcklĂ©, Abhishek Srivastava, and Iryna Gurevych.Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021. 3 [44] Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. Umbrela: Umbrela is the (open- source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06519, 2024. 1, 4 [45] Aaron van den Oord,Oriol Vinyals,and Koray Kavukcuoglu.Neural discrete representation learning. In NeurIPS, 2017. 3 [46] Wessel N. van Wieringen. Lecture notes on ridge regression, 2026. 13 [47] Sagar Vaze, Nicolas Carion, and Ishan Misra.Genecis: A benchmark for general conditional image similarity. In 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 6862â6872. IEEE, 2023. 4 [48] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.Improving text em- beddings with large language models.arXiv preprint arXiv:2401.00368, 2023. 7, 8, 14 [49] Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, et al. A neural corpus indexer for docu- ment retrieval. Advances in Neural Information Processing Systems, 35:25600â25614, 2022. 3 [50] Benjamin Warner, Antoine Chaffin, Benjamin ClaviĂ©, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gal- lagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Grif- fin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional en- coder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526â2547, Vienna, Austria, 2025. As- sociation for Computational Linguistics. 12 [51] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In 2021 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 11302â11312. IEEE, 2021. 1, 3 [52] Xing Wu, Chaochen Gao, Yipeng Su, Jizhong Han, Zhongyuan Wang, and Songlin Hu. Smoothed contrastive learning for unsupervised sentence embedding, 2022. 7, 8, 14 [53] Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang. Fashion captioning: Towards generating accurate descrip- tions with semantic rewards. In Computer Vision â ECCV 2020, pages 1â17, Cham, 2020. Springer International Pub- lishing. 3 [54] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.When and why vision- language models behave like bags-of-words, and what to do about it? 2022. 4 [55] Hansi Zeng, Chen Luo, Bowen Jin, Sheikh Muhammad Sar- war, Tianxin Wei, and Hamed Zamani. Scalable and effective generative information retrieval. In Proceedings of the ACM Web Conference 2024, pages 1441â1452, 2024. 1, 3 [56] Wenhao Zhang, Mengqi Zhang, Shiguang Wu, Jiahuan Pei, Zhaochun Ren, Maarten De Rijke, Zhumin Chen, and Pengjie Ren. Excluir: Exclusionary neural information re- trieval. In Proceedings of the AAAI conference on artificial intelligence, pages 13295â13303, 2025. 3 [57] Tianyu Zhu, Myong Chol Jung, and Jesse Clark. Generalized contrastive learning for multi-modal retrieval and ranking. In Companion Proceedings of the ACM on Web Conference 2025, pages 661â670, 2025. 3