Paper deep dive
SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
Andrea Scarinci, Virginia Negri, Brayan Impata, Suleiman Khan, Victor Martinez, Marcello Federico
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/18/2026, 8:32:08 AM
Summary
The paper introduces SynthAVE, a large-scale benchmark for e-commerce attribute value extraction (AVE) spanning 12,726 products across 229 categories, 792 attributes, and 4 languages. To address the cost of human labeling, the authors propose a multi-LLM arena validation framework using 21 judge configurations (7 model families x 3 prompts). Majority voting among these diverse models achieves 95.2% agreement with human experts (Cohen's kappa = 0.92), demonstrating that synthetic labeling combined with LLM-ensemble validation can provide high-quality, cost-effective ground truth for fine-tuning LLMs.
Entities (19)
Relation Signals (16)
LLM Arena â achievesagreementwith â Human Experts
confidence 95% · The majority vote ensemble agrees with human experts at Cohen's Îș = 0.92
LLM Arena â comprisesjudgesfrom â Nova Pro v1
confidence 95% · Nova (Amazon Artificial General Intelligence, 2024)... Nova Pro v1
LLM Arena â comprisesjudgesfrom â Gemma-3-27B
confidence 95% · Gemma (Gemma Team et al., 2025)... Gemma 3 27B
LLM Arena â comprisesjudgesfrom â Claude 3.5 Sonnet v2
confidence 95% · We selected 7 LLM families... Claude (Anthropic, 2024)... Claude 3.5 Sonnet v2
LLM Arena â comprisesjudgesfrom â gpt-oss-120b
confidence 95% · GPT (OpenAI et al., 2024)... GPT OSS 120B
LLM Arena â comprisesjudgesfrom â Mistral-Large-3
confidence 95% · Mistral (Mistral AI, 2025)... Mistral Large 3
LLM Arena â comprisesjudgesfrom â Qwen3-VL-235B
confidence 95% · Qwen (Bai et al., 2025)... Qwen3 VL 235B
LLM Arena â comprisesjudgesfrom â DeepSeek-R1
confidence 95% · DeepSeek (Guo et al., 2025)... DeepSeek R1
SynthAVE â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families $\times$ 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohen's $\kappa = 0.92$ (95.2% agreement), while individual judges show substantial inter-model agreement (Fleiss' $\kappa = 0.76$). This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review.
Tags
Links
- Source: https://arxiv.org/abs/2607.07469v1
- Canonical: https://arxiv.org/abs/2607.07469v1
Trouble viewing inline? Open PDF directly â
Full Text
56,593 characters extracted from source content.
Expand or collapse full text
SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation Andrea Scarinci1, Virginia Negri1, Brayan Impata1, Suleiman Khan1, Victor Martinez1, Marcello Federico1 1Amazon andscar, vrgngr, biimpata, suleimkh, vicmg, marcfede@amazon.com Abstract Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs (Negri et al., 2025), deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families Ă 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohenâs Îș=0.92Îș=0.92 (95.2% agreement), while individual judges show substantial inter-model agreement (Fleissâ Îș=0.76Îș=0.76). This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review. SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation Andrea Scarinci1, Virginia Negri1, Brayan Impata1, Suleiman Khan1, Victor Martinez1, Marcello Federico1 1Amazon andscar, vrgngr, biimpata, suleimkh, vicmg, marcfede@amazon.com 1 Introduction Fine-tuning and evaluating large language models (LLMs) for e-commerce applications demands massive volumes of high-quality labeled data. Specifically, predicting and evaluating product attribute values from unstructured catalog text (e.g., product titles, descriptions, bullet points) requires labeled examples that maintain representativeness across three key dimensions: product categories (e.g., electronics, clothing, furniture), attributes (e.g., color, material, dimensions), and languages. At catalog scaleâthousands of product categories, thousands of attributes, and multiple languagesâachieving reliable model performance requires hundreds of labeled examples per (product category Ă attribute Ă language) triplet. This combinatorial requirement translates to millions of annotations, making human labeling prohibitively costly. Prior work demonstrates the feasibility of using LLMs to generate synthetic labels for e-commerce attribute value extraction (Negri et al., 2025). However, deploying such approaches at industrial scale introduces a critical challenge: how to validate label quality efficiently across heterogeneous attributes and languages without exhaustive manual review. A scalable solution must integrate synthetic label generation with automated quality control mechanisms. This paper presents SynthAVE (Synthetic data for Attribute Value Extraction), a large-scale human-validated benchmark augmenting prior generation methodology (Negri et al., 2025) with scalable quality assurance. We introduce a multi-LLM auditing framework where each generated label is independently evaluated by 21 judge configurations (7 model families Ă 3 prompts), with final labels determined through majority voting. To mitigate systematic biases, we employ diverse models from different providers alongside varied prompting strategies. Ground truth for calibrating this system is established through human validation of 12,726 products. Empirical evaluation demonstrates that diverse models with different biases aggregate into highly reliable predictions: the ensemble achieves 95.2% agreement with human experts (Cohenâs Îș=0.92Îș=0.92) while enabling cost-effective validation at scale. Our main contributions are: âą SynthAVE: A human-validated benchmark of 12,726 products spanning 229 product categories, 792 attributes, and 4 languages for multilingual attribute extraction research.111The public release will additionally include English-language annotations produced using the same pipeline, extending coverage to 5 languages. âą Multi-LLM Validation Framework: An auditing approach employing majority voting across diverse judge configurations, achieving 95.2% agreement with human evaluation while nabling cost-effective validation at scale. 2 Related Work Information extraction (IE) from unstructured web text is a foundational capability for large-scale e-commerce systems and knowledge base construction. Common applications include product attribute extraction, entity population, catalog enrichment, and fact acquisition from heterogeneous online sources (Zhang et al., 2024; Martinez-Rodriguez et al., 2020; Dagdelen et al., 2024). Recent advances in large language models (LLMs) have substantially lowered the barrier to deploying flexible extraction systems across domains and schemas (Zhu et al., 2024). However, evaluating the quality of extracted information remains a central challenge, primarily due to the scarcity of high-quality human-annotated data and the scale at which modern IE systems operate. Classical evaluation methodologies for IE depend on manually annotated ground truth datasets and report performance using metrics such as precision, recall, and F1 score (Xu et al., 2024; Zhu et al., 2024). While effective in controlled research settings, this paradigm does not scale to real-world web and e-commerce applications. Annotation is expensive, time-consuming, and often requires domain expertise, particularly when extraction targets complex or evolving schemas (Deng et al., 2024; Hsu and Roberts, 2025). As a result, many production pipelines operate with limited or no gold-standard evaluation data, motivating the development of evaluation methods that function under weak or missing supervision. Existing benchmarks for product attribute extraction illustrate this tension between scale and quality. The Multi-source Attribute Value Extraction (MAVE) dataset (Yang et al., 2022) provides 2.2 million attribute-value annotations across 1,257 attributes from Amazon product profilesâachieving impressive scale through automated pipelines. However, MAVEâs quality assurance relies primarily on classifier-based filtering rather than comprehensive human validation of attribute-value annotations. Additionally, MAVE is limited to English and addresses extraction rather than verification tasks. These limitations motivate both the exploration of synthetic evaluation methods and the development of more rigorous validation frameworks. In response to these constraints, recent work has explored evaluation strategies that reduce or eliminate dependence on human-labeled ground truth. Seitl et al. (2024) propose an automatic IE evaluation framework based on synthetic ground truth generation, in which artificially constructed structured facts are injected into natural documents and systems are evaluated on their ability to recover them. Despite their scalability, synthetic evaluation methods introduce limitations related to realismâartificially injected facts may not fully capture the ambiguity, inconsistency, and noise present in naturally occurring e-commerce text (Negri et al., 2025). Another line of work investigates the use of LLMs to generate annotated datasets for downstream evaluation. By leveraging LLMs as annotators, it becomes possible to rapidly construct large quantities of labeled data where human annotation is infeasible (Mohta et al., 2023; Tan et al., 2024). In e-commerce contexts, this supports approximate evaluation of extraction pipelines across diverse product categories (NadaĆ et al., 2025). However, automatically generated annotations may reflect model-specific biases or systematic errors, necessitating careful validation and selective human oversight (De et al., 2025). Pipeline design choicesâdecomposing extraction into multiple stages, structured output formats, and normalization rulesâfurther improve performance (Jaradeh et al., 2023; Sabeh et al., 2024; Satyadharma et al., 2025), but complicate evaluation when standardized benchmarks are unavailable. 3 Task and Dataset Attribute Value Extraction Task. The attribute value extraction (AVE) task requires determining whether a given attribute value can be verified from unstructured product text. Given a productâs unstructured text (title, bullet points, description) and an attribute-value pair, the task is to assign one of three labels: CORRECT, INCORRECT, or UNKNOWN. We illustrate with a concrete example. Given the product title âPortable Laptop Stand, Adjustable Height, Aluminum Construction, Silver Finish, Compatible with 10-17 inch Devicesâ, the label depends on the attribute-value pair: (color,Silver)( color, Silver) is correct since the text states âSilver Finishâ; (color,Black)( color, Black) is incorrect as it contradicts the text; and (weight,1.2kg)( weight, 1.2kg) is unknown since no weight information is present. Training models to perform this three-way classification reliably requires labeled examples of all cases across diverse product categories, attributes, and languages. Our synthetic data generation pipeline produces such examples at scale, and our validation framework ensures their quality. Synthetic Data Generation. We constructed the dataset in two stages. First, we sampled real products from a large e-commerce catalog spanning 4 languages (Spanish, French, Italian, German), ensuring categories exist in all languages with minimum 30 products per category. Second, for each sampled product, we generated synthetic variants using the controlled generation methodology of Negri et al. (2025). This process creates a strict one-to-one correspondence: each synthetic product instance is paired with exactly one target attribute and one value slot. The generation pipeline produces all three label types: correct (value explicitly or implicitly supported by text), incorrect (text contradicts the value), and unknown (attribute cannot be determined from available text). Table 1: Dataset statistics by language ES FR IT DE Total products 3,159 3,501 3,007 3,059 % of dataset 24.8 27.5 23.6 24.0 Unique attributes 636 675 609 673 Label Distribution (%) CORRECT 48.6 48.8 49.1 48.7 INCORRECT 15.3 15.8 14.5 14.3 UNKNOWN 36.1 35.4 36.4 36.9 The synthetic dataset comprises 12,726 products across 229 product categories, covering 792 unique attributes and 2,607 distinct product-attribute combinations. The dataset spans major product domains including electronics, apparel, home & furniture, and sporting goods. Each category contains a minimum of 30 products, with larger categories such as NOTEBOOK_COMPUTER reaching 209 products. Categories are evaluated on domain-specific attributes (e.g., connectivity_technology for electronics, closure.type for apparel). Full category and attribute distributions are provided in Appendix B. Validation Challenge. While the synthetic pipeline achieves 92.6% accuracy in preliminary human analysis, this error rate is problematic for benchmark evaluationâmetrics computed against noisy ground truth yield misleading conclusions. Yet exhaustive human review of 12,726 products across 4 languages is prohibitively expensive. Section 4 presents our solution: a multi-LLM validation framework that achieves human-level accuracy at a fraction of the cost. 4 Multi-LLM Validation Framework Having defined the attribute value extraction task and dataset construction process, we now address the core challenge: validating synthetic labels at scale without exhaustive human review. Building upon established generation methods (Negri et al., 2025), we present a multi-LLM auditing framework that aggregates diverse model judgments into reliable quality estimates. Our framework treats LLM judges as independent evaluators whose collective judgment, when properly aggregated, can approximate human expert consensus. This requires careful attention to two foundational aspects: ensuring independence across judges, and establishing qualification criteria that parallel traditional human auditor standards. Judge Selection and Configuration. For aggregated judgments to be statistically meaningful, individual judges must provide genuinely independent assessments. We enforce this through diversity along two dimensions. Model family diversity. We selected 7 LLM families from different providers (Table 2): Claude (Anthropic, 2024), Nova (Amazon Artificial General Intelligence, 2024), GPT (OpenAI et al., 2024), Mistral (Mistral AI, 2025), DeepSeek (Guo et al., 2025), Qwen (Bai et al., 2025), and Gemma (Gemma Team et al., 2025). This ensures diversity across model scales and training objectives (general-purpose, reasoning-focused, multilingual), with outputs independent of family-specific biases. Table 2: LLM models evaluated in our arena. Model Family Model Name Anthropic Claude 3.5 Sonnet v2 Amazon Nova Pro v1 OpenAI GPT OSS 120B Mistral Mistral Large 3 DeepSeek DeepSeek R1 Qwen Qwen3 VL 235B Google Gemma 3 27B Prompt diversity. For each model, we developed three substantially different prompt versions (detailed in Appendix D). This prevents model outputs from reflecting prompt over-fitting or artificial consistency due to identical phrasing. This design yields 7Ă3=217Ă 3=21 independent judge configurations. Each configuration evaluates all 12,726 products across 4 languages (Spanish, French, Italian, German), producing 267,246 total judgments. Figure 1: Framework for multi-LLM judge configurations. Seven model families combined with three prompt variations yield 21 independent judges. Qualification Criteria. Beyond independence, each LLM judge must meet qualification criteria that parallel traditional standards for human auditors. General competency establishes baseline capability: models must meet minimum performance thresholds on established benchmarks, such as Global Average â„70%â„ 70\% on LiveBench (White et al., 2025), paralleling education or certification requirements for human auditors. Task-specific competency ensures domain relevance: models must demonstrate sufficient understanding of attribute verification through internal benchmark evaluation, analogous to subject matter expertise. Self-consistency addresses output reliability: given known variability in LLM outputs due to floating-point precision issues (Yuan et al., 2025), judges must achieve minimum consistency when generating responses for identical prompts. Instruction adherence ensures usable outputs: models must reliably follow specified output formats, paralleling structured reporting requirements for human evaluators. Evaluation Task. Judges perform the attribute consistency verification task defined in Section 3, classifying each sample as CORRECT, INCORRECT, or UNKNOWN with brief justification. Aggregation through Majority Voting. Given a panel of qualified, independent judges, we aggregate their outputs via majority voting across all 21 configurations. This aggregation mechanism yields several important properties. Higher inter-judge agreement correlates with higher prediction accuracy, as consensus signals clearer cases. Unanimous agreement indicates highest-confidence predictions, while split decisions flag samples warranting closer inspection. Most importantly, diverse judges with moderate individual agreement can achieve high ensemble accuracy through error cancellation. We therefore deliberately seek diversity among judges: moderate inter-model agreement (as measured by Fleissâ Îș) combined with high ensemble-human agreement (Cohenâs Îș) indicates successful diversityâindependent judges with different biases combining into robust collective predictions. Ground Truth Establishment. To evaluate our framework, we established ground truth labels through a disagreement-based annotation strategy that leverages the LLM arena to minimize human effort. Each synthetic sample has an original label assigned by the generation algorithm. We compared this label against the LLM arena majority vote and applied the following protocol: âą Agreement cases: When the majority vote matched the original synthetic label, the sample was accepted as correct without human review. âą Disagreement cases: When the majority vote contradicted the original label, the sample was flagged for manual review by domain experts, who determined the final ground truth label. To validate the agreement assumption, an expert annotator independently triaged a stratified random sample of 400 agreement cases (100 per language, comprising 100 unanimous and 300 mixed-agreement samples). The annotator overturned the majority vote in only 12/400 cases (3.0%, 95% CI [1.3%, 4.7%]). Critically, unanimous casesâwhere all 21 judges agreedâwere never overturned (0/100), and all 12 disagreements occurred exclusively in mixed-agreement samples (4.0% overturn rate). Among majority labels, UNKNOWN showed the highest overturn rate (5.0%) compared to CORRECT (2.2%) and INCORRECT (0.0%), consistent with UNKNOWN being the âhedgeâ vote that occasionally masks determinable cases. This confirms that agreement between the generation algorithm and diverse LLM judges serves as a reliable signal of label correctness, with the majority panel achieving ⌠97% accuracy (full triage analysis in Appendix F). 5 Results We first establish that our validation framework achieves human-level reliability, then present the final dataset features after cleaning. Overall Arena Performance. Table 3 summarizes agreement between LLM Arena majority voting and human expert evaluation. The arena achieves 95.2% overall agreement with human evaluators, with Cohenâs Îș=0.92Îș=0.92 indicating almost perfect inter-rater reliability (Landis and Koch, 1977). Performance is consistent across languages (94.0â96.4%). Individual judges show substantial inter-model agreement (Fleissâ Îș = 0.757), yet the majority vote ensemble achieves almost perfect agreement with humansâvalidating that diverse models with different biases aggregate into highly accurate predictions. Table 3: LLM Arena agreement with human evaluation by language Lang. Products Acc. Cohen Îș Fleiss Îș IT 3,007 96.4% 0.94 0.755 DE 3,059 95.3% 0.92 0.757 ES 3,159 94.7% 0.91 0.757 FR 3,501 94.0% 0.90 0.759 All 12,726 95.2% 0.92 0.757 Data Cleaning Performance. Table 4 summarizes the Arenaâs effectiveness as a data cleaning mechanism. The original synthetic pipeline achieves 92.6% accuracy (946 errors across 12,726 samples). The arena corrects 786 of these errors (83.1% error correction rate), improving accuracy to 95.0%. When framed as binary bad-label detection, the Arena achieves 98.0% precision and 95.2% recall (F1 = 96.6%)âwhen it flags a label as incorrect, it is almost always right. Table 4: LLM Arena data cleaning performance Metric Value Baseline Original synthetic accuracy 92.6% Original errors 946 Arena Correction Arena accuracy 95.0% Errors corrected 786 / 946 (83.1%) Net improvement +2.4% Bad Label Detection Precision 98.0% Recall 95.2% F1 Score 96.6% Per-Class Performance. Table 5 presents per-label metrics. The arena performs best on CORRECT (F1 = 97.3%), followed by UNKNOWN (F1 = 94.5%). INCORRECT labels prove most challenging (F1 = 89.2%), primarily due to lower recallâthe arena tends to classify borderline INCORRECT cases as UNKNOWN, a conservative behavior preferable in production where false negatives are less costly than false positives. Table 5: Per-class performance metrics (averaged across languages) Label Precision Recall F1-Score CORRECT 96.2% 98.5% 97.3% INCORRECT 91.5% 87.0% 89.2% UNKNOWN 95.0% 94.1% 94.5% Performance by Original Label Type. Stratifying by original synthetic label reveals where the arena adds most value (Table 6). Originally CORRECT labels already achieve 97.3% accuracy, with modest arena improvement (+1.2%). Originally INCORRECT labels show the largest gap: synthetic accuracy is only 79.3%, but the arena recovers to 88.7% (+9.4%), demonstrating particular effectiveness at correcting the most error-prone category. Table 6: Arena accuracy (%) by original synthetic label type Original Count Orig Acc Arena Acc Î CORRECT 5,980 97.3 98.5 +1.2 INCORRECT 2,240 79.3 88.7 +9.4 UNKNOWN 4,506 92.8 93.6 +0.8 Agreement Level and Accuracy. Accuracy correlates strongly with agreement level: unanimous consensus (36% of products) yields 100% accuracy, very high agreement (>>85%) yields 98.8%, while low-agreement cases (<<50%, 1.4% of products) achieve only 65.1%, suggesting these warrant human review. Full stratification is provided in Appendix A. Additional Analyses. No individual judge configuration outperformed the majority vote ensemble (95.0%), and performance varied more across model families than across prompt variations, underscoring the importance of model diversity. Figure 2 illustrates this: certain model families consistently form high-agreement clusters (Îșâ0.80Îșâ 0.80â0.900.90) while others show lower mutual agreement (Îșâ0.57Îșâ 0.57â0.650.65), with patterns invariant across languages. Language-specific cleaning shows Italian achieving the highest fix rate (90.2%) and French the lowest (76.1%). Error analysis reveals 80.5% of errors involve confusion with UNKNOWN, while direct CORRECT â INCORRECT misclassification is rare (19.5%). Full details are in Appendix A. Figure 2: Pairwise Cohenâs Îș between judge configurations (German). Within-family agreement is high (diagonal blocks), while cross-family diversity drives ensemble accuracy. Patterns hold across all languages (Appendix C). Cost Analysis. Total validation cost was $290.50 for 12,726 products ($22.83 per 1,000 products, 267,246 API calls).222Pricing based on commercial API rates as of June 2025. Detailed breakdowns are in Appendix G. The SynthAVE Dataset. The resulting dataset comprises 12,726 human-validated products spanning 229 product categories, 792 attributes, and 4 languages (47.7% CORRECT, 15.8% INCORRECT, 36.5% UNKNOWN). All samples carry quality guarantees through LLM-human consensus (97% verified accuracy via expert triage of 400 agreement cases; Appendix F) or direct expert annotation. Unlike MAVE (Yang et al., 2022), SynthAVE provides comprehensive human validation, explicit UNKNOWN labels, and multilingual coverage. The dataset and per-judge predictions will be publicly released. 6 Conclusion We presented SynthAVE, a large-scale human-validated dataset and validation framework for synthetic e-commerce data. Building on prior work in synthetic data generation (Negri et al., 2025), we introduced an LLM-arena framework that enables scalable quality assurance through multi-model evaluation. Our evaluation yields three key findings. First, the LLM arena achieves 95% agreement with human evaluation (Cohenâs Îș = 0.92) across 12,726 products, 229 product categories, 792 attributes, and 4 languages. Second, diversity enables accuracy: individual judges show only substantial agreement (Fleissâ Îș = 0.76), yet the ensemble outperforms any single modelâwith unanimous consensus reaching 100% accuracy. Third, expert triage of 400 agreement cases confirms that the majority panel is overturned only 3% of the time (0% for unanimous cases), validating the disagreement-based annotation strategy. The arena corrects 83.1% of synthetic labeling errors at $0.02 per product, reserving human review only for low-agreement cases. These results support adopting multi-LLM evaluation protocols that incorporate prompt and model diversity, moving beyond single-model approaches. The SynthAVE dataset will be publicly released to support future research in scalable data quality assurance for attribute value extraction. Limitations Several limitations warrant discussion. First, our evaluation covers four European languages with shared Latin script; generalization to languages with different scripts (e.g., Chinese, Arabic), morphological structures, or lower LLM proficiency requires additional validation. Similarly, the framework was evaluated on e-commerce attribute verificationâa task with relatively objective ground truthâand may perform differently on more subjective evaluation tasks. Second, the framework depends on the availability of diverse, high-capability LLMs. As model families evolve, retire, or change pricing, the specific judge configurations may require updating. The finding that some model families significantly underperform others (accuracy ranging from 76% to 93%) raises questions about minimum capability thresholds for judge inclusion. Third, our ground truth construction relied on a disagreement-based annotation strategy where human review was triggered only when the LLM ensemble disagreed with synthetic labels. Expert triage of 400 stratified agreement cases validated this approach (3.0% overturn rate, 0% for unanimous cases), though it does not provide traditional inter-annotator agreement metrics among human experts. Finally, while the framework is designed for offline validation rather than real-time inference, the latency of running 21 judge configurations may still be prohibitive for rapid iteration cycles. Future work could explore distilling ensemble judgments into smaller, specialized models for faster evaluation. Ethics Statement SynthAVE is constructed from synthetic product data generated using the methodology of Negri et al. (2025), with seed products sampled from commercial e-commerce catalogs. All brand names, model identifiers, and potentially identifying product information were anonymized during generation to prevent privacy concerns and mitigate LLM biases from prior knowledge of specific productsâno real customer data or personally identifiable information is present in the released dataset. Human validation was performed by domain experts specialized in e-commerce catalog systems, who established ground truth labels through careful review of disagreement cases between the synthetic generation pipeline and LLM arena predictions. SynthAVE is designed as a test set optimized for evaluation coverage: it spans 2,598 product type-attribute combinations with a minimum of 30 samples per combination, prioritizing breadth over the volume typically required for training. We encourage researchers to leverage this benchmark for downstream applications including validating catalog quality assurance systems, improving attribute coverage in product catalogs, developing robust multilingual extraction methods, and advancing LLM-as-judge evaluation methodologies. We hope SynthAVE contributes to more accurate and reliable e-commerce information systems that benefit both businesses and consumers. References Amazon Artificial General Intelligence (2024) The Amazon Nova family of models: technical report and model card. Technical report Amazon. External Links: Link Cited by: §4. Anthropic (2024) The Claude 3 model family: Opus, Sonnet, Haiku. Note: https://w.anthropic.com/news/claude-3-familyAccessed: 2024-03-14 Cited by: §4. S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-VL technical report. External Links: 2511.21631 Cited by: §4. J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. S. Rosen, G. Ceder, K. A. Persson, and A. Jain (2024) Structured information extraction from scientific text with large language models. Nature Communications 15 (1), p. 1418. Cited by: §2. S. De, D. K. Sanyal, and I. Mukherjee (2025) Fine-tuned encoder models with data augmentation beat ChatGPT in agricultural named entity recognition and relation extraction. Expert Systems with Applications 277, p. 127126. Cited by: §2. S. Deng, Y. Ma, N. Zhang, Y. Cao, and B. Hooi (2024) Information extraction in low-resource scenarios: survey and perspective. In 2024 IEEE International Conference on Knowledge Graph (ICKG), Vol. , p. 33â49. External Links: Document Cited by: §2. Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. RamĂ©, M. RiviĂšre, et al. (2025) Gemma 3 technical report. External Links: 2503.19786 Cited by: §4. D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645 (8081), p. 633â638. External Links: Document Cited by: §4. E. Hsu and K. Roberts (2025) Leveraging large language models for knowledge-free weak supervision in clinical natural language processing. Scientific Reports 15 (1), p. 8241. Cited by: §2. M. Y. Jaradeh, K. Singh, M. Stocker, A. Both, and S. Auer (2023) Information extraction pipelines for knowledge graphs. Knowledge and Information Systems 65 (5), p. 1989â2016. Cited by: §2. J. R. Landis and G. G. Koch (1977) The measurement of observer agreement for categorical data. Biometrics, p. 159â174. Cited by: §5. J. L. Martinez-Rodriguez, A. Hogan, and I. Lopez-Arevalo (2020) Information extraction meets the semantic web: a survey. Semantic Web 11 (2), p. 255â335. Cited by: §2. Mistral AI (2025) Introducing Mistral 3: Mistral Large 3 and the Mistral 3 model family. Note: https://mistral.ai/news/mistral-3Accessed: 2026-02-04 Cited by: §4. J. Mohta, K. Ak, Y. Xu, and M. Shen (2023) Are large language models good annotators?. In Proceedings on âI Canât Believe Itâs Not Better: Failure Modes in the Age of Foundation Modelsâ at NeurIPS 2023 Workshops, Proceedings of Machine Learning Research, Vol. 239, p. 38â48. Cited by: §2. M. NadaĆ, L. DioĆan, and A. Tomescu (2025) Synthetic data generation using large language models: advances in text and code. External Links: 2503.14023 Cited by: §2. V. Negri, V. M. GĂłmez, S. A. Balanya, and S. Rajaram (2025) Attribute-aware controlled product generation with LLMs for e-commerce. External Links: 2601.04200 Cited by: §1, §1, §2, §3, §4, §6, Ethics Statement. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2024) GPT-4 technical report. External Links: 2303.08774 Cited by: §4. K. Sabeh, M. Kacimi, J. Gamper, R. Litschko, and B. Plank (2024) Exploring large language models for product attribute value identification. External Links: 2409.12695 Cited by: §2. S. Satyadharma, F. Sheikholeslami, S. Kaul, A. U. Batur, and S. A. Khan (2025) Auto prompting without training labels: an LLM cascade for product quality assessment in e-commerce catalogs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, S. Potdar, L. Rojas-Barahona, and S. Montella (Eds.), Suzhou, China, p. 937â953. External Links: Link, Document Cited by: §2. F. Seitl, T. KovĂĄĆĂk, S. Mirshahi, J. KryĆĄtĆŻfek, R. Dujava, M. OndreiÄka, H. Ullrich, and P. Gronat (2024) Assessing the quality of information extraction. External Links: 2404.04068 Cited by: §2. Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu (2024) Large language models for data annotation and synthesis: a survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 930â957. External Links: Link, Document Cited by: §2. C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, et al. (2025) LiveBench: a challenging, contamination-limited LLM benchmark. In Proceedings of the International Conference on Learning Representations, Cited by: §4. D. Xu, W. Chen, W. Peng, C. Zhang, T. Xu, X. Zhao, X. Wu, Y. Zheng, Y. Wang, and E. Chen (2024) Large language models for generative information extraction: a survey. Frontiers of Computer Science 18 (6), p. 186357. Cited by: §2. L. Yang, Q. Wang, Z. Yu, A. Kulkarni, S. Sanghai, B. Shu, J. Elsas, and B. Kanagal (2022) MAVE: a product dataset for multi-source attribute value extraction. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM â22, New York, NY, USA, p. 1256â1265. External Links: Document Cited by: §2, §5. J. Yuan, H. Li, X. Ding, W. Xie, Y. Li, W. Zhao, K. Wan, J. Shi, X. Hu, and Z. Liu (2025) Understanding and mitigating numerical sources of nondeterminism in LLM inference. External Links: 2506.09501 Cited by: §4. Y. Zhang, M. Zhong, S. Ouyang, Y. Jiao, S. Zhou, L. Ding, and J. Han (2024) Automated mining of structured knowledge from text in the era of large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 6644â6654. Cited by: §2. Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, and N. Zhang (2024) LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities. World Wide Web 27 (5), p. 58. Cited by: §2, §2. Appendix A Detailed Results A.1 Agreement Level and Accuracy Table 7 shows arena accuracy stratified by the proportion of judges agreeing on the majority vote. Table 7: LLM Arena accuracy by model agreement level Agreement Level Products Accuracy Low (<<50%) 172 65.1% Medium (50â70%) 1,198 79.5% High (70â85%) 1,398 91.1% Very High (>>85%) 9,958 98.8% Unanimous (100%) 4,612 100.0% A.2 Data Cleaning Performance by Language Table 8 presents detailed cleaning performance by language, including fix rates and regression rates. Table 8: Data cleaning performance by language Lang. Orig Acc Arena Acc Fix Regr. IT 91.9% 96.4% 90.2% 3.0% DE 93.8% 95.3% 93.2% 4.6% ES 92.5% 94.7% 75.8% 3.8% FR 92.1% 94.0% 76.1% 4.5% A.3 Error Pattern Distribution Table 9 breaks down the arenaâs 630 errors by confusion pattern. The dominant pattern is INCORRECT â UNKNOWN (27.5%), indicating conservative abstention. Confusion involving UNKNOWN accounts for 80.5% of all errors, while direct CORRECT â INCORRECT misclassification is rare (19.5%). Table 9: Error pattern distribution (Human â Arena) Error Pattern Count % of Errors INCORRECT â UNKNOWN 173 27.5% UNKNOWN â CORRECT 147 23.3% UNKNOWN â INCORRECT 128 20.3% INCORRECT â CORRECT 88 14.0% CORRECT â UNKNOWN 59 9.4% CORRECT â INCORRECT 35 5.6% Total Errors 630 100% Appendix B Dataset Statistics This appendix provides comprehensive statistics for the SynthAVE test set we release with this paper. B.1 Overall Scale âą Total products: 12,726 unique products âą Product categories: 229 distinct types âą Unique attributes: 792 across all languages âą Product-attribute combinations: 2,607 unique pairs âą Average entries per combination: 4.9 âą Languages: 4 (Spanish, French, Italian, German) âą Products per language: 3,007â3,501 âą Judge configurations: 21 (7 models Ă 3 prompts) B.2 Label Distribution Table 10: Overall label distribution (majority vote) Label Count % CORRECT 6,214 48.8 UNKNOWN 4,602 36.2 INCORRECT 1,910 15.0 Total 12,726 100.0 Table 10 presents the overall label distribution based on majority vote across all judges. B.3 Product Category Distribution Table 11 presents the top 20 product categories by volume. Table 11: Top 20 product categories by count (of 229 total) Category Count % NOTEBOOK_COMPUTER 209 1.6 CELLULAR_PHONE 190 1.5 HANDBAG 180 1.4 SHOES 140 1.1 PRINTER 138 1.1 WATCH 133 1.0 CAMERA_DIGITAL 127 1.0 CHAIR 123 1.0 SECURITY_CAMERA 122 1.0 WEARABLE_COMPUTER 121 1.0 DRESS 118 0.9 LIGHT_FIXTURE 115 0.9 CABINET 111 0.9 KICK_SCOOTER 110 0.9 BACKPACK 110 0.9 VIDEO_PROJECTOR 108 0.8 BRA 107 0.8 PERSONAL_COMPUTER 107 0.8 HEADPHONES 107 0.8 MONITOR 106 0.8 âŠ209 additional categories B.4 Attribute Distribution Table 12 presents the top 20 attributes by frequency. Table 12: Top 20 attributes by count (of 792 total) Attribute Count % closure.type 296 2.3 frame.material 290 2.3 brand 237 1.9 outer.material 185 1.5 material 170 1.3 display.type 163 1.3 color 132 1.0 waist.size 121 1.0 battery.average_life 115 0.9 neck.neck_style 115 0.9 handle.material 111 0.9 container.type 108 0.8 top.material 107 0.8 chest.size 106 0.8 sleeve.type 100 0.8 display.size 88 0.7 special_feature 87 0.7 size 86 0.7 weight_capacity.maximum 84 0.7 metal_type 81 0.6 âŠ772 additional attributes B.5 Top Product-Attribute Combinations Table 13 presents the most frequent product-attribute combinations in the dataset. Table 13: Top 20 product-attribute combinations (of 2,607 total) Product Type Attribute N RADIO display.type 27 THERMOMETER outer.material 25 ELEC._CIGARETTE battery.power 25 BLANKET blanket_form 24 SUITCASE handle.type 24 ENVELOPE closure.type 23 CLOTHES_RACK frame.material 23 AIMING_SCOPE has_night_vision 23 WALKING_STICK base.material 22 SEASONING container.type 22 HANDBAG material 22 POWER_BANK battery.cell_comp 22 INCONT._PROT. incont._prot._type 21 OUTBUILDING frame.material 21 WATCH_BAND watch_band_design 21 PICTURE_FRAME frame.material 21 SELF_BAL._VEH. wheel.size 20 WALLET wallet_comp._type 20 WATCH dial.color 20 SUNGLASSES polarization_type 20 B.6 Per-Language Statistics Table 14 provides detailed per-language statistics including unique product types, attributes, and combinations. Table 14: Detailed statistics by language ES FR IT DE Total entries 3,159 3,501 3,007 3,059 Product types 227 229 225 229 Attributes 636 675 609 673 Combinations 1,443 1,505 1,267 1,483 Majority Vote Counts CORRECT 1,536 1,710 1,477 1,491 INCORRECT 482 553 437 438 UNKNOWN 1,141 1,238 1,093 1,130 B.7 Ground Truth Distribution For human validation, we collected ground truth labels for all entries. Table 15 shows the distribution. Table 15: Ground truth label distribution (human-verified) Ground Truth Count % CORRECT 6,073 47.7 UNKNOWN 4,645 36.5 INCORRECT 2,008 15.8 Total 12,726 100.0 Appendix C Inter-Judge Agreement Heatmaps by Language This appendix presents pairwise agreement heatmaps for all four languages. Figure 2 in the main text shows German (Cohenâs Îș). Below we provide the complete set for all languages and both agreement metrics. C.1 Cohenâs Kappa Heatmaps Figures 3, 4, 5 Figure 3: Pairwise Cohenâs Îș between judge configurations for Italian. Clustering patterns are consistent with other languages, with the same model families forming high-agreement and low-agreement groups. Figure 4: Pairwise Cohenâs Îș between judge configurations for French. Despite French showing the lowest overall arena accuracy (94.0%), inter-judge agreement patterns remain consistent with other languages. Figure 5: Pairwise Cohenâs Îș between judge configurations for Spanish. C.2 Raw Agreement Heatmaps Figures 6, 7, 8, 9 Figure 6: Pairwise raw agreement between judge configurations for Spanish. Agreement ranges from 75% to 96% (within-family pairs). Figure 7: Pairwise raw agreement between judge configurations for German. Agreement ranges from approximately 75% (cross-family pairs with lower-performing models) to 96% (within-family pairs). Figure 8: Pairwise raw agreement between judge configurations for Italian. Italian shows the highest overall agreement, consistent with its leading arena accuracy (96.4%). Figure 9: Pairwise raw agreement between judge configurations for French. Appendix D Evaluation Prompt Templates This appendix provides the complete prompt templates used for our LLM Arena evaluation across all three prompt versions. Each version maintains the same evaluation task (attribute consistency verification) while employing different structural approaches and reasoning strategies to ensure IID generation requirements. D.1 Prompt Version 1: Direct Instruction You are an expert product attribute evaluator. Your task is to determine whether a generated attribute label is correct, incorrect, or unknown based on the provided product information. PRODUCT INFORMATION: ID: id | Category: category | Title: title Product Description: description Bullet Points: bullet_points ATTRIBUTE: attribute | GENERATED LABEL: generated_label TASK: Evaluate whether the generated label "generated_label" is accurate for the attribute "attribute" based on the product information provided. Note that some product information (description or bullet points) may not be available. RESPONSE OPTIONS: - CORRECT: The generated label accurately reflects the productâs attribute based on the available information - INCORRECT: The generated label is clearly wrong based on the product information - UNKNOWN: Cannot determine if the label is correct or incorrect from the available product information (including cases where insufficient information is provided) RESPONSE FORMAT: You must respond with exactly one of: "CORRECT", "INCORRECT", or "UNKNOWN". After your choice, in a new line, provide a brief explanation (1-2 sentences) of your reasoning. Example response format: CORRECT The product description explicitly mentions this attribute value. Figure 10: Prompt Version 1: Direct instruction format with minimal structure. D.2 Prompt Version 2: XML-Structured with Examples You are an e-Commerce data analyst specializing in product attribute verification and quality assurance. Assess whether a generated attribute label correctly represents the specified product attribute based on available listing information. <examples> <example> ID: B08N5WRWNW | Category: LUGGAGE Title: <brandname> <makename> Hardside Expandable Luggage with Spinner Wheels, Brushed Anthracite, Carry-On 20-Inch Description: Durable polycarbonate shell construction with scratch-resistant texture Bullet Points: * 100% Polycarbonate shell * Four multidirectional spinner wheels * TSA-approved combination lock Attribute Name: material_type | Generated Label: Polycarbonate Your Response: CORRECT The description and bullet points explicitly confirm polycarbonate shell construction, validating the material type label. </example> <example> ID: B07XYZ123A | Category: CLOTHING Title: Menâs Cotton Blend Casual T-Shirt Blue Medium | Description: [Not provided] | Bullet Points: [Not provided] Attribute Name: sleeve_length | Generated Label: Long Sleeve Your Response: UNKNOWN Title mentions t-shirt but provides no sleeve length information, and additional product details are unavailable. </example> <example> ID: B09ABC456D | Category: ELECTRONICS Title: Wireless Bluetooth Headphones with Noise Cancellation Description: Premium over-ear headphones featuring active noise cancellation technology Bullet Points: * 30-hour battery life * Bluetooth 5.0 connectivity * Foldable design for portability Attribute Name: connectivity_technology | Generated Label: USB-C Your Response: INCORRECT Product clearly indicates Bluetooth connectivity, directly contradicting the USB-C label assignment. </example> </examples> <inputs> <id>id</id> <category>category</category> <item_name>title</item_name> <description>description</description> <bullet_points>bullet_points</bullet_points> <attribute_name>attribute</attribute_name> <generated_label>generated_label</generated_label> </inputs> <instructions> Evaluate the generated attribute label by: 1. Reviewing all available product information for evidence relating to the specified attribute 2. Determining if the generated label matches, conflicts with, or cannot be verified from the product data 3. Handling incomplete information appropriately when description or bullet points are missing Classify as: - CORRECT: Label accurately reflects attribute based on available evidence - INCORRECT: Label contradicts information in product details - UNKNOWN: Insufficient data to verify label accuracy Response format: Respond with exactly one of "CORRECT", "INCORRECT", or "UNKNOWN", followed by a brief explanation. </instructions> Figure 11: Prompt Version 2: XML-structured format with in-context examples. D.3 Prompt Version 3: Role-Based with Systematic Guidelines ## Role You are an e-Commerce data analyst specializing in product attribute verification and quality assurance. Your task is to assess whether a generated attribute label correctly represents the specified product attribute based on available listing information. ## Task Overview Evaluate the accuracy of a generated attribute label by analyzing product listing data and determining if the label is supported, contradicted, or cannot be verified by the available information. ## Evaluation Guidelines Follow these steps systematically: 1. **Review Available Evidence**: Examine all provided product information (title, description, bullet points) for any evidence relating to the specified attribute. 2. **Compare Label Against Evidence**: Determine whether the generated label: - Matches the product information (explicitly or implicitly) - Contradicts the product information - Cannot be verified due to insufficient data 3. **Handle Missing Information**: When description or bullet points are unavailable or incomplete, acknowledge the limitation in your assessment. ## Classification Categories - **CORRECT**: The label accurately reflects the attribute based on available evidence in the product listing - **INCORRECT**: The label contradicts or misrepresents information found in the product details - **UNKNOWN**: Insufficient data exists to verify the labelâs accuracy ## Examples ### Example 1: CORRECT - ID: B08N5WRWNW | Category: LUGGAGE - Title: <brandname> <makename> Hardside Expandable Luggage with Spinner Wheels, Brushed Anthracite, Carry-On 20-Inch - Description: Durable polycarbonate shell construction with scratch-resistant texture - Bullet Points: * 100% Polycarbonate shell * Four multidirectional spinner wheels * TSA-approved combination lock - Attribute: material_type | Label: Polycarbonate Assessment: CORRECT - Description and bullet points explicitly confirm polycarbonate shell construction. ### Example 2: UNKNOWN - ID: B07XYZ123A | Category: CLOTHING - Title: Menâs Cotton Blend Casual T-Shirt Blue Medium | Description: [Not provided] | Bullet Points: [Not provided] - Attribute: sleeve_length | Label: Long Sleeve Assessment: UNKNOWN - Title mentions t-shirt but provides no sleeve length information. ### Example 3: INCORRECT - ID: B09ABC456D | Category: ELECTRONICS - Title: Wireless Bluetooth Headphones with Noise Cancellation - Description: Premium over-ear headphones featuring active noise cancellation technology - Bullet Points: * 30-hour battery life * Bluetooth 5.0 connectivity * Foldable design for portability - Attribute: connectivity_technology | Label: USB-C Assessment: INCORRECT - Product clearly indicates Bluetooth connectivity, contradicting USB-C label. ## Product Information to Evaluate - **ID**: id | **Category**: category | **Title**: title - **Description**: description - **Bullet Points**: bullet_points - **Attribute Name**: attribute | **Generated Label**: generated_label ## Response Format Provide your assessment as: CORRECT/INCORRECT/UNKNOWN followed by a brief explanation (1-2 sentences). Figure 12: Prompt Version 3: Role-based format with systematic guidelines and markdown structure. Appendix E Evaluation Settings E.1 Input Format "id": "X", "category": "FILE_FOLDER", "text_fields": "title": "<brand>, Lot de 100 pochettes...", "features": [ "Les pochettes A4 en polythylne...", "Pochettes transparentes..." ], "description": "Ces chemises en plastique..." , "attribute_of_interest": "name": "tab.position", "value": "Ct" Figure 13: Example input format for LLM judges. Models receive structured product data with target attribute to verify. Figure 13 shows the input format provided to each judge. E.2 Model Configuration All models use a low temperature (0.1) to ensure consistent, reproducible outputs. We leave default settings for Top-p and Top-k parameters. For OpenAIâs GPT OSS reasoning model, we set reasoning_effort="low" to balance computational cost with performance. This parameter controls the depth of the modelâs internal chain-of-thought reasoning before producing a final answer. Even at the âlowâ setting, the model demonstrates strong performance on our classification task while significantly reducing inference costs compared to higher reasoning effort levels. Model-Specific Formats. Each model family requires different API message formats: âą Claude: Uses Anthropicâs message format with anthropic_version specification âą Nova: Requires nested content structure with explicit text fields âą OpenAI: Standard chat completion format with with reasoning_effort parameter âą Mistral: Uses instruction-tuned format with [INST] tags âą DeepSeek/Qwen3/Gemma: Standard chat message format E.3 Response Parsing Models return responses in different formats requiring specialized parsing: âą Standard format (Claude, Nova, Mistral, DeepSeek, Qwen3, Gemma): First line contains the classification label (CORRECT, INCORRECT, or UNKNOWN), followed by reasoning on subsequent lines. âą OpenAI reasoning format: Response contains <reasoning>...</reasoning> tags encapsulating the modelâs chain-of-thought, followed by the final classification label. The parser extracts both components separately. Invalid or unparseable responses default to UNKNOWN with the full response content logged for debugging. E.4 Evaluation Pipeline For each product and judge configuration: 1. Input formatting: Product data formatted according to model-specific API requirements 2. API invocation: Model called via AWS Bedrock with configured parameters 3. Response parsing: Extract classification and reasoning from model output, handling format variations 4. Validation: Verify response contains valid classification (CORRECT, INCORRECT, or UNKNOWN); invalid responses default to UNKNOWN 5. Error handling: API failures and malformed responses are caught and logged, with graceful degradation to UNKNOWN 6. Storage: Save classification, reasoning, model ID, prompt version, and timestamp All models were accessed through AWS Bedrockâs unified API, ensuring consistent infrastructure across different model providers. The pipeline processes 21 judge configurations (7 models Ă 3 prompt variants) per product systematically. Appendix F Human Triage of Agreement Cases To validate the assumption that agreement between the synthetic generation pipeline and the LLM arena majority vote implies label correctness, an expert annotator independently triaged a stratified random sample of 400 agreement cases. The sample was balanced across locales (100 per language) and stratified by agreement level (100 unanimous, 300 mixed-agreement). For each sample, the annotator reviewed the product text, attribute-value pair, and majority vote label, then recorded whether they agree with the majority, disagree (i.e., would overturn it), or are unsure. Only clear disagreements count as overturns in the analysis below; unsure cases (6/400) are excluded from the overturn rate. Overall Results. The annotator disagreed with the majority vote in only 12 out of 400 cases (3.0%), with a 95% confidence interval of [1.3%, 4.7%] (margin of error ± 1.7%). Table 16: Expert triage overturn rates by slice Slice Overturn 95% CI n Overall 3.0% [1.3%, 4.7%] 400 Mixed samples 4.0% [1.8%, 6.2%] 300 Unanimous samples 0.0% â 100 Majority = UNKNOWN 5.0% [1.6%, 8.3%] 161 Majority = CORRECT 2.2% [0.1%, 4.4%] 181 Majority = INCORRECT 0.0% â 58 Breakdown by Locale. Table 17: Expert triage overturn rates by locale Locale Overturns Rate n IT 5 5.0% 100 DE 3 3.0% 100 ES 3 3.0% 100 FR 1 1.0% 100 Key Findings. âą The majority panel achieves a 97% accuracy rate (12/400 overturns) with a tight confidence interval of [1.3%, 4.7%]. âą Unanimous cases have a 0% overturn rate (0/100): when all 21 judges agree, the label is always correct in our sample. âą UNKNOWN is the least reliable majority label at 5.0% overturn rate, compared to CORRECT at 2.2% and INCORRECT at 0.0%. This is consistent with UNKNOWN being the conservative vote that can mask determinable cases. âą All 12 overturns came from mixed-agreement samples (4.0% rate); no unanimous sample was overturned. âą Locale differences range from 1.0% (FR) to 5.0% (IT), but with n=100 per locale the margins of error (2â4%) overlap substantially. Distinguishing locale-level effects would require ⌠1,000+ samples per locale. Implications. The 3% ± 1.7% overturn rate supports the claim that the majority panel correctly labels ⌠97% of agreement cases. Combined with the 0% overturn rate for unanimous cases, this validates the disagreement-based annotation strategy: agreement between the generation pipeline and a diverse 21-judge panel is a reliable signal of label correctness, with residual error concentrated in inherently ambiguous mixed-agreement cases. Appendix G Cost Analysis Details This appendix provides detailed cost breakdowns for the LLM Arena validation framework. All costs are based on AWS Bedrock on-demand pricing as of January 2026.333https://aws.amazon.com/bedrock/pricing/ G.1 Token Statistics Each API call consists of input tokens (the prompt with product data) and output tokens (the modelâs classification and reasoning). Input token counts vary based on prompt version and product text length; output tokens are estimated at 50 tokens per response (classification label plus brief reasoning). Table 18: Average input tokens per prompt by language and version Lang Ver Mean Med. Min Max Std DE V1 727 664 272 2554 308 V2 1095 1031 643 2925 308 V3 1229 1167 776 3060 309 ES V1 688 637 273 2229 285 V2 1056 1005 639 2596 285 V3 1191 1140 778 2733 285 FR V1 716 671 274 2222 291 V2 1084 1039 638 2587 290 V3 1219 1174 775 2723 291 IT V1 714 665 274 2079 301 V2 1082 1032 644 2446 300 V3 1217 1167 780 2581 300 Table 18 presents input token statistics. The three prompt versions differ in complexity: V1 provides basic instructions, V2 adds attribute descriptions, and V3 includes worked examples. This is reflected in token counts, with V3 prompts averaging 70% more tokens than V1. Token counts vary across languages due to differences in product text length and linguistic characteristics. German (DE) has the highest average token counts, while Spanish (ES) has the lowest. The standard deviation of approximately 285â308 tokens reflects the diversity in product description lengths across the dataset. G.2 Full Arena Costs Table 19: Full arena costs by language Lang. Products API Calls $/Prod. Total IT 3,007 63,147 $0.023 $68.80 DE 3,059 64,239 $0.023 $70.69 ES 3,159 66,339 $0.022 $70.76 FR 3,501 73,521 $0.023 $80.25 All 12,726 267,246 $0.023 $290.50 Table 19 shows the per-language cost breakdown for running the full 21-judge arena (7 models Ă 3 prompt versions). Costs scale linearly with dataset size, with French being most expensive due to having the most products. G.3 Scaling Projections Table 20 projects arena costs at various scales. At production scale (1M products), the full arena would cost approximately $22,825âsubstantially less than equivalent human annotation, which would cost $100,000â500,000 at typical rates of $0.10â0.50 per label. Table 20: Projected arena costs at scale Products API Calls Cost $/Prod. 1K 21K $23 $0.023 10K 210K $228 $0.023 100K 2.1M $2,283 $0.023 1M 21M $22,825 $0.023 G.4 Cost-Efficient Configurations For budget-constrained deployments, reduced ensemble configurations offer significant savings with potential accuracy trade-offs. Table 21: Alternative ensemble configurations (per 1,000 products) Configuration Judges $/1K Savings Full Arena (7Ă3) 21 22.83 â Excl. highest-cost 18 11.57 49% Top-3 families (3Ă3) 9 2.08 91% Single family (3 prompts) 3 0.54â11.26 51â97% Single config 1 0.14â4.39 81â99% Table 21 presents alternative ensemble configurations. Cost varies substantially across model families, with open-weight models typically 10â30Ă cheaper than commercial APIs. Using a single model family with all three prompt versions represents 3â49% of full arena cost depending on family choice. The trade-off between cost and accuracy presents optimization opportunities. Unanimous agreement cases (36% of products) could use a smaller initial ensemble, with full arena deployment reserved for uncertain cases. Such adaptive strategies could reduce average cost by 30â50% while maintaining high accuracy on confident predictions. However, ablation studies validating accuracy of reduced configurations are required before deployment.