Paper deep dive
Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a-Judge and Human-in-the-Loop
Ashmi Banerjee, Adithi Satish, Wolfgang Wörndl, Yashar Deldjoo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 8:06:11 AM
Summary
The paper proposes a three-phase calibration framework to evaluate multi-dimensional sustainable city-trip recommendations using LLMs-as-a-Judge. The framework addresses the challenges of evaluating subjective dimensions like relevance, diversity, sustainability, and popularity balance. By comparing LLM judgments (GPT, Gemini, DeepSeek) against human expert evaluations, the authors identify systematic misalignments and use dimension-specific rules and few-shot prompting to calibrate the models. Results show that while calibration improves inter-judge consistency and alignment on certain dimensions like sustainability, it can also lead to 'preference sharpening' where models become more decisive in their specific interpretations.
Entities (11)
Relation Signals (4)
Ashmi Banerjee → affiliatedwith → Technical University of Munich
confidence 100% · Ashmi Banerjee ashmi.banerjee@tum.de Technical University of Munich
LLM-as-a-Judge → evaluates → Sustainability
confidence 100% · We study LLMs-as-Judges for sustainable city-trip lists across four dimensions -- relevance, diversity, sustainability, and popularity balance
GPT-4o-mini → isevaluatedby → LLM-as-a-Judge
confidence 100% · Paired recommendations are generated using gpt-4o-mini and gemini-2.5-flash; pairs with more than three overlapping cities are discarded
SynthTRIPs → providesqueriesfor → LLM-as-a-Judge
confidence 100% · Our query set is derived from SynthTRIPs [3]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating nuanced conversational travel recommendations is challenging when human annotations are costly and standard metrics ignore stakeholder-centric goals. We study LLMs-as-Judges for sustainable city-trip lists across four dimensions -- relevance, diversity, sustainability, and popularity balance, and propose a three-phase calibration framework: (1) baseline judging with multiple LLMs, (2) expert evaluation to identify systematic misalignment, and (3) dimension-specific calibration via rules and few-shot examples. Across two recommendation settings, we observe model-specific biases and high dimension-level variance, even when judges agree on overall rankings. Calibration clarifies reasoning per dimension but exposes divergent interpretations of sustainability, highlighting the need for transparent, bias-aware LLM evaluation. Prompts and code are released for reproducibility: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.24158v1
- Canonical: https://arxiv.org/abs/2604.24158v1
Trouble viewing inline? Open PDF directly →
Full Text
23,581 characters extracted from source content.
Expand or collapse full text
Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a-Judge and Human-in-the-Loop Ashmi Banerjee ashmi.banerjee@tum.de Technical University of Munich Munich, Germany Adithi Satish adithi.satish@tum.de Technical University of Munich Munich, Germany Wolfgang Wörndl woerndl@in.tum.de Technical University of Munich Munich, Germany Yashar Deldjoo yashar.deldjoo@poliba.it Polytechnic University of Bari Bari, Italy Abstract Evaluating nuanced conversational travel recommendations is chal- lenging when human annotations are costly and standard metrics ignore stakeholder-centric goals. We study LLMs-as-Judges for sus- tainable city-trip lists across four dimensions — relevance, diversity, sustainability, and popularity balance, and propose a three-phase calibration framework: (1) baseline judging with multiple LLMs, (2) expert evaluation to identify systematic misalignment, and (3) dimension-specific calibration via rules and few-shot examples. Across two recommendation settings, we observe model-specific biases and high dimension-level variance, even when judges agree on overall rankings. Calibration clarifies reasoning per dimension but exposes divergent interpretations of sustainability, highlighting the need for transparent, bias-aware LLM evaluation. Prompts and code are released for reproducibility 1 . Keywords LLMs, LLM-as-a-Judge, Calibration, Tourism Recommender Sys- tems, Sustainability ACM Reference Format: Ashmi Banerjee, Adithi Satish, Wolfgang Wörndl, and Yashar Deldjoo. 2026. Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a- Judge and Human-in-the-Loop. In 34th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’26), June 08–11, 2026, Gothenburg, Sweden. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3774935. 3812717 1 Introduction and Context Tourism recommender systems operate in a multi-stakeholder set- ting, where personalization must balance sustainability, destination capacity, and community impact. Standard accuracy metrics cap- ture relevance but overlook sustainability, overtourism risk, and the balance between popular and lesser-known destinations [2]. Evaluation is further complicated by the tourism data gap: official 1 https://github.com/ashmibanerjee/trs-llm-calibration This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License. UMAP ’26, Gothenburg, Sweden © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2311-7/2026/06 https://doi.org/10.1145/3774935.3812717 statistics are often too aggregated for operational recommender systems and differ across countries, limiting cross-country com- parability [6]. No canonical benchmark exists for conversational itinerary recommendation that supports multidimensional evalua- tion. Expert evaluation is costly and unscalable, while automated metrics often miss contextual and societal factors. LLMs-as-Judges provide scalable approximations of human reasoning but exhibit model-specific biases, especially on subjective criteria [5]. PHASE 1: Baseline LLM Judging User Query (q ∈ Q) + Recommendation Lists (L 1 , L 2 ) Multiple LLM Judges (e.g., GPT, Gemini, DeepSeek) Baseline Judgments (Scores on R, D, S, PM) ⚙ Relevance (R) Diversity (D) Sustainability (S) ⚖ Popularity Mix (PM) PHASE 2: Expert Evaluation & Misalignment Analysis Human Expert Evaluators Expert Judgment Misalignment Analysis & Bias Identification (Compare LLM vs. Expert) Derive Calibration Rules & Few-Shot examples PHASE 3: Dimension-Specific Calibration Calibration Process (Rule-based, few-shot prompting) Re-evaluation by Calibrated LLMs Calibrated LLM Judgments (Improved alignment & reliability) Figure 1: Three-phase LLM calibration framework: baseline judgments, comparison with human evaluations to identify systematic errors, and dimension-specific calibration to im- prove alignment and reliability. In tourism, this subjectivity is amplified: sustainability varies by seasonality, diversity requires balancing geographic spread with coherence, and popularity must reconcile authenticity with main- stream appeal. Even human evaluators disagree on these dimen- sions, making perfect consensus neither achievable nor desirable. Moreover, evaluation must consider non-participating stakeholders, such as residents affected by overtourism, aligning with emerging notions of societal fairness [1, 2]. To capture these considerations, we evaluate recommendations across four dimensions: Relevance, Diversity, Sustainability, and Popularity Mix. Relevance and diversity reflect established yet in- herently subjective criteria, shaped by user perceptions and multi- stakeholder trade-offs [1, 4]. Sustainability and popularity mix ad- dress the need to balance personalization with destination capac- ity and community well-being, thereby mitigating overtourism risks [2]. In our framework, sustainability is treated as a normative arXiv:2604.24158v1 [cs.AI] 27 Apr 2026 UMAP ’26, June 08–11, 2026, Gothenburg, SwedenBanerjee et al. rubric dimension, assessed by expert judgment rather than strict factual verification, with indicators including seasonality, crowd- ing risk, and the encouragement of less commercialized destina- tions. While recent work, such as SynthTRIPs [3], generates diverse, knowledge-grounded travel queries, it does not address how rec- ommendations should be evaluated across these dimensions. This motivates our central question: How can LLM-based evaluators be calibrated to assess multi-dimensional tourism recommendations in the absence of objective ground truth? Contributions. We present a methodological study of LLM-based evaluation for sustainable city-trip recommendations: •Three-phase calibration framework aligning LLM judges with human experts across relevance, diversity, sustainability, and popularity balance. •Empirical analysis of model biases, showing calibration im- proves inter-judge consistency while preserving legitimate dis- agreement. • Lightweight calibration checklist from human–LLM misalign- ment to reduce bias without fine-tuning. •Reproducible evaluation pipeline generalizable to other multi- dimensional, domain-specific tasks. We retain dimension-level evaluation as our primary focus to keep the trade-offs explicit, even though we also compute confidence- weighted aggregated scores. In multi-stakeholder settings, aggre- gation can obscure trade-offs and mask specific failure modes; an- alyzing dimensions separately keeps disagreements explicit and actionable. 2 Evaluation Framework and Experimental Methodology We propose a three-phase calibration framework to systematically evaluate multi-dimensional European city-trip recommendations using both human experts and LLM judges (Figure 1). In this work, we use the general term Evaluators to refer to both LLMs and human experts. When specifying a particular group, we append either LLM or human explicitly. The term Judges is reserved exclusively for LLM evaluators, while Experts always refers to human evaluators. Figure 2: Web-based survey interface used by human experts for pairwise evaluation of city-trip recommendation lists, including dimension-specific scoring and optional justifica- tions. Each query푞 ∈ Qrepresents a traveler request with explicit constraints. For each query, two LLMs generate ranked lists퐿 1 and퐿 2 of푘=5 cities, each with justifications. Evaluators com- pare these lists using a structured pairwise protocol [7], assigning dimension-specific judgments over the four predefined dimensions (subsection 2.1). Phase 1: Baseline Judging. Paired recommendation lists are inde- pendently evaluated by multiple LLM judges, producing baseline judgments across all dimensions. Phase 2: Expert Evaluation and Misalignment Analysis. Baseline judgments are compared with evaluations from five domain experts with at least 5 years of experience in tourism recommendations. Sys- tematic disagreements reveal recurring error patterns and inform calibration rules [9]. Phase 3: Dimension-Specific Calibration. The derived calibration rules after Phase 2 and a few-shot examples are integrated into refined judge prompts, and the same queries are re-evaluated to an- alyze the updated inter-judge variance and alignment with experts. 2.1 Evaluation Dimensions Our study evaluates city-trip recommendation lists generated by LLMs for a natural language query with explicit constraints. We evaluate across four dimensions grounded in multi-stakeholder fairness literature [1, 4] and tourism-specific concerns [2]: •Relevance (R) measures how well the recommended cities align with the query’s explicit constraints (e.g., budget, season, inter- ests) and implicit preferences inferred from the traveler [1, 3]. •Diversity (D) captures geographic and thematic variety within a list, reflecting the presence of distinct regions and experience types [1, 4]. •Sustainability (S) evaluates responsible travel characteristics such as seasonality, walkability, public transit access, air qual- ity, and destination capacity. Assessments rely on observable, context-dependent indicators rather than generic or unsubstanti- ated sustainability claims [2]. •Popularity Mix (P M ) examines the balance between widely known destinations and lesser-known alternatives [2]. In this paper, popularity, popularity mix, and popularity balance are used interchangeably. 2.2 Experimental Setup Query Selection and Data Generation. Our query set is derived from SynthTRIPS [3], which generates realistic travel requests by combining user preferences with structured constraints (e.g., season, budget, interests, sustainability) and a pool of European cities. We use only the natural-language queries—excluding city lists and constraints—to prompt LLMs (e.g., “European city break in July, walkable, less touristy” ). From 2,302 generated queries, we retain 123 via deduplication and diversity filtering. Paired recommendations are generated using gpt-4o-mini and gemini-2.5-flash; pairs with more than three overlapping cities are discarded, resulting in 100 evaluation queries and one practice query. Experimental Conditions. We compare gemini-2.5-flash (퐿 1 ) and gpt-4o-mini (퐿 2 ), and they are evaluated by both Experts and Judges. The LLM judges (gpt-5, gemini-2.5-pro, deepseek-v3) are configured with휏=0.0, top-푝=0.95, and an 8192-token context window. Five human experts with at least 5 years of experience in the tourism domain evaluate the recommendations through a web-based inter- face that displays side-by-side lists with dimension-specific scoring Multi-Dimensional Evaluation of Sustainable City TripsUMAP ’26, June 08–11, 2026, Gothenburg, Sweden (Figure 2). Each query receives responses from at least three ex- perts, resulting in approximately 15 hours of total annotation time. Three LLM judges evaluate the same 100 queries to enable cross- architecture comparison. Pairwise Comparison Protocol. We use pairwise comparisons to improve judgment stability and reduce scale-related biases [4]. For each query, evaluators assign dimension-specific scores푠 푑 (퐿 1 , 퐿 2 | 푞) ∈ −2,−1,0,+1,+2, with positive values favoring퐿 1 ; responses marked as unsure are excluded from the analysis. This allows us to assess inter-judge reliability and alignment with human evalua- tion [9], capturing the inherent subjectivity of dimensions such as Sustainability and Diversity. In addition to dimension-level judgments, LLM judges report a confidence score푐 푑 ∈ [0,1]for each dimension and select a best list per query to indicate their overall preference. To quantify confidence-aware overall preference, we compute a confidence- weighted aggregated score 푆 agg for each query: 푆 agg = ∑︁ 푑∈R,D,S,P 푀 푠 푑 · 푐 푑 ,(1) where푠 푑 denotes the signed pairwise score for dimension푑and푐 푑 the corresponding confidence. Positive values of푠 푑 favor퐿 1 , while negatives favor 퐿 2 . At the dataset level, we summarize directional preference us- ing two complementary ratios. The Aggregated Score Ratio (ASR) quantifies the total strength of evidence by dividing the sum of all confidence-weighted scores favoring퐿 1 by the sum of those favor- ing퐿 2 . In contrast, the Best List Ratio (BLR) measures the frequency of preference, calculated as the total number of times퐿 1 was explic- itly selected as the ‘best’ list divided by the total selections for퐿 2 . For both ratios, values>1 indicate a net preference for퐿 1 , while values<1 indicate a net preference for퐿 2 . Together, these metrics allow us to examine whether holistic best list choices align with the direction implied by confidence-weighted, dimension-level evi- dence, and to assess consistency between decomposed judgments and overall preference signals. Table 1: Calibration rules used to guide LLM judge behavior. The exact prompt formulations are available in the reposi- tory 2 . RuleGuideline RelevanceMatch explicit constraints using city-level evidence; avoid keyword- only matches. SustainabilityReward verifiable indicators; penalize vague claims; account for season- ality and off-peak travel. PopularityFavor a balanced mix of mainstream destinations and hidden gems; down-weight crowded options unless justified. DiversityEnsure geographic and thematic variety unless restricted by the query. Tie-breakingResolve close cases using constraint coverage, specificity, factual accu- racy, and context. ValidationVerify entity type (city, landmark, country) and factual correctness. Context & Sea- sonality Penalize unsafe, closed, or seasonally unsuitable recommendations. Confidence Credit evidence-backed claims; mark uncertainty only when ambiguity is unavoidable. Calibration Procedure. Calibration is performed through iterative re-prompting [8], informed by recurring patterns of disagreement between LLM judges and human experts. It focuses on systematic 2 https://github.com/ashmibanerjee/trs-llm-calibration rather than marginal disagreement by analyzing only cases with |푠 llm 푑 −푠 human 푑 | ≥2, which indicate clear directional conflicts with the human majority. Since single-point differences are common across both human-human and judge-judge comparisons and generally reflect acceptable subjective variation rather than errors worth correcting, we only include the two-point disagreements in our misalignment analysis. We derive eight calibration rules (Table 1) that capture oper- ational criteria, common failure modes, and corrective examples which can be generalized across queries and models. These rules are incorporated into refined judge prompts, along with targeted few-shot demonstrations. While the two-point threshold is em- pirically justified, future work should examine whether stacking multiple calibration rules introduces noise. Overall, our study shows that calibration improves alignment on subjective dimensions and stabilizes inter-judge behavior. 3 Results and Discussions We analyze LLM judge behavior before and after calibration, focus- ing on alignment with human experts and inter-judge consistency. RQ1. How well do judges align with human experts, and why do they disagree? Note on inter-rater reliability. We do not report traditional inter- rater agreement because the number of raters varies across queries and the task involves inherently subjective, multidimensional judg- ments; instead, we focus on model-specific patterns and calibration effects to analyze consistency and directional preferences. Results. Alignment is measured against the majority vote of humans for each response. Against the human majority (Table 2, “Before”), GPT aligns best on Relevance (0.53) and achieves mod- erate alignment on other dimensions (0.36–0.42). Gemini shows a comparable macro-average alignment (approximately 0.42), while DeepSeek performs lower overall (approximately 0.34), with partic- ularly weak agreement on Relevance (approximately 0.17). Subjec- tive dimensions, especially Diversity and Sustainability, remain the most challenging for judges (mid-0.30s to 0.40s). Human annotators also disagree among themselves, so perfect alignment is neither expected nor always desirable; the objective is to reduce systematic LLM errors that humans rarely make. Table 2: LLM judge agreement with the human majority vote before and after calibration (values denote the proportion of queries where the judge matches the human majority decision). Arrows indicate the direction of change after cali- bration. DeepSeekGPTGemini MetricBeforeAfterBeforeAfterBeforeAfter R0.170.18↑0.530.530.480.54↑ D0.350.29↓0.360.360.370.39↑ S0.360.360.400.400.370.35↓ P M 0.470.43↓0.420.420.450.39↓ Why disagreements occur (evidence and examples). Manual analysis reveals four recurring causes: (i) Type and factual errors UMAP ’26, June 08–11, 2026, Gothenburg, SwedenBanerjee et al. go unpenalized by judges but are reliably caught by experts: Malta, Santorini, and Plitvice Lakes are treated as cities in some lists. (i) Context and seasonality: judges do not flag Lviv during conflict or Keukenhof in October (closed), nor do they penalize Tallinn’s deep winter conditions that diminish trip quality; experts do. (i) Unsure aversion: Gemini never chooses “Unsure,” GPT almost never (one case on Sustainability), while DeepSeek does so more often. Forcing a side when the evidence is insufficient reduces alignment on edge cases. (iv) Attention to explicit constraints vs. surface cues: experts reward concrete coverage of budget, time, and persona, while judges sometimes reward keyword echoes or generic sustainability claims. RQ2. What changes after dimension-specific calibration? Results. Calibration primarily stabilizes inter-judge consistency. Gains are largest for Popularity Balance and Sustainability, where concrete rules (e.g., verifying "green" claims) reduce idiosyncratic interpretations. Subjective dimensions like Relevance show mixed results, indicating that rules cannot fully replace world knowl- edge. Alignment with human experts improves qualitatively (bet- ter reasoning) but quantitative gains are modest (Table 2). Gem- ini improves on Relevance (+0.06), while GPT and DeepSeek re- main static or dip slightly. This indicates calibration enforces strict, evidence-based assessment—often penalizing vague claims that hu- man experts might forgive—rather than simply mimicking human intuition. We performed statistical tests (푝<0.05) on pre–post calibration deltas, showing significant improvements primarily in Sustainability for all three judges, with selective gains for DeepSeek on Relevance and Diversity, while Popularity Balance remained sta- ble, indicating that calibration enhances alignment and inter-judge consistency without introducing unintended variance. Examples. Explicit constraints successfully help judges penalize false entities (e.g., Malta as a city) and unsubstantiated sustainabil- ity claims. However, trade-offs in Diversity (geographic spread vs. thematic coherence) remain inherently subjective, causing valid disagreements to persist. Effects on Directional preferences. We analyze whether calibra- tion reduces systematic bias toward specific lists (퐿 1 vs.퐿 2 ) using confidence-weighted scores (ASR) and best-list selections (BLR) (Table 3). The results clarify that calibration reduces uncertainty (noise) but has divergent effects on directionality: •Correction toward Parity (Gemini) Pre-calibration, Gemini skewed slightly against퐿 1 . Calibration shifts its ASR significantly toward equilibrium (0.71→0.98), aligning with the evidence that the lists are of comparable quality. •Preference Sharpening (DeepSeek) Conversely, DeepSeek am- plifies its initial preference for퐿 1 (1.30→2.88). By removing ambiguity, calibration makes the model more decisive in its spe- cific interpretation of the rules (e.g., strictly rewarding specific keywords), leading to polarization ("sharpening") rather than consensus. • Entrenchment (GPT) GPT shows minimal movement (0.58→ 0.64), suggesting its evaluation heuristics are deeply entrenched and less sensitive to few-shot prompts. Table 3: Directional preference ratios (>1 favors퐿 1 ,<1 favors퐿 2 ) before and after calibration. Arrows indicate the direction of change following calibration, computed using confidence-weighted aggregated scores (ASR) and explicit best-list selections (BLR). JudgeASRBLR BeforeAfterBeforeAfter Gemini0.710.98↑0.911.46↑ GPT-50.580.64↑0.740.89↑ DeepSeek1.302.88↑1.402.81↑ 4 Conclusion and Limitations Tourism recommendation requires metrics that prioritize sustain- ability and diversity over simple accuracy. While LLMs enable scalable multi-dimensional assessment, they exhibit model-specific biases. Calibration stabilizes reasoning and harmonizes heuristics, acting as a “clarifier” rather than a universal corrector: it reduces noise while revealing how models interpret rules—moving some toward parity (Gemini) and others toward sharpened preferences (DeepSeek). This transparency shows that one-size-fits-all consen- sus between judges is unrealistic. Future improvements should refine “Unsure” thresholds and incorporate richer, evidence-linked contextual reasoning. Despite limitations regarding human over- sight and normative assumptions, domain-specific calibration pro- vides a scalable, transferable framework, making assumptions ex- plicit and comparable. Our open-source pipeline offers a reusable foundation for transparent, stakeholder-aware evaluation across domains such as health, education, and policy. References [1] Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato. 2019. Beyond personalization: Research directions in multistakeholder recommendation. arXiv preprint arXiv:1905.01986 (2019). [2]Ashmi Banerjee, Paromita Banik, and Wolfgang Wörndl. 2023. A review on indi- vidual and multistakeholder fairness in tourism recommender systems. Frontiers in big Data 6 (2023), 1168692. [3] Ashmi Banerjee, Adithi Satish, Fitri Nur Aisyah, Wolfgang Wörndl, and Yashar Deldjoo. 2025. SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Data Generation for Personalized Tourism Recommenders. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval. 3743–3752. [4]Michael D Ekstrand, F Maxwell Harper, Martijn C Willemsen, and Joseph A Konstan. 2014. User perception of differences in recommender algorithms. In Proceedings of the 8th ACM Conference on Recommender systems. 161–168. [5]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al.2024. A survey on llm-as- a-judge. arXiv preprint arXiv:2411.15594 (2024). [6]Carmen Lam and Bob McKercher. 2013. The tourism data gap: The utility of official tourism information for the hospitality and tourism industry. Tourism Management Perspectives 6 (2013), 82–94. [7]ChaeHun Park, Minseok Choi, Dohyun Lee, and Jaegul Choo. 2024. Paireval: Open- domain dialogue evaluation metric with pairwise comparisons. In First Conference on Language Modeling. [8]Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. 2022. Planning with large language models via corrective re-prompting. In NeurIPS 2022 Foundation Models for Decision Making Workshop. [9]Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating align- ment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624 (2024).