Paper deep dive
SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
Dae Lee, Mihai Delgeanu, Adel Youssef
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/20/2026, 4:19:11 AM
Summary
The paper introduces SESSE, a training-free framework for LLM-as-judge evaluation that decomposes holistic A/B preference judgments into structured, automatically mined sub-questions. By analyzing the judge's own error cases on a development set, SESSE generates a reusable bank of evaluation criteria (A/B/NA votes). This approach provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes. On the RewardBench dataset, SESSE achieves performance near-parity with chain-of-thought baselines and competes with fine-tuned specialist models like RISE-Judge-32B, while remaining fully training-free.
Entities (8)
Relation Signals (6)
SESSE → achievesperformanceon → RewardBench
confidence 95% · On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline
SESSE → comparedto → Chain-of-Thought
confidence 90% · SESSE achieves near-parity with the chain-of-thought baseline
SESSE → competeswith → RISE-Judge-32B
confidence 90% · SESSE... is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist
SESSE → uses → Gemini 2.5 Flash
confidence 90% · SESSE with Gemini 2.5 Flash reaches 93.3% accuracy
SESSE → uses → Qwen2-VL-7B
confidence 90% · Models: Qwen2-VL-7B... SESSE... Qwen2-VL-7B’s degradation
SESSE → differsfrom → G-Eval
confidence 85% · SESSE differs from all prior work: criteria emerge directly from the judge’s own error distribution... unlike Prometheus... and G-Eval... which require manually specified dimensions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.
Tags
Links
- Source: https://arxiv.org/abs/2608.18303v1
- Canonical: https://arxiv.org/abs/2608.18303v1
Trouble viewing inline? Open PDF directly →
Full Text
24,160 characters extracted from source content.
Expand or collapse full text
SESSE: Sketch, Expand, Sort, Summarize, Evaluate LLM-as-Judge Evaluation via Structured Decomposition Dae Lee, Mihai Delgeanu, Adel Youssef Apple dae_lee@apple.com Abstract LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge’s own error cases — requiring no oracle responses, task- specific rubrics, or fine-tuning. On Reward- Bench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is com- petitive with RISE-Judge-32B (92.7%), a fine- tuned specialist, while remaining fully training- free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label am- biguity and judge failure modes unavailable from a single holistic output token. 1 Introduction LLM-as-judge evaluation has become a standard paradigm for scalable response quality assessment — popularized by MT-Bench (Zheng et al., 2023), Chatbot Arena (Chiang et al., 2024), and Alpaca- Farm (Dubois et al., 2023) — yet it reduces eval- uation to a single holistic A/B preference choice. Even with chain-of-thought prompting, the reason- ing is unstructured free text produced in a single pass — offering no mechanism to decompose qual- ity into auditable dimensions, isolate which aspects drove the preference, or detect systematic annota- tion noise. Existing approaches fall into two paradigms, each with a critical limitation. The first, criterion- based methods — G-Eval (Liu et al., 2023), FActScore (Min et al., 2023), UniEval (Zhong et al., 2022), CheckList (Ribeiro et al., 2020) — decompose quality into predefined dimensions but require manually specified criteria or human- Figure 1: SESSE replaces holistic judgment with structured decomposition — criteria emerge automati- cally from the judge’s own error cases, producing per- criterion vote evidence. designed rubrics. The second, trained special- ist methods, fits directly to labeled data: oracle- dependent approaches such as DeCE (Yu et al., 2025a) and CheckEval (Lee et al., 2025) derive criteria from gold responses, while RISE-Judge (Yu et al., 2025b) and other fine-tuned variants — JudgeLM (Zhu et al., 2024), Prometheus 2 (Kim et al., 2024b), Auto-J (Li et al., 2024) — improve calibration via SFT+DPO at the cost of training data and parameter updates. SESSE differs from all prior work: criteria emerge directly from the judge’s own error distribution on standard pairwise labels, requiring no oracle responses, task-type an- notation, or fine-tuning. We propose SESSE, a training-free framework that replaces holistic judgment with structured de- composition. Error cases from a first-pass holistic run on a dev set drive automatic mining of eval- uation criteria. These criteria are clustered into kgroups and generalized into a reusable A/B/NA (Not Applicable) sub-question bank. Independent per-criterion voting produces an interpretable au- dit trail and a final preference; ordering random- ization across position mitigates positional bias arXiv:2608.18303v1 [cs.AI] 18 Aug 2026 (Wang et al., 2024a), and constraining the judge to discrete tokens A/B/NA yields superior calibration over open-ended scoring (Zhao et al., 2021; Ren et al., 2023). Decomposition addresses the reliability problem central to holistic judgment: a single output token conflates every quality dimension into one binary decision, so two judges — or one judge across two runs — can disagree without any way to lo- calize why. By contrast, each SESSE sub-question is independently verifiable: an annotator, a sec- ond model, or the same judge at a later time can re-check a single A/B/NA vote in isolation, with- out re-deriving the entire holistic judgment. This mirrors why checklist- and rubric-style protocols improve inter-rater agreement over free-form hu- man judgments (Ribeiro et al., 2020): breaking one hard judgment into many easy ones bounds disagreement to specific, inspectable criteria rather than an opaque holistic vote. Our contributions are: (i) SESSE, a fully auto- mated 5-stage pipeline that mines evaluation cri- teria from judge error cases, clusters them into a reusable sub-question bank, and aggregates per- criterion votes — without oracle responses, rubrics, or fine-tuning; and (i) empirical evidence that SESSE is competitive with fine-tuned special- ist evaluators on RewardBench while providing per-criterion diagnostic evidence unavailable from holistic judgment. 2 Method SESSE operates as a five-stage pipeline (Figure 2). Stages 0–1 (Error Collection, Sketch). Stage 0 runs the judge holistically on the dev set, collect- ing cases where the prediction disagrees with the ground-truth label. Stage 1 prompts the judge to generate conditional evaluation criteria from these errors as sub-questions: “If [protasis: the condition under which the criterion applies], which response better satisfies [apodosis: the evaluation property]: A, B, or NA?” Criteria emerge from the judge’s own failure distribution — unlike Prometheus (Kim et al., 2024a) and G-Eval (Liu et al., 2023), which require manually specified dimensions. Stages 2–3 (Expand, Sort). Stage 2 embeds can- didate sub-questions and clusters them intokthe- matic groups. Stage 3 ranks sub-questions within each cluster via greedy farthest-first traversal (Gon- zalez, 1985), producing a diversity-ordered active bank of depthn≤ n max without rerunning cluster- ing, enforcing criterion orthogonality across the bank. This ranked structure enables depth ablation at inference time (Appendix A). Stage 4 (Summarize). An LLM call produces one generalized protasis per cluster, serving as a human-readable cluster label. Near-duplicate clusters are merged via LLM binary equivalence judgment; sub-questions from merged clusters are pooled and top-n re-selected. Stage 5 (Evaluate). Tier 1 presents each ab- stract protasis to the judge and selects only relevant clusters for the current example, skipping inappli- cable clusters to reduce per-example cost. Tier 2 answers each selected question with its original (error-derived) protasis paired with its apodosis, preserving the concrete, instance-specific phrasing from Stage 1. When a criterion does not apply (e.g., a code cluster on a poetry prompt), the judge returns NA; NA votes are excluded from the A/B tally, letting the judge gracefully skip criteria that do not apply. The final preference is the majority of all A and B ballots across selected clusters; if A and B votes are equal, the example is declared a tie and excluded from the accuracy denominator. 3 Experiments 3.1 Setup Dataset: RewardBench (Lambert et al., 2024), 2,985 pairwise preference examples across chat, chat-hard, safety, and reasoning subsets. Dev (n=1,985) for error collection; held-out val (n=1,000) for all reported metrics. The dev/val split is a random partition of the full 2,985-example set: 1,000 examples held out for val, with the re- mainder used as dev. Metric: Pairwise preference accuracy on held-out val. Table 1 reports CoT holistic accuracy split by whether SESSE reaches a decision: non-tie rows (SESSE commits to a preference) and tie rows (equal A/B votes, SESSE has no preference to of- fer); Tie% is the fraction of tied examples. SESSE accuracy is reported on non-tie rows only. Models: Qwen2-VL-7B (Wang et al., 2024b), Gemini Flash Lite, and Gemini 2.5 Flash (Gem- ini Team, 2024). Configuration:k=25clusters (selected by pre- liminary silhouette analysis),n=10, temperature 0. Optimal depthn ∗ per model is selected by dev accuracy. Figure 2: SESSE pipeline: offline bank construction (Stages 0–4, once) and online per-example inference (Stage 5). SystemHolisticSESSETie% non-tietienon-tie Qwen2-VL-7B0.6980.6670.663 ‡ 21.3% Flash Lite0.8800.6790.820 ‡ 2.6% Gemini 2.5 Flash0.9450.6950.9333.5% RISE-Judge (32B) † (Yu et al., 2025b)0.927— Table 1: RewardBench val (n=1,000), dev-optimaln ∗ . ‡ p<0.05 McNemar (non-tie subset). † SFT+DPO. 3.2 Main Results Table 1 presents SESSE results at the dev-optimal n ∗ across all three evaluated models. To ensure a fair comparison, all gaps between the holistic baseline and SESSE are computed on non-tie rows only, matching the McNemar test, which likewise restricts its analysis to the pairs where the two meth- ods disagree, testingH 0 : SESSE and the CoT holis- tic baseline are equally likely to be correct when their predictions diverge. Gemini 2.5 Flash shows a minor 1.3% relative gap, which is not statistically significant (p>0.05). Flash Lite and Qwen2-VL- 7B show statistically significant relative gaps of 6.8% and 5.0%, respectively (p<0.05). Notably, SESSE with Gemini 2.5 Flash reaches 93.3% ac- curacy alongside a low 3.5% tie rate, placing it in the same performance tier as the specialist evalua- tor RISE-Judge (32B) (Yu et al., 2025b) (92.7%); while RISE-Judge requires SFT+DPO fine-tuning, SESSE achieves comparable performance while remaining fully training-free and providing per- criterion evidence. Depth ablation (Appendix A) shows a per-tier pattern: the capable judge (Gemini 2.5 Flash) sat- urates atn=2sub-questions per cluster, the mid- tier judge (Flash Lite) benefits from broader evi- dence and peaks atn=5, while the weakest judge (Qwen2-VL-7B) peaks earlier, atn=3, before de- grading. Qwen2-VL-7B’s degradation pastn=3 reveals a bank quality effect: accuracy peaks at n=3then degrades, because 51.9% of instances are covered by a single dominant protasis in the self-generated bank — beyondn=3, the depth bud- get is consumed by redundant sub-questions from that cluster. A cross-model transfer experiment — regenerating the bank with Gemini while keeping Qwen2-VL-7B as judge — confirms the accuracy ceiling is set by judge capability, not bank quality (Table B.1, Appendix B). SESSE tie rows are harder examples: holistic accuracy on tie rows is consistently lower than on non-tie rows across all three models (Table 1). The gap is most pronounced for capable judges — Gemini 2.5 Flash drops from 94.5% to 69.5% on tie rows, suggesting that ties signal genuine ambiguity rather than arbitrary disagreement. 4 Discussion Training-free parity with fine-tuned judges. SESSE’s near-parity with CoT holistic for Gemini 2.5 Flash (Table 1) represents a minor trade-off: near-parity in exchange for structured, auditable per-criterion evidence at no training cost. CoT holistic itself already outperforms RISE-Judge (Yu et al., 2025b), so SESSE’s comparable accuracy mainly shows that it preserves the base model’s ad- vantage without requiring fine-tuning. For mid-tier judges, the gap is significant but reflects a differ- ent dynamic: Flash Lite’s drop stems primarily from Tier-1 false positives on safety subsets (see Domain-specific signal recovery below), not from the decomposition mechanism itself. Domain-specific signal recovery. SESSE’s im- pact varies substantially across RewardBench sub- sets. Ondonotanswer, Gemini 2.5 Flash gains +20.6% relative over holistic (75.9% → 91.5%) as explicit safety-refusal criteria isolate a signal that holistic judgment conflates with general prefer- ence. Conversely,xstest-should-responddrops 27.1% relative due to Tier-1 false positives on borderline-safe prompts, pointing to Tier-1 preci- sion as the key failure mode for domain-sensitive subsets. When to prefer SESSE. SESSE is preferable when interpretability matters: the per-criterion vote trail enables annotation auditing and exposes label noise, justifying its inference overhead beyond raw accuracy. Tie rate as a deployment-time capability sig- nal. Tie% is a label-free proxy for judge reliability: Qwen2-VL-7B’s 21.3% tie rate — versus 2.6–3.5% for the Gemini models — predicts its accuracy gap, since the judge cannot form a directional preference on roughly one-in-five examples (which score only 66.7% accuracy when forced holistically; Table 1). A high Tie% on a held-out probe thus signals a capability mismatch or domain shift without re- quiring ground-truth labels, letting a practitioner switch judges, retarget the bank, or flag examples for human review. Failure modes. When SESSE diverges from holistic judgment, analysis reveals capability- dependent failure modes (see Table C.1, Ap- pendix C, for the full three-bucket taxonomy). We conducted further analysis on Flash Lite and Gem- ini 2.5 Flash (SESSE accuracy above 80%) to un- derstand failure modes. For Flash Lite, disagree- ments split roughly evenly between genuine la- bel ambiguity (49%, high vote entropy), decom- position failure (26%), and semantic equivalence (26%), reflecting a mid-tier judge without a sin- gle dominant failure signature. For the capable judge (Gemini 2.5 Flash), 48% of disagreements instead reflect semantic equivalence between re- sponses (C3): these pairs have higher response similarity than the dataset average (0.75 vs. 0.69 mean cosine similarity between Sentence-BERT embeddings; Reimers and Gurevych 2019), and the bank correctly returns high NA rates when no meaningful distinction exists. This C3 signature is not a model failure but a dataset property — nearly half of Gemini’s degradation rows encode stylistic preference rather than substantive quality differ- ences, making the per-criterion vote distribution a lightweight benchmark audit tool independent of accuracy measurement. In all cases, per-criterion vote distributions make failure modes explicitly di- agnosable, a feature absent from monolithic A/B scoring. 5 Conclusion We presented SESSE, a training-free framework that decomposes holistic LLM-as-judge evaluation into automatically derived A/B/NA sub-questions mined from the judge’s own error distribution. On RewardBench, SESSE achieves a non-significant gap to CoT holistic for capable judges and is in the same performance range as a fine-tuned special- ist evaluator (RISE-Judge 32B) while remaining training-free. Beyond accuracy, the per-criterion vote distributions provide interpretable diagnostic evidence — surfacing failure modes and bench- mark quality issues that holistic scoring leaves opaque — a use case that scales with judge ca- pability rather than against it. Limitations Bank portability. The question bank is derived from a specific judge model’s error distribution on a specific dev set. Changing the judge model or evaluation domain requires re-running Stages 0–4 (a one-time offline cost). Amortization strategies — cross-domain bank transfer, incremental error pool updates, shared banks across related judge models — are not explored here. Cluster count & benchmark scope. We fix k=25by silhouette analysis; ablation overkand generalization beyond RewardBench (open-ended generation, multi-turn, non-English) are deferred to future work. Inference cost. SESSE atn=10requires∼282 LLM calls per example vs. 1 holistic; capable judges saturate atn=2, reducing cost proportion- ally with no accuracy loss. Decomposition limits. A/B/NA sub-questions cannot capture end-to-end execution evaluation (code compilation, derivation correctness) — the primary source of SESSE degradation on reasoning and code subsets. Ethical Considerations SESSE evaluates LLM outputs using an LLM judge, inheriting any biases present in the judge model’s error distribution. Criteria mined from judge failures may encode systematic biases — users deploying SESSE in high-stakes annotation pipelines should audit the generated question bank for bias before use. All experiments use publicly available models and datasets. References Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anas- tasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chat- bot Arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st In- ternational Conference on Machine Learning, pages 8359–8388. PMLR. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca- Farm: A simulation framework for methods that learn from human feedback. In Advances in Neural Infor- mation Processing Systems. Gemini Team. 2024. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Teofilo F. Gonzalez. 1985. Clustering to minimize the maximum intercluster distance. Theoretical Com- puter Science, 38:293–306. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024a. Prometheus: Inducing fine- grained evaluation capability in language models. In Proceedings of the Twelfth International Conference on Learning Representations. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. Prometheus 2: An open source lan- guage model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Re- wardBench: Evaluating reward models for language modeling. arXiv. Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using check- lists. In Proceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing, pages 15771–15798, Suzhou, China. Association for Computational Linguistics. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2024. Generative judge for evaluating alignment. In Proceedings of the 12th International Conference on Learning Representa- tions. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using gpt-4 with better human align- ment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Jie Ren, Yao Zhao, Tu Vu, Peter J Liu, and Balaji Laksh- minarayanan. 2023. Self-evaluation improves selec- tive generation in large language models. In NeurIPS 2023 Workshop on Robustness of Few-shot and Zero- shot Learning in Foundation Models. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024a. Large language models are not robust multiple choice selectors. In Proceedings of the 12th International Conference on Learning Representa- tions. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-VL: Enhancing vision-language model’s per- ception of the world at any resolution. arXiv preprint arXiv:2409.12191. Fangyi Yu, Nabeel Seedat, Dasha Herrmannova, Frank Schilder, and Jonathan Richard Schwarz. 2025a. Beyond pointwise scores: Decomposed criteria- based evaluation of LLM responses. arXiv preprint arXiv:2509.16093. Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. 2025b. Improve LLM- as-a-judge ability as a general ability. In Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14099–14115, Suzhou, China. Association for Computational Lin- guistics. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Sys- tems, volume 36, pages 46595–46623. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.Towards a unified multi- dimensional evaluator for text generation. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2024. JudgeLM: Fine-tuned large language models are scalable judges. In Proceedings of the 38th AAAI Conference on Artificial Intelligence. A Depth Ablation Table A.1 sweepsn=1–10sub-questions per clus- ter for all three self-generated banks (bold = best per model). All models plateau quickly: accuracy stabilizes within the first few values ofnand shows little change beyond that, suggesting a small ques- tion budget is sufficient once the most informative sub-questions are exhausted. Qwen2-VL-7B is the exception, showing a slight degradation at largern due to limited cluster diversity in the self-generated bank. B Cross-Model Bank Transfer Table B.1 compares Qwen2-VL-7B accuracy when evaluated with a self-generated bank versus a bank whose questions were generated by Gemini 2.5 Flash, holding the judge model fixed. Both banks reach the same peak accuracy (66.3%), showing that substituting a stronger model’s questions does not meaningfully change evaluation performance when the judge model is held fixed. C Failure Mode Taxonomy Table C.1 reports the per-configuration breakdown of degradation rows — examples where SESSE and the holistic baseline disagree. We report this breakdown only for configurations with SESSE ac- curacy above 80% (Flash Lite, Gemini 2.5 Flash); at lower accuracy, degradation rows are too fre- quent to characterize as a small set of distinct fail- ure signatures. Each row is attributed to one of three buckets: C1 (Ambiguous) reflects genuine la- bel difficulty, where sub-question votes are evenly split; C2 (Decomposition failure) indicates the judge is confidently wrong, with≥65% of non-NA votes against the ground-truth label; C3 (Semantic equivalence) indicates the bank correctly declines to pick a winner, returning high NA rates when the two responses are too similar to discriminate. nQwen2-VL-7BGemini 2.5 FlashFlash Lite 10.6260.9230.790 20.6480.9330.804 30.6630.9270.801 50.6440.9320.820 70.6450.9290.817 100.6430.9300.819 Table A.1: Accuracy (excl. ties) vs. n (self-generated banks; bold = n ∗ ). Configk eff SESSE acc (n=10)SESSE (n ∗ ) Qwen self-generated80.6400.663 (n ∗ =3) Qwen+Gemini bank140.6370.663 (n ∗ =2) Table B.1: Qwen2-VL-7B with Gemini-generated vs. self-generated bank (val). ConfigJudgeBankDegradation (% val)C1 (Ambiguous)C2 (Decomposition)C3 (Semantic equiv.) Flash LiteFlash Liteself-generated105 (10.5%)49%26%26% Gemini 2.5 FlashGemini 2.5 Flashself-generated48 (4.8%)29%23%48% Table C.1: Degradation bucket distribution. C1 = genuine ambiguity (high vote entropy); C2 = decomposition failure (SESSE consistently wrong≥65% non-NA votes against label); C3 = semantic equivalence (high NA rate, responses too similar to discriminate). Percentages of degradation rows per config.