Paper deep dive
Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge
Yulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk, Ilya Revin
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/10/2026, 2:17:12 AM
Summary
The paper introduces DualJudge, a hybrid evaluation framework for Large Language Models (LLMs) that combines intuitive direct scoring with structured, deliberative evaluation using a confidence-aware Fuzzy Analytic Hierarchy Process (FAHP). By decomposing assessments into criteria and modeling epistemic uncertainty via triangular fuzzy numbers, the framework improves evaluation reliability and robustness, particularly in logic-intensive domains, as validated on the JudgeBench benchmark.
Entities (5)
Relation Signals (3)
DualJudge → evaluatedon → JudgeBench
confidence 100% · Systematically validated on JudgeBench, our structured approach...
Fuzzy Analytic Hierarchy Process → extends → Analytic Hierarchy Process
confidence 100% · propose a confidence-aware Fuzzy AHP (FAHP) extension
DualJudge → integrates → Fuzzy Analytic Hierarchy Process
confidence 100% · DualJudge... adaptively fuses holistic direct scores with structured AHP outputs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Effective evaluation of large language models (LLMs) remains a critical bottleneck, as conventional direct scoring often yields inconsistent and opaque judgments. In this work, we adapt the Analytic Hierarchy Process (AHP) to LLM-based evaluation and, more importantly, propose a confidence-aware Fuzzy AHP (FAHP) extension that models epistemic uncertainty via triangular fuzzy numbers modulated by LLM-generated confidence scores. Systematically validated on JudgeBench, our structured approach decomposes assessments into explicit criteria and incorporates uncertainty-aware aggregation, producing more calibrated judgments. Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits, with FAHP showing superior stability in uncertain comparison scenarios. Building on these insights, we propose \textbf{DualJudge}, a hybrid framework inspired by Dual-Process Theory that adaptively fuses holistic direct scores with structured AHP outputs via consistency-aware weighting. DualJudge achieves state-of-the-art performance, underscoring the complementary strengths of intuitive and deliberative evaluation paradigms. These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.03742v1
- Canonical: https://arxiv.org/abs/2604.03742v1
Trouble viewing inline? Open PDF directly →
Full Text
55,341 characters extracted from source content.
Expand or collapse full text
11institutetext: St. Petersburg State University, University emb., 7/9, St. Petersburg, Russia 22institutetext: ITMO University, Kronverkskiy av., 49, St. Petersburg, Russia Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge Yulong He Ivan Smirnov Dmitry Fedrushkov Sergey Kovalchuk Ilya Revin Abstract Effective evaluation of large language models (LLMs) remains a critical bottleneck, as conventional direct scoring often yields inconsistent and opaque judgments. In this work, we adapt the Analytic Hierarchy Process (AHP) to LLM-based evaluation and, more importantly, propose a confidence-aware Fuzzy AHP (FAHP) extension that models epistemic uncertainty via triangular fuzzy numbers modulated by LLM-generated confidence scores. Systematically validated on JudgeBench, our structured approach decomposes assessments into explicit criteria and incorporates uncertainty-aware aggregation, producing more calibrated judgments. Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits, with FAHP showing superior stability in uncertain comparison scenarios. Building on these insights, we propose DualJudge, a hybrid framework inspired by Dual-Process Theory that adaptively fuses holistic direct scores with structured AHP outputs via consistency-aware weighting. DualJudge achieves state-of-the-art performance, underscoring the complementary strengths of intuitive and deliberative evaluation paradigms. These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment. Code is available at https://github.com/hreyulog/AHP_llm_judge. 1 Introduction The rapid progress of large language models (LLMs) has created an urgent demand for reliable and scalable evaluation methodologies. Recent research increasingly adopts LLM-as-a-Judge, where language models automatically assess generated outputs as a substitute for costly human evaluation. Such approaches have demonstrated promising alignment with human judgments across diverse tasks [2, 5]. However, despite their growing adoption, existing evaluation paradigms largely rely on a single judging strategy, which introduces systematic biases across domains. Also, this strategy is not robust for different the instructions and prompts. [10]. The dominant paradigm employs holistic evaluation, where an LLM directly produces an overall score or preference through intuitive judgment. Direct scoring and pairwise comparison are efficient but implicitly compress multiple evaluation dimensions into a single reasoning step. Prior studies show that LLM evaluators are sensitive to prompting formats and reasoning styles, resulting in unstable performance across task types [2]. Moreover, LLM-based evaluation approach is inconsistent for different scoring scale [9]. In particular, holistic judgments may fail to capture structured reasoning signals required in logic-intensive domains such as mathematics, coding, and legal analysis. An alternative paradigm introduces structured evaluation, which decomposes judgments into multiple criteria and aggregates them using formal multi-criteria decision-making (MCDM) frameworks such as the Analytic Hierarchy Process (AHP). Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [11, 22, 21, 8], suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment. Nevertheless, the effectiveness of structured hierarchical evaluation as a general paradigm for LLM-as-a-Judge remains insufficiently understood, and systematic comparisons with holistic judging across heterogeneous domains are largely missing. In this work, we conducted a comprehensive empirical study on JudgeBench [17], covering 17 categories spanning mathematical reasoning, programming, scientific knowledge, and soft-knowledge tasks. Our results reveal a clear pattern: structured evaluation based on AHP consistently outperforms direct holistic judging in logic-intensive domains, demonstrating that explicit criterion decomposition improves evaluation reliability. Moreover, we find that modeling judgment uncertainty further enhances performance. By extending AHP with triangular fuzzy numbers to represent preference ambiguity, Fuzzy AHP (FAHP) achieves higher accuracy than traditional crisp AHP, indicating that incorporating uncertainty better reflects the inherently probabilistic nature of LLM-generated judgments. We also treated scoring scale design as a controllable component of the system and showed the impact of the method for different scales. Motivated by the complementary strengths of holistic and structured evaluation, we further introduce DualJudge, a hybrid framework that combines two independent evaluation signals: a direct holistic score and a structured AHP-based score (crisp or fuzzy). Rather than introducing complex interactions, DualJudge aggregates the two scores through simple averaging, yielding consistently improved robustness across domains. Despite its simplicity, this hybrid strategy achieves the best overall performance on JudgeBench. Our contributions are threefold: • We provide a systematic empirical comparison between holistic and structured evaluation paradigms within LLM-as-a-Judge across diverse domains. • We demonstrate that AHP-based structured evaluation significantly outperforms direct judging baselines, and that FAHP further improves performance by modeling judgment uncertainty. • We propose DualJudge, a simple hybrid averaging framework that combines complementary evaluation signals and achieves the strongest overall accuracy. 2 Related Work 2.1 LLM-as-a-Judge The rapid proliferation of LLMs has necessitated scalable evaluation methodologies, leading to the widespread adoption of the LLM-as-a-Judge paradigm [5]. This approach substitutes costly human annotation with automated assessments, demonstrating promising alignment with human preferences across diverse tasks. The dominant strategy in this domain is holistic evaluation, where an LLM generates a single overall score or preference through intuitive judgment. While computationally efficient, holistic methods implicitly compress multiple evaluation dimensions into a single reasoning step. Prior studies indicate that such evaluators are highly sensitive to prompting formats and reasoning styles, resulting in unstable performance and systematic biases, particularly in logic-intensive domains such as mathematics and coding. To mitigate these limitations, recent research has explored structured evaluation paradigms. Unlike holistic judging, structured approaches decompose complex judgments into multiple criteria and aggregate them using formal frameworks [11, 22]. Thus, the widely used prompt-based evaluation solution G-Eval [10] interacts with predefined evaluation criteria and detailed evaluation steps to obtain the aggregate score. This decomposition forces the model to engage with specific quality dimensions systematically, enhancing interpretability and reducing the opacity inherent in black-box verdicts [21]. Although initial work suggests that hierarchical reasoning improves alignment with human assessment [8], systematic comparisons between holistic and structured paradigms across heterogeneous domains remain insufficiently understood. Our work addresses this gap by empirically validating the superiority of structured reasoning in logic-driven tasks. 2.2 Analytic Hierarchy Process The AHP, originally proposed by Saaty [16, 14], is a classical MCDM framework designed to decompose complex decision problems into hierarchical structures. AHP organizes decision elements into multiple levels, typically consisting of a goal layer, a criteria layer, and an alternative layer, enabling systematic reasoning through pairwise comparisons. Given a set of criteria, decision makers construct a pairwise comparison matrix =(aij),aij>0A=(a_ij), a_ij>0, where aija_ij denotes the relative importance of criterion i over criterion j. The priority weight vector w is obtained from the principal eigenvector: =λmax,Aw= _ w, where λmax _ represents the maximum eigenvalue of A. The consistency ratio (CR) is further computed to ensure logical coherence in human judgments. Due to its interpretability and structured reasoning capability, AHP has been widely applied in evaluation, ranking, risk analysis, and resource allocation tasks. In recent years, structured evaluation paradigms inspired by AHP have also been adopted in automated assessment systems and LLM evaluation pipelines, where complex judgments are decomposed into fine-grained criteria [11]. However, classical AHP assumes precise numerical comparisons between criteria. In practice, human judgments—and particularly LLM-generated evaluations—often exhibit uncertainty, vagueness, and linguistic ambiguity. This limitation motivates extensions that incorporate uncertainty modeling. 2.3 Fuzzy Analytic Hierarchy Process To address uncertainty in pairwise comparisons, researchers introduced fuzzy set theory [23] into AHP, resulting in the FAHP [1]. FAHP replaces crisp comparison values with fuzzy numbers, enabling decision makers to express preferences using approximate or linguistic judgments. A common formulation represents comparisons using triangular fuzzy numbers (TFN) [4]: a~ij=(lij,mij,uij), a_ij=(l_ij,m_ij,u_ij), where lijl_ij, mijm_ij, and uiju_ij denote lower, modal, and upper bounds of preference intensity, respectively. Weight estimation is then performed through fuzzy aggregation and subsequent defuzzification procedures[1]. Compared with standard AHP, FAHP provides several advantages: • improved robustness under uncertain or subjective judgments, • better modeling of linguistic reasoning, • reduced sensitivity to inconsistent comparisons. These properties make FAHP particularly suitable for evaluation scenarios involving subjective assessment, incomplete information, or ambiguous criteria definitions. Consequently, FAHP has been increasingly applied in intelligent decision-support systems and complex evaluation environments. Figure 1: Overview of the proposed hybrid AHP-based evaluation framework. The pipeline integrates direct scoring, crisp AHP, and fuzzy AHP through consistency checking, weight derivation, and result aggregation. 3 Methodology This section presents the overall methodology of our study (see Fig 1), which aims to improve the reliability of LLM-based automatic evaluation through a hybrid judging framework. We first introduce the evaluation benchmark used in our experiments, followed by the structured decision-making mechanisms underlying our approach, and finally the proposed DualJudge framework. Our methodology is motivated by the observation that existing evaluation paradigms exhibit complementary strengths: intuitive scoring methods provide holistic judgments but lack interpretability, while structured approaches offer transparency and logical decomposition but may suffer from instability under uncertain reasoning. To address this limitation, we design a unified evaluation pipeline that integrates direct scoring with both crisp and fuzzy AHP formulations, and adaptively fuses their outputs based on consistency-aware reliability estimation. The resulting framework enables context-aware criterion construction, uncertainty modeling in pairwise comparisons, and adaptive aggregation of multiple evaluation signals, forming a principled hybrid judging process. The following subsections describe the dataset, the AHP-based evaluation mechanism, and the DualJudge architecture in detail. 3.1 Dataset We evaluate our framework on JudgeBench [18], a benchmark specifically designed to assess the judgment capability of LLMs. Unlike standard generation benchmarks that focus on output quality, JudgeBench constructs challenging pairwise comparison tasks: for each query, two model responses with minimal quality disparity are provided, along with a reliable ground-truth preference label. This design forces judge models to perform fine-grained, criteria-based analysis rather than relying on superficial heuristics (e.g., response length or formatting), making it an ideal testbed for investigating the limitations of single-paradigm evaluation methods. Data Composition. JudgeBench aggregates questions from LiveBench [20] and MMLU-Pro [19], covering 17 fine-grained categories across STEM, humanities, and professional domains. To enable robust cross-generator validation, the benchmark provides two independent response sets: • GPT Split: Response pairs generated by GPT-4-class models (350 samples total). • Claude Split: Response pairs generated by Claude-3-class models (270 samples total). These splits ensure that observed improvements are not artifacts of a specific response style or generator bias. Table 1 details the category-wise distribution. Table 1: Category distribution of JudgeBench subsets by source and evaluation split. MMLU-Pro categories are aggregated for brevity. Source Categories GPT Claude LiveBench livebench-math 56 34 livebench-reasoning 98 51 livecodebench 42 31 MMLU-Pro 14 disciplines (math, physics, CS, law, etc.) 154 154 Total 17 categories 350 270 3.2 Crisp and Fuzzy AHP Evaluation Figure 2: Despite a tie in direct scoring, the structured AHP-based evaluations (crisp and fuzzy) consistently favor Response A. Criterion importance is estimated using both the classical AHP and its uncertainty-aware extension, the FAHP. Both formulations operate on a shared pairwise comparison structure, differing only in how uncertainty in LLM judgments is modeled and propagated during aggregation. To provide further intuition into how the proposed framework operates in practice, we present a concrete example in Fig. 2. 3.2.1 Pairwise Comparison Matrix Construction Let the evaluation criteria be =c1,c2,…,cK.C=\c_1,c_2,…,c_K\. The LLM produces pairwise importance assessments represented as (ci,cj,sij)(c_i,c_j,s_ij), where sij∈1,…,9s_ij∈\1,…,9\ follows Saaty’s fundamental scale. These comparisons define a reciprocal judgment matrix =[1a12⋯a1K1/a121⋯a2K⋮⋱⋮1/a1K1/a2K⋯1],aij=1,i=j,sij,i<j,1/sji,i>jA= bmatrix1&a_12&·s&a_1K\\ 1/a_12&1&·s&a_2K\\ & & & \\ 1/a_1K&1/a_2K&·s&1 bmatrix,a_ij= cases1,&i=j,\\ s_ij,&i<j,\\ 1/s_ji,&i>j\\ cases (1) 3.2.2 Crisp AHP Weight Estimation Criterion weights are computed using the principal eigenvector method. Let λmax _ denote the dominant eigenvalue of A. The crisp weight vector cw^c satisfies c=λmaxc,Aw^c= _ w^c, followed by normalization wic=vi∑k=1Kvk.w_i^c= v_i _k=1^Kv_k. Logical consistency is quantified using the CR: CI=λmax−K−1,CR=CIRIK,CI= _ -KK-1, CR= CIRI_K, where RIKRI_K denotes the random index for matrix size K. The matrix is considered acceptable when CR≤τCR≤τ. where τ=0.15τ=0.15 [6] is adopted as the consistency threshold. 3.2.3 Automatic Consistency Repair When the consistency constraint is violated, a bounded automatic repair procedure is applied to reduce local inconsistencies in the judgment matrix. The repair enforces approximate multiplicative transitivity, aik≈aijajk,a_ik≈ a_ija_jk, which characterizes consistent pairwise comparisons in the analytic hierarchy process [16]. Entries exhibiting excessive logarithmic deviation from their transitive estimates are locally adjusted while preserving the reciprocity condition: aik←a^ik,aki=1/a^ik.a_ik← a_ik, a_ki=1/ a_ik. The repair strategy is heuristic and performs a limited number of local corrections within a single repair phase rather than iteratively optimizing the entire matrix[3, 24]. Updated entries are projected onto the admissible Saaty scale range to preserve interpretability of comparison intensities [16]. After repair, the CR is recomputed. If the repaired matrix satisfies the consistency requirement, it is accepted for weight estimation. Otherwise, the judgment matrix is considered structurally inconsistent and the pairwise comparisons are regenerated instead of continuing further repair. Matrices exhibiting unstable eigenvalue growth or extreme comparison ratios are likewise treated as structurally inconsistent and trigger regeneration of comparisons. 3.2.4 Fuzzy AHP Representation To capture uncertainty inherent in LLM judgments, each discrete comparison score is mapped to a TFN a~ij=(lij,mij,uij),lij≤mij≤uij. a_ij=(l_ij,m_ij,u_ij), l_ij≤ m_ij≤ u_ij. A predefined mapping function converts discrete scores into TFNs: a~ij=Scale(sij,γij), a_ij=Scale(s_ij, _ij), where the confidence value γij∈[0,1] _ij∈[0,1] controls the degree of uncertainty contraction. Confidence directly shrinks the fuzzy interval toward the modal value while keeping the modal judgment unchanged: lij′=lij+(mij−lij)γij,uij′=uij−(uij−mij)γij.l _ij=l_ij+(m_ij-l_ij) _ij, u _ij=u_ij-(u_ij-m_ij) _ij. The resulting confidence-adjusted TFN is a~ij′=(lij′,mij,uij′) a _ij=(l _ij,m_ij,u _ij), such that higher confidence produces narrower fuzzy ranges, representing reduced epistemic uncertainty in LLM-generated comparisons. Reciprocity is preserved in the fuzzy domain: a~ji′=(1uij′,1mij,1lij′). a _ji= ( 1u _ij, 1m_ij, 1l _ij ). 3.2.5 Fuzzy Weight Computation Criterion importance is estimated using the fuzzy geometric mean method: g~i=(∏j=1Ka~ij)1/K. g_i= ( _j=1^K a_ij )^1/K. Let g~i=(li(g),mi(g),ui(g)) g_i=(l_i^(g),m_i^(g),u_i^(g)) denote the resulting TFN. Fuzzy weights are obtained via modal-based scalar normalization: w~i=(li(g)∑k=1Kmk(g),mi(g)∑k=1Kmk(g),ui(g)∑k=1Kmk(g)). w_i= ( l_i^(g) _k=1^Km_k^(g), m_i^(g) _k=1^Km_k^(g), u_i^(g) _k=1^Km_k^(g) ). Crisp priorities are obtained using centroid defuzzification: wi=li+mi+ui3,w_i= l_i+m_i+u_i3, followed by simplex normalization such that ∑i=1Kwi=1. _i=1^Kw_i=1. 3.2.6 Shared Matrix Principle Both crisp and fuzzy weights are derived from the same repaired comparison matrix. The fuzzy matrix is reconstructed from repaired scores while preserving confidence signals, ensuring that differences between AHP and FAHP arise solely from uncertainty modeling rather than inconsistent evidence. 3.3 DualJudge We introduce DualJudge, an adaptive hybrid evaluation framework designed to address the limitations of single-paradigm LLM assessment. Existing approaches typically rely either on holistic, intuitive judgments (e.g., direct scoring) or structured, decompositional methods (e.g., AHP). Our empirical analysis suggests that these paradigms exhibit complementary error patterns across task domains. DualJudge integrates both within a unified architecture conceptually inspired by Dual-Process Theory [7], which distinguishes between fast, intuitive reasoning (System 1) and slow, deliberative reasoning (System 2). In our framework, the absolute scoring branch corresponds to intuitive evaluation, while the structured AHP branch reflects deliberative reasoning. CR-Aware Adaptive Fusion. Let Sahp∈[0,1]S_ahp∈[0,1] and Sabs∈[0,1]S_abs∈[0,1] denote the evaluation scores produced by the structured and absolute branches, respectively. The final score is computed via an adaptive fusion mechanism. We first incorporate a consistency screening step based on the AHP CR. Following standard AHP practice, structured evaluations with excessive inconsistency are considered unreliable. We therefore adopt the same consistency threshold τ defined in Section 3.2 and treat comparisons satisfying CR≤τCR≤τ as reliable. Beyond binary filtering, we further interpret the CR as a continuous reliability signal [15, 24]. Lower CR values indicate stronger internal agreement among pairwise comparisons and thus higher confidence in the structured evaluation. The fusion weight assigned to the structured branch is defined as α(CR)=exp(−βCR),α(CR)= (-β CR), where β controls the sensitivity to residual inconsistency. This formulation ensures that the structured branch maintains a non-negligible influence even under moderate inconsistency, reflecting the assumption that partially consistent structured reasoning still provides useful signals. Since all observed CR values lie within the reliable regime, β primarily modulates relative confidence rather than correcting severely inconsistent cases. We therefore fix β=7β=7 in all experiments. The final score is computed as Sfinal=α(CR)Sahp+(1−α(CR))Sabs.S_final=α(CR)S_ahp+ (1-α(CR) )S_abs. 4 Experiments 4.1 Experimental Setup All experiments are conducted on JudgeBench using GPT-oss and Qwen3.5 [13] models as the judge backbone. We evaluate two model scales for each model (gpt-oss-20b, gpt-oss-120b) [12] and Qwen3.5-35B-A3B, Qwen3.5-9B across both GPT and Claude response splits to ensure robustness to generator variation. Two scoring granularities are considered (1–10 and 1–5 scales [9]), allowing us to analyze the effect of resolution on structured evaluation stability. Evaluation performance is measured as agreement accuracy with ground-truth pairwise preferences. All LLM interactions are cached to ensure reproducibility and eliminate sampling variance. 4.2 Results Table 2 summarizes model performance across scales, scoring resolutions, while Tables 0.A (Appendix) provide detailed per-domain breakdowns. Table 2: Overall performance comparison across models and settings (Merged GPT/Claude splits) Model Scale Count Direct AHP DualJudge (Baseline) Crisp Fuzzy D+C D+F gpt-oss-20b 1–10 620 69.83% 74.52% 75.35% 76.95% 77.60% 1–5 620 71.77% 75.48% 76.45% 78.06% 78.55% gpt-oss-120b 1–10 620 75.81% 80.49% 80.97% 82.10% 82.10% 1–5 620 75.00% 78.06% 78.23% 78.87% 78.71% qwen3.5-9b 1–10 620 82.70% 81.44% 83.55% 84.07% 84.03% 1–5 620 81.91% 82.40% 83.87% 84.66% 84.21% qwen3.5-35b 1–10 620 87.38% 83.65% 85.47% 87.19% 87.19% 1–5 620 85.73% 83.60% 86.24% 86.95% 86.69% Structured Reasoning vs. Direct Scoring. Across the aggregated results , structured evaluation methods (AHP and DualJudge) generally outperform direct scoring, with the magnitude of improvement inversely correlated with base model capability. For weaker evaluators like gpt-oss-20b, structured frameworks yield substantial gains: under the 1–10 scale, accuracy improves from 69.83% (Direct) to 75.35% (Fuzzy AHP) and peaks at 77.60% (DualJudge D+F). The trend persists for gpt-oss-120b (+4.7–6.3 p) and qwen3.5-9b (+1.4–2.0 p). However, for the strongest evaluator qwen3.5-35b under the 1–10 scale, direct scoring (87.38%) remains competitive, with structured variants achieving 85.47–87.19%. This suggests that decomposed reasoning provides the most value when base judgments are noisy or poorly calibrated, while highly capable models may already internalize multi-criteria trade-offs. Fuzzy vs. Crisp AHP. Fuzzy AHP consistently matches or exceeds Crisp AHP across model families and scales, supporting the hypothesis that explicit uncertainty modeling better accommodates the stochastic nature of LLM outputs. The advantage is most pronounced for mid-tier models: e.g., gpt-oss-20b (1–5 scale) improves from 75.48% (Crisp) to 76.45% (Fuzzy), and qwen3.5-9b (1–10) from 81.44% to 83.55%. For qwen3.5-35b, the gap narrows but Fuzzy still leads in 3 of 4 settings (e.g., 1–5 scale: 86.24% vs. 83.60%). The sole exception occurs for qwen3.5-35b (1–10), where both AHP variants underperform Direct scoring, indicating that fuzzy logic cannot fully compensate when the base evaluator’s holistic judgments are already near-optimal. DualJudge Performance. The hybrid DualJudge approaches (D+C and D+F) achieve peak accuracy in 7 of 8 configurations, demonstrating the benefit of fusing direct and structured signals. Notably, hybrid approaches show the most consistent gains on the Claude data split, which generally contains harder-to-distinguish response pairs [18]. Gains are largest for weaker models: gpt-oss-20b sees a +7.77 p improvement (69.83% → 77.60%) under the 1–10 scale, while gpt-oss-120b gains +6.29 p (75.81% → 82.10%). For stronger models, improvements diminish but remain positive: qwen3.5-9b gains +1.37–2.75 p, and qwen3.5-35b shows marginal gains (+0.19 p) or parity under 1–10. Notably, D+F (Fuzzy fusion) tends to lead for 20B-class models, while D+C (Crisp fusion) is competitive or superior for 9B/35B-class models, suggesting that the optimal fusion strategy may depend on the evaluator’s inherent uncertainty calibration. Scale Sensitivity and Task Generalization. The 1–5 scale generally yields slightly higher absolute accuracy than 1–10 across methods (e.g., gpt-oss-20b Direct: 71.77% vs. 69.83%), likely due to reduced granularity easing the judgment burden. However, the relative gains from structured methods remain consistent across scales, indicating robustness to scoring protocol. Task-level analysis (see Appendix 0.A) reveals that gains concentrate on reasoning-intensive benchmarks: LiveCodeBench and MMLU-Pro mathematics/physics show 8–12 p improvements for weaker models under DualJudge, while simpler factual categories exhibit smaller deltas. This pattern reinforces that structured decomposition is most valuable for cognitively complex, multi-attribute evaluation tasks. Collectively, these aggregated results substantiate the central hypothesis: structured reasoning frameworks complement direct scoring, with adaptive fusion (DualJudge) yielding the most robust performance. The inverse relationship between base model capability and structured-method gains suggests a practical guideline: employ lightweight Direct scoring for state-of-the-art evaluators on well-defined tasks, but prioritize AHP/DualJudge pipelines for smaller models, ambiguous criteria, or high-stakes evaluation scenarios. The consistency across model families, scales, and evaluator backends (GPT vs. Claude splits in Appendix) underscores the generalizability of the proposed approach. 5 Discussion The empirical results presented in Section 4.2 validate the core premise of our work: that hybrid evaluation frameworks can effectively mitigate the inherent instability of LLM-based judgment. The fact that DualJudge consistently outperforms other approaches across model scales and dataset splits suggests that combining fast, intuitive reasoning (System 1, based on the absolute scoring branch) with slower, deliberative reasoning (System 2, based on the structured AHP branch) creates something greater than the sum of its parts — the big-picture view of direct scoring covers for the blind spots of rigid step-by-step decomposition, and the other way around. Structural Scaffolding for Weaker Evaluators. The magnitude of performance gains is inversely correlated with base model capability, indicating that structured reasoning acts as a cognitive scaffold. For gpt-oss-20b, DualJudge improves accuracy by +7.77 p (1–10 scale), whereas gpt-oss-120b and qwen3.5-9b see gains of +6.29 p and +1.37 p, respectively. At the upper end, qwen3.5-35b already achieves 87.38% with direct scoring, and structured variants yield parity or marginal changes. Weaker models, which often lack calibrated internal representations for complex evaluation tasks, benefit significantly from the explicit decomposition and consistency enforcement of AHP. Unlike standard chain-of-thought prompting, which relies on implicit step generation, our framework imposes mathematical constraints (e.g., consistency ratio checks) and explicit criterion weighting, providing a more rigorous scaffold for judgment calibration. Modeling Epistemic Uncertainty. Fuzzy AHP consistently matches or exceeds Crisp AHP across model families and scales, underscoring the probabilistic nature of LLM outputs. LLMs do not produce deterministic truths; their token distributions reflect varying degrees of confidence. Crisp AHP forces these stochastic judgments into discrete Saaty scales, potentially discarding valuable nuance. By mapping pairwise comparisons to triangular fuzzy numbers (TFNs) modulated by confidence scores, Fuzzy AHP preserves uncertainty, allowing the aggregation mechanism to down-weight less reliable judgments. This advantage is most pronounced for mid-tier models and ambiguous tasks. However, when base evaluators are highly calibrated (e.g., qwen3.5-35b), the overhead of fuzzy decomposition yields diminishing returns, and direct scoring remains competitive. This suggests that future evaluation frameworks should adopt uncertainty-aware aggregation but may require adaptive routing to avoid unnecessary complexity for near-optimal baselines. Domain Sensitivity and Adaptive Fusion. Performance gains are heterogeneous across domains, confirming that no single paradigm is universally optimal. On cognitively complex benchmarks like LiveCodeBench and MMLU-Pro mathematics/physics, DualJudge yields substantial improvements (e.g., +11.9 p for gpt-oss-20b on LiveCodeBench), likely because multi-criteria decomposition aligns with the inherent structure of code correctness, efficiency, and style. In contrast, factual or recall-heavy domains (e.g., history, law) exhibit smaller deltas, as direct scoring already captures salient signals efficiently. Our adaptive fusion mechanism—which dynamically weights the structured branch based on consistency ratios and confidence—partially addresses this variability. Nevertheless, the results suggest that domain-aware routing or task-specific criterion weighting could further optimize the trade-off between evaluation fidelity and computational cost. Practical Implications and Future Directions. Collectively, these findings offer actionable guidelines for LLM evaluation: (1) employ lightweight direct scoring for state-of-the-art models on well-defined tasks; (2) prioritize DualJudge pipelines for smaller models, high-stakes assessments, or ambiguous criteria; and (3) leverage fuzzy aggregation when evaluator calibration is uncertain or task complexity is high. Future work should explore dynamic thresholding for consistency checks, multimodal criterion decomposition, and real-time adaptive routing to minimize overhead while preserving robustness. As LLMs continue to scale, structured evaluation frameworks will remain essential for ensuring reliable, transparent, and reproducible judgment in automated assessment pipelines. 6 Limitations and Future Work While our results demonstrate the efficacy of the DualJudge framework, several limitations warrant acknowledgment and provide directions for future research. Computational Overhead. The primary limitation of the proposed methodology is increased computational cost. The AHP component requires pairwise comparisons for criteria, and DualJudge necessitates running both direct and structured pipelines. For real-time evaluation or large-scale RLHF data processing, this latency may be prohibitive. Future Work: We plan to investigate efficiency optimizations, such as sparse pairwise comparison (evaluating only the most critical criterion pairs) or employing a cascade strategy where the lightweight Direct score is used first, and the expensive AHP pipeline is triggered only for low-confidence or high-stakes samples. Dependency on Criteria Quality. Our experiments utilize a Category Mode where shared criteria are derived for each dataset category. The performance of AHP is inherently bound to the relevance and completeness of these criteria. If the predefined criteria fail to capture key aspects of response quality, the structured evaluation may be systematically biased. Future Work: We aim to explore dynamic criterion generation, where the LLM generates task-specific evaluation criteria on a per-sample basis before constructing the pairwise matrix. This would enhance adaptability to novel or out-of-distribution tasks. Model and Dataset Scope. Our study focuses on the gpt-oss and qwen3.5 families and the JudgeBench benchmark. While JudgeBench is rigorous, it is limited to pairwise comparison tasks with ground truth. The generalizability to single-response scoring (e.g., reward modeling for RLHF) or open-weight models (e.g., Llama, Qwen) remains to be fully verified. Future Work: We intend to extend DualJudge to reward modeling pipelines, using the hybrid scores as fine-tuning signals for smaller reward models. Additionally, we will validate the framework across a broader spectrum of open-source architectures to ensure the findings are not model-specific. Hyperparameter Sensitivity. The fusion mechanism relies on fixed hyperparameters. While these values performed robustly in our settings, optimal parameters may vary across different model families or task complexities. Future Work: Developing a learnable fusion module that adapts the weighting strategy based on task embeddings or model characteristics could further optimize performance without manual tuning. 7 Conclusion In this work, we introduced DualJudge, a hybrid evaluation framework that unifies intuitive direct scoring with structured AHP reasoning. To validate the effectiveness of each component, we conducted a series of systematic comparative experiments across Direct Scoring, Crisp AHP, Fuzzy AHP, and the hybrid DualJudge framework. Our extensive experiments on JudgeBench demonstrate a clear performance hierarchy: structured methods (AHP/FAHP) consistently outperform direct scoring, with Fuzzy AHP further improving stability over Crisp AHP by explicitly modeling epistemic uncertainty. Ultimately, DualJudge achieves the highest accuracy by adaptively fusing these signals, yielding significant gains particularly in reasoning-intensive domains and for smaller model scales. The findings underscore a critical insight for the field of automated evaluation: reliability is not solely a function of model scale, but also of reasoning architecture. By explicitly modeling the evaluation process through decomposed criteria and uncertainty-aware aggregation, we can extract more reliable signals from existing models. As LLMs are increasingly deployed in high-stakes decision-making scenarios, frameworks like DualJudge offer a principled path toward more transparent, robust, and trustworthy automated evaluation systems. References [1] J.J. Buckley (1985) Fuzzy hierarchical analysis. Fuzzy Sets and Systems 17 (3), p. 233–247. External Links: ISSN 0165-0114, Document, Link Cited by: §2.3, §2.3. [2] C. Chiang and H. Lee (2023-12) A closer look into using large language models for automatic evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 8928–8942. External Links: Link, Document Cited by: §1, §1. [3] G. Crawford and C. Williams (1985) A note on the analysis of subjective judgment matrices. Journal of Mathematical Psychology 29 (4), p. 387–405. External Links: ISSN 0022-2496, Document, Link Cited by: §3.2.3. [4] D.J. Dubois (1980) Fuzzy sets and systems: theory and applications. Mathematics in Science and Engineering, Academic Press. External Links: ISBN 9780080917726, Link Cited by: §2.3. [5] E. Emirtekin (2025) Large language model-powered automated assessment: a systematic review. Applied Sciences 15 (10). External Links: Link, ISSN 2076-3417, Document Cited by: §1, §2.1. [6] A. S. José, Alonso, T. Mł, and Lamata (2006) Consistency in the analytic hierarchy process: a new approach. Int. J. Uncertain. Fuzziness Knowl. Based Syst. 14, p. 445–459. External Links: Link Cited by: §3.2.2. [7] D. Kahneman (2011) Thinking, fast and slow. macmillan. Cited by: §3.3. [8] V. Kampourakis, G. Kavallieratos, G. Spathoulas, V. Gkioulos, and S. Katsikas (2025) LLM-assisted ahp for explainable cyber range evaluation. External Links: 2512.10487, Link Cited by: §1, §2.1. [9] W. Li, M. Zhao, W. Dong, J. Cai, Y. Wei, M. Pocress, Y. Li, W. Yuan, X. Wang, R. Hou, K. Lou, W. Zeng, Y. Yang, Y. Du, and M. Wang (2026) Grading scale impact on llm-as-a-judge: human-llm alignment is highest on 0-5 grading scale. External Links: 2601.03444, Link Cited by: §1, §4.1. [10] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-eval: nlg evaluation using gpt-4 with better human alignment. External Links: 2303.16634, Link Cited by: §1, §2.1. [11] X. Lu, J. Li, K. Takeuchi, and H. Kashima (2024-11) AHP-powered LLM reasoning for multi-criteria evaluation of open-ended responses. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 1847–1856. External Links: Link, Document Cited by: §1, §2.1, §2.2. [12] OpenAI (2025) Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925, Link Cited by: §4.1. [13] Qwen Team (2026-02) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.1. [14] R.W. Saaty (1987) The analytic hierarchy process—what it is and how it is used. Mathematical Modelling 9 (3), p. 161–176. External Links: ISSN 0270-0255, Document, Link Cited by: §2.2. [15] T.L. Saaty and L.G. Vargas (2012) Models, methods, concepts & applications of the analytic hierarchy process. International Series in Operations Research & Management Science, Springer. External Links: ISBN 9781461435969, Link Cited by: §3.3. [16] T.L. Saaty (1980) The analytic hierarchy process: planning, priority setting, resource allocation. Advanced book program, McGraw-Hill International Book Company. External Links: ISBN 9780070543713, LCCN 79041060, Link Cited by: §2.2, §3.2.3, §3.2.3. [17] S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. Popa, and I. Stoica (2025) JudgeBench: a benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. [18] S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. A. Popa, and I. Stoica (2024) JudgeBench: a benchmark for evaluating llm-based judges. External Links: Link Cited by: §3.1, §4.2. [19] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen (2024) MMLU-pro: a more robust and challenging multi-task language understanding benchmark. External Links: 2406.01574, Link Cited by: §3.1. [20] C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, Shubh-Agrawal, S. S. Sandha, S. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum (2025) LiveBench: a challenging, contamination-limited llm benchmark. External Links: 2406.19314, Link Cited by: §3.1. [21] H. Wu, S. Zhou, H. Zhang, and W. Chen (2026) Doc2AHP: inferring structured multi-criteria decision models via semantic trees with llms. External Links: 2601.16479, Link Cited by: §1, §2.1. [22] J. Xie, Y. Li, X. Yin, and X. Wan (2025) DSGram: dynamic weighting sub-metrics for grammatical error correction in the era of large language models. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’25/IAAI’25/EAAI’25. External Links: ISBN 978-1-57735-897-8, Link, Document Cited by: §1, §2.1. [23] L.A. Zadeh (1965) Fuzzy sets. Information and Control 8 (3), p. 338–353. External Links: ISSN 0019-9958, Document, Link Cited by: §2.3. [24] X. Zeshui and W. Cuiping (1999) A consistency improving method in the analytic hierarchy process1research supported by nsf of china and shandong.1. European Journal of Operational Research 116 (2), p. 443–449. External Links: ISSN 0377-2217, Document, Link Cited by: §3.2.3, §3.3. Appendix 0.A Full Results Performance comparison between gpt-oss-20b and gpt-oss-120b across dataset splits under two scoring scales. Values denote accuracy (%). Higher is better. Category Split N gpt-oss-20b gpt-oss-120b 1–10 Scale 1–5 Scale 1–10 Scale 1–5 Scale Dir Cri Fuz DC DF Dir Cri Fuz DC DF Dir Cri Fuz DC DF Dir Cri Fuz DC DF LiveBench math GPT 56 82.182.1 87.587.5 89.389.3 87.587.5 87.587.5 75.075.0 87.587.5 89.389.3 87.587.5 87.587.5 87.587.5 89.389.3 89.389.3 92.992.9 92.992.9 82.182.1 85.785.7 85.785.7 87.587.5 87.587.5 Claude 34 67.667.6 79.479.4 91.291.2 76.576.5 91.291.2 79.479.4 79.479.4 79.479.4 79.479.4 79.479.4 76.576.5 82.482.4 82.482.4 82.482.4 82.482.4 76.576.5 82.482.4 82.482.4 82.482.4 82.482.4 reasoning GPT 98 73.573.5 73.573.5 73.573.5 77.677.6 77.677.6 80.680.6 85.785.7 85.785.7 88.888.8 88.888.8 88.888.8 90.890.8 91.891.8 91.891.8 91.891.8 87.887.8 87.887.8 87.887.8 88.888.8 88.888.8 Claude 51 80.480.4 80.480.4 78.478.4 88.288.2 86.386.3 86.386.3 70.670.6 76.576.5 82.482.4 82.482.4 84.384.3 94.194.1 94.194.1 94.194.1 94.194.1 88.288.2 90.290.2 90.290.2 92.292.2 90.290.2 livecodebench GPT 42 83.383.3 88.188.1 90.590.5 92.992.9 95.295.2 85.785.7 83.383.3 83.383.3 92.992.9 92.992.9 92.992.9 90.590.5 90.590.5 95.295.2 95.295.2 90.590.5 95.295.2 95.295.2 97.697.6 97.697.6 Claude 31 74.274.2 74.274.2 71.071.0 90.390.3 90.390.3 71.071.0 80.680.6 80.680.6 87.187.1 87.187.1 67.767.7 83.983.9 87.187.1 90.390.3 90.390.3 74.274.2 80.680.6 80.680.6 80.680.6 80.680.6 MMLU-Pro biology GPT 11 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 45.545.5 45.545.5 45.545.5 45.545.5 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 Claude 11 54.554.5 63.663.6 63.663.6 54.554.5 63.663.6 45.545.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 business GPT 11 72.772.7 63.663.6 63.663.6 63.663.6 63.663.6 81.881.8 54.554.5 54.554.5 63.663.6 63.663.6 90.990.9 63.663.6 63.663.6 81.881.8 81.881.8 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 54.554.5 72.772.7 72.772.7 72.772.7 72.772.7 54.554.5 63.663.6 72.772.7 63.663.6 63.663.6 chemistry GPT 11 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 81.881.8 81.881.8 81.881.8 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 CS GPT 11 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 90.990.9 81.881.8 81.881.8 90.990.9 90.990.9 economics GPT 11 54.554.5 63.663.6 63.663.6 72.772.7 72.772.7 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 72.772.7 72.772.7 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 Claude 11 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 63.663.6 72.772.7 63.663.6 63.663.6 63.663.6 72.772.7 81.881.8 81.881.8 81.881.8 72.772.7 engineering GPT 11 72.772.7 63.663.6 63.663.6 63.663.6 63.663.6 45.545.5 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 90.990.9 90.990.9 100.0100.0 100.0100.0 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 45.545.5 63.663.6 63.663.6 63.663.6 63.663.6 36.436.4 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 81.881.8 81.881.8 81.881.8 81.881.8 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 health GPT 11 54.554.5 72.772.7 72.772.7 72.772.7 72.772.7 63.663.6 90.990.9 90.990.9 90.990.9 90.990.9 54.554.5 81.881.8 81.881.8 72.772.7 72.772.7 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 Claude 11 90.990.9 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 history GPT 11 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 54.554.5 54.554.5 63.663.6 63.663.6 72.772.7 63.663.6 63.663.6 72.772.7 72.772.7 Claude 11 54.554.5 36.436.4 36.436.4 45.545.5 45.545.5 63.663.6 45.545.5 45.545.5 45.545.5 45.545.5 63.663.6 63.663.6 63.663.6 72.772.7 72.772.7 63.663.6 63.663.6 63.663.6 54.554.5 54.554.5 law GPT 11 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 63.663.6 63.663.6 72.772.7 54.554.5 54.554.5 63.663.6 45.545.5 45.545.5 54.554.5 45.545.5 45.545.5 45.545.5 54.554.5 Claude 11 36.436.4 72.772.7 72.772.7 72.772.7 72.772.7 36.436.4 54.554.5 54.554.5 45.545.5 54.554.5 45.545.5 72.772.7 72.772.7 72.772.7 72.772.7 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 math GPT 11 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 81.881.8 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 other GPT 11 72.772.7 63.663.6 63.663.6 72.772.7 63.663.6 63.663.6 63.663.6 63.663.6 72.772.7 72.772.7 45.545.5 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 Claude 11 54.554.5 72.772.7 72.772.7 63.663.6 63.663.6 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 philosophy GPT 11 72.772.7 54.554.5 54.554.5 63.663.6 63.663.6 45.545.5 63.663.6 63.663.6 63.663.6 63.663.6 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 63.663.6 63.663.6 63.663.6 63.663.6 Claude 11 27.327.3 54.554.5 54.554.5 36.436.4 36.436.4 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 72.772.7 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 physics GPT 11 36.436.4 63.663.6 63.663.6 63.663.6 63.663.6 45.545.5 90.990.9 90.990.9 81.881.8 81.881.8 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 36.436.4 72.772.7 72.772.7 72.772.7 72.772.7 Claude 11 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 100.0100.0 90.990.9 100.0100.0 100.0100.0 90.990.9 100.0100.0 100.0100.0 100.0100.0 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 psychology GPT 11 45.545.5 72.772.7 81.881.8 63.663.6 63.663.6 45.545.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 63.663.6 63.663.6 54.554.5 54.554.5 36.436.4 63.663.6 63.663.6 63.663.6 63.663.6 Claude 11 36.436.4 45.545.5 45.545.5 36.436.4 36.436.4 36.436.4 27.327.3 27.327.3 27.327.3 27.327.3 27.327.3 36.436.4 36.436.4 36.436.4 36.436.4 36.436.4 54.554.5 27.327.3 27.327.3 27.327.3 Overall GPT 350 71.771.7 76.076.0 76.976.9 78.678.6 78.678.6 73.173.1 80.080.0 80.680.6 82.682.6 82.982.9 79.479.4 81.481.4 82.082.0 83.483.4 83.483.4 76.076.0 79.779.7 79.779.7 80.980.9 81.181.1 Claude 270 67.467.4 72.672.6 73.373.3 74.874.8 76.376.3 70.070.0 69.669.6 71.171.1 72.272.2 73.073.0 71.171.1 79.379.3 79.679.6 80.480.4 80.480.4 73.773.7 75.975.9 76.376.3 76.376.3 75.675.6 Performance comparison between Qwen3.5-9B and Qwen3.5-35B-A3B across dataset splits under two scoring scales. Values denote accuracy (%). Higher is better. Category Split N Qwen3.5-9B Qwen3.5-35B-A3B 1–10 Scale 1–5 Scale 1–10 Scale 1–5 Scale Dir Cri Fuz DC DF Dir Cri Fuz DC DF Dir Cri Fuz DC DF Dir Cri Fuz DC DF LiveBench math GPT 56 91.191.1 87.587.5 89.389.3 87.587.5 87.587.5 87.587.5 82.182.1 82.182.1 89.389.3 87.587.5 89.389.3 82.182.1 83.983.9 87.587.5 87.587.5 83.983.9 87.587.5 89.389.3 89.389.3 89.389.3 Claude 34 85.385.3 85.385.3 88.288.2 88.288.2 88.288.2 91.291.2 88.288.2 91.291.2 94.194.1 94.194.1 91.291.2 88.288.2 91.291.2 88.288.2 88.288.2 94.194.1 79.479.4 85.385.3 94.194.1 94.194.1 reasoning GPT 98 95.995.9 90.890.8 91.891.8 92.992.9 93.993.9 94.994.9 89.889.8 89.889.8 90.890.8 89.889.8 93.993.9 85.785.7 89.889.8 91.891.8 91.891.8 93.993.9 89.889.8 93.993.9 92.992.9 92.992.9 Claude 51 88.288.2 88.288.2 92.292.2 92.292.2 94.194.1 88.288.2 88.288.2 90.290.2 92.292.2 92.292.2 96.196.1 96.196.1 96.196.1 98.098.0 98.098.0 100.0100.0 92.292.2 92.292.2 92.292.2 92.292.2 livecodebench GPT 42 78.678.6 95.295.2 97.697.6 95.295.2 95.295.2 73.873.8 88.188.1 85.785.7 85.785.7 83.383.3 92.992.9 85.785.7 88.188.1 92.992.9 92.992.9 85.785.7 92.992.9 92.992.9 92.992.9 92.992.9 Claude 31 58.158.1 77.477.4 77.477.4 80.680.6 80.680.6 67.767.7 80.680.6 83.983.9 83.983.9 83.983.9 77.477.4 87.187.1 87.187.1 93.593.5 90.390.3 87.187.1 90.390.3 90.390.3 90.390.3 90.390.3 MMLU-Pro biology GPT 11 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 81.881.8 81.881.8 72.772.7 72.772.7 72.772.7 63.663.6 72.772.7 72.772.7 72.772.7 Claude 11 63.663.6 45.545.5 45.545.5 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 54.554.5 72.772.7 72.772.7 72.772.7 72.772.7 business GPT 11 81.881.8 63.663.6 72.772.7 72.772.7 63.663.6 72.772.7 54.554.5 63.663.6 63.663.6 63.663.6 81.881.8 63.663.6 63.663.6 63.663.6 63.663.6 81.881.8 72.772.7 72.772.7 72.772.7 72.772.7 Claude 11 90.990.9 72.772.7 81.881.8 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 chemistry GPT 11 81.881.8 90.990.9 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 72.772.7 72.772.7 72.772.7 Claude 11 90.990.9 72.772.7 72.772.7 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 81.881.8 81.881.8 90.990.9 77.377.3 77.377.3 81.881.8 81.881.8 CS GPT 11 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 Claude 11 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 economics GPT 11 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 90.990.9 100.0100.0 100.0100.0 100.0100.0 Claude 11 72.772.7 72.772.7 81.881.8 72.772.7 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 engineering GPT 11 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 81.881.8 81.881.8 81.881.8 81.881.8 100.0100.0 81.881.8 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 Claude 11 90.990.9 90.990.9 100.0100.0 90.990.9 90.990.9 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 90.990.9 90.990.9 90.990.9 81.881.8 health GPT 11 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 54.554.5 63.663.6 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 Claude 11 90.990.9 72.772.7 72.772.7 72.772.7 72.772.7 90.990.9 90.990.9 100.0100.0 90.990.9 90.990.9 81.881.8 72.772.7 81.881.8 81.881.8 81.881.8 86.486.4 81.881.8 81.881.8 86.486.4 86.486.4 history GPT 11 90.990.9 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 Claude 11 72.772.7 45.545.5 54.554.5 45.545.5 45.545.5 54.554.5 72.772.7 72.772.7 72.772.7 72.772.7 81.881.8 63.663.6 63.663.6 72.772.7 72.772.7 52.652.6 52.652.6 63.263.2 63.263.2 63.263.2 law GPT 11 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 63.663.6 72.772.7 72.772.7 72.772.7 90.990.9 72.772.7 72.772.7 81.881.8 81.881.8 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 Claude 11 100.0100.0 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 90.990.9 100.0100.0 81.881.8 81.881.8 90.990.9 90.990.9 90.990.9 math GPT 11 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 81.881.8 90.990.9 90.990.9 90.990.9 Claude 11 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 90.990.9 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 95.595.5 100.0100.0 100.0100.0 100.0100.0 other GPT 11 54.554.5 63.663.6 63.663.6 63.663.6 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 72.772.7 90.990.9 90.990.9 90.990.9 90.990.9 Claude 11 54.554.5 54.554.5 63.663.6 63.663.6 63.663.6 72.772.7 72.772.7 72.772.7 72.772.7 72.772.7 63.663.6 81.881.8 81.881.8 81.881.8 81.881.8 63.663.6 72.772.7 81.881.8 81.881.8 81.881.8 philosophy GPT 11 81.881.8 90.990.9 90.990.9 100.0100.0 90.990.9 81.881.8 72.772.7 72.772.7 72.772.7 72.772.7 90.990.9 72.772.7 72.772.7 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 81.881.8 Claude 11 54.554.5 45.545.5 45.545.5 54.554.5 54.554.5 54.554.5 36.436.4 36.436.4 36.436.4 36.436.4 81.881.8 90.990.9 90.990.9 90.990.9 90.990.9 90.590.5 57.157.1 61.961.9 76.276.2 76.276.2 physics GPT 11 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 100.0100.0 Claude 11 100.0100.0 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 90.990.9 100.0100.0 100.0100.0 100.0100.0 100.0100.0 100.0100.0 90.990.9 90.990.9 90.990.9 100.0100.0 90.990.9 psychology GPT 11 54.554.5 45.545.5 45.545.5 45.545.5 45.545.5 54.554.5 63.663.6 63.663.6 72.772.7 72.772.7 54.554.5 63.663.6 63.663.6 72.772.7 72.772.7 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 Claude 11 36.436.4 45.545.5 45.545.5 45.545.5 45.545.5 45.545.5 36.436.4 45.545.5 45.545.5 45.545.5 63.663.6 54.554.5 54.554.5 54.554.5 54.554.5 54.554.5 59.159.1 59.159.1 54.554.5 54.554.5 Overall GPT 350 85.185.1 85.185.1 86.686.6 86.986.9 86.686.6 83.783.7 83.183.1 83.783.7 85.185.1 84.384.3 88.688.6 82.382.3 84.684.6 87.187.1 87.187.1 86.686.6 86.386.3 88.988.9 88.388.3 88.388.3 Claude 270 79.679.6 76.776.7 79.679.6 80.480.4 80.780.7 79.679.6 81.581.5 84.184.1 84.184.1 84.184.1 85.885.8 85.485.4 86.686.6 87.387.3 87.387.3 84.684.6 80.180.1 82.882.8 85.285.2 84.684.6