Paper deep dive

PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

Cheng Chang, Yining Mao, Peng Qi

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 103

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 9/30/2026, 7:54:49 AM

Summary

The paper introduces PADMÉ, a data synthesis method for meta-evaluating Language Model (LM) agent evaluators. It addresses the difficulty of aligning LM evaluator decisions with human judgment by reformulating meta-evaluation as a preference alignment problem. PADMÉ uses small language models to generate trajectory pairs with known quality contrasts, filters them using SLM judges, and synthesizes a dataset without human involvement. Experiments show PADMÉ improves agreement with human judgment from 73% to 85% and demonstrates correlations between evaluation performance and model characteristics.

Entities (11)

PADMÉ · method · 98%Meta-Evaluation · concept · 92%communication clarity · criterion · 90%friendliness · criterion · 90%task resolution · criterion · 90%τ3-bench · dataset · 90%gpt-oss-120b · model · 85%GPT-OSS-20B · model · 85%nemotron-lightning-3.5 · model · 85%Qwen3-30B-A3B-Instruct · model · 85%Fireworks · service · 80%

Relation Signals (7)

PADMÉ → synthesizes → Dataset

confidence 95% · We build a prototype of PADMÉ and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria.

PADMÉ → uses → Small Language Models

confidence 95% · PADMÉ uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget.

PADMÉ → buildsupon → τ3-bench

confidence 90% · Building on τ3-bench, we develop an instance of PADMÉ and use it to synthesize 1,000 data pairs

PADMÉ → improves → agreement with human judgment

confidence 90% · Human validation on a 150-sample subset demonstrates that PADMÉ improves agreement with human judgment from 73% to 85% over a naive baseline.

Qwen3-30B-A3B-Instruct → usedfor → User Simulation

confidence 85% · User interactions are simulated via qwen3-30b-a3b-instruct.

gpt-oss-120b → usedin → PADMÉ pipeline

confidence 85% · Steering instructions are generated by gpt-oss-120b. Filtering employs K=2 judges (J1=nemotron-lightning-3.5 and J2=gpt-oss-120b).

nemotron-lightning-3.5 → usedin → PADMÉ pipeline

confidence 85% · Filtering employs K=2 judges (J1=nemotron-lightning-3.5 and J2=gpt-oss-120b).

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem of evaluating this alignment Meta-Evaluation. Tackling it directly is difficult: collecting human data is expensive, absolute scoring is hard to align, and using an LM meta-evaluator recurses the question of trustworthiness. We adopt a reformulation of meta-evaluation as a preference judgment problem: rather than comparing human and LM evaluator scores of a trajectory, we ask whether their implied preferences align. Building on this, we introduce PADMÉ, a data synthesis method that generates reliable criterion-based meta-evaluation data for agentic settings. PADMÉ uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget. We build a prototype of PADMÉ and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria. Human validation on a 150-sample subset demonstrates that PADMÉ improves agreement with human judgment from 73% to 85% over a naive baseline. Meta-evaluating 25 common models with our dataset demonstrates the correlations between evaluation performance and scoring granularity, leniency, and model size, among other factors.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

99,636 characters extracted from source content.

Expand or collapse full text

TAE (Trust-AI-Eval): Can We Trust AI Evaluation? PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators Cheng Chang Affiliation: Uniphore Email: cheng.chang@uniphore.com Yining Mao Affiliation: Uniphore Email: yining.mao@uniphore.com Peng Qi Affiliation: Uniphore Email: peng.qi@uniphore.com Abstract Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem of evaluating this alignment Meta-Evaluation. Tackling it directly is difficult: collecting human data is expensive, absolute scoring is hard to align, and using an LM meta-evaluator recurses the question of trustworthiness. We adopt a reformulation of meta-evaluation as a preference judgment problem: rather than comparing human and LM evaluator scores of a trajectory, we ask whether their implied preferences align. Building on this, we introduce PADMÉ, a data synthesis method that generates reliable criterion-based meta-evaluation data for agentic settings. PADMÉ uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget. We build a prototype of PADMÉ and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria. Human validation on a 150-sample subset demonstrates that PADMÉ improves agreement with human judgment from 73% to 85% over a naive baseline. Meta-evaluating 25 common models with our dataset demonstrates the correlations between evaluation performance and scoring granularity, leniency, and model size, among other factors. 1 Introduction Language models (LMs) are routinely used to evaluate LM agent systems11 1 Here, we broadly define an LM agent system as a computer program that uses language models to hold multi-turn conversations with users, call tools (other computer programs) to aid its work, and change the state of a virtual environment on behalf of its users. Critically, it produces a trajectory rather than a single response during one user session. in research and commercial use cases [1, 2, 3], and the evaluations are frequently applied over a set of distinct performance metrics [4, 5, 6]. For example, the developers of a customer-service agent on a shopping website would reasonably want to know how the agent scores, across thousands of conversations, on evaluation criteria such as friendliness, factuality, and answer relevance. The evaluation criteria can be stylistic, open-ended, or demand an intelligent understanding of whole conversations and their outcomes to be properly evaluated, so the environment lacks natural and deterministic signals for assessing them [2, 7]. In these cases, LM evaluators may be the only viable option for automated evaluations. Figure 1: Overview of criterion-specific meta-evaluation. Given multi-step user-agent interaction trajectories evaluated under a specific criterion c (e.g., communication clarity), an LM evaluator f assigns a continuous score sf∈[0,1]s_f∈[0,1]. Meta-evaluation asks the question “Which of the evaluators (s1,s2,s3s_1,s_2,s_3) should I trust?”, which PADMÉ attempts to solve through preference alignment. For the scores provided by the LM evaluator to be useful, they must reliably reflect the actual performance of the agent over the criteria. For agentic systems that mostly serve human users through conversations and environment interactions, the alignment between human and LM evaluator judgments is a natural estimate of the reliability of the LM evaluator [1, 3, 7]. We call the assessment of this alignment the Meta-Evaluation problem (Figure 1). Meta-evaluation is well established for single-turn response evaluations and reinforcement learning with human feedback [8, 9, 10, 11]. It has only recently reached agentic settings in limited scenarios [2, 12, 13]. A direct method would involve collecting human annotations for each evaluated agent over each criterion of interest [1, 7, 14], but human labeling is costly, which makes it especially prohibitive for rapidly developing agent platforms with a massive number of agent designs, use cases, and datasets. Additionally, maintaining consistency when rating items on an absolute scale is hard for human annotators [15, 16]. Another direct solution is to use a more capable model, a “meta-evaluator”, to judge the evaluation of the LM evaluator. For example, finetuned evaluators are commonly benchmarked against frontier LMs [17, 18, 19]. However, this method requires access to stronger models, which can be prohibitive in itself. It also recurses the meta-evaluation problem, since nothing guarantees the reliability of the LM meta-evaluator itself [7, 10]. In this paper, we present a simple, reliable, and inexpensive method for automatic meta-evaluation of arbitrary LM evaluators. We avoid the difficulties mentioned above with two designs. First, instead of asking whether humans would give a trajectory the same score as the LM evaluator does, we hand the evaluator two trajectories to score separately, and ask whether the preference implied by the score difference agrees with the human preference (Section 3.2). This is an established practice in prior research (Section 2). Second, we define an algorithm that synthesizes labeled trajectory pairs for a given agent system and an arbitrary set of criteria, using small language models (SLMs) no more capable than the ones the existing evaluator and agent system rely on (Section 3.1). No human input is needed during data synthesis or at evaluation time. We call the algorithm PADMÉ, or Preference Alignment Data synthesis for Meta-Evaluation. Building on τ3τ^3-bench [20, 21, 22], we develop an instance of PADMÉ and use it to synthesize 1,000 data pairs across four task domains and three criteria (Section 5.1), validate 150 of them with six human annotators (Section 5.2), and evaluate 25 common models as evaluators (Section 5.3). We claim three contributions. • A data synthesis algorithm that builds meta-evaluation data for agent systems and any evaluation criteria, as opposed to a fixed benchmark (Section 3). • A program that implements this algorithm for a specific use case (Section 4), together with a synthetic dataset generated by that program (Section 5.1). • Experiments showing that the algorithm is data- and cost-efficient, and a human study showing that the preference labels this algorithm constructs agree with human judgment in this scenario (Section 5). We release our code and the synthesized data at https://github.com/chc012/padme. 2 Related Work Judging Agents, and Meta-Evaluating the Judges LLM-as-a-judge is the paradigm for open-ended evaluation [1, 3]. Increasingly, LM evaluators target multi-turn behaviors rather than single-turn responses [2, 13, 12]. Meta-evaluation of these evaluators has evolved from coarse response-level preferences [8, 9, 10] toward finer-grained units, such as skill decompositions [4], checklists [23, 14, 24], rubrics [5, 25], instruction constraints [11], and crowdsourced criterion labels [26]. For meta-evaluation in agentic settings, Agent-as-a-Judge [2] and AJ-Bench [12] use verifier scripts and manual requirement annotations, so the label exists only in use cases where completion rules are predetermined. They explore meta-evaluation for code generation [2], search, data-system manipulation, and GUI interaction [12]. Two costs remain in all of these works: labels rely heavily on human curation, and data generation relies on frontier models. Constructed Quality Differences LLMBar [27] is an early instance of this idea. It releases 419419 hand-curated output pairs in which one response follows the instruction and the other deviates. FBI [28] injects hand-authored perturbations that degrade one capability and asks whether an evaluator notices. It requires manual label verification. RubricEval [25] samples responses from a mixed model pool to elicit performance differences. Its labels are judgments of binary questions regarding each trajectory. REFLECT [29] meta-evaluates judges of deep research agents. It derives controlled perturbations of agent trajectories from a taxonomy of failure patterns. Its pipeline requires human experts for validation. All of these works build their data with frontier models or human annotations, which restricts extension to other agentic use cases. We attempt to offer a more broadly applicable data synthesis recipe with only SLMs (Section 4). Preferences Versus Scores Eliciting pairwise preference from pointwise scores is an established practice. Comparative elicitation recovers a latent scale from pairwise judgments in psychology and statistics [30, 31]. In natural language processing, it yields more reliable human labels than rating scales [15, 16]. Pairwise ranking aligns LLM evaluators with human judgment better than direct scoring [32]. Some reward-model benchmarks rate each completion independently and compare score differences [9]. Within LLM-as-a-judge, Zheng et al. [1] convert single-answer grades into pairwise comparisons in order to measure agreement with human votes. Consequently, we acquire ground-truth data via preferences while prompting the evaluator to score each trajectory pointwise (Eq. (3)), which mirrors deployment conditions. Alternatively, many meta-evaluation studies do measure scoring alignment by correlating evaluator scores directly against human ratings [33, 17, 4, 5]. SLMs as Judges Deploying small language models as judges serves as a premise for our work. SLM judges are competitive across model families and parameter scales [34], specialized mini-evaluators have emerged [17, 35], and lightweight models are well-suited for repetitive agentic sub-tasks [36]. However, SLMs are more susceptible to assertiveness and verbosity confounds [37]. For SLM meta-evaluation, SLMJury [34] meta-evaluates SLM judges, but it uses existing public datasets rather than building agent- and criterion-specific data. 3 Method Figure 2: Overview of the PADMÉ pipeline and meta-evaluation architecture. For each task-criterion cell (t,c)(t,c) from τ3τ^3-bench, steering prompts generate trajectory pairs p=(x−,x+)p=(x^-,x^+) across three quality levels (good, ok, bad). Two LM filter judges discard pairs lacking visible quality contrast to yield dataset KD_K, validated against a human annotation subset. Target evaluators rate individual trajectories (s=f⁡(x,c)∈[0,1]s=f(x;c)∈[0,1]), and meta-evaluation assesses their preference alignment across verified pairs. As mentioned in Section 1, meta-evaluating an evaluator requires assessing its alignment with human judgment. Through the pointwise-to-pairwise reframing of the evaluation objective, we try to assess whether an evaluator will rank two agent trajectories correctly by scoring them independently. PADMÉ synthesizes trajectory pairs with known labels over arbitrary criteria through steering agent prompts. This shifts the core challenge from annotation to data collection. To improve label fidelity, PADMÉ applies filtering and retries after initial trajectory generation. We define the algorithm below. Figure 2 gives an overview of the method. 3.1 Data Curation Generation We define the input of PADMÉ as a cell (t,c)(t,c), consisting of task parameters t and evaluation criterion c. t may include textual descriptions of the task (e.g., “rebook a plane ticket”), tools and resources available to the agent for the task (e.g., functions list_purchased_tickets and buy_ticket), information about the simulated user’s intention of the task (e.g., ‘if rebooking is not possible, cancel the ticket’), and so on22 2 The selection of information available here is up to the developer of the specific agent system. Care must be taken here to ensure that no unwanted information leakage happens inside the task parameters that would reveal the solution of the task directly to the trajectory-generating agent.. c consists of the name and a description of the criterion. The criterion can be any performance axis of the agent system, such as “task completion rate”, “user satisfaction”, “tool-use relevance”, etc. The descriptions should ideally be detailed and align with application scenarios. As an example, see Appendix H.1 for the criteria descriptions in our experiment. For good coverage of behavior, the source dataset should be a benchmark used to evaluate the targeted agent system, though any dataset that provides sets of tts and ccs would apply. We use each cell as a seed datum to generate a trajectory pair p=(x−,x+)p=(x^-,x^+) with two different steering levels, ℓ,ℓ′∈ℒ=bad≺ok≺good , =\ bad ok good\. Here, a trajectory denotes a complete record of interactions between a user and an agent in a task execution session. Trajectories are sampled using a rollout function RollRoll: xℓ∼Roll⁡(t,sc,t,ℓ,θ),p=(x−,x+):=(xℓ,xℓ′)for ℓ≺ℓ′.x (t,s_c,t, ;θ ), p=(x^-,x^+):= (x ,x ) . (1) The steering instruction sc,t,ℓs_c,t, is generated by a WriteSteer function, which uses an SLM conditioned on the criterion (c), task (t), and the steering intent (ℓ ). The instruction is then appended to the agent’s system prompt (Appendix H.3). Each cell randomly selects two distinct steering levels, with the three resulting contrasts (bad,ok\ bad, ok\, ok,good\ ok, good\, and bad,good\ bad, good\) stratified equally across the dataset. For example, a bad steering instruction on “friendliness” may ask the agent to be cold and concise in its response, an ok one to be neutral in its tone, and a good one to be warm and comforting in its replies. Each instruction is tailored to the specific task and carries different information. The generator prompt that produces these instructions is in Appendix H.2. All rollouts within a cell share a fixed environment context θ, which may include the agent model, user simulator, domain configuration, and decoding seed. While fixing θ ensures the steering instruction is the primary deliberate variable, LM stochasticity and dynamic environment simulations naturally introduce trajectory variance. Filtering Each generated trajectory pair p=(x−,x+)p=(x^-,x^+) is sequentially evaluated by a cascade of K SLM preference judges, J1,…,JKJ_1,…,J_K (Appendix H.4). Given a pair p in randomized order (Appendix C.2) and its associated criterion c, each judge returns its own preference over the pair. Its verdict Jk(p,c)∈0,1J_k(p,c)∈\0,1\ records whether that preference agrees with the constructed synthetic label. Judges have veto power: a score of 1 retains the pair, while 0 discards it. With 0D_0 representing the initial set of generated pairs and kD_k the subset surviving stage k, the cascade progresses as: k=p∈k−1∣Jk(p,c)=1,k=1,…,K.D_k= \\,p _k-1\ \ J_k(p,c)=1\, \, k=1,…,K. (2) This construction forms a nested sequence K⊆⋯⊆1⊆0D_K ·s _1 _0, ensuring a pair is preserved only if approved by all K judges. We set K=2K=2 and evaluate intermediate depths (Table 2). Retry To improve data efficiency, a cell rejected by the filtering cascade is resampled with the same steering prompts and fresh rollouts, up to R retries. Resampled trajectories reuse the steering instructions to ensure level contrast stratification. Since retries target cells previously rejected by the filter, subsequent attempts operate on inherently harder tasks, leading to an expected drop in retention rate but raising the overall data collection count. Thus, R serves as a hyperparameter trading increased dataset yield against compute cost. We set R=2R=2 in our experiment and analyze this trade-off in Table 1. The complete dataset curation pipeline, integrating generation, filtering, and retries, is summarized in Algorithm 1 (Appendix A). 3.2 Meta-Evaluating an Evaluator An evaluator under test is a scoring function f⁡(x,c)∈[0,1]f(x;c)∈[0,1] that rates a single trajectory x against a criterion c. Because f evaluates each trajectory independently without viewing trajectory pairs, its pairwise preference over p=(x−,x+)p=(x^-,x^+) is derived directly from the score gap: Δf(p)=f⁡(x+,c)−f⁡(x−,c),y^f(p)=sign⁡Δf(p)∈+1,−1,0, _f(p)=f(x^+;c)-f(x^-;c), y_f(p)=sign _f(p)∈\+1,-1,0\, (3) where y^f(p)=0 y_f(p)=0 indicates a tie in score. This setup mirrors application scenarios where evaluators score individual execution traces without access to counterfactual rollouts. Because synthetic ground truth ⋆ always prefers x+x^+, an evaluator aligns with ground truth if and only if it assigns a strictly higher score to x+x^+ (Δf(p)>0 _f(p)>0). Ties (Δf(p)=0 _f(p)=0) count as disagreements because a deployed evaluator that cannot separate x+x^+ from x−x^- supplies no usable signal. Evaluator accuracy is therefore defined as the agreement between the evaluator and the synthetic ground truth: Agr(f,⋆;)=1||∑p∈[Δf(p)>0].Agr(f, ;D)= 1|D| _p 1 [ _f(p)>0 ]. (4) More broadly, we can formalize any pairwise decision source as a preference provider r with decision space y^r(p)∈+1,−1,0 y_r(p)∈\+1,-1,0\. Evaluators may output neutral ties (y^f=0 y_f=0), whereas human annotations and synthetic ground truth are strictly binary in −1,+1\-1,+1\. Agreement between any two providers r and r′r over a dataset D is: Agr(r,r′;)=1||∑p∈[y^r(p)=y^r′(p)].Agr(r,r ;D)= 1|D| _p 1 [\, y_r(p)= y_r (p)\, ]. (5) In the experiments below, we assess evaluator accuracy Agr⁡(f,⋆,)Agr(f, ;D) (Section 5.3), synthetic label accuracy against human majority (majmaj) annotations Agr⁡(maj,⋆,k)Agr(maj, ;D_k) (Section 5.2), and fine-grained evaluations across criteria, domains, level contrasts, and agent models (Appendices E and F.1). 4 Experimental Setup Agent System & Criteria We build an agent trajectory collection system upon a variant of the τ3τ^3-bench repository [20, 21, 22]. We rely on its task definitions, agent framework, user simulator, tool sets, and domain-specific environments, but use our own LM evaluator system. This setup mimics the expected usage of the PADMÉ algorithm in real-world scenarios, where developers of an agent system bring in the agents, task parameters, and evaluation criteria of interest, and meta-evaluate the evaluator over them with PADMÉ. The base version of τ3τ^3-bench comprises 375 tasks across four task domains: airline (50), banking (97), retail (114), and telecom (114), each featuring different agent prompts, environments, and tools. We evaluate three runtime criteria: friendliness, communication clarity, and task resolution. Detailed descriptions of the criteria are in Appendix H.1. The task-criterion matrix contains 1,125 evaluation cells, with quality level contrasts stratified evenly across the dataset. Pipeline Models & Data Curation All pipeline components are driven by open-weight SLMs ranging from 3B to 5.1B active parameters (21B to 117B total parameters). We test agent trajectory pairs generated using gpt-oss-20b, gpt-oss-120b, and nemotron-lightning-3.5, with both rollouts in a pair generated by the same model. User interactions are simulated via qwen3-30b-a3b-instruct. Steering instructions are generated by gpt-oss-120b. Filtering employs K=2K=2 judges (J1=nemotron-lightning-3.5J_1=nemotron-lightning-3.5 and J2=gpt-oss-120bJ_2=gpt-oss-120b). Each data point is under a retry budget of R=2R=2. Running the 1,125 initial cells under this setup yields 1,000 kept pairs, distributed over the four dataset axes as Table 6 in Appendix B shows. For all open-weight models, we use the Fireworks model deployment and serverless access service33 3 https://fireworks.ai/models. (Appendix I). Meta-Evaluation Sweep We evaluate 25 LMs as evaluators across open-weight and proprietary model families on all 1,000 kept pairs across 3 runs (Appendix I). The open-weight models contain 2B to 2.8T parameters. The evaluator prompt can be found in Appendix H.5. Model evaluations run under vendor-default reasoning budgets and at temperature 0, unless temperature cannot be set44 4 This refers to the more recent OpenAI GPT family of proprietary models.. Three of the 25 evaluators overlap with models used in the generation and filtering pipeline and are explicitly flagged in downstream analyses for potential contamination (Appendix F.2). Human Validation Protocol Human validation is conducted on a stratified sample of 150 pairs from the initial generation attempts (r=0r=0) across all criteria and domains (representativeness checks in Appendix D.4). It serves exclusively to validate the data curation pipeline rather than operating as part of the pipeline itself. Six human annotators form two panels of three on disjoint sets of 75 pairs, generating 450 binary preference choices along with confidence ratings. The majority vote across the three annotators serves as the human reference label (majmaj). Additional information can be found in Appendix D. 5 Results and Analysis 5.1 Dataset Out of 1,125 initial task-criterion cells, 1,000 trajectory pairs survive both filter stages under a retry budget of R=2R=2, requiring 1,673 total generation draws (Table 1). Per-attempt retention decays predictably across retries, as subsequent retries operate exclusively on previously rejected cells. Retries yield an additional 230 validated pairs (+30%) at the cost of 548 supplementary trajectory rollouts. Across all subsets, the first filter judge J1J_1 accounts for the vast majority of rejections. Detailed subset breakdowns are provided in Appendix C.1. The dataset curation pipeline generates 1,000 validated trajectory pairs at a total cost of $23.63, or $0.024 per kept pair. Details of cost are in Appendix G. Table 1: Data curation funnel over 1,125 cells across retry attempts. retry (r) draws kept J1J_1 rejected J2J_2 rejected retention rate (%) cumulative retention rate (%) 00 1,125 770 266 89 68.4 68.4 11 355 161 146 48 45.4 82.8 22 193 69 100 24 35.8 88.9 total 1,673 1,000 512 161 59.8 88.9 5.2 Human Study Annotator Confidence and Dataset Difficulty Annotators report their confidence on each pair as 0 (a guess), 1 (leaning), or 2 (certain). We use the per-pair mean over the three annotators as a proxy for how difficult that pair is to judge. Mean self-reported annotator confidence increases by +0.02+0.02 on the 0 to 2 scale (1%) across filtering stages, though the influence of J1J_1 and J2J_2 differs (Appendix D.3). This suggests that filtering does not significantly trivialize the resulting data. Human Alignment with Labels Filtering monotonically increases alignment between the synthetic label (⋆ ) and human majority vote (majmaj), as computed by synthetic label accuracy Agr⁡(maj,⋆,k)Agr(maj, ;D_k) with Eq. (5). Passing pairs through the filters boosts label validity from 73.3% to 84.6% while shifting human-label agreement (Krippendorff’s α) from +0.468+0.468 to +0.694+0.694 (Table 2; panel reliability in Appendix D.2). The improvement appears to plateau at K=2K=2. On the subset of pairs rejected by J1J_1 (0∖1D_0 _1), label validity falls to 38.9% (14/36), which confirms that the filter selectively removes misaligned or noisy trajectories. Label validity among J2J_2-rejected pairs is 80.0% (8/10), close to the 84.6% among retained pairs. Detailed subset breakdowns are provided in Appendix E. Table 2: Agreement between synthetic labels and human majority vote across filter depths D_k. Yield indicates the proportion of the 150 annotated pairs surviving at each depth. Synthetic label accuracy reports Agr⁡(maj,⋆,k)Agr(maj, ;D_k) with bootstrap 95% confidence intervals in brackets. α is Krippendorff’s α for the same label-versus-majority decision. dataset |||D| yield synthetic label accuracy α 0D_0 (unfiltered) 150 100% 73.3% [66–80] +0.468 1D_1 (J1J_1) 114 76% 84.2% [76–90] +0.686 2D_2 (J1&J2J_1\&J_2) 104 69% 84.6% [76–90] +0.694 5.3 Meta-Evaluation Sweep To establish a performance spectrum across common language models, we benchmark 25 LMs with the same basic evaluator prompt (Appendix H.5). We have each model score all 2,000 trajectories (1,000 pairs) independently across three separate runs. Table 3 lists 12 model results out of 25 for readability. Two numbers directly inform the performance of each model: accuracy (Equation 4), our most important performance metric, and average score standard deviation (SD)55 5 Note that average score SD is not the standard deviation of all scores assigned by each evaluator across the dataset, but the average of the standard deviation of each data point score across 3 runs for each evaluator., which helps us understand the stability of the evaluator’s scoring. All analyses in this section are conducted over the full 25 model results, which are reported in Table 18 in the appendix. Table 3: Performance of evaluators across 1,000 trajectory pairs (n=3n=3 runs). 12 of 25 evaluators are selected to span the accuracy range and model families (full view in Table 18). All reported statistics are computed across all 25 evaluators. Released indicates public release month. Temperature = 0 except where unavailable‡. Preferences derive from individual trajectory score gaps; ties count as incorrect. Accuracy and score denote benchmark level accuracy and raw evaluator score, respectively. SD is standard deviation. Leniency is mean emitted score. Average score SD measures per data point scoring stability across 3 runs. Bold indicates best performance per column across all 25 evaluators. Accuracy and average score SD serve as the two primary evaluator metrics. parameters accuracy (%) leniency average tie by domain (%) by criterion (%) # distinct # evaluator released total active reas. mean ± SD (mean score) score SD (%) airl. bank. retail telec. clar. friend. task scores 1 claude-opus-5 2026-07 closed closed Yes 85.1 ± 0.93 0.324 0.023 4.1 84 82 86 87 73 96 85 69 3 kimi-k3 2026-07 2.8T 104B Yes 83.6 ± 0.12 0.508 0.036 6.0 83 80 84 87 71 97 81 37 6 glm-5p2 2026-06 753B ∼ 30B Yes 81.5 ± 0.50 0.521 0.051 10.1 84 81 81 82 78 92 73 31 8 gpt-5.4-nano‡ 2026-03 closed closed No 79.4 ± 1.16 0.631 0.052 7.8 81 78 78 81 74 84 80 61 10 gpt-5.6-sol‡ 2026-07 closed closed Yes 78.7 ± 0.40 0.543 0.035 4.9 79 77 81 77 64 97 74 86 11 claude-haiku-4-5 2025-10 closed closed No 78.7 ± 0.10 0.505 0.000 11.6 81 79 76 81 76 85 74 32 13 nemotron-3-ultra 2026-06 549B 55B Yes 76.8 ± 0.76 0.558 0.044 14.2 75 80 70 81 71 88 71 28 14 llama3.1-70b 2024-07 70.6B dense No 70.0 ± 0.55 0.632 0.016 22.8 67 73 67 72 64 69 77 14 15 gemini-3.7-flash 2026-08 closed closed Yes 69.5 ± 0.45 0.615 0.026 23.4 71 70 69 69 57 95 54 34 17 gemma-4-31b 2026-03 32.2B dense Yes 68.1 ± 0.15 0.657 0.023 27.4 63 67 66 74 47 92 63 14 18 deepseek-v4-pro 2026-08 1.6T 49B Yes 67.0 ± 0.67 0.505 0.045 20.4 62 70 69 65 61 90 49 23 25 qwen3-4b 2025-08 4.4B dense No 36.8 ± 0.65 0.804 0.029 56.9 36 34 32 44 30 25 55 17 Mean, all 25 evaluators 70.1 68.7 69.9 69.0 72.2 62.0 79.5 67.9 Subset Analysis We discuss four subsets: evaluation criteria (friendliness, communication clarity, task resolution), domain (retail, telecom, banking, airline), level contrast (bad-ok, ok-good, bad-good), and agent model (nemotron-lightning-3.5, gpt-oss-120b, gpt-oss-20b). The bad–ok contrast (73.2%) and the bad–good contrast (73.6%) have similar accuracy, while the accuracy of the ok–good subset is much lower, at 63.4% (Table 18). A hypothesis is that, compared to detecting bad trajectories, it is harder for evaluators to distinguish the relative performance of two acceptable trajectories. gpt-oss-120b’s trajectories have an average accuracy of 74.9%, gpt-oss-20b 70.5%, and nemotron-lightning-3.5 65.1% (Table 18). This suggests that the model backbone of the agent impacts the difficulty of evaluating the agent trajectories. Correlation Analysis Table 4 shows the correlation analyses between accuracy and seven factors. Accuracy is highly correlated with scoring resolution. Tie rates range from 3.4% to 56.9% and correlate strongly with accuracy (Spearman’s ρ=−0.946ρ=-0.946). In many cases, models fail due to coarse scoring granularity rather than misjudging. Across models, distinct score count correlates with accuracy at ρ=+0.797ρ=+0.797, showing that score resolution is important for discriminative ability. Table 4: Seven evaluator properties against accuracy, sorted by Spearman’s correlation coefficients. pholmp_holm is the Holm-Bonferroni corrected p-value across the seven tests, and bolded where it indicates significance. n=15n=15 on some rows because ten closed models do not publish parameter counts. Reasoning is a binary factor. factor n ρ pholmp_holm tie rate 25 −0.946-0.946 <0.0001<0.0001 distinct score values 25 +0.797+0.797 <0.0001<0.0001 leniency (mean score) 25 −0.725-0.725 0.00020.0002 total parameters 15 +0.725+0.725 0.00890.0089 factor n ρ pholmp_holm active parameters 15 +0.518+0.518 0.1347 reasons by default 25 +0.404+0.404 0.1347 release date 25 +0.359+0.359 0.1347 The mean score an evaluator assigns to data points, which is a way of quantifying leniency, shows a significant anticorrelation with the evaluator performance (ρ=−0.725ρ=-0.725). It is possible that a harsher evaluator holds a longer internal list of expectations of the agent behavior, and is thus better at distinguishing nuanced differences between two trajectories. An evaluator that is easily satisfied, i.e., assigns scores close to 1 easily, risks conflating good and great agent behaviors. Total parameter (ρ=+0.725ρ=+0.725) and active parameter counts (ρ=+0.518ρ=+0.518, not significant) show some level of correlation with accuracy. For proprietary models, an exception is that gpt-5.4-nano and gpt-5.4-mini, which have smaller expected parameter counts, outperform gpt-5.6-sol. 6 Limitations and Future Work Extensibility to More Evaluation Criteria: Future work should apply PADMÉ to a larger and more diverse set of criteria. It would especially benefit from a stress test of criteria that, through steering in a direction, would go against critical instruction-following training of the agent models (such as toxicity level or answer safety). Broadening Agent and Domain Coverage: The current study limits agent trajectory generation to τ3τ^3-bench and its domains. To further validate the proposed method, it should be generalized across diverse agent architectures, operation environments, and broader domain benchmarks beyond customer-service agent operations. Score Granularity and Calibration: Given that evaluator accuracy is heavily driven by score resolution, future work should systematically evaluate controlled scoring regimes (e.g., discrete Likert scales versus continuous [0,1][0,1] bounds constrained to fixed decimal precision) to explore how output formatting affects model ties and discrimination. Other Reliability Signals: For criteria with programmatic rewards, reliability can be assessed without human annotations. We briefly discuss the alignment between verifiable rewards with synthetic labels in Appendix C.3, but it warrants further investigation. 7 Conclusion Meta-evaluation is the evaluation of LM evaluator reliability. Adopting preference alignment in place of absolute score alignment, we demonstrate that meta-evaluation data can be synthesized rather than manually annotated. The proposed framework, PADMÉ, steers an agent’s trajectories along criteria axes, uses the steering intents as the initial labels, and applies peer-sized judges to veto trajectory contrasts. Using models at or below 5.1B active parameters, PADMÉ constructs a 1,000-pair benchmark across four domains and three criteria at a low cost. Filtering boosts human-label agreement from 73.3% to 84.6%. The resulting dataset separates 25 candidate evaluators across a 48.3-percentage-point accuracy spread. The data can be regenerated whenever criteria are modified or agents are updated without human intervention. We thus present a generalizable data synthesis and meta-evaluation recipe rather than a static benchmark. Acknowledgments and Disclosure of Funding All authors are employees at Uniphore. All funding is provided by Uniphore. The authors would like to thank Ishika Agarwal, Bowen He, Tommy Li, Ethan Soon, Artin Tajdini, and Ming Xin (ordered by last names) for their help as human annotators. References [1] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=uccHPGDlao. [2] Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Nn9POI9Ekt. [3] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. The Innovation, 7(6):101253, 2026. ISSN 2666-6758. doi: https://doi.org/10.1016/j.xinn.2025.101253. URL https://w.sciencedirect.com/science/article/pii/S2666675825004564. [4] Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=CYmF38ysDa. [5] Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5877–5919, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.303. URL https://aclanthology.org/2025.naacl-long.303/. [6] Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7421–7454, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.401. URL https://aclanthology.org/2024.acl-long.401/. [7] Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706288. doi: 10.1145/3654777.3676450. URL https://doi.org/10.1145/3654777.3676450. [8] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 1755–1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.96. URL https://aclanthology.org/2025.findings-naacl.96/. [9] Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=fb0G86Dewb. [10] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=G0dksFayVq. [11] Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling, Ying Zhang, Pei Ke, Hongning Wang, and Minlie Huang. IF-RewardBench: Benchmarking judge models for instruction-following evaluation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23816–23843, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1092. URL https://aclanthology.org/2026.acl-long.1092/. [12] Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi GU, Hui Su, Xunliang Cai, and Xiangnan He. AJ-bench: Benchmarking agent-as-a-judge for environment-aware evaluation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 25371–25413, San Diego, California, United States, July 2026a. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.1269. URL https://aclanthology.org/2026.findings-acl.1269/. [13] Hyogon Ryu, Jeonghwan Kim, Yewon Lim, Chaeun Lee, Jeongwook Kim, and Donghoon Ham. Online agent-as-a-judge: Situation-generating evaluation for interactive agents. In Trustworthy AI for Good (AI4GOOD) Workshop @ ICML 2026, 2026. URL https://openreview.net/forum?id=YrcMknjgyx. [14] Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. Rocketeval: Efficient automated LLM evaluation via grading checklist. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=zJjzNj6QUe. [15] Svetlana Kiritchenko and Saif Mohammad. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-2074. URL https://aclanthology.org/P17-2074/. [16] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. RankME: Reliable human ratings for natural language generation. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 72–78, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2012. URL https://aclanthology.org/N18-2012/. [17] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4334–4353, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.248. URL https://aclanthology.org/2024.emnlp-main.248/. [18] Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xsELpEPn4A. [19] Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=5Nn2BLV7SB. [20] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. τ-bench: A benchmark for Tool-Agent-User interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=roNSXZpUDN. [21] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik R Narasimhan. τ2τ^2-bench: Evaluating conversational agents in a dual-control environment. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=OC2z7iSQKa. [22] Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik R Narasimhan, and Victor Barres. τ-knowledge: Evaluating conversational agents over unstructured knowledge. In Forty-third International Conference on Machine Learning, 2026b. URL https://openreview.net/forum?id=XHZK5abtw2. [23] Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve llm evaluation and generation, 2024. URL https://arxiv.org/abs/2410.03608. [24] Karen Zhou and Chenhao Tan. AutoChecklist: Composable pipelines for checklist generation and scoring with LLM-as-a-judge. In Greg Durrett and Ping Jian, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 515–525, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-392-0. doi: 10.18653/v1/2026.acl-demo.51. URL https://aclanthology.org/2026.acl-demo.51/. [25] Tianjun Pan, Xuan Lin, Wenyan Yang, Qianyu He, Shisong Chen, Licai Qi, Wanqing Xu, Hongwei Feng, Bo Xu, and Yanghua Xiao. Rubriceval: A rubric-level meta-evaluation benchmark for llm judges in instruction following, 2026. URL https://arxiv.org/abs/2603.25133. [26] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 1474–1501. Curran Associates, Inc., 2024b. doi: 10.52202/079017-0047. URL https://proceedings.neurips.c/paper_files/paper/2024/file/02fd91a387a6a5a5751e81b58a75af90-Paper-Datasets_and_Benchmarks_Track.pdf. [27] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.07641. [28] Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Sshubam Verma, and Mitesh M Khapra. Finding blind spots in evaluator LLMs with interpretable checklists. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16279–16309, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.911. URL https://aclanthology.org/2024.emnlp-main.911/. [29] Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, and Arman Cohan. Time to reflect: Can we trust llm judges for evidence-based research agents?, 2026. URL https://arxiv.org/abs/2605.19196. [30] L. L. Thurstone. A law of comparative judgment. Psychological Review, 34(4):273–286, 1927. doi: 10.1037/h0070288. [31] Ralph A. Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029. [32] Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=9gdZI7c6yr. [33] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.153. URL https://aclanthology.org/2023.emnlp-main.153/. [34] Anish Laddha, Nitesh Pradhan, and Gaurav Srivastava. Slmjury: Can small language models judge as well as large ones?, 2026. URL https://arxiv.org/abs/2606.07810. [35] Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, Toby Drane, and Young Sun Park. Atla selene mini: A general purpose evaluation model, 2025. URL https://arxiv.org/abs/2501.17195. [36] Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai, 2025. URL https://arxiv.org/abs/2506.02153. [37] Tuhina Tripathi, Manya Wadhwa, Greg Durrett, and Scott Niekum. Pairwise or pointwise? evaluating feedback protocols for bias in LLM-based evaluation. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=uyX5Vnow3U. [38] Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, and Yoshitaka Ushiku. Am i more pointwise or pairwise? revealing position bias in rubric-based llm-as-a-judge, 2026. URL https://arxiv.org/abs/2602.02219. Appendix A Notations and the Algorithm Table 5: Notation, grouped by pipeline stage in the order the paper uses it. Generation t a task from the dataset, carrying its domain policy, tools, and user goal. c a criterion, supplied at runtime as free text: a name and a detailed description. ℓ∈ℒ a steer level, ℒ=bad≺ok≺goodL=\ bad ok good\, ordered by intended quality. θ everything held fixed while the steer level varies: agent model, user simulator, domain, decoding seed. sc,t,ℓs_c,t, the steering instruction, written by a small model from c, t, and ℓ , Eq. (1). x a trajectory: the complete trace generated by an interactive dialogue between a user simulator and an agent executing a task, with tool calls and its outcome. p=(x−,x+)p=(x^-,x^+) a pair, where x+x^+ is generated at the higher steer level. Slot order is randomized whenever the pair is shown to an LLM filter or a human annotator. Curation K the number of filter judges applied in sequence. R the retry budget: a cell may be drawn up to 1+R1+R times. JkJ_k the k-th filter judge, Jk(p,c)∈0,1J_k(p,c)∈\0,1\. kD_k the pairs surviving the first k judges, 0≤k≤K0≤ k≤ K, Eq. (2); 0D_0 is every drawn pair. Evaluation f the evaluator under test, f⁡(x,c)∈[0,1]f(x;c)∈[0,1], applied to one trajectory at a time. Δf(p) _f(p) its score gap on a pair, Eq. (3). r a preference provider: any source of a pairwise decision. Instances used here are the evaluator under test f (preference derived from Δf(p) _f(p)), a filter judge JkJ_k, a human annotator a, the panel majority majmaj of three annotators, and the synthetic label ⋆ . y^r(p) y_r(p) the preference of provider r on pair p: +1+1 for x+x^+, −1-1 for x−x^-, 00 for none. ⋆ the constructed synthetic label, treated as a preference provider: y^⋆≡+1 y_ ≡+1 by definition. Reporting Agr⁡(r,r′,)Agr(r,r ;D) how often two preference providers agree over D, Eq. (5). |zD|_z D restricted to a subset z: one criterion, domain, level contrast, or agent. Table 5 collects the notation used in the main paper, grouped by pipeline stage. Algorithm 1 PADMÉ data curation algorithm. Each cell yields at most one kept pair. 1: cells (t,c)\(t,c)\, fixed context θ, judges J1,…,JKJ_1,…,J_K, retry budget R 2: k←∅D_k← for k=0,…,Kk=0,…,K 3: for all cells (t,c)(t,c) do 4: take the level contrast ℓ≺ℓ′ for this cell from the stratified schedule 5: for attempt=1attempt=1 to 1+R1+R do 6: sc,t,ℓ←WriteSteer(c,t,ℓ)s_c,t, ← WriteSteer(c,t, ), sc,t,ℓ′←WriteSteer(c,t,ℓ′)s_c,t, ← WriteSteer(c,t, ) 7: x−∼Roll⁡(t,sc,t,ℓ,θ)x^- (t,\,s_c,t, ;\,θ), x+∼Roll⁡(t,sc,t,ℓ′,θ)x^+ (t,\,s_c,t, ;\,θ) 8: p←(x−,x+)p←(x^-,x^+), 0←0∪pD_0 _0∪\p\ 9: k←0k← 0 10: while k<Kk<K and Jk+1(p,c)=1J_k+1(p,c)=1 do 11: k←k+1k← k+1, k←k∪pD_k _k∪\p\ ⊳ a veto stops the cascade 12: end while 13: if k=Kk=K then 14: break ⊳ cell filled; only a rejected cell is ever redrawn 15: end if 16: end for 17: end for 18: return KD_K ⊳ 0,…,K−1D_0,…,D_K-1 are kept for the filter-depth ablation The complete PADMÉ dataset curation pipeline, integrating generation, filtering, and retries, is summarized in Algorithm 1. Appendix B Dataset Composition and Evaluation Protocols Table 6 outlines the composition of the 1,000-pair shipping dataset across its four principal axes. Table 7 details the input constraints, task formulations, and output formats for each preference provider. The pipeline naturally produces a balanced dataset across criteria, domains, level contrasts, and agent models without requiring artificial quotas or post-hoc rebalancing, maintaining strong yield consistency across subsets (detailed further in Table 8 and Appendix C.1). The filter judge (JkJ_k) and the human annotator (a) operate under identical information availability. Both receive identical contextual fields, perform pairwise trajectory comparisons, and remain strictly blinded to steering metadata. This symmetry ensures that human label alignment serves as a direct validation of the cascade filter rather than an artifact of information asymmetry. In contrast, the evaluator under test (f) operates pointwise on single trajectories without cross-trajectory visibility, which mirrors realistic deployment conditions. Table 6: A glossary of dataset subsets, along with the distribution of the 1,000 pairs across axes. Each pair carries exactly one level per axis, resulting in subtotal sums of 1,000 per block. Comprehensive retention rates per subset appear in Table 8. Evaluation criterion descriptions reflect the full runtime definitions, also provided in Appendix H.1. Stratified experimental breakdowns are reported in Appendix E. level pairs description Evaluation criterion friendliness 355 Warmth and consideration toward the customer: whether the agent acknowledges their situation and how they feel about it, delivers unwelcome news with care, and leaves them feeling attended to. Judge the manner, not whether the request was resolved. task resolution 329 Whether the customer’s actual problem was settled: did the agent establish what was needed, take the actions that would resolve it, and leave the customer with the outcome they came for. A correct refusal counts as resolution – if the request was not permitted, saying so plainly and explaining why resolves it, while quietly doing it anyway does not. Judge the outcome, not the manner or how well it was explained. communication clarity 316 How easily the customer can follow the agent: whether the main point is findable, whether technical or policy language is explained, whether multi-part information is organised, and whether the customer is left knowing what is true and what happens next. Judge the presentation, not the warmth or the outcome. Domain retail 309 114 tasks; 1,158-word policy, 16 agent tools telecom 294 114 tasks; 3,715-word policy, 13 agent tools, 30 user-side tools banking 262 97 tasks; 926-word policy, 16 agent tools, including retrieval airline 135 50 tasks; 1,313-word policy, 14 agent tools Level contrast bad–good 359 wider gap ok–good 328 narrower gap bad–ok 313 narrower gap Agent model nemotron-lightning-3.5 341 32B total, 3B active gpt-oss-120b 334 116.8B total, 5.1B active gpt-oss-20b 325 20.9B total, 3.6B active Table 7: A glossary of information exposure, task objectives, and output specifications across preference providers. Symmetrical blinding ensures human annotators directly benchmark filter judge decisions, whereas evaluators under test operate in single-trajectory pointwise mode to match real-world deployment. filter judge JkJ_k human annotator a evaluator under test f purpose curates the dataset validates the dataset the object of measurement is given the criterion name and definition, the agent’s tools and the domain policy, and both trajectories the same five fields, rendered in a purpose-built interface (Appendix D) the criterion name and definition, the agent’s tools and the domain policy, and one trajectory is not given the steering prompts, the target levels, or which side was steered better the same, and no filter verdict the same, and never the other trajectory is asked which of the two is better on the criterion which of the two is better on the criterion to score the agent on the criterion from 0.00.0 to 1.01.0 slot order randomized per pair and per judge randomized per pair no order arises: one trajectory per call returns a forced choice of one side, with at most three sentences of reasoning a forced choice of one side, with a confidence rating of 0,1,2\0,1,2\ a score in [0,1][0,1], with at most three sentences of reasoning preference the side it chose; the pair survives only when that matches the synthetic label the side it chose; the majority vote of three is majmaj the sign of the score gap (Eq. (3)) Appendix C Dataset Audits and Yield Dynamics C.1 Generation Yield and Retention Generation retention is reported per experimental subset rather than artificially constrained. Because no post-hoc rebalancing is applied to force quota targets, Table 8 directly reflects where trajectory pairs easily satisfy judge filtering versus where narrower quality gaps require higher generation compute and retries. Retention yield strictly tracks two primary factors: the target level contrast and the specific evaluation criterion. First, wider level contrasts yield higher retention rates: bad–good contrasts complete 96% of targeted cells at 1.38 draws per retained pair, compared to 84% completion at 1.90 draws per pair for narrower bad–ok contrasts. Second, criteria evaluating overt linguistic style achieve higher retention than structural task completion metrics: friendliness reaches 95% cell retention, outperforming task_resolution (88%) and communication_clarity (84%). Across domains, retention rates remain tightly clustered within a 4-percentage-point band, with telecom recording the lowest completion rate (86%) due to its underlying tool and policy complexity (Table 16). Table 8: Cell retention metrics by subset after 2 retries (R=2)(R=2). The retention rate measures the proportion of retained final pairs against the total number of generation draws across all attempts. The cumulative retention rate denotes the proportion of retained final pairs relative to the original set of target cells prior to retries. Draws per retention measures the average number of generation attempts required to yield one retained pair. axis subset cells filled retention rate cum. retention rate draws per retention By criterion friendliness 355/375 70.3% 95% 1.42× task resolution 329/375 58.9% 88% 1.70× communication clarity 316/375 51.9% 84% 1.93× By domain airline 135/150 61.4% 90% 1.63× banking 262/291 61.4% 90% 1.63× retail 309/342 63.1% 90% 1.59× telecom 294/342 54.9% 86% 1.82× By level contrast bad–good 359/375 72.4% 96% 1.38× ok–good 328/375 56.5% 88% 1.77× bad–ok 313/375 52.5% 84% 1.90× By agent model gpt-oss-120b 334/367 66.9% 91% 1.49× nemotron-lightning-3.5 341/384 57.3% 89% 1.74× gpt-oss-20b 325/374 56.1% 87% 1.78× C.2 Position Balance Audit To counter positional bias [38], the presentation order of trajectories in each pair is randomized during meta-evaluation (Table 7). Position balance serves as a diagnostic audit to verify that this randomization prevents positional confounds in filtering and human annotation. As detailed in Table 9, the target trajectory appears in slot two in 511 of the 1,000 final pairs (51.1%), with all experimental subsets remaining within a 45.5–54.3% range and displaying no significant departure from uniform parity. Table 9: Position balance audit of the target (better-steered) trajectory across data subsets. Across all groups, slot distribution remains statistically indistinguishable from uniform parity (binomial p>0.10p>0.10), confirming that position bias is not a baseline confounder. axis slot-2 share range binomial p range criterion 48.7–53.8% 0.17–0.91 domain 46.9–54.1% 0.27–0.73 agent model 45.5–54.3% 0.11–0.22 level contrast 49.2–54.0% 0.14–0.82 whole dataset 511/1,000 = 51.1% C.3 Environment Reward Verification Table 10 evaluates dataset pairs against τ3τ^3-bench programmatic task success rewards [0,1][0,1], which track database transitions, disclosure verification, and action matching independently of model annotation. Omitting the LLM judge component renders this audit entirely deterministic. Under this setup, steering should produce a strong environment reward shift for direct task execution (task_resolution) and a weak or non-significant shift for auxiliary stylistic criteria (communication_clarity and friendliness). The empirical results confirm this progression. Steering for task_resolution produces the largest reward increase (+0.122+0.122, p<0.001p<0.001), whereas friendliness exhibits no significant shift (+0.017+0.017, p=0.488p=0.488), demonstrating clear orthogonality to task outcome. communication_clarity occupies an intermediate position (+0.082+0.082, p<0.001p<0.001), reflecting a real-world dependency where unclear communication occasionally impedes execution workflows. Since the reward shift is smaller than that of explicit task resolution steering, communication clarity remains a distinct behavioral dimension. Overall, the vast majority of retained pairs are not confounded by outcome variation. Among the 1,000 pairs evaluated on both trajectories, 836 (84%) yield identical environment rewards, with the higher-steered trajectory scoring higher in 11.8% of pairs and lower in 4.6%. Table 10: Environment reward verification comparing deterministic τ3τ^3-bench task success reward gaps across evaluation criteria. The reward gap represents the score of the higher-steered trajectory minus that of the lower-steered trajectory. Among the 1,000 pairs evaluated on both trajectories, 836 pairs (84%) exhibit identical environment rewards. criterion pairs mean reward gap better / worse sign test p task resolution 329 +0.122+0.122 49 / 9 <0.001<0.001 communication clarity 316 +0.082+0.082 40 / 14 <0.001<0.001 friendliness 355 +0.017+0.017 29 / 23 0.488 Appendix D Human Panel Reliability and Sampling Robustness D.1 The Annotation Task Figure 3: The interface used by the human panel. The two trajectories of a pair are presented together with the criterion being judged; the steering instruction and target levels are withheld, and the annotator returns a forced binary choice together with a confidence rating. Annotator Panel The panel comprises 3 software engineers and 3 computer science researchers with experience developing language model agents or using them for software development. Two are full-time employees and four are interns, all within the same company as the authors. Two are native English speakers and four have professional English fluency. Annotators review criteria definitions and guidelines prior to evaluation. The panel is partitioned into two independent groups of three annotators. Each group evaluates the same 75 trajectory pairs. Each annotator receives a $50 stipend for completing the annotations. The annotation tasks involved evaluating benign model outputs and contained no offensive, sensitive, or harmful material. To minimize psychological fatigue and potential distress, participation was entirely voluntary, annotators were allowed to opt out or take breaks at any time without penalty. No personally identifiable information (PII) was collected. Task Workflow For each pair p=(x−,x+)p=(x^-,x^+), annotators select the superior trajectory given the target criterion c. Choice is forced (y^a(p)∈−1,+1 y_a(p)∈\-1,+1\), yielding the majority vote reference majmaj used in Section 5.2. Annotators also record confidence on a three-point scale (00 for guess, 11 for leaning, 22 for certain), analyzed in Appendix D.3. Annotators receive the same rendered input as the filtering judges and the evaluators under test (Figure 3; guidelines in Appendix H.6), with trajectory slot positions randomized and target steering instructions sc,t,ℓs_c,t, hidden. D.2 Inter-Annotator Agreement Table 11 and Table 12 quantify inter-annotator reliability across the human annotation panel and compare human consistency against the synthetic ground truth label ⋆ . As shown in Table 11, evaluating Krippendorff’s α across the human-only panel (a1,a2,a3)(a_1,a_2,a_3) yields a baseline reliability of +0.350+0.350 on unfiltered data (0D_0), which increases slightly to +0.383+0.383 on fully filtered pairs (2D_2). Replacing a single human rater with the synthetic label ⋆ and averaging across panel permutations (ai,aj,⋆)(a_i,a_j, ) substantially improves inter-rater reliability, raising Krippendorff’s α to +0.496+0.496 on 2D_2 (Δ=+0.113 =+0.113). This increase indicates that the synthetic ground truth label provides a more consistent central consensus than individual human annotators. Table 12 evaluates the reliability of the majority vote baseline majmaj across the 150 annotated pairs in 0D_0. The overall panel achieves a pooled Krippendorff’s α of +0.350+0.350. Exactly 51% of pairs (77/150) achieve unanimous agreement (3–0 vote), while the remaining 49% represent 2–1 split decisions. Pairwise inter-annotator agreement varies widely, with Cohen’s κ ranging from +0.63+0.63 down to +0.12+0.12 across panel configurations. Table 11: Krippendorff’s α over a panel of three raters. The first row is the human panel (a1,a2,a3)(a_1,a_2,a_3). The second row substitutes one human rater with synthetic label ⋆ and reports mean ± SD across permutations (a1,a2,⋆)(a_1,a_2, ), (a1,a3,⋆)(a_1,a_3, ), and (a2,a3,⋆)(a_2,a_3, ). Δ represents the difference between hybrid and human-only panels. rater pool 0D_0 1D_1 2D_2 (a1,a2,a3)(a_1,a_2,a_3) +0.350+0.350 +0.369+0.369 +0.383+0.383 (ai,aj,⋆)(a_i,a_j, ), mean ± SD +0.372±0.055+0.372± 0.055 +0.479±0.069+0.479± 0.069 +0.496±0.065+0.496± 0.065 Δ +0.022+0.022 +0.110+0.110 +0.113+0.113 Table 12: Reliability metrics for majority vote reference majmaj across the 150 rated pairs in 0D_0 (75 per panel). Unanimous indicates a 3–0 vote. Lower blocks report pairwise agreement metrics for two-annotator panel subsets. measure pooled (150) panel 1 (75) panel 2 (75) Krippendorff’s α +0.350+0.350 [+0.24, +0.45] +0.451+0.451 [+0.29, +0.60] +0.240+0.240 [+0.09, +0.38] unanimous (3–0) 77/150 = 51% 44/75 = 59% 33/75 = 44% Agreement between every two annotators of a panel agreement Cohen’s κ both said “certain” panel 1, (a1,a2)(a_1,a_2) 81.3% (61/75) +0.63+0.63 85% (28/33) panel 1, (a1,a3)(a_1,a_3) 65.3% (49/75) +0.32+0.32 75% (24/32) panel 1, (a2,a3)(a_2,a_3) 70.7% (53/75) +0.41+0.41 83% (30/36) panel 2, (a1,a2)(a_1,a_2) 66.7% (50/75) +0.34+0.34 81% (13/16) panel 2, (a1,a3)(a_1,a_3) 65.3% (49/75) +0.27+0.27 68% (15/22) panel 2, (a2,a3)(a_2,a_3) 56.0% (42/75) +0.12+0.12 73% (27/37) mean 67.6% +0.35+0.35 D.3 Confidence and Task Difficulty Table 13 investigates whether filter judges select for human confidence or agreement. Annotators rate each pair on a three-point scale (00 for guess, 11 for leaning, 22 for certain), where the per-pair confidence rating is the mean of three annotator scores. Human agreement correlates positively with confidence: Krippendorff’s α reaches +0.545+0.545 on the 41 pairs where all three annotators indicate “certain”, compared to +0.270+0.270 on the remaining 109 pairs where at least one annotator is unsure. However, filter judges do not select for higher annotator confidence. Mean confidence moves by only +0.02+0.02 on a 00–22 scale between unfiltered 0D_0 and dual-filtered 2D_2. Furthermore, neither J1J_1 nor J2J_2 retains a significantly more confident pair subset than the one it removes (p=1.000p=1.000 for J1J_1, p=0.172p=0.172 for J2J_2). Given the sample sizes of removed pairs (n=36n=36 for J1J_1 and n=10n=10 for J2J_2), the minimum detectable confidence difference is approximately 0.200.20. These results indicate that filter judges discard pairs where annotators split on majority consensus rather than pairs annotators find inherently difficult. Consequently, filtering raises label agreement without rendering the underlying evaluation task trivial: mean annotator confidence remains 1.511.51 out of 2.002.00 on 2D_2, reflecting non-trivial judgment calls for human evaluators. Table 13: Human annotator confidence and agreement metrics across filter depths kD_k. Per-pair confidence reflects the mean of three annotator ratings (0=guess0=guess, 1=leaning1=leaning, 2=certain2=certain). Statistical significance (p) is computed via permutation tests over 100,000 shuffles. α is Krippendorff’s α. subset |||D| mean confidence SD 0D_0 (unfiltered) 150 1.49 0.43 1D_1 (J1J_1) 114 1.49 0.44 2D_2 (J1&J2J_1\&J_2) 104 1.51 0.44 Kept against removed, per filter judge judge kept removed difference (p) J1J_1 1.49 (1D_1, 114) 1.49 (0∖1D_0 _1, 36) +0.00+0.00 (1.000) J2J_2 1.51 (2D_2, 104) 1.30 (1∖2D_1 _2, 10) +0.21+0.21 (0.172) Agreement does track confidence, which the judges do not select for subset of 0D_0 |||D| α all three said “certain” 41 +0.545+0.545 anyone was unsure 109 +0.270+0.270 D.4 Sampling Robustness Table 14 verifies that the 150 human-annotated pairs accurately represent the full 1,000-pair dataset. In the human evaluation study, pairs are sampled uniformly across domain and criterion combinations. However, the full dataset features non-uniform domain allocations (e.g., airline comprises 13.5% of the 1,000 pairs but 23.1% of the annotated sample). To evaluate potential sampling bias, we post-stratify synthetic label accuracy using actual population weights across four axes: domain, criterion, agent model, and level contrast. Re-weighting shifts reported synthetic label accuracy by less than 1.0pt across all configurations, well within the sampling standard error of ±3.7pt± 3.7pt. Furthermore, dataset retention rates under filtering in the human sample match full population proportions closely: the 150-pair sample survives J1J_1 at 76.0% (114/150) and dual filtering at 69.3% (104/150), compared to 76.4% (859/1,125) and 68.4% (770/1,125) across the full dataset. This confirms that the annotated subset is representative of overall pipeline behavior. Table 14: Synthetic label accuracy (agreement with human majority vote) across raw sample estimates and population re-weighted strata (kD_k). Post-stratification shifts headline accuracy by less than 1.0pt, confirming sample representativeness within a ±3.7pt± 3.7pt standard error. subset as reported domain-weighted criterion-weighted agent-weighted contrast-weighted 0D_0 (unfiltered) 73.3% 73.1% 73.3% 73.4% 73.7% 1D_1 (J1J_1) 84.2% 83.4% 83.8% 84.6% 84.6% 2D_2 (J1&J2J_1\&J_2) 84.6% 83.9% 84.2% 85.0% 85.3% Appendix E Filtering Gains and Human Alignment Dynamics Table 15 breaks down filtering gains for agreement between synthetic labels ⋆ and human majority vote across four dataset axes. Filtering yields positive accuracy gains across all subsets, with the largest increases occurring where evaluation criteria are most difficult to judge without oversight. Specifically, communication clarity achieves a +19+19pt gain, though only 52% of clarity pairs survive dual-filter verification. This indicates that the filtering pipeline discards a larger proportion of borderline trajectories to enforce the target criteria alignment. Task resolution yields a +10+10pt gain with a 72% retention rate. Friendliness exhibits a modest +3+3pt gain with 84% retention, starting from a high unfiltered baseline of 90% and leaving minimal room for further optimization. Two structural axes exhibit uniform gains across subsets. Across agent models, filtering improvements remain stable (+10pt to +13pt), indicating that the effect stems from dataset construction rather than specific architectures. Similarly, gains across level contrasts range tightly between +10pt and +12pt, proving filtering refines fine contrast pairs as effectively as wide ones without relying on gross trajectory contrasts. The telecom domain is the sole exception, yielding a +1pt gain compared to +12pt to +17pt elsewhere. Table 16 suggests a structural mechanism: telecom features a 3,715-word policy, 30 user-side tools, a median length of 48 messages, and tool errors in 48% of simulations, leaving filter judges with the least clear signal. Alternatively, telecom starts with the highest unfiltered accuracy at 79%, leaving minimal room to gain. Cells hold 24 to 50 pairs throughout, supporting reliable subset ordering. Table 17 summarizes the second filter (J2J_2) performance. J2J_2 removes 10 of the 114 pairs passed by J1J_1, yielding a marginal +0.4+0.4pt gain in label purity at the cost of a −7-7pt drop in total yield. Human annotators support J1J_1 filter overrulings in 61% of cases (22/36), but only 20% (2/10) for J2J_2. Among the 10 pairs removed by J2J_2, the ground truth label ⋆ is human-verified as correct in 8 instances. At this sample size, the second filter stage yields diminishing returns. It incurs notable yield loss without providing a statistically meaningful improvement in label purity or alignment quality. The full dataset retains depth 2D_2, with ablation metrics reported for completeness. Table 15: Agreement between synthetic labels ⋆ and human majority vote across filter depths kD_k and dataset subsets. Gain reflects the accuracy change from two filters (2D_2) versus zero filters (0D_0); retention represents the proportion of target pairs retained after two filtering stages. axis subset unfiltered (0D_0) 1 filter (1D_1) 2 filters (2D_2) gain retention By criterion friendliness 90% (45/50) 93% (42/45) 93% (39/42) +3+3pt 84% task resolution 68% (34/50) 77% (30/39) 78% (28/36) +10+10pt 72% communication clarity 62% (31/50) 80% (24/30) 81% (21/26) +19+19pt 52% By domain airline 75% (27/36) 92% (23/25) 92% (22/24) +17+17pt 67% retail 67% (26/39) 80% (24/30) 84% (21/25) +17+17pt 64% banking 72% (26/36) 85% (22/26) 84% (21/25) +12+12pt 69% telecom 79% (31/39) 82% (27/33) 80% (24/30) +1+1pt 77% By agent model gpt-oss-20b 76% (35/46) 90% (27/30) 89% (24/27) +13+13pt 59% nemotron-lightning-3.5 71% (36/51) 81% (34/42) 83% (29/35) +12+12pt 69% gpt-oss-120b 74% (39/53) 83% (35/42) 83% (35/42) +10+10pt 79% By level contrast bad–good (widest) 80% (37/46) 90% (35/39) 92% (33/36) +11+11pt 78% ok–good 75% (38/51) 86% (31/36) 87% (26/30) +12+12pt 59% bad–ok 66% (35/53) 77% (30/39) 76% (29/38) +10+10pt 72% Table 16: Domain complexity metrics across evaluation environments. Tool name overlap between domain pairs ranges from 0.03 to 0.11 by Jaccard index; banking represents the only domain featuring active retrieval. domain policy words agent tools user tools sims median msgs median tool calls tool-error sims mean reward airline 1,313 14 0 440 18 5 21% 0.434 banking 926 16 2 855 30 7 3% 0.037 retail 1,158 16 0 980 22 6 34% 0.353 telecom 3,715 13 30 1,072 48 10 48% 0.243 Table 17: Trade-off analysis for the second-stage filter J2J_2. Among the 10 pairs eliminated by stage two, human majority vote supports the original ground truth label in 8 cases. human panel pairs removed label was right purity yield 1 6 4 86.0 → 88.2% (+2.3+2.3pt) −8-8pt 2 4 4 82.5 → 81.1% (−1.3-1.3pt) −5-5pt pooled 10 8 84.2 → 84.6% (+0.4+0.4pt) −-7pt Appendix F The Full Evaluator Sweep and Contamination Diagnostic F.1 The Full Sweep Table 18 reports performance metrics across all 25 evaluators, serving as the unabridged version of Table 3 in the main text (which presents a 12-evaluator subset selected to span accuracy ranges and model families). All summary statistics, subset means, and correlation analyses in Section 5.3 (including Table 4) are computed over the full 25-evaluator roster. Evaluator accuracy is measured as Agr⁡(f,⋆,)Agr(f, ;D) across all 1,000 pairs in 2D_2 with ties treated as incorrect, reported as the mean ± SD across three independent runs. Accuracy spans a 48.3pt range, from claude-opus-5 at 85.1% down to qwen3-4b at 36.8%. Table 18 introduces two additional column blocks omitted from the main text due to space constraints: accuracy breakdown by level contrast and accuracy breakdown by agent model. Similar to the domain and criterion blocks, each of these blocks partitions the 1,000 pairs such that their weighted average equals the total dataset accuracy. Row indicators denote specific evaluator conditions: † identifies the two models where parse failures are marked as incorrect, while ∗* marks the three models utilized in dataset construction (analyzed further in Appendix F.2). Table 18: The complete sweep: all 25 evaluators across 1,000 data pairs, repeat n=3n=3. Every statistic reported in Section 5.3 is computed over these 25 rows. Table 3 in the main paper shares the row numbering. Leniency is the mean of every score the evaluator emits. It strongly correlates with accuracy (Table 4). Level contrast indicates the two steering levels each trajectory pair has: b–o is bad–ok (n=313n=313), o–g is ok–good (n=328n=328), b–g is bad–good (n=359n=359). Agent model columns are the three trajectory generators of Table 6: oss-20 is gpt-oss-20b (n=325n=325), oss-120 is gpt-oss-120b (n=334n=334), nemo. is nemotron-lightning-3.5 (n=341n=341). Note that agent model is a property of the data point being judged rather than of the evaluator. The final row is the unweighted mean over all 25 evaluators. Released is the public release month. Temperature = 0 except where unavailable‡. Preference is derived from individual trajectory score gaps; ties count as incorrect. Accuracy and score denote benchmark-level accuracy and raw score output of the evaluators, respectively. SD is standard deviation. ∗Contaminated by self-assessment bias because the model is used during data synthesis (Appendix F.2). †Parse failures on 192 and 119 calls after repeated retries, counted incorrect. Bolded numbers indicate best performance within the column. parameters accuracy (%) average tie by domain (%) by criterion (%) by level contrast (%) by agent model (%) distinct # evaluator released total active reas. mean ± SD leniency score SD (%) airl. bank. retail telec. clar. friend. task b–o o–g b–g oss-20 oss-120 nemo. scores 1 claude-opus-5 2026-07 closed closed Yes 85.1 ± 0.93 0.324 0.023 4.1 84 82 86 87 73 96 85 85 82 88 85 88 82 69 2 claude-sonnet-5 2026-06 closed closed Yes 83.8 ± 0.72 0.458 0.040 5.8 82 81 84 87 73 95 82 85 81 86 81 89 82 34 3 kimi-k3 2026-07 2.8T 104B Yes 83.6 ± 0.12 0.508 0.036 6.0 83 80 84 87 71 97 81 87 78 86 82 87 81 37 4 gpt-5.6-terra‡ 2026-07 closed closed Yes 81.7 ± 0.49 0.560 0.043 3.4 81 81 82 82 69 96 78 86 75 84 79 86 80 88 5 gpt-5.6-luna‡ 2026-07 closed closed Yes 81.6 ± 0.64 0.563 0.045 3.8 82 81 81 83 69 93 81 87 75 83 81 85 79 83 6 glm-5p2 2026-06 753B ∼ 30B Yes 81.5 ± 0.50 0.521 0.051 10.1 84 81 81 82 78 92 73 81 78 85 80 85 79 31 7 gpt-oss-120b∗ 2025-08 116.8B 5.1B Yes 80.5 ± 0.40 0.624 0.050 9.9 80 81 80 80 78 87 75 83 74 84 78 87 77 45 8 gpt-5.4-nano‡ 2026-03 closed closed No 79.4 ± 1.16 0.631 0.052 7.8 81 78 78 81 74 84 80 82 72 83 80 84 75 61 9 gpt-5.4-mini‡ 2026-03 closed closed No 79.4 ± 0.79 0.628 0.053 6.1 74 79 80 81 71 91 75 85 72 82 79 84 76 89 10 gpt-5.6-sol‡ 2026-07 closed closed Yes 78.7 ± 0.40 0.543 0.035 4.9 79 77 81 77 64 97 74 84 72 80 77 82 77 86 11 claude-haiku-4-5 2025-10 closed closed No 78.7 ± 0.10 0.505 0.000 11.6 81 79 76 81 76 85 74 75 77 84 76 85 75 32 12 qwen3p8-max 2026-08 2.4T 95B Yes 78.0 ± 0.36 0.521 0.037 11.7 74 78 77 81 70 94 68 81 73 80 76 82 76 30 13 nemotron-3-ultra 2026-06 549B 55B Yes 76.8 ± 0.76 0.558 0.044 14.2 75 80 70 81 71 88 71 79 70 81 76 81 73 28 14 llama3.1-70b 2024-07 70.6B dense No 70.0 ± 0.55 0.632 0.016 22.8 67 73 67 72 64 69 77 73 61 76 73 79 58 14 15 gemini-3.7-flash 2026-08 closed closed Yes 69.5 ± 0.45 0.615 0.026 23.4 71 70 69 69 57 95 54 72 64 73 69 74 66 34 16 gpt-oss-20b∗ 2025-08 20.9B 3.6B Yes 68.7 ± 1.01 0.625 0.056 21.7 71 70 68 67 74 80 51 69 67 70 68 74 64 31 17 gemma-4-31b 2026-03 32.2B dense Yes 68.1 ± 0.15 0.657 0.023 27.4 63 67 66 74 47 92 63 75 56 73 71 75 59 14 18 deepseek-v4-pro 2026-08 1.6T 49B Yes 67.0 ± 0.67 0.505 0.045 20.4 62 70 69 65 61 90 49 69 63 69 63 70 68 23 19 mistral-large2 2024-07 123B dense No 64.8 ± 0.40 0.717 0.014 28.1 59 63 62 72 57 63 74 71 53 70 70 72 53 12 20 nemotron-lightning-3.5∗ 2026-08 32B 3B Yes 62.4 ± 0.93 0.705 0.060 28.6 61 58 60 69 47 75 64 67 52 68 66 69 52 22 21 gemini-3.5-flash-lite 2026-07 closed closed No 56.1 ± 0.68 0.723 0.055 36.3 53 58 47 65 52 59 58 63 41 64 64 62 44 16 22 llama4-maverick 2025-04 400B 17B No 55.6 ± 1.56 0.765 0.039 35.9 47 56 54 61 52 55 60 58 51 58 61 60 45 15 23 llama3.1-8b† 2024-07 8.0B dense No 43.1 ± 0.47 0.622 0.005 43.7 43 46 48 35 28 43 57 44 39 47 42 46 42 12 24 qwen3-1p7b† 2025-04 2.0B dense Yes 42.8 ± 0.35 0.760 0.076 31.9 44 44 42 42 43 47 38 47 33 48 43 45 40 15 25 qwen3-4b 2025-08 4.4B dense No 36.8 ± 0.65 0.804 0.029 56.9 36 34 32 44 30 25 55 43 27 40 43 41 26 17 Mean, all 25 evaluators 70.1 68.7 69.9 69.0 72.2 62.0 79.5 67.9 73.2 63.4 73.6 70.5 74.9 65.1 F.2 Contamination Diagnostic Three evaluators in our sweep contribute directly to benchmark construction: nemotron-lightning-3.5 serves as filter judge J1J_1, gpt-oss-120b serves as J2J_2, and these two models in addition to gpt-oss-20b generate 325–341 of the evaluated trajectory pairs. While filter judge influence is intrinsic to dataset definition, generation influence can be isolated. Table 19 evaluates potential self-preference bias by comparing evaluator accuracy on self-generated trajectories versus external trajectories. The diagnostic indicates that self-preference is not systematic. Only gpt-oss-120b exhibits higher accuracy on self-generated outputs (+9.1pt+9.1pt), gpt-oss-20b displays negligible shift (−0.6pt-0.6pt), and nemotron-lightning-3.5 performs −15.7pt-15.7pt on its own trajectories. This result is primarily driven by trajectory difficulty shifts: because agent assignment is deterministic per cell rather than random, excluding self-generated pairs confounds underlying trajectory difficulty with evaluator bias. As reported in Table 18 under the “by agent model” block, these trajectory difficulty shifts are revealed in the mean accuracy across all 25 evaluators: 74.9%74.9\% on gpt-oss-120b trajectories, 70.5%70.5\% on gpt-oss-20b trajectories, and drops to 65.1%65.1\% on nemotron-lightning-3.5 trajectories. The observed gaps in Table 19 closely track benchmark-wide trajectory difficulty shifts rather than systematic self-preference bias. Notably, all three generator models belong to the sub-7B active parameter MoE tier, highlighting a parameter tier constraint when evaluating lightweight model architectures. Table 19: Contamination diagnostic comparing evaluator accuracy on self-generated versus external trajectory pairs. Values represent means across three independent evaluation runs. accuracy (%) evaluator its own pairs the other pairs gap (pt) gpt-oss-120b 86.5 (n=334n=334) 77.4 (n=666n=666) +9.1+9.1 gpt-oss-20b 68.3 (n=325n=325) 68.9 (n=675n=675) −0.6-0.6 nemotron-lightning-3.5 52.1 (n=341n=341) 67.8 (n=659n=659) −15.7-15.7 Appendix G Cost and Caching The dataset curation pipeline generates 1,000 validated trajectory pairs at a total cost of $23.63, or $0.024 per kept pair (Table 20). Steering instruction generation and filter judges account for 5.0% of total token volume and 12.4% of dollar costs. Table 20: Computational overhead and curation cost. All costs are computed or estimated with the model service provider’s token-based pricing model. Pipeline Stage API Calls Tokens Token Share Uncached Cost Effective Cost 1. Steering Generation 1,125 2.56M 0.3% $0.85 $0.67 2. Trajectory Generation 77,980 709.4M 95.0% $51.16 $18.36 3. Filter Judges (K=2K=2) 2,879 34.8M 4.7% $4.42 $2.27 Total Pipeline 81,984 746.7M 100.0% $56.43 $23.63 Prompt caching is the primary driver of generation cost reduction. In an agentic loop, the full conversation history is re-sent at each turn, making almost every request carry a prefix the server already holds. Across the generation run, 98% of all processed tokens are prompt tokens. Table 21 reports the prompt cache hit rate across all 77,980 trajectory calls. The overall hit rate reaches 90.5%, ranging from 87.9% to 95.4% across individual models. This caching efficiency reduces total API costs by a factor of two to four depending on model pricing. Table 21: Prompt cache hit rates across all 77,980 trajectory generation calls. model prompt tokens cached rate nemotron-lightning-3.5 350,084,755 312,862,256 89.4% gpt-oss-20b 137,504,779 125,776,651 91.5% qwen3-30b-a3b-instruct 107,519,458 102,523,608 95.4% gpt-oss-120b 98,571,871 86,652,136 87.9% total 693,680,863 627,814,651 90.5% Appendix H Prompts This section reproduces every pipeline prompt verbatim in execution order: shared criterion definitions (Appendix H.1), steering instruction generation (Appendix H.2), agent system prompt wrapping (Appendix H.3), filter judging (Appendix H.4), model evaluation (Appendix H.5), and human annotation instructions (Appendix H.6). Variable names enclosed in braces (e.g., criterion_name) represent runtime substitution slots. Agent and user simulator system prompts are retained directly from τ3τ^3-bench without modification; only the steering wrapper (Appendix H.3) is appended to agent instructions. H.1 Criterion Definitions Below is the verbatim text for the three evaluation criteria c. The exact same criterion names and descriptions are supplied to the generator, filter judges, model evaluators, and human annotators, ensuring information parity across all pipeline components. Table 6 provides abridged summaries for reference. ⬇ name: friendliness description: Warmth and consideration toward the customer: whether the agent acknowledges their situation and how they feel about it, delivers unwelcome news with care, and leaves them feeling attended to. Judge the manner, not whether the request was resolved. name: communication_clarity description: How easily the customer can follow the agent: whether the main point is findable, whether technical or policy language is explained, whether multi-part information is organised, and whether the customer is left knowing what is true and what happens next. Judge the presentation, not the warmth or the outcome. name: task_resolution description: Whether the customer’s actual problem was settled: did the agent establish what was needed, take the actions that would resolve it, and leave the customer with the outcome they came for. A correct refusal counts as resolution -- if the request was not permitted, saying so plainly and explaining why resolves it, while quietly doing it anyway does not. Judge the outcome, not the manner or how well it was explained. H.2 Steering Instruction Generator The generator synthesizes three steering instructions sc,t,ℓs_c,t, for a given task cell (t,c)(t,c) in a single call, corresponding to levels ℓ∈bad,ok,good ∈\ bad, ok, good\. Task specifics (task_description and tool_names) are populated dynamically. Rule 5 prevents the generator from encoding unstated task outcomes, while Rule 1 ensures level contrasts reflect behavioral differences rather than varying degrees of task detail. ⬇ You are designing an experiment about how well automatic evaluators judge the quality of AI agent behaviour. For ONE specific task and ONE specific quality metric, write three system-prompt instructions that steer an agent to perform at three levels on that metric: BAD, OK and GOOD. ## The metric Name: criterion_name What it means: criterion_description ## The task What the user is trying to get done: task_description Tools the agent can use: tool_names ## Rules for the instructions you write 1. BE SPECIFIC TO THE KIND OF WORK THIS TASK INVOLVES -- NOT TO ITS PARTICULARS. Generic advice that would fit any task is one failure mode. Naming the task’s details is the other, and it is worse. Do refer to which tools matter here, what kind of information the user needs, and the shape this interaction will take. Do NOT restate the specifics you were given: no names, no user or account or booking identifiers, no retelling of this person’s circumstances. The agent learns those from the conversation itself. Putting them in its system prompt hands it information out of band, and makes your three instructions differ by how much detail they carry rather than by the metric. Wrong: "You are assisting Emma Kim (user ID emma_kim_9957) with cancelling reservation EHGLP3, and she was out of town relying on prior insurance." Right: "When the customer explains why they are asking, acknowledge the circumstances they raise before you get to the outcome." 2. MAKE THE THREE LEVELS SEPARATE. Someone reading the three resulting transcripts should be able to rank them on this metric without being told which is which. If two of your instructions would produce similar behaviour, rewrite them. 3. FOR BAD, AIM AT THE OPPOSITE OF THE METRIC -- do not merely withhold good behaviour. Work out what the active opposite of this metric is, on this task, and instruct the agent to pursue it. If the metric is friendliness, BAD is not neutral or terse: it is cold, dismissive, impatient. If the metric is clarity, BAD is actively confusing, not just unpolished. If the metric is relevant tool use, BAD uses tools in ways that actively do not serve the request. Name the opposite explicitly and tell the agent to do it. 4. DESCRIBE OK IN ITS OWN TERMS, not as "somewhat good" or "slightly bad", and not as a milder version of BAD. OK is what unremarkable, adequate, uncared-for work looks like on this metric -- the agent is neither trying nor failing. 5. YOU DO NOT KNOW THE CORRECT OUTCOME, AND MUST NOT ASSUME ONE. You have not been told how this task should be resolved, and you may not guess. Do not write anything that presupposes an answer -- not "state the refusal", not "confirm it is not permitted". Where the outcome matters to how the agent should behave, phrase it conditionally: "if it turns out you cannot do what they are asking, ...". 6. STEER CONDUCT, NEVER DICTATE WORDS. Do not write the agent’s reply for it, do not include a sentence for it to say, and never use "respond with", "say exactly" or a quoted message. You are shaping HOW it works, not WHAT it concludes. 7. WRITE ONLY THE INSTRUCTION BODY. Do not tell the agent to keep the instruction secret or that the objective overrides its defaults -- that is added separately. Two to five sentences each, addressed to the agent as "you". Nothing is off limits in how badly the BAD level may perform on this metric. If performing the opposite of this metric means leaving work out, being unhelpful, being wrong, or failing the task outright, say so plainly. The experiment needs a genuinely poor example, and a BAD level that is merely lukewarm is the most common way this goes wrong. Return JSON with exactly these keys: - "reasoning": what good, adequate and poor criterion_name concretely look like for this task, and how you will make the three levels separate - "bad": the instruction body for the BAD level - "ok": the instruction body for the OK level - "good": the instruction body for the GOOD level H.3 Steering Wrapper The following instruction is padded around each steering instruction when it is injected into the original agent prompt. ⬇ <behaviour> For this conversation you have one additional objective. It takes strict precedence over your default behaviour, including when it produces handling that is less polished, less thorough or less successful than you would normally deliver. Following it exactly is what makes your handling of this conversation correct. instruction Do not mention, quote or allude to this objective, and do not reuse its wording. </behaviour> H.4 Filter Judge The filter judge prompt Jk(p,c)J_k(p,c) is identical for J1J_1 and J2J_2, differing only in the underlying model deployment. The prompt slot contains shared context (tool definitions, knowledge bases, and domain policies), while response_a and response_b contain the rendered trajectory pair in randomized slot order. Filter judges output a forced choice, and pairs are retained only when both judges match the synthetic ground truth label ⋆ . ⬇ You are evaluating two AI assistant responses on a specific criterion. Criterion: criterion_name Definition: criterion_description User prompt: prompt Response A: response_a Response B: response_b Which response better fulfills the criterion? Return JSON with exactly two keys: - "reasoning": at most 3 sentences comparing the two responses on this criterion - "answer": "A" or "B" State the answer last, and state it even if the choice is close. H.5 Evaluator Under Test The prompt template for evaluators under test f⁡(x,c)f(x;c). Evaluators process a single trajectory x in isolation, assigning a score in [0,1][0,1] without access to the paired execution. Pairwise preference predictions y^f(p) y_f(p) are derived post hoc from the sign of the score gap (Eq. (3)). Evaluators receive the identical shared context prompt provided to filter judges and human annotators, eliminating information asymmetry across evaluation modes. ⬇ You are evaluating one customer-service conversation on a single named criterion. CRITERION: criterion_name criterion_description Judge this criterion and nothing else. Ignore everything the criterion does not mention. A conversation can fail at its task and still be better on the criterion. Judge the AGENT’s conduct only. The customer’s turns were produced by a simulator and are not being evaluated. REFERENCE DOCUMENTS (the agent’s tools, and the policy it had to follow). Consult them if you need to check whether the agent was permitted to do something. You will not need them for most conversations. prompt CONVERSATION Turns are numbered. "Agent calls X(...)" is a tool call the agent made, and "Tool result ->" is what came back. response Score the agent on criterion_name from 0.0 (worst) to 1.0 (best). Use the full range: reserve 0.0 and 1.0 for clear cases and place ordinary ones in between. Return JSON with exactly two keys: - "reasoning": at most 3 sentences, citing what in the conversation drove the score - "score": a number between 0.0 and 1.0 H.6 Human Annotator Instructions The instructions of the human annotation interface (Figure 3). Annotators review these instructions prior to their initial trial and retain permanent access during annotation. On-screen confidence levels (00, 5050, and 100100) are mapped to numerical values (00, 11, and 22) as used throughout Appendix D.3. ⬇ Before you start Pairwise annotation of agent conversations. Read this once. Judge one criterion, and nothing else Each pair is judged on a single named criterion. Only that criterion counts. Ignore everything else. In an extreme scenario, a conversation can fail the task and still be the better one on the metric. The flow -- three steps per pair 1. Pick a side. Click the conversation that better fits the criterion, or press the left / right arrow key. 2. Rate your confidence: 1 for 0 (a guess), 2 for 50 (leaning one way), 3 for 100 (certain). 3. Press Next (Enter) to submit and move on. Backspace goes back to revise. There is more to read if you need it Under the header are buttons labelled "Additional documents to help you decide": the agent’s tool list, the policy it had to follow, and the knowledge base where one exists. They are closed by default and open on a click, one at a time. Tool results and searches inside a conversation open the same way. You do not need any of it for most pairs. Reach for the policy when the question is whether the agent was allowed to do something. How much to deliberate Use your best judgment, but it might hurt to think too much. In my experience giving each example about a minute should be sufficient. If the two really are hard to separate, still pick a side -- then rate it 0. That records the pair honestly as a coin-flip instead of hiding a guess among your real judgments. [ Start annotating ] ---- shown once at the start of each criterion’s block of pairs ---- You are now judging <criterion name> <criterion description> Press any key or click to begin this section ---- the standing question in the header of every pair ---- Which conversation is better on this criterion? Appendix I Assets, Licenses, and Terms of Use Table 22 lists every existing asset this work builds on, its role in the pipeline, and its license and terms of use. All open-weight models are accessed through a single hosted inference provider rather than downloaded, and all proprietary models are accessed through their vendors’ paid APIs, so each model is used under both its own license and the serving provider’s terms of service. Table 22: Existing assets used in this work. Model roles refer to the pipeline stages of Table 7 and the evaluator sweep of Section 5.3. asset role in this work license terms of use Benchmark substrate and serving infrastructure τ3τ^3-bench, sierra-research/ tau2-bench [20, 21, 22] Task definitions, domain environments, tool sets, agent framework, and user simulator MIT License Free commercial and non-commercial use, modification, and redistribution provided the MIT notice (Copyright (c) 2025 Sierra Research) is preserved; evaluation runs are additionally subject to third-party LLM API terms Fireworks AI serverless inference Hosted inference for every open-weight model in the pipeline and the sweep not applicable (service) Fireworks AI terms of service Open-weight models gpt-oss-20b, gpt-oss-120b Trajectory generation; 120b also generates steering instructions and serves as filter judge J2J_2; both are evaluators Apache 2.0 model license and Fireworks AI terms of service nemotron-lightning-3.5 Filter judge J1J_1, trajectory generation, and evaluator NVIDIA OpenMDW-1.1 as above nemotron-3-ultra Evaluator NVIDIA OpenMDW-1.1 as above qwen3-30b-a3b-instruct User simulator Apache 2.0 as above qwen3-1p7b, qwen3-4b Evaluators Apache 2.0 as above qwen3p8-max Evaluator Apache 2.0 as above llama3.1-8b, llama3.1-70b Evaluators Llama 3.1 Community License as above llama4-maverick Evaluator Llama 4 Community License as above gemma-4-31b Evaluator Apache 2.0 as above mistral-large2 Evaluator Mistral Research License as above deepseek-v4-pro Evaluator MIT License as above glm-5p2 Evaluator MIT License as above kimi-k3 Evaluator custom Kimi K3 license as above Proprietary models (weights not released; API access only) claude-opus-5, claude-sonnet-5, claude-haiku-4-5 Evaluators closed weights, no public license Anthropic API terms of service gpt-5.6-terra, gpt-5.6-luna, gpt-5.6-sol, gpt-5.4-nano, gpt-5.4-mini Evaluators closed weights, no public license OpenAI API terms of service gemini-3.7-flash, gemini-3.5-flash-lite Evaluators closed weights, no public license Google Gemini API terms of service

Instant research discovery