Paper deep dive

Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation

HyunJoon Jung, William Na

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 45

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 4/2/2026, 3:28:01 AM

Summary

This paper investigates the scaling properties of LLM-based agent judges in evaluating conversational AI. Through a study of 960 sessions, the authors identify a 'score-coverage dissociation' where scoring reliability improves logarithmically with panel size, while unique issue discovery follows a sublinear power law. The study demonstrates that structured persona conditioning is essential for ensemble diversity, allowing agent panels to achieve human-level evaluation performance by aggregating diverse perspectives rather than relying on individual judge consistency.

Entities (4)

HyunJoon Jung · researcher · 100%Big Five personality framework · framework · 99%LLM-as-Judge · methodology · 98%Score-coverage dissociation · phenomenon · 95%

Relation Signals (2)

LLM-as-Judge → exhibits → Score-coverage dissociation

confidence 95% · we identify a score–coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law

Big Five personality conditioning → enhances → Ensemble diversity

confidence 92% · The mechanism traces to ensemble diversity—Big Five personality conditioning makes agents probe different quality dimensions

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,039 characters extracted from source content.

Expand or collapse full text

Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation HyunJoon Jung Corresponding author: hjung@mphora.ai MPhora.ai William Na MPhora.ai Abstract LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score–coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law—both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity—Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties. Figure 1: Overview. 32 persona-based agent judges evaluate conversational AI across 960 sessions. A Turing test confirms agent scores are indistinguishable from human scores (p=0.38p=0.38). The scaling analysis reveals a score–coverage dissociation: scoring reliability improves logarithmically while issue discovery follows a sublinear power law (b≈0.69b≈ 0.69). An ablation shows structured persona conditioning is necessary. 1 Introduction Evaluating conversational AI now relies heavily on LLM-as-Judge approaches (Zheng et al., 2023; Liu et al., 2023; Kim et al., 2024), which offer scalability but have well-documented biases in single-judge settings (Wang et al., 2024). A natural response is to use multiple evaluators—analogous to crowdsourced annotation, where aggregating multiple raters improves label reliability (Sheng et al., 2008; Snow et al., 2008). But unlike human annotators labeling static examples, agent judges actively converse with the system under test. Each agent judge conducts a multi-turn interaction from a distinct persona—varying in expertise, communication style, and evaluation focus—and produces both a quality score and qualitative insights. The standard assumption from crowdsourcing research (Sheng et al., 2008; Snow et al., 2008) is that more raters yield more reliable consensus. We show that this assumption is incomplete for agent-based evaluation. In crowdsourcing, all annotators label the same item; in agent-based evaluation, each judge generates a different interaction, observing different facets of the same system. This distinction changes what “adding more raters” can achieve. Evaluating conversational AI well requires both reliable scoring—can we trust the quality rating?—and broad coverage—have we observed enough nondeterministic behavior to understand where the system fails? These seem naturally aligned: more judges should improve both. To test this, we conducted a large-scale study using 32 persona-based agent judges—each with distinct expertise, Big Five personality profile (Costa and McCrae, 1992), and interaction style—across 15 tasks spanning 5 domains and 3 complexity levels (480 sessions per model pair, 960 total). A Turing-style validation with 50 human raters confirms that agent–human score differences are indistinguishable from human–human differences (p=0.38p=0.38). With this established, the scaling analysis reveals a score–coverage dissociation: scoring reliability improves logarithmically with panel size, while unique issue discoveries follow a sublinear power law—both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. An expertise effect sharpens this picture: expert agent judges score lower (Cohen’s d=0.24d=0.24) by posing harder queries that expose quality limitations invisible to simpler interactions. Panel reliability, meanwhile, arises not from agents agreeing with each other, but from agents probing different aspects of the same system—their diverse errors cancel out while the shared quality signal accumulates. A controlled ablation confirms that these properties require structured persona conditioning, not simple prompting. The contributions of this study are as follows: (1) a Turing-style test (50 human raters, 86 sessions) confirms agent–human score differences are indistinguishable from human–human differences; (2) we identify the score–coverage dissociation—scoring reliability improves logarithmically while issue discovery follows a sublinear power law—and show it holds across two model pairs; (3) a variance decomposition reveals the ensemble diversity mechanism behind panel reliability, and we show that expert judges function as adversarial probes, discovering 1.9×1.9× more issue categories; and (4) a controlled ablation demonstrates that structured persona conditioning—not simple prompting—is required to produce these scaling properties. 2 Related Work Multi-rater aggregation. Dawid and Skene (1979) introduced the EM framework for estimating true labels from noisy annotators. Snow et al. (2008) showed that 4–5 non-expert crowdworkers can match expert quality, and Sheng et al. (2008) demonstrated that selectively acquiring additional labels improves data quality. These studies established that more raters generally help—but all assume raters label the same static item. We extend this question to agent-based evaluation, where each rater generates a different interaction, influencing how aggregation works. Beyond scoring: why coverage matters. Most evaluation frameworks focus on quality scores, but scores miss what issues the system has. In usability testing, Nielsen and Landauer (1993) showed different users discover different problems; Faulkner (2003) and Schmettow (2012) agreed: coverage follows a different curve than reliability. For generative AI, this gap matters more—nondeterministic outputs mean a single interaction cannot capture the full range of behavior. In ecology, species accumulation curves (Colwell & Coddington, 1994) formalize a similar phenomenon: common species are found first, rare species require more sampling. To our knowledge, no prior work has applied this framing to agent-based evaluation or formally disentangled scoring and coverage scaling within the same framework. From LLM-as-Judge to Agent-as-Judge. Zheng et al. (2023) introduced LLM-as-Judge; follow-up work tackled biases (Wang et al., 2024, 2025), multi-agent debate (Chan et al., 2024), and pairwise discussion (Li et al., 2024). The Agent-as-a-Judge paradigm (Zhuge et al., 2025) (ICML 2025) and IntellAgent (Levi and Kadar, 2025) extended this to interactive evaluation; Verga et al. (2024) showed a panel of smaller LLMs outperforms a single GPT-4 judge; Tseng et al. (2025) found discussion on the same item yields marginal gains—our agents independently generate different interactions. Surveys (Yu, 2025; You et al., 2026) identify panel composition and scaling as open problems we address directly. Persona-based agents. Park et al. (2023) demonstrated realistic persona agents; Serapio-García et al. (2023) and Jiang et al. (2024) showed LLMs can embody Big Five traits with behavioral consistency. Zou et al. (2025) found self-report personality correlates weakly with interaction behavior—we sidestep this by measuring evaluation behavior directly. Most recently, Lu et al. (2025) deployed persona-aligned LLM agents to simulate usability testing of web designs. Our focus is on the statistical properties of agent panels—how scores and discoveries scale with panel size. Reliability theory and ensembles. The Spearman-Brown formula (Spearman, 1910) predicts reliability growth assuming tau-equivalent raters. Shrout and Fleiss (1979) formalized ICC for various designs. In machine learning, Krogh and Vedelsby (1994) showed ensemble accuracy depends on member diversity, not just individual accuracy—a decomposition later unified by Wood et al. (2023). We draw an explicit analogy: agent-judge panel reliability arises from diverse perspectives rather than consistent agreement, just as ensemble prediction accuracy arises from diverse errors rather than individual accuracy. 3 Method 3.1 Persona-Based Agent Judge Design Conversational AI systems behave differently depending on who is interacting with them: an expert asks probing follow-ups that expose depth limitations, while a novice accepts surface-level answers that mask the same limitations. A single generic judge—or a panel of identical judges—cannot capture this context dependence. What is needed are contextual judges: evaluators whose backgrounds, expertise, and communication styles shape the interaction itself, so that the evaluation reflects the range of real user experiences. Persona-based design addresses this directly. Rather than generating diversity through random sampling or temperature variation, we ground each agent judge in a structured persona that determines how it interacts, what it pays attention to, and how strictly it evaluates. We use the Big Five personality framework (Costa and McCrae, 1992)—the most widely replicated dimensional model of personality, and one that recent work shows LLMs can reliably embody (Serapio-García et al., 2023; Jiang et al., 2024). We chose the Big Five over alternatives (e.g., MBTI) because it is continuous (enabling fine-grained differentiation), orthogonal (the five dimensions are statistically independent), and empirically grounded in decades of cross-cultural validation. Big Five conditioning is what makes this diversity systematic: different personality profiles lead agents to ask different questions, tolerate different failure modes, and weight different quality dimensions. We deploy 32 agent judges with diverse profiles: 8 expert, 17 intermediate, and 7 novice—reflecting the natural skew of real user populations toward intermediate skill levels—spanning 9 geographic regions and ages 22–68 (full pool in Appendix A). Panel coordination. A coordinator agent analyzes each task’s domain and complexity, then ranks the 32 agents by task fitness—a composite of domain expertise match, personality suitability (e.g., high-Conscientiousness agents rank higher for complex multi-step tasks), and expertise level. Panels of size N select the top-N from this ranking. This task-aware selection ensures that even small panels contain contextually relevant judges, and smaller panels are strict subsets of larger ones—so observed differences between sizes reflect marginal contributions, not resampling variance. Dual-role interaction. Each agent operates in two interleaved roles per session: (1) conversationalist—conducting multi-turn interaction conditioned on persona, task context, conversation history, and internal emotional state; and (2) evaluator—recording per-turn diary entries invisible to the target. Each diary entry captures: a response quality score, a satisfaction rationale explaining the score from the persona’s perspective, specific issues or strengths (categorized as functionality, accuracy, helpfulness, clarity, or safety), and updated emotional state. To illustrate, consider two agents receiving the same target response—a generic troubleshooting guide for an account access error: Expert CTO diary (turn 3, score: 0.4): “The response lists generic steps (clear cache, restart browser) but ignores the 403 status code I mentioned. No RBAC-level diagnosis offered.” Issue: accuracy—ignores user-provided technical details. Novice Student diary (turn 3, score: 0.8): “Clear instructions, easy to follow. Would be nice if it explained why this happens.” Issue: helpfulness—lacks educational context. The same response produces a 0.4-point score gap and surfaces different issues—a direct consequence of persona diversity. The diary mechanism serves two purposes beyond recording scores. First, it forces the agent to articulate why it scored as it did, producing the qualitative insights that constitute our discovery data. Second, the accumulated diary entries form a persona-consistent self-narrative that is fed back into the agent’s context at each turn, reinforcing the assigned personality and reducing drift over longer conversations. Emotional state dynamics. Real users do not evaluate each response in isolation: frustration compounds, trust erodes, and patience runs out over a conversation. Each agent maintains a persistent emotional state across turns comprising five dimensions: trust, frustration, engagement, patience, and fatigue. Each dimension evolves after every turn based on the quality of the target’s response, modulated by the agent’s Big Five profile. The emotional state is re-engaged at every turn through the personality-modulated update, acting as a recurring anchor to the assigned persona—a high-Neuroticism agent cannot drift toward calm behavior because its frustration state keeps being pushed upward. Each agent also maintains a structured session memory of all prior messages, target responses, diary entries, and emotional trajectory, enabling it to build on earlier observations and probe identified issues from different angles. 3.2 Experimental Design The task set consists of 15 evaluation scenarios in a 5×35× 3 factorial design: five domains crossed with three complexity levels. The five domains—SaaS/IT, Developer, E-Commerce, Education, Healthcare—exercise distinct failure modes (factual precision, code accuracy, personalization, pedagogy, safety reasoning), each at three complexity levels (Simple/Medium/Complex: max 4/10/20 turns). To test generalizability, we run the full experiment with two cross-family model pairs (judge ≠ target provider) to prevent self-evaluation bias (Zheng et al., 2023): a proprietary pair (GPT-5.4 target, Sonnet 4.6 judge) and an open-source pair (DeepSeek V3.2 target, GLM-5 judge, served via Friendli proxy). All other settings—personas, tasks, panel construction, meta-evaluators—are held constant. Each pair produces 32×15=48032× 15=480 sessions (960 total), at panel sizes N∈1,2,3,4,5,8,12,16,24,32N∈\1,2,3,4,5,8,12,16,24,32\. Cost: $145 (proprietary, 20.6M tokens) + $84 (open-source, 16.4M tokens). Three meta-evaluators from different providers (Gemini 3.1 Pro, Grok 4.1, Claude Opus 4.6) independently score panel reports; all core findings use raw session-level data. A reproducibility test (one task evaluated twice by all 32 agents, 64 sessions) enables stability analysis across runs. For human ground truth, 50 participants (Prolific, 10 per domain) each completed 2 tasks with the same target (GPT-5.4), rating quality on the same 0–1 scale; after cleaning, 43 participants and 86 sessions were retained (Section 4.1). 3.3 Statistical Methods Scoring reliability. ICC(2,k) under the two-way random effects model (Shrout and Fleiss, 1979) is computed from the 32×1532× 15 quality score matrix. We chose this variant because both judges and tasks are sampled from larger populations, making random effects appropriate. We compare logarithmic, linear, power law (ICC=a⋅kbICC=a· k^b), and hyperbolic (ICC=1−a/kICC=1-a/k) fits using AIC, and interpret values following Koo and Li (2016): below 0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, >>0.90 excellent. Variance decomposition. One-way ANOVA with ω2ω^2 for multi-level factors (domain, complexity, expertise); Cohen’s d for pairwise comparisons. A two-way random-effects ANOVA decomposes total variance into between-task (σ^τ2 σ^2_τ), between-judge (σ^π2 σ^2_π), and residual interaction (σ^ε2 σ^2_ ) components. This decomposition is the key to explaining the ensemble mechanism in Section 4.3. Semantic deduplication. Raw insight counts overstate true discovery because agents phrase equivalent observations differently. Following SemDeDup (Abbas et al., 2023), we embed all insights using text-embedding-3-small, apply agglomerative clustering (average linkage) at cosine similarity threshold θ=0.65θ=0.65, and report a confidence band from θ=0.60θ=0.60 to 0.700.70 (Appendix F). 4 Results We first establish that agent judges produce human-grade evaluations (Section 4.1), then present the scaling patterns this enables (Section 4.2), explain the mechanism behind them (Sections 4.3–4.4), and show that structured persona conditioning is required (Sections 4.5–4.6). 4.1 Can Agent Judges Match Human Evaluators? Table 1: Turing test for evaluation: pairwise score differences across three comparison groups. Agent–human differences fall within the range of natural human disagreement (paired t(14)=−0.91t(14)=-0.91, p=0.379p=0.379). Agents are more internally consistent than humans, yet produce indistinguishable quality assessments. Comparison Mean ||diff|| d Interpretation Agent–Agent 0.143 — Internally consistent Human–Agent 0.188 −0.18-0.18 Indistinguishable from H–H Human–Human 0.201 — Natural disagreement Before analyzing scaling behavior, we must establish whether agent judges produce evaluations comparable to human raters. We recruited 50 human raters (Prolific, 10/domain); after cleaning, 43 participants and 86 sessions across all 15 tasks were retained. Each chatted with the same target (GPT-5.4) and rated quality on the same 0–1 scale used by agent judges. We frame this as a Turing test on evaluation behavior. For each of the 15 tasks, we compute the mean absolute score difference within two groups: human–human (H–H) and human–agent (H–A). This yields 15 paired observations—one per task—which we compare with a paired t-test, preserving independence across tasks. The result: the mean H–A difference (0.188) is not significantly different from the mean H–H difference (0.201; paired t(14)=−0.91t(14)=-0.91, p=0.379p=0.379). After Bonferroni correction for per-task comparisons, 13 of 15 tasks show no significant difference; the two exceptions differ in opposite directions, suggesting task-specific variation rather than systematic bias. Overall d=−0.18d=-0.18 (p=0.12p=0.12). Agent judges are more consistent with each other than human raters are (Table 1), yet their scores remain indistinguishable from human scores. Behavioral patterns also converge: humans averaged 4.7 turns per session versus 5.1 for agents, and average message length was 98 characters for humans versus 214–293 for agents. Despite these surface-level differences, the two groups arrive at statistically indistinguishable quality assessments—different conversation paths leading to the same evaluative conclusion. When shown agent diaries for the same task (Figure 3 in Appendix), 41% of participants reported the AI “found issues I missed,” while only 19% reported the reverse. This complementarity—agents and humans surfacing different issues from the same system—motivates the question we turn to next: how do the number of agent judges affect what a panel can measure and discover? 4.2 The Score–Coverage Dissociation Table 2: ICC(2,k) with 95% CIs for both model pairs. Both follow a logarithmic curve (R2>0.97R^2>0.97). Proprietary Open-source k ICC 95% CI Interp. ICC 95% CI Interp. 1 .290 [.00, .70] Poor .243 [.00, .67] Poor 4 .621 [.16, .86] Mod. .562 [.07, .83] Fair 8 .766 [.42, .92] Good .719 [.33, .90] Good 16 .868 [.64, .96] Exc. .837 [.57, .94] Exc. 32 .929 [.80, .98] Exc. .911 [.75, .97] Exc. Figure 2: The score–coverage dissociation. Left: Scoring reliability (ICC) improves logarithmically. Right: Issue discovery follows a sublinear power law (b=0.69b=0.69, R2=0.999R^2=0.999; shaded band: θ=0.60θ=0.60–0.700.70). Gray diamonds show raw observation counts before semantic deduplication; the 75% gap at N=32N\!=\!32 illustrates the importance of deduplication for measuring true discovery. Both dimensions exhibit diminishing returns, but scores saturate ∼2× \!2× faster. With agent-level quality established, we now examine how panel size affects two distinct evaluation outputs: scoring reliability and issue discovery. Scoring reliability (Table 2, Figure 2, left) follows a logarithmic curve in both model pairs (R2=0.974R^2=0.974 and 0.9850.985), substantially better than linear (R2<0.73R^2<0.73) or power law (R2≈0.91R^2≈ 0.91) fits. The open-source pair tracks 0.03–0.06 ICC points lower with a similar slope, likely reflecting less between-task variance. Issue discovery, by contrast, follows a sublinear power law (Figure 2, right). After embedding-based semantic deduplication (θ=0.65θ=0.65), unique findings scale as y=7.8⋅N0.69y=7.8· N^0.69 (R2=0.999R^2=0.999), with a confidence band of b∈[0.62,0.76]b∈[0.62,0.76] across θ=0.60θ=0.60–0.700.70. This exponent is robustly sublinear: across all seven thresholds tested (θ=0.50θ=0.50–0.800.80), the power-law exponent ranges from b=0.47b=0.47 to b=0.90b=0.90, never reaching linearity. The open-source pair confirms the same pattern—sublinear growth with a similar exponent—despite using a different judge backbone (GLM-5 vs. Sonnet 4.6) and target (DeepSeek V3.2 vs. GPT-5.4), suggesting this is a property of agent-based evaluation rather than a specific model pairing. A diverse 4-judge panel discovers 3.3×3.3× more unique issues than a single agent, while producing a nearly identical mean quality score. This is the score–coverage dissociation: scoring reliability reaches “good” at N=8N\!=\!8 (82% of maximum), while discovery reaches only 42% of its N=32N\!=\!32 value—scores saturate roughly twice as fast. We hypothesize this reflects a power law distribution of the finding space: critical issues (“head”) are discovered by small panels, while corner cases (“tail”) require larger panels—analogous to species accumulation curves in ecology (Colwell & Coddington, 1994). Severity-weighted analysis confirms ∼53% \!53\% of findings are high-impact at all panel sizes—larger panels do not merely accumulate low-priority observations. 4.3 The Ensemble Diversity Mechanism What mechanism produces reliable panel scores from unreliable individuals? Individual agent judges are nearly unreproducible (r=0.003r=0.003), yet the 32-judge panel mean differs by only Δ=0.03 =0.03 between independent runs. The variance decomposition explains why: between-agent variance is near zero (0.3%/1.0%)—structured persona conditioning controls individual tendencies—while residual interaction dominates (70–75%). The remaining 24–29% is between-task variance: judges agree on which tasks are harder but disagree on absolute quality. Source Proprietary Open-source Residual (judge × task) 70.6% 74.7% Between-task 29.0% 24.3% Between-judge 0.3% 1.0% This matches the ensemble diversity mechanism (Krogh and Vedelsby, 1994; Wood et al., 2023): each agent probes from a different angle; when averaged, probe-specific noise cancels while the shared quality signal accumulates. ANOVA confirms that what is evaluated matters far more than who evaluates (Table 3): task domain and complexity together explain 8–12% of score variance across both model pairs, while judge expertise explains under 1%. Who evaluates always matters—but structured persona conditioning controls this variable enough that the quality signal dominates. Table 3: What drives score variation? Task properties dominate; judge identity contributes <<1%. Variance explained (ω2ω^2) averaged across both model pairs. p∗<.001^*p<.001. Factor Variance (ω2ω^2) Sig. Interpretation Task complexity 4–8% ∗ Harder tasks produce lower, more variable scores Task domain 4% ∗ Different domains test different failure modes Judge expertise <<1% n.s. Controlled by structured persona conditioning 4.4 Expertise as Adversarial Probing The ensemble mechanism explains scoring reliability, but what drives discovery breadth? Expert agents score lower than novices (proprietary d=0.24d=0.24, open-source d=0.17d=0.17; experts lower in 10–11/15 tasks). To test whether this is a conversation difference rather than calibration, an independent LLM (GLM-5) scored all 960 transcripts blind—no persona, expertise, or original scores. The gradient persists: expert-led conversations receive lower holistic scores (M=0.909M=0.909 vs. novice M=0.934M=0.934, d=0.19d=0.19). Expert Intermediate Novice Real-time (turn-by-turn) 0.807 0.832 0.837 Post-hoc (independent LLM) 0.909 0.931 0.934 The gap between real-time and post-hoc grows with conversation length (+0.07+0.07 simple, +0.14+0.14 complex), consistent with accumulated emotional experience invisible to a holistic reader. The expertise difference is in breadth: high-impact proportion is identical across levels (52–54%), but experts surface 1.9×1.9× more issue categories (114 vs. 60). This breadth advantage is what pushes discovery into the tail of the finding distribution: experts probe quality dimensions that novices never reach. 4.5 Ablation: Structured vs. Simple Persona The preceding results depend on a structured persona engine. Does a simpler approach suffice? We ran 96 sessions across 4 conditions on 3 tasks (simple/medium/complex), 8 agents each: Structured Simple No Persona Repeated Score SD 0.087 0.160 0.164 0.151 Insights/session 13.2 9.0 8.6 12.8 Expertise d −0.35-0.35 −0.17-0.17 −1.03-1.03 — Structured agents produce half the score variance of simple prompting (0.087 vs. 0.160 SD) and 47% more insights per session. Without persona, the expertise effect explodes (d=−1.03d=-1.03); structured conditioning moderates it to a controlled d=−0.35d=-0.35. Repeating the same agent 8 times shows unreliability (SD 0.151) that diverse panels resolve through ensemble diversity. In short, the dissociation requires structured persona conditioning: it controls evaluator variance (enabling logarithmic ICC convergence) while maximizing insight production (enabling sublinear discovery scaling). 4.6 Persona Mechanism Validation If the Big Five personality conditioning produces genuinely different evaluation behavior—not just superficially different phrasing—then established relationships between personality traits and emotional responses should emerge in the agent data. We tested three such hypotheses: Hypothesis r p Verdict Trust gain ∼ Agreeableness +0.754+0.754 <0.001<0.001 Confirmed Peak frustration ∼ Neuroticism +0.756+0.756 <0.001<0.001 Confirmed Engagement ∼ Extraversion +0.057+0.057 0.7570.757 Not confirmed High-Agreeableness agents develop trust more readily, and high-Neuroticism agents experience sharper frustration peaks—both consistent with decades of personality research (Costa and McCrae, 1992). The Engagement–Extraversion link was not confirmed: engagement in a goal-directed task appears to be driven by task progress (whether the target’s responses are helpful) rather than by the social stimulation that extraversion measures. Mean engagement varies only 4.4% across agents, compared to 9.6% for trust and 32.8% for frustration, supporting this interpretation. Emotional signals also carry predictive value: peak frustration correlates negatively with goal achievement (ρ=−0.257ρ=-0.257, p<0.001p<0.001). 5 Discussion The human validation establishes that agent judges, when built on structured persona conditioning, produce evaluations indistinguishable from human raters (p=0.38p=0.38). This is not a trivial result—it depends on the persona engine controlling evaluator variance while preserving diverse evaluation perspectives, as the ablation confirms (Section 4.5). Equally important, agent and human evaluators are complementary: 41% of participants reported the AI found issues they missed, while only 19% reported the reverse. Agent panels are most valuable not as replacements for human testing, but as a first pass that surfaces candidate issues for human review. With human-grade quality established, the score–coverage dissociation offers a new perspective on a long-standing question: how many evaluators are enough? Nielsen and Landauer (1993) argued that five users find most usability problems; critics countered that this is far too few (Spool and Schroeder, 2001). Our data suggest both sides were answering different questions. Scoring reliability follows a logarithmic curve because each additional judge averages out noise in a bounded signal (q∈[0,1]q∈[0,1]), so the marginal gain diminishes. Issue discovery follows a sublinear power law (b≈0.69b≈ 0.69): each agent traverses a different interaction path through the target’s behavior space, but the most critical issues—the “head” of the finding distribution—are discovered first by small panels, while larger panels progressively uncover corner cases in the “tail.” This is consistent with species accumulation curves in ecology, where common species are found first and rare species require more sampling effort. The redundancy among agents is not wasted effort—when independently configured agents converge on the same issues, that convergence is itself evidence the issues are real properties of the target system. That both scaling laws—logarithmic scoring and sublinear discovery—hold across two model pairs with different judge backbones and targets strengthens the case that the dissociation is a structural property of panel-based evaluation, not an artifact of a particular model combination. The dissociation does not manifest uniformly across domains. Closed-domain tasks (e.g., Developer) produce near-zero score variance—all agents assign similar scores to consistently adequate responses—while open-domain tasks (Healthcare, Education) produce wider spreads. The practical implication: when scoring converges quickly, the primary evaluation value shifts from measurement to discovery. These findings translate into a natural deployment strategy. A small panel of four agents is sufficient for continuous monitoring—scoring reliability is moderate at this size, but enough to catch regressions against a known baseline. For more thorough evaluation, expanding to 8–12 agents reaches good reliability and substantially broadens issue coverage, making it suitable for periodic audits. For milestone evaluations such as product launches, the agent panel should be paired with a targeted human study, since agent judges and human raters surface different kinds of issues. Across all tiers, panel composition matters more than panel size: mixing expertise levels produces both tighter scores and broader discovery than uniform panels of the same size. 6 Limitations and Conclusion Limitations and future work. The human study covers all 15 tasks but with unequal per-task sample sizes; a balanced recruitment would strengthen task-level claims. The sublinear discovery exponent depends on the deduplication threshold, though b<1b<1 holds across all thresholds tested (Appendix F). Task-adaptive panel selection may inflate ICC relative to random composition. One of three personality–emotion links was not confirmed, and all experiments are in English. Finally, agent judges may inherit shared biases from the backbone LLM—such as sycophancy or positional bias—that are invisible in the variance decomposition because all agents share them; characterizing such shared biases is an active direction of our research. Conclusion. Agent-based evaluation panels exhibit a score–coverage dissociation: scores converge logarithmically while discoveries follow a sublinear power law—both show diminishing returns, but scores saturate faster. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while larger panels progressively uncover corner cases in the tail. An ablation confirms structured persona conditioning is necessary, and a Turing test confirms agent scores are indistinguishable from human scores. We hope this work encourages practitioners and researchers to think about evaluation panels not as a reliability problem alone, but as a two-dimensional tradeoff between measurement precision and discovery breadth—where the composition of the panel matters more than its size. Ethics Statement This study uses LLM-based agent judges to evaluate a conversational AI system. All LLM usage (agent judge backbone, meta-evaluation, semantic deduplication, post-hoc re-scoring) is disclosed. The human validation study (50 participants recruited via Prolific, 43 retained) involved informed consent; participants could withdraw at any time and were compensated at above-minimum-wage rates. Agent judge personas simulate diverse demographics (9 regions, ages 22–68) but do not represent real individuals and do not include personas with disabilities or elderly users over 70. Agent-based evaluation should supplement, not replace, human testing for products serving vulnerable populations. Reproducibility Statement All experiments use publicly available LLM APIs (OpenAI, Anthropic, Google, xAI, Zhipu AI via Friendli proxy). Model versions: GPT-5.4, Claude Sonnet 4.6, DeepSeek V3.2, GLM-5 (744B), Gemini 3.1 Pro, Grok 4.1. Temperature: 0.7 for targets, default for judges. The 32 agent personas are defined by Big Five profiles (Appendix A). ICC computed via two-way random effects model; semantic deduplication uses text-embedding-3-small embeddings with agglomerative clustering (θ=0.65θ=0.65, average linkage). Total cost: $229 for 960 sessions. References Abbas et al. (2023) M. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos. SemDeDup: Data-efficient learning at web-scale via semantic deduplication. In ACL, 2023. Colwell & Coddington (1994) R. K. Colwell and J. A. Coddington. Estimating terrestrial biodiversity through extrapolation. Phil. Trans. R. Soc. B, 345(1311):101–118, 1994. Wood et al. (2023) D. Wood, T. Mu, A. M. Webb, H. W. J. Reeve, M. Lujan, and G. Brown. A unified theory of diversity in ensemble learning. JMLR, 24(359):1–49, 2023. Chan et al. (2024) C.-M. Chan et al. ChatEval: Towards better LLM-based evaluators through multi-agent debate. In ICLR, 2024. Koo and Li (2016) T. K. Koo and M. Y. Li. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2):155–163, 2016. Costa and McCrae (1992) P. T. Costa and R. R. McCrae. Revised NEO Personality Inventory Professional Manual. Psychological Assessment Resources, 1992. Dawid and Skene (1979) A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. JRSS Series C, 28(1):20–28, 1979. Faulkner (2003) L. Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3):379–383, 2003. Jiang et al. (2024) H. Jiang et al. PersonaLLM: Investigating the ability of LLMs to express personality traits. In Findings of NAACL, 2024. Kim et al. (2024) S. Kim et al. Prometheus: Inducing fine-grained evaluation capability in language models. In ICLR, 2024. Krogh and Vedelsby (1994) A. Krogh and J. Vedelsby. Neural network ensembles, cross validation, and active learning. In NeurIPS, 1994. Levi and Kadar (2025) E. Levi and I. Kadar. IntellAgent: A multi-agent framework for evaluating conversational AI systems. arXiv:2501.11067, 2025. Li et al. (2024) R. Li et al. PRD: Peer rank and discussion improve LLM-based evaluations. TMLR, 2024. Liu et al. (2023) Y. Liu et al. G-Eval: NLG evaluation using GPT-4 with better human alignment. In EMNLP, 2023. Nielsen and Landauer (1993) J. Nielsen and T. K. Landauer. A mathematical model of the finding of usability problems. In INTERACT/CHI, p. 206–213, 1993. Park et al. (2023) J. S. Park et al. Generative agents: Interactive simulacra of human behavior. In UIST, 2023. Schmettow (2012) M. Schmettow. Sample size in usability studies. CACM, 55(4):64–70, 2012. Serapio-García et al. (2023) G. Serapio-García et al. Personality traits in large language models. arXiv:2307.00184, 2023. Sheng et al. (2008) V. S. Sheng et al. Get another label? Improving data quality and data mining using multiple, noisy labelers. In KDD, p. 614–622, 2008. Shrout and Fleiss (1979) P. E. Shrout and J. L. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420–428, 1979. Snow et al. (2008) R. Snow et al. Cheap and fast—but is it good? Evaluating non-expert annotations for NLP tasks. In EMNLP, p. 254–263, 2008. Spearman (1910) C. Spearman. Correlation calculated from faulty data. British Journal of Psychology, 3(3):271–295, 1910. Lu et al. (2025) Y. Lu et al. UXAgent: A system for simulating usability testing of web design with LLM agents. arXiv:2504.09407, 2025. Spool and Schroeder (2001) J. M. Spool and W. Schroeder. Testing web sites: Five users is nowhere near enough. In CHI Extended Abstracts, p. 285–286, 2001. Tseng et al. (2025) Y.-M. Tseng et al. Evaluating large language models as expert annotators. In COLM, 2025. Verga et al. (2024) P. Verga et al. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796, 2024. Wang et al. (2024) P. Wang et al. Large language models are not fair evaluators. In ACL, 2024. Wang et al. (2025) Q. Wang et al. Assessing judging bias in large reasoning models: An empirical study. In COLM, 2025. You et al. (2026) R. You et al. Agent-as-a-Judge. arXiv:2601.05111, 2026. Yu (2025) F. Yu. When AIs judge AIs: The rise of agent-as-a-judge evaluation for LLMs. arXiv:2508.02994, 2025. Zhuge et al. (2025) M. Zhuge et al. Agent-as-a-Judge: Evaluate agents with agents. In ICML, 2025. Zheng et al. (2023) L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. Zou et al. (2025) H. Zou et al. Can LLM “self-report”? Evaluating the validity of self-report scales in measuring personality design in LLM-based chatbots. In COLM, 2025. Appendix A Full Agent Judge Pool Table 4 lists all 32 agent judges used in the experiment, with Big Five profiles, expertise levels, and demographic backgrounds. Table 4: Complete pool of 32 agent judges. O/C/E/A/N = Big Five dimensions (0–1). Expertise: Expert, Intermediate, Novice. # Background Exp. O C E A N Region 1 Accountant, 52 I .40 .70 .35 .40 .49 US 2 IT Project Mgr, 45 E .38 .72 .32 .42 .51 Asia 3 Retired Nurse, 62 N .42 .68 .38 .58 .54 Latam 4 HR Specialist, 38 I .40 .70 .41 .60 .54 EU 5 UX Designer, 29 I .70 .68 .38 .58 .54 Asia 6 Journalist, 34 I .72 .65 .35 .56 .55 EU 7 Systems Architect, 41 E .72 .70 .32 .38 .49 EU 8 AI Researcher, 37 E .70 .68 .35 .40 .49 US 9 Mechanic, 27 I .42 .47 .38 .40 .48 US 10 Game Developer, 33 I .45 .45 .35 .42 .49 Asia 11 Art Student, 24 N .45 .45 .41 .60 .54 US 12 Graphic Designer, 28 I .47 .47 .38 .62 .56 EU 13 Creative Writer, 26 N .72 .42 .38 .62 .56 US 14 Musician, 31 I .70 .40 .35 .64 .58 EU 15 Physicist, 30 E .75 .42 .32 .38 .49 EU 16 CS Student, 23 I .72 .45 .35 .40 .49 Asia 17 Sales Manager, 35 I .42 .42 .71 .42 .38 US 18 Marketing Asst, 29 N .45 .45 .68 .44 .39 EU 19 College Freshman, 21 N .42 .40 .74 .62 .44 US 20 Event Planner, 25 N .45 .42 .71 .64 .46 Latam 21 Marketing Creative, 28 I .70 .40 .71 .60 .44 EU 22 Product Designer, 32 I .72 .42 .68 .58 .44 Asia 23 Startup Founder, 36 E .72 .40 .68 .40 .38 US 24 Data Analyst, 27 I .75 .38 .65 .42 .39 EU 25 Operations Dir, 48 I .40 .70 .65 .40 .39 US 26 QA Manager, 55 I .42 .72 .62 .38 .39 EU 27 Teacher, 43 I .42 .70 .68 .62 .46 US 28 Small Business, 50 N .45 .68 .65 .60 .46 Africa 29 Medical Educator, 39 E .68 .68 .68 .62 .46 Africa 30 NGO Director, 44 I .65 .65 .71 .60 .44 Latam 31 Fortune 500 CTO, 46 E .70 .72 .71 .38 .36 US 32 Tech VP, 38 E .68 .70 .68 .40 .38 Asia Appendix B Per-Task Human–Agent Comparison Table 5: Per-task comparison between human raters and agent judges. prawp_raw: uncorrected; pBonfp_Bonf: Bonferroni-corrected (×15× 15). After correction, only 2 of 15 tasks remain significant. Task NHN_H H PSA d prawp_raw pBonfp_Bonf dev-howto-medium 6 .67 .84 −0.80-0.80 .020 .300 dev-info-simple 3 .60 .81 −1.49-1.49 .068 1.00 dev-troubleshoot-complex 5 .83 .78 +0.29+0.29 .610 1.00 ecom-creative-complex 4 .73 .91 −1.16-1.16 .000 .005* ecom-decision-medium 4 .86 .80 +0.48+0.48 .394 1.00 ecom-howto-simple 6 .82 .68 +0.89+0.89 .039 .585 edu-info-simple 6 .87 .87 +0.00+0.00 .990 1.00 edu-learning-complex 8 .64 .65 −0.05-0.05 .901 1.00 edu-troubleshoot-medium 6 .77 .76 +0.05+0.05 .915 1.00 health-decision-complex 7 .82 .87 −0.35-0.35 .442 1.00 health-howto-simple 3 .87 .79 +1.01+1.01 .063 .945 health-info-medium 6 .80 .92 −1.00-1.00 .002 .030* saas-decision-complex 6 .79 .84 −0.32-0.32 .549 1.00 saas-info-simple 10 .73 .59 +0.80+0.80 .035 .525 saas-troubleshoot-medium 6 .77 .87 −0.90-0.90 .024 .360 Overall 86 .77 .80 −0.18-0.18 .119 — Appendix C Participant Demographics 50 participants were recruited via Prolific (10 per domain); 43 were retained after removing incomplete and duplicate submissions. Demographics of the 43 retained participants: Age 18–24: 4, 25–34: 12, 35–44: 14, 45–54: 9, 55+: 4 Self-assessed expertise Expert: 6, Intermediate: 14, Novice: 23 Domain familiarity M=2.9M=2.9 (SD=1.1SD=1.1), scale 1–5 Appendix D Human Ratings of Agent Diaries After completing each task, participants were shown 3 agent judge diaries (expert, intermediate, novice) for the same task and rated how well each matched their own experience. Figure 3: Left: distribution of “matches my experience” ratings (1–5). 45% of ratings are ≥4≥ 4; mean = 3.07. Right: 41% of participants reported the AI found issues they missed, while only 19% found issues the AI missed—agent judges provide complementary, not redundant, coverage. Appendix E Discovery Scaling: Head-Torso-Tail Analysis Figure 4: Left: Sublinear discovery scaling with head/torso/tail zones annotated. Marginal contribution per judge decreases from 4.7 (N=1–4) to 2.0 (N=16–32), consistent with diminishing returns. The confidence band spans θ=0.60θ=0.60–0.700.70. Right: Hypothesized power law distribution of the finding space. Critical findings (head) have high per-session discovery probability and are found by small panels; moderate findings (torso) require more diverse perspectives; corner cases (tail) are reached only by large panels. This distribution explains the sublinear exponent b≈0.69b≈ 0.69. Appendix F Deduplication Threshold Robustness Figure 5: Power law exponent b as a function of cosine similarity threshold θ. Across all seven thresholds (θ=0.50θ=0.50–0.800.80), the exponent remains below 1.0, confirming that the sublinear conclusion is not an artifact of a particular deduplication setting. The recommended range (θ=0.60θ=0.60–0.700.70, shaded) is chosen by cluster quality inspection: at θ=0.65θ=0.65, semantically equivalent insights from different agents are correctly grouped while distinct issues remain separated. Appendix G Expert vs. Novice Conversation Examples To illustrate how expertise shapes conversations (not just scores), we compare an expert and novice agent on the same Developer task (Intermittent Timeout Errors). Expert (score 0.94, 5 turns): Turn 1: “I’ve been dealing with these intermittent timeouts that are driving me absolutely crazy—they’re not consistent, which makes them WAY harder to diagnose. I need to actually get to the root cause…” [Provides: API gateway → inventory service, 6–7 hour clustering pattern, connection pool stats] Novice (score 0.91, 17 turns): Turn 1: “Hello, I have been experiencing intermittent timeouts on my account and I need help figuring out what is causing them and getting them fixed. It has been happening on and off for a while now…” [No technical details provided] The expert provides specific infrastructure details in turn 1, forcing the target to give precise diagnostic advice—or fail. The novice describes the same problem in general terms, which the target handles well with generic troubleshooting. Both score high, but the expert’s conversation tests the target’s depth while the novice’s tests its accessibility—different quality dimensions from the same task. Appendix H Human–Agent Score Distributions Figure 6: Score distributions by domain. Red histograms: agent judges (N=96 per domain). Blue lines: individual human raters. KS p-values shown per domain; most domains show no significant distributional difference. Appendix I Per-Domain Effect Sizes Figure 7: Effect sizes (Cohen’s d) between human raters (50 recruited, 43 retained, 86 sessions) and agent judges (N=480) by domain. Green: non-significant (p>0.05p>0.05). The overall effect is small (d=−0.18d=-0.18, p=0.12p=0.12).

Instant research discovery