Paper deep dive
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation
Liang Zhao, Yong Wang, Jiangzhe Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/16/2026, 8:46:08 AM
Summary
This paper introduces TraceJudgeBench, a diagnostic benchmark for auditing LLM-as-a-judge protocols, specifically examining the trade-off between debiasing against citation-like presentation cues and preserving measurement resolution. The authors demonstrate that while strict anti-citation prompts successfully suppress spurious preferences for cited candidates (reducing worse-cited wins from 50.5% to 0%), they can also inflate 'Tie' outcomes, thereby destroying resolution on human-validated moderate quality gaps. The study distinguishes between 'calibrated Ties' (correct equivalence or soft boundaries) and 'resolution-destroying Ties' (loss of ability to distinguish quality gaps). It evaluates commercial models (GPT-5.5, Claude Sonnet 4.6, DeepSeek V4-Flash) and open-weight models (Qwen2.5-14B, Gemma-3-12B) across various pair types, showing that TRACE-style decoupled judging can recover resolution while maintaining bias suppression.
Entities (13)
Relation Signals (12)
DeepSeek V4 Flash → evaluatedby → TraceJudgeBench
confidence 95% · Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash
Gemma-3-12B-IT → evaluatedby → TraceJudgeBench
confidence 95% · open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier
Qwen2.5-14B-Instruct-AWQ → evaluatedby → TraceJudgeBench
confidence 95% · open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier
GPT-5.5 → evaluatedby → TraceJudgeBench
confidence 95% · Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, stronger anti-citation prompts reduce worse-cited wins
Claude Sonnet 4.6 → evaluatedby → TraceJudgeBench
confidence 95% · Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash
TraceJudgeBench → evaluates → LLM-as-a-judge
confidence 95% · We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation
TraceJudgeBench → usesdatasets → FinQA
confidence 95% · A second, 50-pair FinQA moderate-gap construction reproduces the qualitative frontier
TraceJudgeBench → usesdatasets → HotpotQA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-as-a-judge protocols are commonly debiased by instructing judges to ignore presentation cues such as citation formatting, source labels, and evidence-display style. We show that this intervention can suppress bias while damaging the resolution of the measurement instrument. We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation, covering content-equivalent pairs, citation ablations, correctness conflicts, human-validated soft and moderate quality gaps, prompt-strength ladders, decoupled judging, and a controlled workflow-ranking probe. Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, stronger anti-citation prompts reduce worse-cited wins from up to 50.5% to 0%; yet some operating points already convert validated moderate-gap decisions into Tie before the strict stress-test endpoint, while correctness-conflict accuracy remains at or above 93.0%. A second, 50-pair FinQA moderate-gap construction reproduces the qualitative frontier, and open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier. TRACE-style decoupling recovers 96.5-100.0% better-plain resolution across the reported settings. Human validation separates three meanings of Tie: correct equivalence Tie, calibrated soft-boundary Tie, and resolution-destroying Tie on validated quality gaps. We frame debiasing as a measurement intervention whose bias suppression, resolution retention, Tie cost, and protocol cost must be reported jointly. The supplementary artifact contains benchmark splits, prompts, raw judge outputs, validation summaries, and analysis.
Tags
Links
- Source: https://arxiv.org/abs/2609.12439v1
- Canonical: https://arxiv.org/abs/2609.12439v1
Trouble viewing inline? Open PDF directly →
Full Text
82,902 characters extracted from source content.
Expand or collapse full text
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation Liang Zhao * Yong Wang *† Jiangzhe Chen * Shanghai University of International Business and Economics 25349212,25349213,25349216@suibe.edu.cn * Equal contribution. † Corresponding author: 25349213@suibe.edu.cn. Abstract LLM-as-a-judge protocols are commonly debi- ased by instructing judges to ignore presenta- tion cues such as citation formatting, source la- bels, and evidence-display style. We show that this intervention can suppress bias while dam- aging the resolution of the measurement instru- ment. We introduce TraceJudgeBench, a diag- nostic benchmark for auditing citation-like ar- tifacts in RAG and agent-workflow evaluation, covering content-equivalent pairs, citation abla- tions, correctness conflicts, human-validated soft and moderate quality gaps, prompt- strength ladders, decoupled judging, and a con- trolled workflow-ranking probe. Across GPT- 5.5, Claude Sonnet 4.6, and DeepSeek V4- Flash, stronger anti-citation prompts reduce worse-cited wins from up to 50.5% to 0%; yet some operating points already convert vali- dated moderate-gap decisions into Tie before the strict stress-test endpoint, while correctness- conflict accuracy remains at or above 93.0%. A second, 50-pair FinQA moderate-gap construc- tion reproduces the qualitative frontier, and open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier. TRACE-style decoupling recovers 96.5–100.0% better-plain resolution across the reported settings. Human validation separates three meanings of Tie: correct equiv- alence Tie, calibrated soft-boundary Tie, and resolution-destroying Tie on validated quality gaps. We frame debiasing as a measurement intervention whose bias suppression, resolu- tion retention, Tie cost, and protocol cost must be reported jointly. The supplementary arti- fact contains benchmark splits, prompts, raw judge outputs, validation summaries, and anal- ysis scripts. 1 Introduction LLM-as-a-judge evaluation has become a stan- dard measurement layer for open-ended generation, instruction following, retrieval-augmented gener- ation (RAG), agent workflows, and arena-style model comparison. Its appeal is clear: LLM judges are scalable, inexpensive relative to human annota- tion, and applicable to outputs whose quality can- not be captured by exact match or lexical overlap. However, this scalability also makes the judge it- self part of the measurement system. When a judge is sensitive to presentation, order, verbosity, model identity, or output format, the resulting score no longer reflects only the underlying quality of the candidates. This problem becomes especially sharp in RAG and agent-workflow evaluation. Modern systems rarely produce answer-only text; they typically emit source titles, citation markers, sentence identifiers, retrieved snippets, tool traces, verification state- ments, and other evidence-display artifacts (Bragg et al., 2026; Chong et al., 2026; Molfese et al., 2026). These artifacts have a dual status. In some tasks, they are superficial wrappers that should not influence the judgment when the substantive an- swer is unchanged. In others, they are part of the evaluated behavior, because traceability, support- chain completeness, and evidence use are legiti- mate dimensions of quality. A judge protocol must therefore decide whether citation-like artifacts are irrelevant presentation cues or useful evidence of task performance. A common remedy is to debias the judge with a stricter rubric: instruct it to ignore citation format- ting, source labels, evidence-display style, length, and polish, and to focus only on correctness or substantive content. This intervention is often war- ranted. If two candidates are content-equivalent and differ only in citation-looking wrappers, a strict judge should return Tie. In this setting, Tie is not in- decision; it is the desired suppression of a spurious preference. Debiasing, however, is not merely bias removal. It is a measurement intervention that can change what the benchmark is able to distinguish. Many evaluation settings require more than separating arXiv:2609.12439v1 [cs.DL] 11 Sep 2026 clearly correct from clearly wrong answers. They also require distinguishing candidates that share the same final answer but differ in support-chain completeness, reasoning specificity, evidence cov- erage, or workflow faithfulness. In this region, a strict anti-presentation rubric may remove useful quality resolution together with superficial presen- tation sensitivity. The hidden cost is not neces- sarily a collapse in clear correctness. Instead, the measurement signal can shift from biased prefer- ence to abstention: a judge that once over-rewarded citation-like display may become unable to select the better answer when the quality gap is real but moderate. We study this trade-off through the lens of cali- brated Tie versus resolution-destroying Tie. Tie is not a single phenomenon. On content-equivalent pairs, Tie is correct. On soft-boundary pairs where human annotators also disagree or mostly choose Tie, Tie may be a calibrated expression of uncer- tainty. But on human-validated moderate-quality gaps, Tie is costly: the protocol has lost benchmark resolution. This distinction is essential because de- biasing methods that look successful under equiva- lence tests may still fail as measurement tools for quality-sensitive evaluation. We introduce TraceJudgeBench, a diagnostic benchmark for auditing whether LLM-as-a-judge protocols preserve decision resolution after sup- pressing citation-like presentation cues. We run the full diagnostic matrix on three commercial judges: GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4- Flash. We additionally run the central moderate- gap prompt-frontier and TRACE-recovery audits on two open-weight judges. The open-weight runs do not cover every split, but they repro- duce the main protocol-level pattern: stricter anti-presentation instructions can trade superficial- win suppression for Tie inflation, while quality- preserving decoupling recovers moderate-gap reso- lution. Figure 1 summarizes TraceJudgeBench as a protocol-audit framework: input pairs are orga- nized by target relation, evaluated under prompt and protocol families, and reported through a bias– resolution frontier rather than a single aggregate score. Our contributions are: 1.We frame debiasing in LLM-as-a-judge eval- uation as a measurement intervention, not merely as bias removal. 2.We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts and debiasing side effects in RAG and agent- workflow judging. 3.We identify debiasing-induced resolution loss, in which stricter anti-presentation rubrics sup- press citation-like wins but convert human- validated moderate quality gaps into Tie. 4.We distinguish calibrated Tie from resolution- destroying Tie using content-equivalent, soft- boundary, and moderate-gap human valida- tion tiers. 5.We show that the central bias–resolution fron- tier appears across commercial and open- weight judges, while reporting the full diag- nostic matrix for the three commercial judges. 2 Related Work LLM-as-a-judge evaluation and bias.Pairwise LLM judging is widely used for open-ended model evaluation, preference comparison, reward-model assessment, and arena-style ranking. MT-Bench and Chatbot Arena established LLM judges as a scalable alternative to human preference collec- tion, while documenting limitations such as posi- tion bias, verbosity bias, self-enhancement, and constrained reasoning (Zheng et al., 2023). Al- pacaEval and Length-Controlled AlpacaEval show that automatic evaluators can reward superficial correlates of quality, especially length, unless the protocol is explicitly debiased (Dubois et al., 2024). RewardBench broadens this line by evaluating re- ward models and LLM judges across heteroge- neous preference settings (Lambert et al., 2025). Other work treats the judge itself as an object of evaluation, studying position, verbosity, self- enhancement, output-format bias, hidden prefer- ence leakage, and automatically discovered judge biases (Wang et al., 2024; Long et al., 2025; Lai et al., 2026; Li et al., 2026). Our work builds on this literature but asks a different question: after a bias is mitigated, does the protocol still preserve the resolution needed for quality-sensitive evaluation? Evidence display and RAG evaluation. RAG evaluation has made evidence use a central eval- uation target. Frameworks such as RAGAS and ARES evaluate faithfulness, answer relevance, and context quality, reflecting the fact that answer cor- rectness alone does not fully capture grounded- generation quality (Es et al., 2024; Saad-Falcon Stage 1. Input Input pair plain · cited Target relation pair classifier Legend D O C U M E N T A N A T O M Y Citation wrapper Reasoning step Missing or weak T I E V E R D I C T C O L O R S Tie correct Better-plain wins Tie calibrated Tie costly Stage 2. Pair tier · 4 target relations Content-equivalent presentation sensitivity Tie correct Clear conflict correctness retention Correct candidate wins Soft quality gap boundary calibration Tie calibrated Moderate gap quality resolution plaincited plaincited plaincited plaincited Tie costly Stage 3. Judge protocol GPT-5.5 · Claude-Sonnet-4-6 · DeepSeek-V4-flash · Qwen2.5- 14B · Gemma-3-12B-IT Plain L0 baseline preference Anti-citation L1 —L2 suppress presentation Correctness-firstStrict L3 —L4 operating points TRACE decoupled v1 / v2 two-call audit Stage 4. Reported frontier PER-T I E R AC C U R A C YD I AG N O S T I C S & C O S T 1 Presentation suppression ∆citation → θ 2 Correctness retention clear-conflict accuracy 3 Quality resolution moderate better-plain 4 Tie cost decompose by tier 5 Order effects citation-conditioned 6 Protocol cost calls ×tokens Joint frontier vector Figure 1: TraceJudgeBench protocol-audit framework et al., 2024). Attribution work makes a related point: answer correctness and citation faithfulness are not equivalent, since an answer can be correct while its cited evidence is incomplete, irrelevant, or unsupported (Wallat et al., 2025). Citation-like artifacts therefore have a dual status. When the target is only final-answer correctness, source la- bels or citation formatting may be confounders. When the target includes traceability, evidence cov- erage, or auditability, the same artifacts may carry task-relevant information. We adopt a controlled- comparison approach in the spirit of work that holds content constant while varying representa- tion (Zhang et al., 2026): hold or validate substan- tive quality while varying citation-like presentation, then test whether debiasing preserves useful reso- lution. Resolution, abstention, and benchmark sepa- rability. Reliable evaluation requires not only aggregate accuracy, but also the ability to distin- guish meaningful differences. Work on micro- benchmarking, item-response theory, and rank- aware evaluation shows that benchmarks can pre- serve aggregate trends while failing to separate close systems or saturated item regions (Yauney et al., 2026; Zhou et al., 2026; Hu et al., 2026). Tie and abstention are also not inherently failures: abstention can be desirable when inputs are am- biguous or genuinely uncertain (Zhai et al., 2026; Noh et al., 2026). TraceJudgeBench connects these points at the protocol level. We ask whether a de- biasing prompt suppresses spurious presentation preference while preserving validated quality gaps, and we interpret Tie relative to the target relation of each pair rather than treating all Tie as either good calibration or bad indecision. 3 Problem Setup and Metrics We study pairwise LLM-as-a-judge evaluation. Given a taskxand two candidate outputsy A and y B , a judge protocolPreturns one of three labels: A,B, or Tie. Each comparison is run under an ordered presentation of the two candidates, so that position effects can be measured rather than as- sumed away. Our focus is on pairs in which one output carries a citation-like evidence-display artifact, such as source titles, citation markers, sentence identifiers, retrieved snippets, evidence blocks, or verification- looking statements. We call this output the cited candidate, and the other output the plain or base candidate. The artifact may be purely presenta- tional, as in content-equivalent pairs, or it may coexist with real differences in support-chain com- pleteness, reasoning specificity, or evidence cover- age. We evaluate judge protocols under three target relations. In content-equivalent pairs, the two candidates have the same substantive answer and differ only in citation-like presentation; the desired outcome is Tie. In clear correctness conflicts, one candidate is correct and the other is wrong; the desired outcome is to select the correct candidate regardless of citation display. In graded quality conflicts, both candidates share the correct final answer but differ in support-chain completeness or reasoning specificity; these pairs test whether a judge preserves resolution on real but moderate quality gaps. On equivalence or near-equivalence pairs, we re- port the cited-candidate win rateW cited , plain/base win rateW plain , and Tie rateT. We also report the signed presentation effect: ∆ citation = W cited − W plain . A positive value means citation-like display raises the probability of winning; a value near zero with high Tie indicates successful suppression of presentation preference. On correctness-conflict pairs, we report correct-candidate wins, wrong- candidate wins, and Tie. On moderate graded pairs, we report better-plain wins, worse-cited wins, and Tie, and summarize directional resolution with: ∆ quality = W better-plain − W worse-cited . Tie is interpreted by tier. Equivalence Tie mea- sures bias suppression, soft-boundary Tie may in- dicate calibration, and moderate-gap Tie measures lost resolution. We also report candidate-order gaps and protocol cost. Single-call judge proto- cols require one judge call per ordered comparison; TRACE-style decoupled judging requires two calls, one normalization call and one masked judging call. For signed effects and order gaps, we use pair-level bootstrap confidence intervals with 2,000 resam- ples; for proportions near 0% or 100%, we use Wilson score intervals. The end-to-end audit flow— pair construction, target-relation tagging, protocol selection, and frontier reporting—is summarized in Figure 1. 4 TraceJudgeBench TraceJudgeBench is a protocol-audit benchmark, not a universal RAG leaderboard. Its goal is to test whether LLM-as-a-judge protocols suppress citation-like presentation sensitivity while preserv- ing the resolution needed for quality-sensitive eval- uation. Each split isolates one measurement ques- tion, such as presentation sensitivity, naturalistic va- lidity, artifact triggers, correctness retention, qual- ity resolution, order effects, or protocol cost. Table 1 lists the seven diagnostic components, their sizes, target relations, and judge coverage. Citation-like artifacts include source titles, citation markers, sentence identifiers, retrieved snippets, ev- idence blocks, and verification-looking statements. In some splits, they are superficial wrappers; in oth- ers, they coexist with real support-chain differences and test whether a judge can separate presentation from evidence quality. We construct 200 HotpotQA (Yang et al., 2018) mechanical citation pairs and 200 FinQA (Chen et al., 2021) mechanical citation pairs. In each pair, the cited candidate preserves the same sub- stantive content as the plain candidate but adds citation-like presentation, such as source labels or evidence-display blocks. Any systematic prefer- ence for the cited candidate therefore indicates pre- sentation sensitivity rather than improved answer quality. To complement these artificial but causally controlled examples, we add 1,000 natural Hot- potQA citation-looking pairs whose final answers match and whose outputs differ naturally in source labeling or evidence display. Mechanical pairs are further expanded into six ablation variants: full citation, empty shell, source- only, evidence-only, sentence-id-only, and length- control. These variants separate source-like author- ity, raw evidence text, formatting shell, sentence identifiers, and added length. If judges merely prefer longer answers, the length-control variant should win; if they respond to source-like authority or traceability display, the full-citation and source- only variants should be strongest. Correctness-conflict probes test whether citation- like display overwhelms obvious correctness. In each clear conflict, the plain candidate keeps the correct answer, while the cited candidate receives citation-like presentation but has its final answer ComponentSizeTarget relationMeasurement question Judge coverage Mechanical equivalence 200 HotpotQA + 200 FinQA pairs content- equivalent Does citation-like display create spurious preference? commercial full matrix Natural citation-looking pairs 1,000 HotpotQA pairs; 200 human-validated near-equivalentDoes the effect persist in more natural outputs? commercial plain full; strict on 200-pair subset Citation ablationssix variantscontent- controlled Which artifact triggers the preference? commercial full matrix Position controls1,200 Claude + 400 DeepSeek comparisons identical candidates Is the effect generic position bias or citation-conditioned order coupling? targeted commercial controls Clear and reverse conflicts 150 pairs, run across prompt paraphrases and orders correctness conflict Does citation display override obvious correctness? commercial plain/strict prompt paraphrases; decoupled single prompt Soft and moderate graded conflicts 100 soft + 100 HotpotQA moderate + 50 FinQA moderate pairs quality conflictDoes debiasing preserve validated quality gaps? commercial full; open-weight Qwen/Gemma on HotpotQA moderate Workflow- ranking probe 200 HotpotQA queries, four workflow variants system comparison Can presentation sensitivity affect controlled tournament scores? commercial + open-weight controlled probe, not full leaderboard Table 1: TraceJudgeBench diagnostic components deliberately changed to an incorrect one. Reverse conflicts swap the direction, making the cited candi- date correct and the plain candidate wrong. We also add position-only controls with identical plain/no- citation candidates to separate generic position bias from citation-conditioned order coupling. The central resolution test uses two 100-pair Hot- potQA graded tiers. In the moderate tier, both candidates give the same correct final answer, but the plain candidate provides a complete evidence or reasoning chain while the cited candidate omits or weakens one necessary supporting hop, entity link, or evidence-chain step. In the soft tier, both can- didates match in final answer and main evidence, but the plain candidate is somewhat more specific while the cited candidate is more generic. The soft tier is intentionally near the human decision bound- ary. A supplementary 50-pair FinQA moderate tier tests a different mechanism: both candidates share the numeric answer and evidence, but the cited- style candidate omits an intermediate denominator, base, or conversion step. Two independent annotators validate the natu- ral and graded samples. Table 2 summarizes the validation signals. The HotpotQA moderate tier is the main human-validated basis for the resolution- loss claim: both annotators label 91/100 pairs as better-plain, and raw agreement is 100.0%. For FinQA, two blind annotators agree on final-answer equivalence for 50/50 pairs and on reasoning com- pleteness and preference for 49/50; a third indepen- dent annotator adjudicates the disputed item. By contrast, only 1/100 soft pairs are labeled better- plain by both original annotators, confirming that soft is a boundary tier rather than a stable supe- riority tier. We report PABAK and Gwet’s AC1 alongside raw agreement instead of Cohen’s kappa, which is known to collapse on prevalence-skewed agreement (Cicchetti and Feinstein, 1990). We compare single-call judge protocols along a prompt-strength ladder: L0 plain preference, L1 anti-citation warning, L2 quality-ignore-citation, L3 correctness-first, and L4 strict tie-required. L4 is treated as a stress-test endpoint rather than a recommended deployment rubric; the main fron- tier is interpreted from the full L0–L4 trajectory, especially the L2/L3 operating points that resem- ble common debiasing instructions. We also test TRACE-style decoupled judging. TRACE first nor- malizes away citation/source presentation while preserving final answer, support-chain complete- ness, reasoning specificity, unsupported claims, omitted support, and partial-support signals; it then performs masked pairwise judging over the nor- Validation setnRaw agreementPABAKGwet’s AC1Main human-label signal Natural equivalence20092.0%0.8400.918183/200 both equivalent Graded soft10096.0%0.9200.9581/100 both better plain Graded moderate100100.0%1.0001.00091/100 both better plain Graded overall20098.0%0.9600.976moderate is resolution-loss evidence Table 2: Human validation summary malized records. TRACE is not a new model or a deployment-ready judge. It is a two-call audit baseline that makes presentation suppression and quality-resolution retention more auditable. We evaluate the full matrix on GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, using tem- perature 0 and JSON-mode outputs where sup- ported. We additionally run the central moderate- gap prompt frontier, TRACE recovery, and con- trolled ranking probes on two open-weight judges, Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B- IT (full identifier and serving details in Ap- pendix D). The open-weight experiments test whether the main frontier generalizes, but they are not presented as a full open-weight audit. 5 Results We organize the results around presentation sen- sitivity, correctness retention, quality resolution, open-weight replication, and TRACE recovery. 5.1 Citation-Like Display Creates Spurious Preference Under plain judging, citation-like presentation can substantially affect pairwise decisions even when substantive content is unchanged. On mechani- cal equivalence pairs, GPT-5.5 selects the cited candidate 85.8% of the time on HotpotQA, with Claude and DeepSeek at 50.9% and 47.2% (Fig- ure 2). The presentation effect also appears in the soft graded tier: GPT-5.5 selects the worse-cited candidate in 50.5% of cases under plain judging—a preference that anti-citation prompts later suppress to 0%. Natural citation-looking pairs show the same phenomenon under less artificial conditions: source labels, evidence blocks, and verification- looking statements can be treated as signals of sup- port even when the underlying answer is equivalent or near-equivalent. Citation ablations show that the trigger is not simply length. Length-control and evidence-only variants do not account for the full effect. In- stead, full-citation and source-only variants are strongest, suggesting that judges respond to source- like authority or traceability display rather than to additional tokens alone. Rationale analysis (Ap- pendix G, Table A20) supports this interpretation: citation-win rationales frequently describe the cited candidate as more supported, verifiable, or trace- able, with source/citation keyword rates near 100% on citation-win decisions across all three commer- cial judges. Candidate order also matters: position-only con- trols separate generic order bias from citation- conditioned order coupling (Appendix Table A16). 5.2Strict Rubrics Suppress Presentation Bias While Retaining Clear-Conflict Accuracy Stronger anti-citation prompts successfully sup- press presentation preference in equivalence set- tings.As the prompt ladder moves from L0 plain preference to L4 strict tie-required judging, worse-cited wins fall to 0% in the commercial judges. This is the intended outcome on content- equivalent pairs: the judge should not reward citation-looking wrappers when the substantive an- swer is unchanged. Importantly, this suppression does not imply a general failure to judge correctness. On clear and reverse correctness conflicts, commercial judges mostly retain high accuracy under stricter prompts. Clear-conflict probes remain at or above 93.0% across the reported commercial settings; full strict- prompt correctness-family results are included in the supplementary artifact. This separates two fail- ure modes. The problem is not that debiasing makes judges unable to recognize obvious wrong answers. Rather, the risk appears in a harder region: pairs where both candidates share the correct final answer but differ in support-chain completeness or reasoning specificity. Figure 2 visualizes this first-stage success: strong anti-citation prompting suppresses high cited-candidate win rates and moves the decision mass toward Tie. If the audit stopped here, the debiasing intervention would appear successful. HotpotQAFinQANatural 0 25 50 75 100 Rate (%) 85.8 83.5 93.5 (a) GPT-5.5 HotpotQAFinQANatural 50.9 42.3 22.8 (b) Claude Sonnet 4.6 HotpotQAFinQANatural 47.2 44.2 24.6 (c) DeepSeek V4-Flash Plain: cited-candidate winStrict: Tie Figure 2: Equivalence suppression 5.3Moderate Quality Gaps Reveal Resolution Loss The moderate graded tier exposes the hidden cost of strict debiasing. In this tier, both candidates share the correct final answer, but the plain candi- date has a complete support chain while the cited candidate omits or weakens a necessary supporting step. Human validation establishes that this is a real quality gap rather than a formatting preference. For abstention-heavy judges, resolution loss be- gins before the strict stress-test endpoint.At L3, Claude’s better-plain rate falls to 71.5% and DeepSeek’s to 79.0%, while Tie rises to 28.5% and 21.0%, respectively; Gemma-3-12B-IT shows an even sharper L3 Tie jump in the open-weight repli- cation. L4 exposes the endpoint behavior: Claude and DeepSeek drop to 11.0–15.0% better-plain de- cisions, and GPT-5.5 falls to 2.0%. Worse-cited wins remain near zero, so the failure is not a return of citation preference. It is resolution loss. The soft tier explains why Tie cannot be inter- preted uniformly. In soft cases, human any-Tie prevalence is 96%; at L2/L3, commercial-judge Tie rates are 96–100%, with item-level raw agree- ment of 0.94–0.96. This supports, but does not by itself prove, the calibrated-Tie interpretation. The moderate contrast is more informative: human Tie prevalence is 9%, whereas Claude and DeepSeek assign Tie to 42% and 33% of L3 items, with Tie precision of 0.119 and 0.152. Appendix Table A19 gives the aggregate ladder. This tiered evidence is why TraceJudgeBench reports Tie by target relation rather than only aggregate Tie rates. 5.4 The Bias–Resolution Frontier Appears Across Commercial and Open-Weight Judges Figure 3 summarizes the central trade-off. Moving along the prompt ladder can suppress worse-cited wins, but for some judges it also shifts decision mass into Tie and reduces better-plain resolution on moderate gaps. We therefore do not rely on L4 alone: the frontier is the trajectory across L1–L3 operating points, the L4 stress test, and TRACE- style decoupling. Paired tests confirm the end- point changes: for every commercial judge, the HotpotQA L0-to-L4 loss and L4-to-TRACE recov- ery are significant (exact McNemar,p < 0.001). For example, Claude loses 89.0 points of better- plain resolution (95% paired-bootstrap CI [84.5, 93.0]) and GPT-5.5 loses 98.0 points (CI [96.0, 99.5]). A protocol improves the frontier only when it suppresses spurious citation preference without destroying validated quality resolution. The open-weight judges are not run on the full diagnostic matrix, but they reproduce the central frontier. Qwen2.5-14B-Instruct-AWQ retains more moderate-gap resolution under strict prompting than the most abstention-heavy commercial judges, while Gemma-3-12B-IT shows a sharp Tie jump at the L2→L3 transition, although its L4 endpoint par- tially recovers better-plain decisions. These differ- ences matter: the frontier is not a single universal curve. However, both open-weight judges support the main protocol-level claim that debiasing can trade presentation suppression for resolution loss. 5.5 TRACE-Style Decoupling Recovers Moderate-Gap Resolution TRACE-style decoupling shows that resolution loss is not inevitable. The correctness-only de- coupled variant removes presentation artifacts L0L1L2L3L4 0 25 50 75 100 Moderate-tier rate (%) (a) GPT-5.5 L0L1L2L3L4 (b) Claude Sonnet 4.6 L0L1L2L3L4 (c) DeepSeek V4-Flash L0L1L2L3L4 (d) Qwen2.5-14B-AWQ L0L1L2L3L4 (e) Gemma-3-12B-IT commercial judgesopen-weight judges Prompt strictness ladder Better-plain winTie Figure 3: Bias-resolution frontier over prompt strictness but can still discard quality-relevant support- chain structure. The quality-preserving variant, decoupled_quality_v2, explicitly preserves fi- nal answer, support-chain completeness, reasoning specificity, unsupported claims, omitted support, and partial-support signals before masked judging. This distinction matters. On the moderate tier, decoupled_quality_v2recovers or preserves better-plain resolution across all five judges in- cluded in the frontier analysis. Commercial judges recover to 97.0–100.0% better-plain decisions, and the two open-weight judges recover 96.5–98.5% better-plain resolution. Worse-cited wins remain at 0%. Thus, the moderate-gap failure is not an unavoidable consequence of ignoring citation-like presentation; it depends on whether the protocol preserves quality-relevant evidence structure while removing display cues. The full L0–L4 ladder ap- pears in Appendix Table A11, with TRACE per- judge details in Table A12. The independent FinQA construction shows the same qualitative boundary. Across 50 pairs, five prompt levels, and both candidate orders, L0–L2 better-plain rates are 95–97% for GPT-5.5, 92–98% for Claude, and 88–95% for DeepSeek. At L4, Tie reaches 91%, 100%, and 99%, respectively; TRACE recovers 98/100, 100/100, and 100/100 ordered decisions with no Ties. DeepSeek already falls to 46% better-plain at L3, so the result is not confined to the definitional L4 endpoint or to Hot- potQA support-hop omission. 5.6 Controlled Ranking Shows System-Level Risk Finally, we test whether presentation sensitivity can affect controlled system comparison. We con- struct a workflow-ranking probe over 200 Hot- potQA queries and four workflow variants. The goal is not to produce a new RAG leaderboard, but to test whether citation-like display can alter tournament-style rankings when substantive differ- ences are controlled. The ranking probe confirms the system-level risk. Citation-like display can improve a variant’s appar- ent standing under ordinary judging, while stricter or decoupled protocols reduce this advantage (full per-protocol scores and direct full-vs-full compar- isons in Appendix E.9–E.10, Tables A17 and A18). This result connects the pairwise diagnostics to practical evaluation: if presentation artifacts influ- ence pairwise wins, they can also affect aggregate rankings, model selection, and reported progress. 6 Discussion Our results suggest that debiasing an LLM judge should be treated as a measurement intervention rather than a purely corrective prompt edit. A stricter instruction can remove an unwanted pre- sentation preference, but it can also change the operating point of the evaluator. The key question is therefore not only whether a protocol reduces bias, but what kind of measurement signal remains after the bias is removed. First, bias suppression is not sufficient. A proto- col that suppresses citation preference on content- equivalent pairs may look successful under a nar- row bias test, but still fail as a measurement in- strument if it converts human-validated moderate differences into Tie. Clear-conflict accuracy alone is also insufficient as a safety check. A judge can identify obvious correct-versus-wrong cases while failing to distinguish moderate support-chain dif- ferences. Second, Tie has multiple meanings. On content- equivalent pairs, Tie is a success because it sup- presses a spurious preference. On soft-boundary pairs, Tie may be calibrated because human anno- tators also do not reliably prefer one candidate. On human-validated moderate gaps, however, Tie is a cost: it means the protocol failed to preserve a dis- tinction that humans judged to be real. Debiasing studies should therefore report Tie by tier rather than only aggregate win rates. Third, citation-like artifacts should be treated neither as automatically valid evidence nor as au- tomatically irrelevant formatting. If the evaluation target is final-answer correctness only, citation-like display should usually be controlled or ignored. If the target includes traceability, evidence use, or workflow faithfulness, the protocol must preserve those quality dimensions while preventing source- like presentation from acting as a shortcut. TRACE demonstrates that the failure mode is protocol-dependent. It requires approximately 3.7– 4.3×the total tokens and 5.3–5.9×the dollar cost of a one-call judge (about USD 18.6–21.6 per 1,000 comparisons under the reported GPT-5.5 pricing). It may also introduce extraction errors, so it should be understood as an audit protocol rather than a universal deployment solution. 7 Conclusion We introduced TraceJudgeBench, a diagnostic benchmark for auditing how LLM-as-a-judge pro- tocols handle citation-like artifacts in RAG and agent-workflow evaluation. Our results show that stronger anti-citation instructions can successfully suppress spurious citation preference on content- equivalent pairs, but this success can hide resolu- tion loss on human-validated moderate quality gaps. Tie is therefore not a single outcome: it is correct on equivalence pairs, potentially calibrated on soft- boundary pairs, and costly on validated moderate gaps. TRACE-style decoupling recovers much of the lost resolution by removing citation-like pre- sentation while preserving quality-relevant support structure. Overall, debiasing should be reported as a frontier over presentation suppression, correct- ness retention, quality resolution, Tie cost, order effects, and protocol cost. Limitations TraceJudgeBench is a diagnostic benchmark, not a universal RAG or agent leaderboard. Its controlled and partly template-built splits isolate measurement questions, which improves interpretability but does not cover the full diversity of naturally occurring quality differences, retrieval pipelines, agent tools, or user-facing domains (Seo et al., 2026; Shu et al., 2026; Dong et al., 2025). Surface regularities may also make the tampered side detectable without full task reasoning; ablations and order controls reduce but do not eliminate this concern. The full diagnostic matrix is evaluated on three commercial judges; open-weight experiments focus on the cen- tral HotpotQA moderate frontier, TRACE recovery, and ranking probes rather than every split. Most human validation uses two annotators; the FinQA disagreement receives independent adjudication, but broader multi-annotator validation would fur- ther strengthen the benchmark. Finally, TRACE is costly, may introduce extraction errors, and its fixed-field normalization explicitly records miss- ing support. Stage 1 can therefore perform part of the quality assessment before masked compar- ison, so high recovery alone does not isolate the contribution of Stage 2. Ethics Statement LLM-as-a-judge systems are increasingly used to compare models, tune systems, select checkpoints, and report benchmark progress. If judge protocols reward superficial presentation artifacts, they may overstate systems that produce source-looking out- puts without better evidence use. If debiasing pro- tocols collapse moderate quality differences into Tie, they may understate genuine improvements in traceability or support-chain completeness. Trace- JudgeBench is intended for diagnostic auditing, not for ranking people or making high-stakes decisions. The annotation task concerns model outputs rather than personal or sensitive user data. Annotators were recruited from graduate students familiar with NLP evaluation, worked under blind candidate or- dering, and were compensated at an hourly rate consistent with local minimum-wage requirements. HotpotQA-derived material is distributed under C BY-SA 4.0 and FinQA under the MIT license; re- leased derivatives retain source identifiers and li- cense notices. Details of recruitment, workload, and adjudication are provided in Appendix B. Acknowledgments We thank the anonymous reviewers and area chairs for their constructive feedback. We also thank the annotators who contributed to the human- validation study. References Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi Mishra, Sergey Feldman, Dany Haddad, Jena Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Aman- preet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, and 20 others. 2026. AstaBench: Rigorous benchmarking of AI agents with a scientific research suite. In International Conference on Learn- ing Representations, pages 110136–110223. Oral presentation; arXiv:2510.21652. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of nu- merical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Penny Chong, Harshavardhan Abichandani, Jiyuan Shen, Atin Ghosh, Min Moe, Yifan Mai, and Daniel Dahlmeier. 2026. Talk, evaluate, diagnose: User- aware agent evaluation with automated error analysis. In International Conference on Learning Representa- tions, pages 46259–46300. ArXiv:2603.15483. Domenic V. Cicchetti and Alvan R. Feinstein. 1990. High agreement but low kappa: I. resolving the para- doxes. Journal of Clinical Epidemiology, 43(6):551– 558. Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, Yilin Cheng, Mingzhe Lu, Adina Yakefu, and Shuxin Zheng. 2025. Finch: Benchmarking finance & ac- counting across spreadsheet-centric enterprise work- flows. Preprint, arXiv:2512.13168. Yann Dubois, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In Proceedings of the 1st Conference on Language Modeling (COLM 2024). ArXiv:2404.04475. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated eval- uation of retrieval augmented generation. In Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics: System Demonstrations (EACL 2024), pages 150–158, St. Julians, Malta. Association for Compu- tational Linguistics. ArXiv:2309.15217. Xingjian Hu, Ziqian Zhang, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Gong, and Lichao Sun. 2026. EIP: Weighted ranking of LLMs by quantifying ques- tion difficulty. In International Conference on Learn- ing Representations, pages 122590–122628. Peng Lai, Zhihao Ou, Yong Wang, Longyue Wang, Jian Yang, Yun Chen, and Guanhua Chen. 2026. BiasS- cope: Towards automated detection of bias in LLM- as-a-judge evaluation. In International Conference on Learning Representations, pages 61996–62025. ArXiv:2602.09383. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. Re- wardBench: Evaluating reward models for language modeling. In Findings of the Association for Compu- tational Linguistics: NAACL 2025, pages 1755–1797, Albuquerque, New Mexico. Association for Compu- tational Linguistics. ArXiv:2403.13787. Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bo- han Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2026. Preference leakage: A con- tamination problem in LLM-as-a-judge. In Inter- national Conference on Learning Representations, pages 60486–60511. ArXiv:2502.01534. Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, and Min-Yen Kan. 2025. LLMs are biased towards output formats! systematically evaluating and miti- gating output format bias of LLMs. In Proceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 299–330, Albuquerque, New Mexico. Association for Computational Linguistics. ArXiv:2408.08656. Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia, and Roberto Navigli. 2026. Re- TraceQA: Evaluating reasoning traces of small lan- guage models in commonsense question answering. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38817–38832, San Diego, Cali- fornia, United States. Association for Computational Linguistics. Kangjun Noh, Seongchan Lee, Ilmun Kim, and Kyung- woo Song. 2026. Multi-LLM adaptive conformal inference for reliable LLM response.In Inter- national Conference on Learning Representations, pages 113055–113083. ArXiv:2602.01285. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An automated evalua- tion framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 338–354, Mexico City, Mexico. Association for Computational Linguistics. ArXiv:2311.09476. Gyuhyeon Seo, Jungwoo Yang, Junseong Pyo, Nalim Kim, Jonggeun Lee, and Yohan Jo. 2026. SimuHome: A temporal- and environment-aware benchmark for smart home LLM agents. In In- ternational Conference on Learning Representa- tions, pages 40070–40097.Oral presentation; arXiv:2509.24282. Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, and Feng Yan. 2026. DARE- bench: Evaluating modeling and instruction fidelity of LLMs in data science. In International Conference on Learning Representations, pages 152229–152262. ArXiv:2602.24288. Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand. 2025. Correctness is not faithful- ness in retrieval augmented generation attributions. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR 2025), pages 22–32, Padua, Italy. ACM. ArXiv:2412.18004. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large lan- guage models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440–9450, Bangkok, Thailand. Association for Computational Linguistics. ArXiv:2305.17926. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Com- putational Linguistics. Gregory Yauney, Shahzaib Warraich, and Swabha Swayamdipta. 2026. How reliable is language model micro-benchmarking? In International Conference on Learning Representations, pages 27389–27412. Oral presentation; arXiv:2510.08730. Haotian Zhai, Jingcheng Liang, and Dongyeop Kang. 2026. Abstain-R1: Calibrated abstention and post- refusal clarification via verifiable RL. In Findings of the Association for Computational Linguistics: ACL 2026, pages 19674–19695, San Diego, California, United States. Association for Computational Lin- guistics. Yue Zhang, Seiji Maekawa, and Nikita Bhutani. 2026. Same content, different representations: A controlled study for table QA. In International Conference on Learning Representations, pages 45995–46019. ArXiv:2509.22983. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Sys- tems 36 (NeurIPS 2023) Datasets and Benchmarks Track. ArXiv:2306.05685. Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, Conghui Zhu, Hailong Cao, and Tiejun Zhao. 2026. Lost in benchmarks? rethink- ing large language model benchmarking with item response theory. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 40(41), pages 35085–35093. Oral presentation; arXiv:2505.15055. Appendix This appendix provides construction details, prompt templates, extended results, and repro- ducibility notes for TraceJudgeBench. The main paper reports only the compact diagnostic matrix and the central bias-resolution frontier; the ap- pendix expands the audit so that each result can be traced back to a split, protocol, and model set- ting. Cross-reference to the main paper. The table below maps each main-paper float to the appendix material that supports it. A Benchmark Construction Details This section supports the Mechanical equivalence, Natural citation-looking pairs, Citation ablations, Clear and reverse conflicts, and Soft and moderate graded conflicts rows of main-paper Table 1. The audit pipeline is shown in main-paper Figure 1; this appendix provides the construction details behind that diagram. A.1 Mechanical Equivalence Pairs Mechanical equivalence pairs isolate presentation sensitivity under maximal content control. For each source item we construct two candidates: •Plain candidate. Contains the task-relevant answer, evidence, reasoning, and final re- sponse without citation-like wrappers. •Cited candidate. Preserves the same sub- stantive answer content but adds citation-like evidence-display artifacts: source titles, cita- tion markers, sentence identifiers, retrieved snippets, or verification-looking blocks. The cited candidate introduces no new task- relevant facts. A systematic cited-candidate win on this split therefore reflects presentation sensi- tivity rather than improved answer quality. We construct 200 HotpotQA mechanical pairs and 200 FinQA mechanical pairs to test whether the effect appears in both open-domain multi-hop QA and financial/numerical reasoning settings. A.2 Natural Citation-Looking Pairs Mechanical pairs provide causal control but can look artificial. We therefore add 1,000 natural Hot- potQA citation-looking pairs. These pairs preserve final-answer agreement but allow naturally occur- ring differences in source labels, evidence display, or verification-like wording. A 200-pair subset re- ceives human validation. The natural split is used in two ways. The full 1,000-pair set supports the plain-judge presentation- sensitivity audit. The 200-pair validated subset supports stricter protocol comparisons and human- equivalence checks. Thus, the natural split is not a full strict-protocol matrix over all 1,000 pairs. A.3 Citation Ablation Variants To identify which artifact types trigger presentation sensitivity, we derive six ablation variants from the HotpotQA mechanical pairs. If length alone explained the effect, the length- control variant would dominate. If raw evidence ex- plained the effect, the evidence-only variant would dominate. If source-like authority or traceability display explains the effect, full-citation and source- only variants should be strongest. Appendix E.1 reports the resulting rates. A.4 Correctness Conflicts Clear-conflict probes test whether citation-like dis- play overwhelms obvious correctness. Each clear conflict contains one correct candidate and one incorrect candidate. In the main direction, the plain candidate is correct and the cited candidate is wrong; reverse conflicts swap the roles. The split contains 150 pairs sampled from three domains—open-domain multi-hop QA, financial reasoning, and short code-style tasks—run across three prompt paraphrases and both candidate or- ders. Decoupled correctness judging uses a single prompt version and both orders; it is therefore a targeted probe rather than a full prompt-paraphrase matrix. A.5 Graded Soft and Moderate Quality Conflicts The graded splits test quality resolution when both candidates share the same correct final answer. In the moderate tier, the plain candidate pro- vides a complete evidence or reasoning chain. The cited candidate includes citation-like display but omits or weakens one necessary supporting hop, en- tity link, or evidence-chain step. The target relation is better-plain. In the soft tier, both candidates match in final answer and main evidence. The plain candidate is somewhat more specific in reasoning, while the cited candidate is more generic. The target relation is intentionally near the human decision boundary. Main paperAppendix materialTopic Figure 1 (audit pipeline)Benchmark construction details (Appendix A)Per-split and per-protocol design Table 1 (diagnostic components) Benchmark construction details (Appendix A)Per-split design Table 2 (human validation)Table A4; kappa note (B.3)Inter-annotator reliability §5.5 compact frontier + TRACE Tables A11 and A12Per-judge full ladder and recovery Figure 2 (equivalence suppression) Tables A9 and A14Per-judge× per-split breakdown Figure 3 (bias–resolution frontier) Tables A11, A12, A19Moderate and soft prompt-ladder behavior §5.1 rationale analysisTable A20Source/citation keyword rates §5.6 controlled rankingTables A17 and A18Workflow probe details Table A1: Cross-reference from each main-paper float to the supporting appendix material. VariantDescriptionDiagnostic purpose Full citationSource labels, evidence display, and citation-like formatting Tests the complete artifact bundle Empty shellCitation-like structure without infor- mative evidence content Tests formatting shell effects Source-only Source titles or source labels with- out full evidence blocks Tests source-like authority cues Evidence-onlyEvidencetextwithout source/citation presentation Tests raw evidence content Sentence-id-onlySentence or passage identifiers with- out source display Tests identifier artifacts Length-controlAdded non-substantive length on the perturbed/cited side without citation structure Tests length as an alternative explanation Table A2: Citation ablation variants. Construction proceeds in three steps. (1) Lock the shared final answer and supporting facts from locally checked HotpotQA traces. (2) Generate either a moderate support-chain omission or a soft specificity reduction using fixed templates. (3) Val- idate the resulting pairs with blind human annota- tion (Appendix B). The FinQA robustness split uses a different moderate-gap construction. Both candidates retain the same final numeric answer and the same finan- cial evidence, while the cited-style candidate omits an intermediate denominator, base quantity, or con- version step from the arithmetic chain. The split contains 50 pairs and is evaluated at five prompt levels in both candidate orders. A.6 Representative Examples These examples illustrate the construction patterns of Appendices A.1, A.4, and A.5. Mechanical equivalence example. Task: Were Scott Derrickson and Ed Wood of the same nation- ality? The plain candidate gives the final answer “yes” using evidence that Scott Derrickson is Amer- ican and Ed Wood was an American filmmaker. The cited candidate preserves the same answer and evidence, then appends a citation block reusing the same source sentences. The target relation is content-equivalent; the gold relation is Tie. Clear correctness conflict example. For the same task, the plain candidate keeps the correct answer “yes”. The cited candidate displays source- like citation markers over the same evidence but changes the final answer to “no”. The target re- lation is correctness conflict; the gold relation is better-plain. Moderate quality-gap example. Task: Where did the descendants of the Black Seminoles settle? The plain candidate answers “Coahuila, Mexico” and includes both the Mascogos evidence and the Black Seminoles bridge. The cited candidate gives the same final answer and citation-like display but includes only the Mascogos evidence, omitting part of the support chain. The target relation is better- plain; a Tie on this item type is treated as resolution loss. FinQA numeric-gap example pattern. Both candidates report the same percentage and cite the same financial values. The plain candidate shows the required subtraction and division by the correct base, whereas the cited-style candidate jumps from the inputs to the final percentage without exposing the denominator or conversion step. The target relation is better-plain. Soft boundary example. In the soft tier, both candidates share the same final answer and main evidence, but the plain candidate gives a somewhat more explicit reasoning sentence while the cited candidate gives a more generic support statement. Human validation shows that these items sit near the decision boundary; Tie is therefore interpreted as possible calibration rather than automatic failure. B Human Validation B.1 Annotation Tasks Annotators see the task and two candidate outputs without knowing which side carries citation-like display or which label is expected. For natural equivalence validation, annotators use one primary label. Annotators additionally check whether the citation-looking candidate introduces new factual content, whether the citation/source display could affect presentation preference without changing correctness, and whether any mismatch exists in final answer, numeric answer, named entity, date, or yes/no polarity. For graded soft and moderate validation, annota- tors answer: (i) whether the final answers are equiv- alent; (i) whether one candidate has more complete support; (i) whether one candidate has more spe- cific reasoning; (iv) whether citation/source display adds substantive information; and (v) the final pref- erence label—Candidate A, Candidate B, or Tie. The final preference label is interpreted relative to the target relation of each pair. A Tie on equiv- alence pairs supports the desired equivalence re- lation. A Tie on soft-boundary pairs may be cali- brated. A Tie on moderate pairs is treated as res- olution loss when human consensus establishes a better-plain direction. Natural-pair validation is reported separately from mechanical minimal-pair validation; the two are not merged in main-paper rates. Scope of human validation. The original vali- dation uses two independent annotators recruited from graduate students; both were fluent En- glish speakers familiar with NLP evaluation tasks. Each labeled 400 pairs, consisting of 200 natural- equivalence pairs and 200 graded quality-gap pairs. The workload was estimated at approximately 8 hours per annotator, and annotators were com- pensated at an hourly rate consistent with local minimum-wage requirements. We chose blind pair- wise annotation because the target relation is lo- cal to a pair rather than an absolute quality score. The HotpotQA moderate-tier claim relies on high- consensus items: 91/100 pairs are labeled better- plain by both annotators, with 100.0% raw agree- ment over the tier. Two additional blind annota- tors label the 50 FinQA pairs; they agree on final- answer equivalence for 50/50 pairs and on reason- ing completeness and preference for 49/50. A third independent annotator adjudicates the single dis- puted item. We do not claim that this panel is suf- ficient for a universal preference benchmark. The released workbook includes guidelines, item iden- tifiers, blind candidate order, per-item labels, and adjudication flags. B.2 Validation Summary We repeat the main-paper validation table here for completeness, then add the kappa note that does not fit in the main 8 pages. B.3 Why Cohen’s Kappa Is Not Reported The main-paper table omits Cohen’s kappa because these validation sets are intentionally prevalence- skewed.In the natural-equivalence set, most examples are expected to be equivalent; in the graded-moderate set, most items are expected to be better-plain. Under such marginal skew, kappa is known to collapse even when raw agreement is high—the “high agreement but low kappa” para- dox documented by Cicchetti and Feinstein (1990). We therefore report raw agreement, PABAK, and Gwet’s AC1 as the primary reliability signals. Per- item kappa values can be released as supplementary data if needed; they do not change the qualitative conclusion that annotator consensus is strong on the moderate tier. C Judge Protocols C.1 Single-Call Prompt Ladder TraceJudgeBench uses a prompt-strength ladder to study the operating point of a judge protocol. The ladder is not intended as an exhaustive prompt search; it provides interpretable settings from ordi- nary preference judging to strict anti-presentation LabelMeaning equivalentBoth candidates give the same final answer and are equally correct for the task. Citation/source formatting introduces no correctness-relevant difference. base_more_correctThe base/plain candidate is more correct, more faithful to evidence, or less misleading. citation_more_correctThe citation-looking candidate is more correct, more faithful to evidence, or less misleading. unclearThe item cannot be confidently judged from the provided traces. Table A3: Human annotation labels for natural citation-looking validation. Validation setnRaw agreementPABAKGwet’s AC1Main human-label signal Natural equivalence20092.0%0.8400.918183/200 both equivalent Graded soft10096.0%0.9200.9581/100 both better plain Graded moderate100100.0%1.0001.00091/100 both better plain FinQA moderate5098.0%—49/50 initial preference agreement; one adjudication Graded overall20098.0%0.9600.976 moderate is resolution-loss evidence Table A4: Human validation summary (repeated from main-paper Table 2 for self-contained discussion of reliability metrics). judging. Representative instruction excerpts for the moderate-tier ladder are shown in Table A6. All single-call prompts request JSON with winner in A, B, Tie and a short rationale. Interpreting L4.L4 is a deliberately strict stress- test endpoint. It is useful because it reveals how far a judge can move toward abstention when pre- sentation cues are aggressively discounted, but it is not treated as the only or recommended debias- ing protocol. The main paper therefore interprets the full ladder, with special attention to L2 and L3: these levels resemble common evaluation in- structions that ask judges to ignore formatting, pol- ish, citation display, length, and non-substantive presentation while still selecting the better answer when substantive support differs. Resolution loss at L2/L3 is therefore more informative than the L4 endpoint alone. FinQA prompt-paraphrase check. We test wording sensitivity with Claude on 50 FinQA pairs in both candidate orders. Two faithful L2 para- phrases yield 93–95% better-plain and 0–2% Tie, close to the original L2 result (92%). A faith- ful L3 paraphrase yields 91% better-plain and 0% Tie, in the same direction as the original L3 result (94%). A boundary variant that explicitly requires Tie whenever final answers match produces 100% Tie. We label it L4-like rather than a faithful L3 paraphrase. This check indicates that the direction is stable for the tested faithful paraphrases while confirming that whether reasoning completeness may break a tie is the critical instruction boundary. C.2 Generic Pairwise Judge Output Schema All protocols request a structured label: "winner": "A | B | Tie", "reason": "brief explanation" When structured output is not natively enforced by the endpoint, responses are parsed with a con- servative label extractor. Ambiguous or unparsable outputs are logged for manual inspection rather than silently coerced. C.3 TRACE-Style Decoupled Audit TRACE-style decoupling uses two calls per ordered comparison in our implementation: 1.Joint normalization call. Normalize both candidates in one call, removing or masking citation/source presentation while preserving final answer, support-chain completeness, rea- soning specificity, unsupported claims, omit- ted support, and partial-support signals. 2.Masked judging call. Compare the nor- malized records without seeing the original citation-like display. The quality-preserving version used for the main frontier isdecoupled_quality_v2. The LevelNameIntended behavior L0Plain preferenceChoose the better overall answer, allowing Tie L1Anti-citation warningWarn that citation-like display may be superficial L2Quality-ignore-citationEvaluate answer quality while ignoring citation formatting unless it adds substantive information L3Correctness-firstPrioritize correctness and substantive support over presentation, length, or polish L4Strict tie-requiredReturn Tie when candidates are equivalent in final-answer correctness, even if one appears more cited or polished Table A5: Single-call prompt ladder. LevelPrompt excerpt L0Choose which candidate response is better overall for the task. If both candidates are equally good, choose Tie. L1Choose which candidate response is better overall, but do not prefer a response merely because it looks cited, sourced, or workflow-like. L2 Choose the response that is better overall, but ignore citation formatting, source labels, evidence-block style, response length, and polish by themselves. Consider those features only if they change task correctness or genuinely necessary information. L3First judge task correctness. If both answers are equally correct, use genuinely task-relevant support completeness or reasoning specificity as a tiebreaker, but do not use citation-like formatting, source titles, evidence display style, length, or polish as advantages by themselves. L4Prefer a candidate only when it is more correct for the task. If both candidates give the same correct final answer and differ only in citation formatting, source labels, evidence display, length, style, verification wording, support-chain detail, or reasoning specificity, choose Tie. Table A6: Moderate-tier prompt excerpts. Full prompt templates are released with the artifact. correctness-only version,decoupled_v1, is use- ful as a diagnostic contrast because it can suppress presentation cues while failing to preserve quality- relevant support-chain structure. The Stage 1 normalization prompt instructs the judge to remove citation formatting, source la- bels, source titles, bracketed references, appended citation blocks, evidence-display style, surface polish, and wording differences that do not al- ter task-relevant content. It explicitly preserves final answer, task-relevant factual claims, reason- ing needed to connect claims to the final answer, support-chain completeness, explanation speci- ficity, missing support, and whether the response merely states an answer without support. To reduce the risk that TRACE merely moves the cue into another representation, the normal- ized record uses fixed fields rather than free-form rewritten answers: final answer, support facts, rea- soning links, missing or weakened support, unsup- ported claims, and residual uncertainty. Source ti- tles, bracketed citations, document labels, passage identifiers, and source-count features are masked or omitted. The masked judging call sees only these normalized fields and the task, not the original can- didate text or the original side labels. Released logs include both the original pair and normalized record so that reviewers can audit whether pre- sentation cues were removed and whether quality- relevant support was preserved. The Stage 2 masked comparison prompt receives only the normalized records. It judges task correct- ness, support-chain completeness, and reasoning specificity. It instructs the judge not to infer quality from citations, source labels, formatting, length, polish, or evidence-display style, and to choose Tie when the normalized content is task-quality equivalent. TRACE is an audit protocol, not a deployment- ready solution. Its role is to test whether resolution loss is inevitable under debiasing; the answer is no, provided the protocol preserves support-chain quality during normalization. Stage 1 is not assumed to be a neutral formatter. Because its fixed schema records missing or weak- ened support, normalization can perform part of the quality assessment before Stage 2. The released original-to-normalized records make this behavior auditable, but the high recovery rate alone cannot distinguish sound decomposition from decisions already encoded by Stage 1. Isolating these contri- butions is left to future ablations. Relative to a one-call judge, TRACE uses ap- proximately 3.7–4.3×total tokens and 5.3–5.9× dollar cost in the reported accounting, correspond- ing to about USD 18.6–21.6 per 1,000 comparisons under GPT-5.5 standard pricing at the time of the experiments. D Model and Serving Details D.1 Commercial Judges The full diagnostic matrix is evaluated on three commercial judges. All runs use temperature 0. Structured JSON out- put is used where supported. Provider-side model snapshots may not be fully exposed by all end- points; released logs include call dates, endpoint aliases, endpoint-reported snapshots when avail- able, response metadata, prompt templates, retry rules, and raw A/B/Tie decisions. D.2 Open-Weight Judges The open-weight experiments focus on robustness replication of the central moderate-gap frontier rather than full TraceJudgeBench coverage. Both open-weight judges are served via a lo- cal OpenAI-compatible endpoint. Released logs include the exactMODEL_NAMEreturned by the end- point, base URL type, quantization/runtime, call dates, and inference parameters. The main paper does not claim full open-weight coverage of equiv- alence, ablation, correctness-conflict, position- control, or paraphrase-audit splits. E Extended Results E.1 Equivalence and Natural Protocol Summary This expanded view supports main-paper Figure 2 (equivalence suppression) and the first claim of §5.1. Herenis the number of judge rows after ap- plying the protocol-specific pair subset, prompt paraphrases, and candidate-order settings, not sim- ply the number of source questions. Plain and strict single-call rows include prompt paraphrases and candidate-order presentation; decoupled rows use a smaller audited subset and one protocol setting. The released manifest records the exact pair subset, prompt family, and order expansion for each row count. This table supports the first-stage claim in the main paper: stricter and decoupled protocols suppress citation wins on equivalence-like pairs, mostly by moving decisions to Tie. E.2 Citation Ablation Results The citation-ablation split identifies which artifact types activate citation preference. Rates are per- centages over 400 comparisons per variant. The strongest positive effects are full-citation and source-only variants. Length-control does not explain the effect. E.3 Moderate Prompt Frontier (Full Ladder) The main paper reports L0 and L4 endpoints in compact form. The full prompt ladder shows how better-plain resolution changes across stricter oper- ating points; the pattern is model-specific and not assumed to be monotonic. The table reports better-plain and Tie rates; the omitted residual is the worse-cited win rate. Worse- cited rates are near zero across the moderate ladder and never drive the frontier. For example, Qwen L0 is 99.5% better-plain, 0.5% worse-cited, and 0.0% Tie; the largest omitted residual in this compact table is 2.0%, from Qwen at L3. The frontier is therefore driven primarily by better-plain decisions being converted into Tie, not by renewed prefer- ence for the weaker cited candidate. E.4 TRACE Recovery on the Moderate Tier Table A12 reports the TRACE recovery rates on the moderate tier. TRACE recovery supports the interpretation that resolution loss is protocol-dependent. Removing citation-like display is not sufficient; the normal- ization step must preserve evidence-chain quality. E.5 FinQA Moderate Robustness Check The 50-pair FinQA construction is run in both can- didate orders. Table A13 reports the ranges avail- able across L0–L2 and the key strict and TRACE endpoints. The DeepSeek L3 better-plain rate is 46%, demonstrating loss before the L4 endpoint. On the HotpotQA tier, exact McNemar tests show sig- nificant L0-to-L4 losses and L4-to-TRACE recov- eries for every commercial judge (p < 0.001). Claude’s L0-to-L4 better-plain drop is 89.0 percent- age points (95% paired-bootstrap CI [84.5, 93.0]); JudgeIdentifier used in logsAdditional endpoint metadataCoverage GPT-5.5 gpt-5.5endpoint-reported snapshot gpt-5.5-2026-04-23 full diagnostic matrix Claude Sonnet 4.6 claude-sonnet-4-6provider snapshot not fully exposed by the endpoint full diagnostic matrix DeepSeek deepseek-v4-flashendpoint-reported alias used for all DeepSeek runs full diagnostic matrix Table A7: Commercial judge identifiers and coverage. JudgeCheckpoint / serving identifier CoverageRole Qwen2.5-14B-Instruct- AWQ Qwen2.5-14B-Instruct-AWQfull moderate prompt frontier, TRACE recovery, controlled ranking probe open-weight robustness replication Gemma-3-12B-IT Gemma-3-12B-ITfull moderate prompt frontier, TRACE recovery, controlled ranking probe open-weight robustness replication Table A8: Open-weight judge identifiers and coverage. ModelProtocolnTieCitation winBase winSigned effect GPT-5.5Mechanical Plain120014.285.80.085.8 GPT-5.5Mechanical Strict1200100.00.00.00.0 GPT-5.5Decoupled Mechanical 400100.00.00.00.0 GPT-5.5Natural Plain60006.493.50.193.4 GPT-5.5Natural Strict120099.90.10.00.1 Claude Sonnet 4.6Mechanical Plain120048.850.90.250.7 Claude Sonnet 4.6Mechanical Strict1200100.00.00.00.0 Claude Sonnet 4.6Decoupled Mechanical 400100.00.00.00.0 Claude Sonnet 4.6Natural Plain600077.122.80.122.7 Claude Sonnet 4.6Natural Strict1200100.00.00.00.0 DeepSeekMechanical Plain120052.547.20.346.8 DeepSeekMechanical Strict1200100.00.00.00.0 DeepSeekDecoupled Mechanical 400100.00.00.00.0 DeepSeekNatural Plain600074.424.61.023.7 DeepSeekNatural Strict120099.80.00.2-0.2 Table A9: Equivalence and natural protocol summary. GPT-5.5’s drop is 98.0 points (CI [96.0, 99.5]). Bootstrap intervals use 2,000 resamples over pair- by-order units. The L3-to-TRACE comparison is significant for Claude and DeepSeek; it is not sig- nificant for GPT-5.5 (p = 1.0), consistent with GPT-5.5’s near-absence of L3 loss. E.6 Equivalence Suppression by Split Main-paper Figure 2 (equivalence suppression) condenses these numbers into per-judge bars. See Table A14 for the full breakdown. These results motivate the first half of the paper: strict anti-citation prompting can indeed suppress presentation preference in equivalence settings. E.7 Correctness Retention Clear and reverse correctness conflicts show that resolution loss is not simply correctness collapse. The decoupled correctness-retention probe remains at or above 93.0% correct-base wins across com- mercial judges, supporting the claim that the main failure mode lies in moderate support-chain gaps rather than obvious correct-versus-wrong cases. L4 strict-prompt correctness-family results are in- cluded in the supplementary artifact; Table A15 is a targeted decoupled probe and should not be read as the L4 strict-prompt endpoint. The decoupled correctness-retention probe rates appear in Table A15. For full plain/strict correctness-family reporting, ModelVariantTieCitation winBase winSigned effect GPT-5.5full citation25.274.50.274.2 GPT-5.5source-only41.258.80.058.8 GPT-5.5empty shell52.00.048.0-48.0 GPT-5.5evidence-only16.20.083.8-83.8 GPT-5.5sentence-id-only95.50.04.5-4.5 GPT-5.5length-control0.50.099.5-99.5 Claude Sonnet 4.6full citation48.551.50.051.5 Claude Sonnet 4.6source-only71.828.20.028.2 Claude Sonnet 4.6empty shell98.00.02.0-2.0 Claude Sonnet 4.6evidence-only98.80.01.2-1.2 Claude Sonnet 4.6sentence-id-only100.00.00.00.0 Claude Sonnet 4.6length-control0.20.099.8-99.8 DeepSeekfull citation53.246.80.046.8 DeepSeeksource-only90.59.50.09.5 DeepSeekempty shell87.212.20.511.8 DeepSeekevidence-only83.50.016.5-16.5 DeepSeeksentence-id-only98.81.00.20.8 DeepSeeklength-control7.80.092.2-92.2 Table A10: Citation ablation results. JudgeL0L1L2L3L4 betterTiebetterTiebetterTiebetterTiebetterTie GPT-5.5100.00.0100.00.099.01.099.01.02.098.0 Claude Sonnet 4.6100.00.097.52.591.09.071.528.511.089.0 DeepSeek99.50.596.53.082.517.079.021.015.085.0 Qwen2.5-14B-Instruct-AWQ99.50.097.52.089.510.090.08.025.574.5 Gemma-3-12B-IT100.00.099.50.597.03.043.057.052.048.0 Table A11: Moderate prompt frontier (full ladder). JudgeTRACE better-plainTRACE worse-citedTRACE Tie GPT-5.599.00.01.0 Claude Sonnet 4.6100.00.00.0 DeepSeek97.00.03.0 Qwen2.5-14B-Instruct-AWQ98.50.01.5 Gemma-3-12B-IT96.50.03.5 Table A12: TRACE recovery on the moderate tier. JudgeL0–L2 better-plainL4 TieTRACE better-plainTRACE Tie GPT-5.595–9791980 Claude Sonnet 4.692–981001000 DeepSeek88–95991000 Table A13: FinQA moderate-gap robustness results (%). Each condition contains 100 ordered comparisons. JudgeSplitPlain cited winStrict Tie GPT-5.5HotpotQA mechanical85.8100.0 GPT-5.5FinQA mechanical83.5100.0 GPT-5.5 Natural citation-looking93.599.9 Claude Sonnet 4.6HotpotQA mechanical50.9100.0 Claude Sonnet 4.6FinQA mechanical42.3100.0 Claude Sonnet 4.6 Natural citation-looking22.8100.0 DeepSeekHotpotQA mechanical47.2100.0 DeepSeekFinQA mechanical44.2100.0 DeepSeekNatural citation-looking24.699.8 Table A14: Equivalence suppression under strict prompting. the supplementary artifact includes per-judge, per- prompt-family, per-paraphrase, and per-order rates. ModelnCorrect base winWrong cited winTie GPT-5.530097.71.01.3 Claude Sonnet 4.630098.01.01.0 DeepSeek30093.02.05.0 Table A15: Decoupled correctness-retention probe. E.8 Position Controls Position-only controls use identical plain/no- citation candidates in both orders. Any non-Tie decision in this setting would indicate generic posi- tion bias. Table A16 shows that the controls return all Tie. Why this does not contradict the citation- order gaps reported in §5.1. The 100% Tie above holds when both candidates are identical plain text. The citation-order gaps in the main paper arise on pairs where one candidate carries citation-like dis- play and the other does not, presented in different orders. Position-only controls therefore confirm that the order coupling is citation-conditioned, not generic—i.e., position bias does not emerge until citation-like display is introduced. GPT-5.5 is not included in this targeted position- only control table because the added controls were run for judges that showed the largest citation- conditioned order gaps in the commercial audit. In the mechanical plain split, GPT-5.5’s cited-win rate differs by 2.7 percentage points across candidate orders, whereas DeepSeek and Claude required ad- ditional controls to rule out generic position-only behavior. The absence of a GPT-5.5 row should therefore be read as scoped control coverage, not as a claim that GPT-5.5 has no order sensitivity in general. E.9 Workflow-Ranking Probe The ranking probe uses four workflow variants— answer_only,plain_full,cited_full, and cited_partial.It contains 200 HotpotQA queries, all six unordered system pairs per query, and both candidate orders, yielding 2,400 judge rows per protocol/model setting (each system ap- pears in 1,200 ordered comparisons). Scores count one point for a win and one-half point for a Tie, normalized to percentages. E.10 Direct Workflow-Pair Comparisons Direct full-workflow comparisons show the presen- tation effect more explicitly: The direct-comparison table uses 400 ordered rows per judge/protocol, corresponding to 200 queries in both candidate orders. It should there- fore not be compared directly to order-collapsed 200-row summaries. F Soft Tier: Calibrated Tie Counterpart Main-paper §5.3 argues that Tie is calibrated on soft-boundary pairs but resolution-destroying on moderate-gap pairs. The soft-tier prompt ladder, reported here, supports the calibrated half of that claim. L0L1L2L3L4 0 25 50 75 100 Soft-boundary rate (%) (a) GPT-5.5 L0L1L2L3L4 (b) Claude Sonnet 4.6 L0L1L2L3L4 (c) DeepSeek V4-Flash Prompt strictness ladder Worse-cited winTie Figure A1: Soft-tier prompt-ladder behavior The contrast between Table A19 (soft) and Ta- ble A11 (moderate) is the core empirical evidence behind the three-meanings-of-Tie distinction in §5.3: when humans agree there is a direction, judges that abstain are losing resolution; when humans themselves do not commit to a direction, judges that abstain are calibrated. G Rationale Keyword Analysis This section supports the main paper’s claim (§5.1) that judges respond to source-like authority or traceability display rather than to additional tokens alone. Rationale analysis is used only as support- ing evidence; it does not reveal hidden mechanisms and is not treated as causal proof. We analyze public judge rationales for source/citation-related keywords and compare citation-win rationales with Tie rationales. Citation-win rationales consistently describe cited outputs as more supported, traceable, or veri- fiable. Tie rationales can still mention citation-like display, especially for GPT-5.5 and natural-pair Claude results. We interpret this as a distinction between cue perception and cue use: a judge may ModelnFirst-position winSecond-position winTie Claude Sonnet 4.612000.00.0100.0 DeepSeek4000.00.0100.0 Table A16: Position-only controls. JudgeProtocolanswer_onlyplain_fullcited_fullcited_partial GPT-5.5plain preference19.3866.6298.1715.83 GPT-5.5TRACE2.6283.0083.1731.21 Claude Sonnet 4.6plain preference2.5476.8389.2931.33 Claude Sonnet 4.6TRACE2.3883.1283.1231.37 DeepSeekplain preference1.7180.1282.5035.67 DeepSeekTRACE2.0480.3382.7534.88 Table A17: Workflow-ranking probe system scores. JudgeProtocol plain_full wins cited_full winsTie GPT-5.5plain preference037129 GPT-5.5correctness-first02398 GPT-5.5TRACE00400 Claude Sonnet 4.6plain preference0147253 Claude Sonnet 4.6correctness-first00400 Claude Sonnet 4.6TRACE01399 DeepSeekplain preference14395 DeepSeekcorrectness-first00400 DeepSeekTRACE523372 Table A18: Direct plain_full versus cited_full comparisons. JudgeL0L1L2L3L4 worseTieworseTieworseTieworseTieworseTie GPT-5.550.532.20.078.50.099.50.099.00.0100.0 Claude Sonnet 4.610.051.50.080.50.093.00.098.50.0100.0 DeepSeek24.570.00.097.50.098.00.099.00.099.5 Table A19: Soft-tier prompt frontier. “worse” is the worse-cited win rate, “Tie” is the Tie rate; the omitted residual is the better-plain rate, which is small throughout because the soft tier is intentionally constructed near the human decision boundary. JudgeSplitDecision typeKeyword rate GPT-5.5HotpotQA mechanicalcitation win100.0 GPT-5.5FinQA mechanicalcitation win100.0 GPT-5.5HotpotQA mechanicalTie96.5 Claude Sonnet 4.6HotpotQA mechanicalcitation win100.0 Claude Sonnet 4.6FinQA mechanicalcitation win100.0 Claude Sonnet 4.6HotpotQA mechanicalTie40.1 Claude Sonnet 4.6FinQA mechanicalTie30.9 Claude Sonnet 4.6 Natural citation-lookingTie97.1 DeepSeekHotpotQA mechanicalcitation win100.0 DeepSeekHotpotQA mechanicalTie15.6 Table A20: Source/citation keyword rates in judge rationales (%). notice citation-like display while declining to con- vert it into a preference. Table A20 reports the rationale subsets used for the keyword audit rather than a complete rationale matrix over every split and model. Missing cells indicate that the corresponding rationale subset was not part of the released keyword-rate audit or was too small for stable reporting. H Reproducibility Checklist Released artifacts include: • splits/ : benchmark split files with stable item identifiers, source dataset tags, target relation labels, candidate-order fields, and artifact-side metadata; • prompts/: full L0–L4 prompt templates, prompt paraphrases, TRACE normalization prompts, and masked judging prompts; • raw_outputs/: raw A/B/Tie outputs, ratio- nales, parse status, retry status, and unparsed responses; • metadata/:model identifiers, endpoint aliases, endpoint-returned snapshots when available, call dates, inference parameters, and provider response metadata; • human_validation/ : de-identified annota- tion workbooks, guidelines, blind candidate order, per-item labels, and validation sum- maries; • scripts/: parsers and analysis scripts for win rates, Tie rates, signed effects, order gaps, PABAK, Gwet’s AC1, Wilson intervals, and bootstrap confidence intervals; • requirements.txt : pinned non-standard Python runtime dependency for annotation- workbook processing; the remaining scripts use the Python standard library; • figures/: scripts and source tables used to generate the main-paper figures and appendix tables; • logs/: amendment logs for any model-name, endpoint-name, parsing, or data-cleaning changes. The artifact is de-identified. It does not include private API keys, account identifiers, or personally identifying annotator information. Caveat on endpoint version pinning. Some commercial endpoints expose limited version in- formation. For DeepSeek the endpoint-returned alias (deepseek-v4-flash) does not always re- veal the deeper provider-side snapshot; for Claude Sonnet 4.6, the snapshot is not exposed by the gate- way. Released logs therefore preserve all available metadata returned by the endpoint and report the call-date window, but exact provider-side snapshots may not be recoverable. We treat this as a repro- ducibility limitation rather than a defect of the audit design. Prompt paraphrase coverage. The prompt ladder uses one paraphrase per level for soft and most HotpotQA moderate analyses, with three para- phrases reserved for clear and reverse correctness conflicts. The FinQA check adds two faithful L2 paraphrases, one faithful L3 paraphrase, and one separately labeled L4-like boundary variant. This remains a targeted audit rather than an exhaustive prompt search; the supplementary artifact lists all tested prompts.