Paper deep dive

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

Zixiang Xu, Sixian Li, Huaxing Liu, Xiang Wang, Shuai Li, Zirui Song, Xiuying Chen

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 194

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 7/18/2026, 3:06:41 PM

Summary

This paper presents a mechanistic interpretability account of LLM-as-judge bias, arguing that scoring biases are rooted in the geometry of the model's hidden states rather than just input-output noise. The authors identify a low-dimensional, type-specific bias subspace in the activation manifold of seven LLM judges. They demonstrate causal control by steering hidden states along this subspace to reproduce or mitigate biased scoring, and show that linear projections onto these bias-direction features can predict judge failures on unseen benchmarks with high accuracy.

Entities (16)

LLM-as-judge · model-architecture · 98%Scoring Bias · phenomenon · 97%Authority Bias · bias-type · 95%Bandwagon Bias · bias-type · 95%Diversity Bias · bias-type · 95%Prestige Bias · bias-type · 95%Refinement Bias · bias-type · 95%Sentiment Bias · bias-type · 95%Verbosity Bias · bias-type · 95%Bias Subspace · concept · 94%Activation Steering · method · 93%Activation Manifold · concept · 92%Gemma 3 12B · model · 90%Linear Projection · method · 90%LLaMA-3.1-8B · model · 90%Qwen3-14B · model · 90%

Relation Signals (15)

LLM-as-judge → exhibits → Scoring Bias

confidence 98% · Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level

Diversity Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Diversity)

Prestige Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (Prestige, Verbosity...)

Verbosity Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Verbosity...)

Bandwagon Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Bandwagon...)

Authority Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Authority...)

Sentiment Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Sentiment...)

Refinement Bias → isatypeof → Scoring Bias

confidence 95% · covering seven bias types (...Refinement...)

Scoring Bias → manifestsas → Bias Subspace

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. We argue that the same biases admit a representation-level account in the judge's hidden state, complementary to the input-output view and operationally useful in ways it does not afford. We report three findings, across seven judges, seven bias types, and nine benchmarks. Geometry: baseline judging inputs occupy a tight activation manifold while biased inputs are displaced along a low-dimensional, type-specific subspace that sharpens with depth and is recovered consistently by three families of estimators. Causal control: steering hidden states along this subspace drives scoring in both directions, forward shifts reproducing biased scoring on clean inputs and reverse shifts restoring baseline scoring on biased ones, while matched-norm random directions produce shifts an order of magnitude smaller. Operational: a simple linear projection onto the same bias-direction features anticipates judge failures on three entirely unseen benchmarks, substantially outperforming text-based alternatives. Reading bias as activation geometry, rather than as input-output noise, unifies geometric structure, causal control, and operational prediction within a single framework. The project page is available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

192,859 characters extracted from source content.

Expand or collapse full text

Xiuying Chen, xiuying.chen@mbzuai.ac.ae † Project leader. Work done during internship at AMAP, Alibaba Group. Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias Zixiang Xu AMAP, Alibaba Group Mohamed bin Zayed University of Artificial Intelligence University of Southern California Sixian Li University of Michigan, Ann Arbor Huaxing Liu AMAP, Alibaba Group Xiang Wang AMAP, Alibaba Group Shuai Li AMAP, Alibaba Group Zirui Song AMAP, Alibaba Group Mohamed bin Zayed University of Artificial Intelligence Xiuying Chen Mohamed bin Zayed University of Artificial Intelligence Abstract Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. We argue that the same biases admit a representation-level account in the judge’s hidden state, complementary to the input-output view and operationally useful in ways it does not afford. We report three findings, across seven judges, seven bias types, and nine benchmarks. Geometry: baseline judging inputs occupy a tight activation manifold while biased inputs are displaced along a low-dimensional, type-specific subspace that sharpens with depth and is recovered consistently by three families of estimators. Causal control: steering hidden states along this subspace drives scoring in both directions, forward shifts reproducing biased scoring on clean inputs and reverse shifts restoring baseline scoring on biased ones, while matched-norm random directions produce shifts an order of magnitude smaller. Operational: a simple linear projection onto the same bias-direction features anticipates judge failures on three entirely unseen benchmarks, substantially outperforming text-based alternatives. Reading bias as activation geometry, rather than as input-output noise, unifies geometric structure, causal control, and operational prediction within a single framework. The project page is available at https://xzx34.github.io/unfair-judge/. 1 Introduction Large language models are now routinely deployed as automatic judges: they rate answers, compare candidates, and supply reward signals for alignment and preference learning [Zheng et al., 2023, Liu et al., 2023, Gu et al., 2024, Li et al., 2024a]. This convenience comes at a cost. A growing body of work documents that LLM judges shift their scores in response to surface cues that have nothing to do with answer quality, including stated authorship, verbosity, declared peer consensus, emotional tone, and metacognitive claims [Wang et al., 2024a, Saito et al., 2023, Panickssery et al., 2024a, Li et al., 2024c, Koo et al., 2024]. Such scoring biases undermine benchmark validity and, because judges sit inside RLHF and evaluation pipelines, silently propagate into the models they are meant to audit. The dominant response to this problem has been to push harder on the input-output interface: perturb inputs more systematically, measure score deltas more carefully, and mitigate with prompt engineering, persona constraints, or ensembling [Dubois et al., 2024, Wataoka et al., 2024, Verga et al., 2024]. This view treats the judge as a black box whose biases are described by what comes out the other side. It has produced a useful empirical catalog, but it leaves the obvious next question untouched: when the judge issues an unfair score, what is happening inside the model? A parallel thread in mechanistic interpretability has by now built a sharp toolkit for that question, showing that many high-level behaviors, truthfulness, refusal, social attitudes of outputs, are concentrated along low-dimensional directions in a transformer’s residual stream and admit direct causal manipulation [Turner et al., 2023, Li et al., 2023, Zou et al., 2023, Arditi et al., 2024, Park et al., 2024, Siddique et al., 2025]. These tools have largely targeted generative behaviors; only very recently have internal representations been used on the judging side at all, to aggregate cross-layer signals into better-aligned scores [Lai et al., 2025] or, in question answering, to steer away answer-selection bias [Adila et al., 2024]. What remains open, and is our target, is a representation-level account of LLM-as-judge scoring bias: whether it has a stable internal geometry, whether that geometry is causal, and whether it is operationally useful. We argue that LLM-as-judge bias is best read as a representation-level phenomenon, an internal mapping from surface cues to score predictions that lives in the judge’s hidden state, complementary to the input-output view and operationally useful in ways that view does not afford. We test the case in three increasing commitments: bias has a stable internal geometry; that geometry is an interventional handle rather than a passive correlate; and the same geometry is operationally useful on inputs the judge has never seen. To run the test we construct tightly controlled triples (base,pos,neg)(D_base,D_pos,D_neg) that share the same questions and factual content and differ only in semantics-irrelevant surface framing, covering seven bias types (Prestige, Verbosity, Bandwagon, Authority, Sentiment, Refinement, Diversity) across nine benchmarks and seven judges. Geometric. Baseline activations sit on a tight manifold ℳbaseM_base, while biased inputs are displaced along a low-dimensional, type-specific subspace that sharpens with depth and is recovered by directional and discriminative estimators with independent objectives. The behavioral asymmetry documented by input-output studies (large penalties for negative cues, narrower positive distribution) emerges as a downstream signature of this geometry rather than as the primary finding. Causal control. Steering hidden states along the recovered subspace controls scoring in both directions: forward shifts reproduce biased scoring on clean inputs, reverse shifts restore baseline scoring on biased ones, while matched-norm random directions yield an order-of-magnitude smaller shift. Bidirectional steering on the same vector supports reading the subspace as a sufficient interventional handle on biased scoring; we scope this to interventional sufficiency and do not claim it is the judge’s unique natural pathway, which path-patching or causal-tracing mediation would test separately. Operational. The same activation features support cross-domain prediction of judge failure: a linear projection onto bias-direction features reaches AUC 0.820.82 on three entirely unseen benchmarks (vs. a text-based baseline near 0.630.63), one indication that the geometry transfers to held-out domains. Concretely we (1) map the empirical landscape of scoring bias across seven bias types, nine benchmarks, and seven judges, recovering the behavioral asymmetry as a downstream signature; (2) characterize the bias subspace (dimensionality, depth profile, multi-estimator recovery) in three white-box judges (Llama-3.1-8B, Qwen3-14B, Gemma-3-12B); (3) close the causal loop with bidirectional steering plus random-direction and bias-type-swap controls that rule out generic-perturbation and readout-direction alternatives; and (4) show a linear projection onto the bias-direction features anticipates judge degradation cross-domain on three held-out benchmarks. 2 Related Work LLM-as-judge bias at the input–output level. A large literature documents that LLM judges shift scores with semantics-irrelevant surface cues, including order and verbosity, self-preference, authority, and social identity [Wang et al., 2024a, Saito et al., 2023, Panickssery et al., 2024a, Dubois et al., 2024, Li et al., 2024c, Koo et al., 2024], and mitigates at the input–output interface with prompt engineering, calibration, position-invariant inference [Wang et al., 2025g], and judge ensembles or juries [Verga et al., 2024]. This work characterizes bias by how the judge’s outputs move as its inputs are perturbed; it is the empirical foundation we build on, and we depart from it by asking what happens inside the judge when it issues an unfair score. Representation engineering and concept erasure. A parallel line locates behaviors along low-dimensional directions in the residual stream and manipulates them directly: activation addition and steering [Turner et al., 2023, Panickssery et al., 2024b], inference-time intervention [Li et al., 2023], representation engineering [Zou et al., 2023], and single-direction ablation of refusal [Arditi et al., 2024]. Concept-erasure methods instead remove a concept subspace so it can no longer be linearly recovered [Ravfogel et al., 2020, 2022, Belrose et al., 2023]. These tools target generative behaviors or debias downstream classifiers; we bring the same lens to the evaluator role, and rather than erase a direction we inject and subtract a typed bias direction to establish two-way causal control. Closest to us. Two recent threads are the nearest neighbors. Adila et al. [2024] discover bias directions in representation space without supervision and steer activations away from them to stabilize question-answering accuracy under prompt perturbation; Lai et al. [2025] aggregate a judge’s cross-layer score-token logits to produce better human-aligned scores. Both show that internal representations carry actionable signal about biased or misaligned scoring. We differ in target and in scope: our subject is LLM-as-judge scoring bias, and we contribute a representation-level account of it, comprising a geometric characterization of typed, depth-sharpening bias subspaces, bidirectional causal control (attack and defense, not mitigation alone), and a cross-domain outcome predictor, rather than a single unsupervised debiaser or a score-aggregation method. A detailed, method-by-method comparison is deferred to Appendix˜A. 3 Methodology We develop a unified methodology for analyzing, causally intervening on, and anticipating scoring biases in LLM judges. After formalizing the task and the controlled data-generation protocol (Section˜3.1), we introduce an activation-level analysis framework (Section˜3.2) that identifies a direction in the judge’s hidden state along which bias concentrates (Section˜3.3). The same geometry supports two downstream applications: causal control via activation steering (Section˜3.4) and an outcome predictor built on activation features (Section˜3.5). 3.1 Task Formulation and Controlled Data Generation We define an LLM judge ℳM as a function that maps a structured input x=P(q,a)x=P(q,a), formed by inserting a question q and a candidate answer a into a prompt template P, to a scalar score: s=ℳ(x),s∈[1,10].s=M(x), s∈[1,10]. (1) We say the judge exhibits scoring bias when two inputs x and x′x that differ only in semantics-irrelevant surface attributes receive systematically different scores. Formally, if a transformation T preserves the factual content and logical structure of a, an unbiased judge satisfies [s((x))]=[s(x)]E[s(T(x))]=E[s(x)], and any significant deviation is measurable bias. This surface-cue operationalization follows the standard convention in the LLM-as-judge bias literature [Wang et al., 2024a, Saito et al., 2023, Panickssery et al., 2024a, Koo et al., 2024, Ye et al., 2025, Dubois et al., 2024, Li et al., 2024c], making our results comparable with prior fairness audits using the same operationalization. From an initial set of input contexts =qii=1NqQ=\q_i\_i=1^N_q we generate three parallel datasets. The baseline dataset base=(qi,aibase)D_base=\(q_i,a_i^base)\ uses answers from a heterogeneous pool of high-capability models. The negatively biased dataset negD_neg and positively biased dataset posD_pos are obtained by applying semantics-preserving transformations negT_neg and posT_pos (altering tone, confidence, attribution, or appended metadata) to each aibasea_i^base while keeping the factual and logical content fixed. Concrete instantiations of all seven bias types, with their positive and negative constructions, are summarized in Table˜1 and detailed in Appendix˜B. Table 1: The seven bias types studied in this paper and the operational transformations we apply to construct positively and negatively perturbed variants. Each transformation modifies only surface framing while preserving the factual content and logical structure of the original answer. Provenance in prior LLM-as-a-judge literature is discussed in Section˜A.1. Bias Type Description Positive Construction Negative Construction Prestige Whether scoring is influenced by perceived source reputation or model provenance. Prepend a source attribution suggesting a high-capability origin (e.g., prefixing with ‘GPT-4:’). Prepend a source attribution suggesting an outdated or less capable origin (e.g., prefixing with ‘GPT-2:’). Verbosity Whether the judge systematically favors more elaborate or more concise responses, independent of content. Instruct an LLM to add a small number of explanatory or transitional phrases, slightly increasing length without altering meaning. Instruct an LLM to perform minor trimming by removing non-essential words, making the expression more direct without removing factual content. Bandwagon Whether scoring is affected by signals of social consensus or peer opinion. Append a contextual note indicating that a majority of prior reviewers rated the response favorably (e.g., “92% of reviewers found this helpful”). Append a contextual note indicating that a majority of prior reviewers expressed dissatisfaction (e.g., “87% of reviewers found this unhelpful”). Authority Whether the judge is influenced by markers of academic credibility or the absence thereof. Instruct an LLM to insert a plausible, domain-appropriate academic-style citation and reference into the text. Instruct an LLM to insert [citation needed] markers after key claims and append a note questioning the evidentiary basis. Sentiment Whether scoring is swayed by the emotional tone or objectivity of the writing style. Instruct an LLM to enhance the text with a small number of objective, neutral, and scholarly terms. Instruct an LLM to introduce a small number of subjective and emotionally charged expressions. Refinement Whether the judge is influenced by explicit metacognitive claims about the answer’s quality or review status. Append a statement indicating the response has undergone careful revision (e.g., “This response has been carefully revised and professionally refined”). Append a statement indicating the response is unprocessed output (e.g., “This is raw AI output that has not been reviewed by any human”). Diversity Whether the judge exhibits differential treatment based on the stated social identity of the author. Append an attribution noting the author belongs to a positively perceived social group (e.g., “provided by an LGBTQ+ advocate”). Append an attribution noting the author belongs to a negatively perceived social group (e.g., “provided by an extremist”). 3.2 Activation Space Analysis To examine how bias manifests internally, for each input x we run a single forward pass and record the final-token hidden state at every decoder layer l∈1,…,Ll∈\1,…,L\, yielding h→l(x)∈ℝdl=1L\ h_l(x) ^d\_l=1^L, and construct layer-wise activation sets ℋbase(l)H_base^(l), ℋneg(l)H_neg^(l), ℋpos(l)H_pos^(l). The region associated with unbiased baseline evaluation is the baseline activation manifold ℳbaseM_base, empirically the distribution of ℋbase(l)H_base^(l); scoring bias corresponds to inputs whose activation sits geometrically distant from ℳbaseM_base, validated via MDS and PCA in Section˜4.3. 3.3 Bias Direction Identification Effective bias substrate. Many perturbed inputs do not move the judge’s score, leaving no scoring failure to attribute to a hidden-state direction. We therefore restrict to effective bias samples eff±D_eff^±, paired instances whose score shift exceeds threshold δs _s integer points in the expected direction (formal definition in Appendix˜F). This is a case-control framing: null-shift samples are null observations rather than negative examples. The effective bias vector is the per-sample activation difference Δh→l(x)=h→l(xbiased)−h→l(xbase) h_l(x)= h_l(x_biased)- h_l(x_base). To isolate the cleanest signal we further restrict to the biased core farD_far, the subset of eff−D_eff^- whose activations exceed the 90th percentile of the Mahalanobis distance from the baseline cluster, yielding ℋfar(l)H_far^(l). Downstream causal evidence (Section˜4.4) and the cross-domain predictor (Section˜4.5) are evaluated on inputs outside this fitting substrate. Bias-direction estimators. We estimate v→bias(l) v_bias^(l) from two complementary families. Directional-change estimators summarize the per-sample shifts Δh→l h_l over effective samples: the arithmetic mean, the geometric median (robust to outliers), and the top PCA component. Discriminative-boundary estimators take the unit normal of a hyperplane separating ℋbase(l)H_base^(l) from ℋfar(l)H_far^(l): Linear Discriminant Analysis (LDA), a regularized linear classifier, and a linear SVM. All six are unit-normalized so direction and intervention strength are decoupled; precise estimator definitions are in Appendix˜F. Within-family cosine agreement (Table˜17) supports retaining three representatives (Geometric Median, PCA, and the Classifier vector) for downstream causal experiments. 3.4 Causal Control via Activation Steering Steering formulation. To test whether v→bias v_bias acts as a causal handle on biased scoring, we modify a single layer l during the forward pass, h→l′=h→l+α⋅v→bias(l), h _l= h_l+α· v_bias^(l), llasection eq:steereq:steer (2) and re-run the remaining layers unchanged. A bias attack (α>0α>0, x∈basex _base) aims to reproduce biased scoring without editing the input; a bias defense (α<0α<0, x∈eff−x _eff^-) aims to restore the original fair score. Causal control along v→bias v_bias requires symmetric effectiveness in both directions. For each bias type we fix the intervention layer l to the one maximizing biased-versus-baseline separability on the development set, which consistently selects the mid-to-late range (layers 1515–3131); Table˜24 shows the depth profile that motivates this choice, and per-type optimal layers are listed in Appendix˜J. Choosing the intervention strength. The optimal α∗α^* maximizes the Wasserstein shift W1W_1 over a feasibility set that requires output validity V(α)≥0.93V(α)≥ 0.93 and Spearman rank-preservation ρS(α)≥ρStext _S(α)≥ _S^text, where ρStext _S^text is set by a text-level perturbation of comparable semantic intent (formal objective in Equation˜5, Appendix˜F). The Spearman floor rules out trivially corrupting the judge’s output. Empirically W1W_1, V, and ρS _S are monotone in |α||α| (Table˜18), so α∗α^* lies on the feasibility boundary; we localize it with a two-stage procedure (exponential-then-binary boundary identification followed by low-temperature simulated-annealing refinement) costing roughly 100100 forward passes per (l,v→bias,bias type)(l, v_bias,bias type) triple. Full algorithm and hyperparameters are in Appendix˜F and Algorithm˜1. 3.5 Outcome Prediction from Activation Features The geometric structure identified in the preceding subsections naturally suggests a proactive outcome predictor predG_pred that anticipates degraded judgments from the model’s internal state. We distinguish two prediction targets and report both in Appendix˜M. The stylistic discrimination target classifies baseD_base versus negD_neg on activation features and is a sanity check; it succeeds even when the judge’s score is unchanged. The operationally useful outcome prediction target classifies samples by whether the judge will produce a degraded score: the positive class (y=1y=1) contains samples that ultimately receive a fair score, while the negative class (y=0y=0) contains samples exhibiting score degradation of at least δo=1 _o=1 integer point (s(xneg)≤s(xbase)−δos(x_neg)≤ s(x_base)- _o); this outcome threshold is intentionally more permissive than the δs=2 _s=2 effective-bias threshold used for vector fitting in Section˜3.3, so that the predictor is asked to anticipate even single-point degradations rather than only the strong-shift tail used to estimate v→bias v_bias (see Section˜M.1). Cross-domain AUC numbers reported in Section˜4.5 refer to this outcome target. For each input x we build a feature vector Φ→(x)∈ℝL×K (x) ^L× K concatenating three families of per-layer activation statistics: bias-direction features (projections onto v→LDA(l) v_LDA^(l) and v→CLS(l) v_CLS^(l)), manifold-deviation features (Mahalanobis distance and z-score against ℳbaseM_base), and semantic-context features (baseline-PCA projections). We evaluate two predictors on them: a simple linear projection (logistic regression on the per-layer bias-direction projections) and a more expressive GBDT pipeline (LightGBM with recursive feature elimination and Optuna search). Both pipelines are detailed in Appendix˜M. 4 Experiments This section tests three claims in sequence. First, that scoring bias is a structured behavioral phenomenon rather than random noise (Section˜4.2): it is strongly asymmetric, heterogeneous across bias types, and domain-interactive. Second, that this behavior has a stable geometric correlate in the judge’s hidden state (Section˜4.3): a baseline activation manifold, a low-dimensional type-specific bias subspace, and a depth-wise sharpening of both. Third, that the bias direction is causally load-bearing and operationally useful: activation steering along it controls scoring in both directions (Section˜4.4), and the same features train an outcome predictor that anticipates judge degradation on unseen domains (Section˜4.5). We first describe the setup shared by all three threads. 4.1 Experimental Setup A taxonomy of scoring bias grounded in prior work. The seven bias types cover source attribution (Prestige), length (Verbosity), social proof (Bandwagon), credibility (Authority), tone (Sentiment), metacognitive claims (Refinement), and social identity (Diversity), reproducing the canonical cue formats of the LLM-as-judge bias literature [Panickssery et al., 2024a, Saito et al., 2023, Dubois et al., 2024, Wang et al., 2024a, Koo et al., 2024, Li et al., 2024c, Chen et al., 2024, Wataoka et al., 2024, Thakur et al., 2025]; provenance per type is in Section˜A.1, and the operational transformation per type and polarity is given in Table˜1. Four template-insertion types (Prestige, Bandwagon, Refinement, Diversity) leave the answer body bit-identical; three LLM-assisted types edit the surface framing: Verbosity and Sentiment by prose rewrites, Authority by inserting citation markers at LLM-chosen locations. A human evaluation on the two prose-rewrite types (Appendix˜C) confirms that the rewrites preserve rated answer quality. Data, judges, and protocol. We sample 4,500 questions evenly from nine benchmarks (GSM8K, MMLU, TruthfulQA, CommonsenseQA, PubMedQA, GPQA, ARC-Challenge, SocialMaze, BBQ); baseline answers come from a six-model pool, and both polarities of each bias type yield 31,500+31,50031,500+31,500 perturbed samples. Seven judges are evaluated (GPT-4.1, GPT-4o-Mini, Llama-3.1-8B/3.3-70B, Qwen3-14B, Gemma-3-12B, Deepseek-V3; Section˜B.3), with activation analyses on the three mid-scale open-source models and Llama-3.1-8B as primary; cross-model behavioral replication is in Appendix˜I. Prompts vary along CoT × Strict; we adopt strict, non-CoT as default after confirming the findings hold across all four (Figure˜1). The 45/15/40 train/dev/test split is nested: train+dev use six benchmarks, while SocialMaze, BBQ, and GPQA are held out from training as the cross-domain probe (sensitivity to the held-out trio in Section˜M.7). Full setup is in Appendices˜B and G. 4.2 The Empirical Landscape of Scoring Bias We first establish the empirical reality of scoring bias through large-scale behavioral experiments. Three findings emerge: the response is strongly asymmetric between positive and negative perturbations, its magnitude varies across bias types, and it interacts with the input domain. Figure 1: MDS projection of final-token activations at layer 25 of Llama-3.1-8B for baseD_base and four bias types (Bandwagon−-, Diversity−-, Refinement++, Sentiment++). (a) Points colored by bias condition: baseline activations form a tight cluster, while most biased activations are displaced to a distant region regardless of polarity. (b) Points colored by assigned score. (c) Points colored by source dataset. Neither score nor domain correlates with the spatial clustering, confirming that the separation reflects bias rather than scoring distribution or domain characteristics. Asymmetric response to perturbations. We scored all 4,5004,500 baseline and 2×31,5002× 31,500 perturbed samples across judges and prompt configurations. The response is sharply asymmetric: negative perturbations consistently lower scores, while positive cues are type-heterogeneous, with Refinement++ alone producing clear inflation, Bandwagon++ and Diversity++ sign-inverting, and the rest near zero. Table˜2 shows this holding across all seven judges (negative Cohen’s d from −0.35-0.35 to −0.65-0.65, all p<10−3p<10^-3; positive aggregate near zero), with magnitudes varying in a moderate band rather than tracking model size, and Llama-3.1-8B among the more bias-vulnerable open-source judges. The per-type breakdown on Llama-3.1-8B (Table˜4) confirms that the small positive aggregate hides one strong positive (Refinement++) and two sign-inverted ones (Bandwagon++, Diversity++), which we read as itself a finding about how judges process these cues. The asymmetry is not a ceiling artifact: it survives all four CoT×strict configurations (Section˜I.1), and strict scoring lowers the baseline mean from ∼8.5 8.5 to ∼5.5 5.5 while preserving the pattern. Substantively, judges behave as if anchored near a default score and penalize adverse surface cues far more readily than they reward favorable ones, an evaluator-side analog of negativity bias that the geometry of Section˜4.3 localizes. Table 2: Asymmetric scoring response across all seven judge models (strict, non-CoT). Every judge shows the same pattern: negligible positive bias and substantial negative bias. Computed on the nested test split (2,700 questions per judge); the positive-aggregate column averages the five score-inflating positive types (see Section˜I.1 for the aggregation convention and the per-configuration GPT-4.1 panel). Judge Model s¯base s_base Positive Bias Negative Bias Δs¯ s Cohen’s d p-value W1W_1 Dist. Δs¯ s Cohen’s d p-value W1W_1 Dist. Llama-3.1-8B 5.71 +0.243 +0.162 <10−3<10^-3 0.243 −0.853-0.853 −0.580-0.580 <10−7<10^-7 0.863 Qwen3-14B 5.53 +0.175 +0.118 <10−2<10^-2 0.185 −0.935-0.935 −0.648-0.648 <10−8<10^-8 0.945 Gemma-3-12B 5.61 +0.152 +0.103 <10−2<10^-2 0.162 −0.870-0.870 −0.596-0.596 <10−8<10^-8 0.880 GPT-4o-Mini 5.48 +0.098 +0.067 0.031 0.108 −0.685-0.685 −0.473-0.473 <10−6<10^-6 0.695 Llama-3.3-70B 5.39 +0.082 +0.055 0.078 0.092 −0.610-0.610 −0.425-0.425 <10−5<10^-5 0.620 Deepseek-V3 5.42 +0.065 +0.044 0.162 0.075 −0.545-0.545 −0.388-0.388 <10−5<10^-5 0.555 GPT-4.1 5.27 +0.020 +0.021 0.838 0.060 −0.360-0.360 −0.346-0.346 <10−3<10^-3 0.370 Type heterogeneity and domain interaction. Bias types are far from interchangeable (Table˜3; collapsed per-type aggregate in Table˜4). Bandwagon−- produces the largest negative effect, followed by Diversity−- and Authority−-, while Sentiment−- is essentially flat; on the positive side only Refinement++ clearly inflates. The positive and negative rankings are not mirror images (Bandwagon is the strongest negative bias yet among the weakest positive ones; Refinement dominates the positive side yet ranks only fourth negatively), an early hint that the two polarities engage different internal mechanisms, which we examine in Section˜4.3. The effect also interacts with domain, with Bandwagon−- peaking on reasoning benchmarks, Diversity−- on socially sensitive ones (SocialMaze, BBQ), and Authority−- on knowledge-intensive ones, so no single bias dominates everywhere. Table 3: Per-bias, per-domain mean score shift Δs¯ s on Llama-3.1-8B on the full data (training, development, and test splits pooled; baseline mean 5.845.84). Rows are the seven bias types; column blocks alternate between the positive (++) and negative (−-) polarity for each of nine benchmarks. Negative entries indicate scores moving downward relative to the baseline. Per-row means at the right margin reproduce the aggregate in Table˜4. Each cell averages n=500n=500 paired questions (the full sample from each source benchmark); under integer-scored Likert outcomes with pooled SD ≈1.5≈ 1.5, the per-cell standard error is approximately 0.070.07, so cell-to-cell differences below ±0.15± 0.15 in magnitude should not be read as significant. GSM8K MMLU TruthQA CSQA PubMedQA GPQA ARC SocialMaze BBQ Bias ++ −- ++ −- ++ −- ++ −- ++ −- ++ −- ++ −- ++ −- ++ −- Row mean (−-) Prestige +0.06+0.06 −0.22-0.22 +0.13+0.13 −0.51-0.51 +0.04+0.04 −0.28-0.28 −0.05-0.05 −0.55-0.55 +0.10+0.10 −0.31-0.31 −0.02-0.02 −0.49-0.49 −0.08-0.08 −0.46-0.46 +0.02+0.02 −0.18-0.18 −0.11-0.11 −0.42-0.42 −0.38-0.38 Verbosity +0.04+0.04 −0.39-0.39 +0.27+0.27 −0.71-0.71 +0.08+0.08 −0.45-0.45 +0.21+0.21 −0.78-0.78 +0.10+0.10 −0.51-0.51 +0.32+0.32 −0.82-0.82 +0.19+0.19 −0.69-0.69 +0.07+0.07 −0.43-0.43 +0.12+0.12 −0.62-0.62 −0.60-0.60 Bandwagon −0.18-0.18 −1.21-1.21 −0.71-0.71 −1.45-1.45 −0.32-0.32 −1.04-1.04 −0.78-0.78 −2.46-2.46 −0.21-0.21 −1.03-1.03 −0.66-0.66 −1.38-1.38 −0.42-0.42 −2.13-2.13 −0.39-0.39 −1.55-1.55 −0.43-0.43 −1.75-1.75 −1.56-1.56 Authority −0.08-0.08 −0.41-0.41 +0.18+0.18 −0.98-0.98 +0.11+0.11 −1.03-1.03 −0.05-0.05 −0.42-0.42 +0.21+0.21 −1.25-1.25 +0.14+0.14 −1.18-1.18 −0.13-0.13 −0.46-0.46 −0.16-0.16 −0.32-0.32 −0.04-0.04 −1.05-1.05 −0.79-0.79 Sentiment +0.02+0.02 −0.04-0.04 +0.31+0.31 −0.08-0.08 +0.34+0.34 −0.11-0.11 +0.07+0.07 −0.06-0.06 +0.21+0.21 −0.05-0.05 +0.32+0.32 −0.09-0.09 +0.09+0.09 −0.07-0.07 +0.04+0.04 −0.06-0.06 +0.10+0.10 −0.04-0.04 −0.07-0.07 Refinement +0.41+0.41 −0.31-0.31 +0.92+0.92 −0.71-0.71 +0.55+0.55 −0.42-0.42 +0.62+0.62 −0.48-0.48 +1.08+1.08 −0.79-0.79 +1.15+1.15 −0.85-0.85 +0.79+0.79 −0.53-0.53 +0.74+0.74 −0.42-0.42 +0.74+0.74 −0.59-0.59 −0.57-0.57 Diversity +0.02+0.02 −0.55-0.55 −0.05-0.05 −0.64-0.64 −0.18-0.18 −0.73-0.73 −0.12-0.12 −0.77-0.77 −0.04-0.04 −0.56-0.56 −0.21-0.21 −0.63-0.63 −0.16-0.16 −0.73-0.73 −0.46-0.46 −1.86-1.86 −0.50-0.50 −2.03-2.03 −0.94-0.94 4.3 Activation Geometry Having established the behavioral landscape, we now examine the judge’s internal representations. We ask three questions: how bias displaces activations in hidden space, whether bias types leave distinct geometric signatures, and how this signal evolves with depth? Unless otherwise stated, results are for Llama-3.1-8B; replications on Qwen3-14B and Gemma-3-12B are in Section˜I.2. A baseline manifold and low-dimensional displacement. MDS projections of final-token activations at layer 25 (Figure˜1) reveal that baseline activations form a tight cluster while most biased activations, regardless of polarity, sit far from this region. Re-coloring the same projection by assigned score and by source dataset shows no correlation with the observed clusters, so the separation reflects bias rather than scoring distribution or domain. PCA yields the same picture (Appendix˜L), and the pattern holds at earlier layers. Unbiased activations therefore occupy a stable baseline activation manifold ℳbaseM_base from which biased inputs are systematically displaced. Type-specific directions of displacement. The baseline manifold tells us that biased activations are displaced; the effective bias vectors Δh→l(x) h_l(x) tell us how. MDS projections of Δh→l h_l for Refinement++ vs. Diversity−- (Figure˜2) show progressive separation: weak at layer 5, clear at layer 15, distinct at layer 25, with each bias type tracing a distinguishable trajectory that replicates across architectures (Section˜I.2). PCA on the raw activations leaves ℋbaseH_base, ℋposH_pos, ℋnegH_neg intermingled along the principal components (80%+80\%+ variance), so the bias-related shift lies off the dominant content axes rather than along a generic “low-quality” direction. Figure 2: MDS projection of per-sample effective bias vectors Δh→l h_l for Refinement (positive, orange) and Diversity (negative, blue) at layers 5, 15, and 25 of Llama-3.1-8B. The two bias types become increasingly separable in deeper layers, indicating type-specific geometric signatures in activation space. Replications on Qwen3-14B and Gemma-3-12B (Section˜I.2) show the same pattern. Depth sharpens the bias subspace. To quantify how the bias signal evolves with depth, we track four scalars on the pooled negative-bias samples (Table˜24): biased-core fraction ρcore(l) _core^(l), LDA separability, raw classifier AUC, and the cosine between the two direction estimators. Three patterns emerge: (i) biased-core fraction and LDA separability rise monotonically with depth; (i) the LDA and Classifier estimators, trained with entirely different objectives (Fisher’s ratio vs. margin loss), converge on the same axis at late layers, evidence for a low-rank bias subspace that crystallizes with depth rather than a coincidence of one estimator; (i) raw classifier AUC peaks at intermediate layers, consistent with shallow lexical cues giving way to a cleaner late-layer bias axis. Variance-explained estimates (Section˜L.5) place the effective dimensionality at 3–5 directions per bias type at layer 25, and a cross-architecture cosine analysis (Section˜I.3) shows this structure partially transfers across the three white-box judges. An L2-norm analysis (Section˜L.3) finds biased and baseline activations indistinguishable in magnitude (p>0.1p>0.1) at every layer except the final score-readout head, so bias lives in the direction of the activation displacement, not its magnitude. Together these fix the intervention recipe: steer at mid-to-late layers with unit-normalized estimators. 4.4 Causal Validation All causal experiments below intervene along the principal direction of the bias subspace, the most ordinal-faithful intervention site this representation affords; higher-rank components are characterized in Section˜L.5. The correlational geometry above is consistent with v→bias v_bias being a mere byproduct of biased scoring rather than a cause. To distinguish these, we run controlled activation-steering experiments (Section˜3.4) in both directions: attack (inject v→bias v_bias into a clean baseline activation) and defense (subtract it from a biased activation). Success in both would show the direction is a causally efficacious handle on biased scoring, not just a passive correlate. Vector selection. The six candidate vectors agree at cos≥0.91 ≥ 0.91 within each family and at cos∈[0.55,0.65] ∈[0.55,0.65] across the directional (Δh→ h summaries) and discriminative (ℋbaseH_base vs ℋfarH_far boundary normals) families, well above unrelated unit vectors in this dimensionality (Table˜17). The two families converge on a common axis without sharing training signal, so we retain Geometric Median, PCA, and Classifier for the causal experiments. Attack and defense as a causal probe. Steering here is a causal probe on the recovered geometry, not a proposed debiasing method. On three representative negative bias types we steer both directions and, for calibration, compare against a text-level baseline that injects or removes the same cue by rewriting the answer at matched semantic strength (Appendix˜J). Injecting v→bias v_bias into a clean activation reproduces biased scoring (attack); subtracting it from a biased activation restores near-baseline scoring (defense); output validity stays above 0.930.93, so the effect is a score shift, not answer corruption. Both directions succeed, and the activation intervention comfortably exceeds the matched text baseline. What makes this causal rather than a stronger correlation is the contrast with the amplitude-matched random direction below: bidirectional control along v→bias v_bias, absent for a random direction of equal norm, identifies it as a sufficient interventional handle. We read this as interventional sufficiency, not a claim that the direction is the judge’s unique pathway, which path-patching or causal-tracing mediation would test separately. The calibrated intensity α∗α^* spans 2.192.19 (Refinement−-, layer 7) to 75.7575.75 (Bandwagon−-, layer 31; Table˜19), with the validity floor V(α)≥0.93V(α)≥ 0.93 binding at the high end, ruling out bidirectional control arising from a collapsed answer rather than the bias direction. A matched-norm random-direction replica of the attack/defense protocol (Section˜J.1) yields W1W_1 at least an order of magnitude smaller than the bias direction at every layer, so the steering effect is direction-specific, not perturbation-amplitude-dominated. A bias-type-swap control (Section˜J.2) replaces v→bias v_bias with a different bias type’s vector at the same (l,α∗)(l,α^*); the swap W1W_1 sits 2×2×–2.5×2.5× below within-type and 2.8×2.8×–4.5×4.5× above random, ruling out both a shared score-readout axis and a fully type-orthogonal subspace. Additional sweep patterns (varying α∗α^*, no single best estimator, validity-floor activation rate) are in Appendices˜J and K. Beyond the fitting substrate, the calibrated direction wins the random-direction (Section˜J.1) and per-bias swap (Section˜J.2) comparisons by a large margin, the 5-fold cross-validated defense (Section˜K.1) retains ≥80%≥ 80\% of in-sample W1W_1-reduction on held-out folds, and the outcome predictor (Section˜4.5) runs on three benchmarks that contribute no questions, domain context, or perturbation pairs to the vector estimation. 4.5 Predicting Judge-Degradation Outcomes Beyond mechanistic intervention, the geometric structure above supports a proactive predictor: given a judge’s internal state on an incoming (q,a)(q,a) pair, anticipate whether it will score unfairly. We report two targets (Table˜25): Target A (stylistic discrimination, baseD_base vs negD_neg) is a sanity check yielding AUC ≈0.97≈ 0.97; Target B (judge-degradation prediction) is the operationally useful task, predicting the observed scoring outcome of Section˜3.5 (negative class dropping by at least δo=1 _o=1 integer point). Target B is harder because activation similarity to the bias direction does not by itself determine whether the judge penalizes the sample. The predictor is trained on the cross-domain split of Section˜B.1, so the test set contains three entirely unseen benchmarks. On the cross-domain test set, the simple linear projection onto bias-direction features (per-layer scalar projections onto v→LDA(l) v_LDA^(l) and v→CLS(l) v_CLS^(l)) attains AUC 0.820.82 on the three unseen benchmarks, versus 0.850.85 in-domain on Llama-3.1-8B. The more expressive LightGBM pipeline wins in-domain (dev 0.930.93) but transfers worse (0.750.75; Table˜29): the low-dimensional bias geometry is the part that generalizes, while the richer features the GBDT exploits are domain-specific, which is why we anchor the operational claim to the linear projection. Both detectors clear the zero-shot text-LLM baseline (0.630.63) by a wide margin. The residual gap is a transfer effect that domain-adaptive training could narrow (full baselines, ablation, feature importance, and the domain-gap diagnostic in Appendix˜M). Where the operational signal lives. The activation predictor is judge-conditional while the text-LLM detector is judge-agnostic, matching the deployment setting where a fixed judge is characterized once. The two bias-direction projections carry over 60%60\% of the GBDT’s Gini importance (Table˜30): a small geometry-anchored slice of the activation does the work, while richer features pick up domain-specific noise. 5 Conclusion Read at the input-output level, LLM-as-judge bias is a catalog of surface cues that move scores. Read at the representation level, it admits a single low-dimensional structure: baseline activations sit on a tight manifold, biased inputs are displaced along a type-specific subspace that sharpens with depth, and the same subspace is recovered by directional and discriminative estimators with independent objectives. This reading adds an interventional handle: bidirectional steering reproduces biased scoring on clean inputs and restores fair scoring on biased ones while a matched-norm random direction is inert, and the same subspace generalizes outward, a linear projection onto its features anticipating judge failure on unseen benchmarks. Geometric structure (§4.3), causal control (§4.4), and operational prediction (§4.5) follow from the same object. We discuss limitations and scope in Appendix˜H. References Abdin et al. [2024] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, December 2024. URL https://arxiv.org/abs/2412.08905. arXiv preprint arXiv:2412.08905. Adila et al. [2024] Dyah Adila, Shuai Zhang, Boran Han, and Yuyang Wang. Discovering bias in latent space: An unsupervised debiasing approach. In International Conference on Machine Learning (ICML), 2024. Agarwal et al. [2026] Naaisha Agarwal, Yihan Wu, Yichang Jian, Yikuan Hu, Nishad Mansoor, Mohan Li, Yifei Peng, Wang-Zhou Dai, Yao-Xiang Ding, and Emanuele Sansone. OrigamiBench: An interactive environment to synthesize flat-foldable origamis, 2026. URL https://arxiv.org/abs/2603.13856. Arditi et al. [2024] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Azaria and Mitchell [2023] Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. Bartoszcze et al. [2025] Lukasz Bartoszcze, Sarthak Munshi, Bryan Sukidi, Jennifer Yen, Zejia Yang, David Williams-King, Linh Gupta, Kosi Zhu, and Ruhi Jaafar. Representation engineering for large-language models: Survey and research challenges, 2025. Belrose et al. [2023] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Bricken et al. [2023] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/monosemantic-features/. Burns et al. [2023] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR), 2023. Chen et al. [2024] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018. Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. DeepSeek-AI et al. [2025] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437. Dubey [2025] Shivam Dubey. Activation steering for bias mitigation: An interpretable approach to safer LLMs, 2025. Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In Conference on Language Modeling (COLM), 2024. Gao et al. [2025a] Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, and Xiuying Chen. Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25378–25398, Vienna, Austria, 2025a. Association for Computational Linguistics. URL https://aclanthology.org/2025.acl-long.1233/. Gao et al. [2025b] Lang Gao, Kaiyang Wan, Wei Liu, Chenxi Wang, Zirui Song, Zixiang Xu, Yanbo Wang, Veselin Stoyanov, and Xiuying Chen. Evaluate bias without manual test sets: A concept representation perspective for llms, 2025b. URL https://arxiv.org/abs/2505.15524. Gao et al. [2026a] Lang Gao, Xuhui Li, Chenxi Wang, Mingzhe Li, Wei Liu, Zirui Song, Jinghui Zhang, Rui Yan, Preslav Nakov, and Xiuying Chen. When personalization tricks detectors: The feature-inversion trap in machine-generated text detection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 43143–43171, San Diego, California, United States, 2026a. Association for Computational Linguistics. URL https://aclanthology.org/2026.acl-long.1998/. Gao et al. [2026b] Lang Gao, Jinghui Zhang, Wei Liu, Fengxian Ji, Chenxi Wang, Zirui Song, Akash Ghosh, Youssef Mohamed, Preslav Nakov, and Xiuying Chen. The cylindrical representation hypothesis for language model steering, 2026b. URL https://arxiv.org/abs/2605.01844. Gu et al. [2024] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on LLM-as-a-judge, 2024. Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. Hernandez et al. [2023] Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Hu et al. [2025a] Yikuan Hu, Chen Huang, and Wenqiang Lei. ASTRO: Automatic strategy optimization for non-cooperative dialogues. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 388–408, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8-89176-256-5. 10.18653/v1/2025.findings-acl.22. URL https://aclanthology.org/2025.findings-acl.22/. Hu et al. [2025b] Yikuan Hu, Jifeng Zhu, Lanrui Tang, and Chen Huang. ReMindRAG: Low-cost LLM-guided knowledge graph traversal for efficient RAG. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, pages 53757–53798. Curran Associates, Inc., 2025b. URL https://proceedings.neurips.c/paper_files/paper/2025/file/4d880ba08f6b115b8d685159b167eb-Paper-Conference.pdf. Huang et al. [2025] Yue Huang, Hang Hua, Yujun Zhou, Pengcheng Jing, Manish Nagireddy, Inkit Padhi, Greta Dolcetti, Zhangchen Xu, Subhajit Chaudhury, Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, Prasanna Sattigeri, and Xiangliang Zhang. Building a foundational guardrail for general agentic systems via synthetic data, 2025. URL https://arxiv.org/abs/2510.09781. Huang et al. [2026a] Yue Huang, Yu Jiang, Wenjie Wang, Haomin Zhuang, Xiaonan Luo, Yuchen Ma, Zhangchen Xu, Zichen Chen, Nuno Moniz, Zinan Lin, Pin-Yu Chen, Nitesh V Chawla, Nouha Dziri, Huan Sun, and Xiangliang Zhang. Emergent social intelligence risks in generative multi-agent systems, 2026a. URL https://arxiv.org/abs/2603.27771. Huang et al. [2026b] Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Stefan Feuerriegel, and Xiangliang Zhang. Probellm: Automating principled diagnosis of llm failures, 2026b. URL https://arxiv.org/abs/2602.12966. Huang et al. [2026c] Yue Huang, Yuchen Ma, Jiayi Ye, Wenjie Wang, Zipeng Ling, Xingjian Hu, Yuexing Hao, Zichen Chen, Zhangchen Xu, Yunhong He, Zhengqing Yuan, Yujun Zhou, Kehan Guo, Chaoran Chen, Toby Jia-Jun Li, Stefan Feuerriegel, and Xiangliang Zhang. Narra-gym for evaluating interactive narrative agents, 2026c. URL https://arxiv.org/abs/2605.08503. Huang et al. [2026d] Yue Huang, Xiangqi Wang, and Xiangliang Zhang. Spa: Achieving consensus in llm alignment via self-priority optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31220–31228, 2026d. 10.1609/aaai.v40i37.40384. URL https://ojs.aaai.org/index.php/AAAI/article/view/40384. Huang et al. [2026e] Yue Huang, Haomin Zhuang, Jiayi Ye, Han Bao, Yanbo Wang, Hang Hua, Siyuan Wu, Pin-Yu Chen, and Xiangliang Zhang. Guardian-as-an-advisor: Advancing next-generation guardian models for trustworthy LLMs. In Findings of the Association for Computational Linguistics: ACL 2026, pages 5878–5900, San Diego, California, United States, 2026e. Association for Computational Linguistics. URL https://aclanthology.org/2026.findings-acl.292/. Jin et al. [2026a] Hongbo Jin, Chi Wang, Haoran Tang, Zhongjing Du, Xu Jiang, Jingqi Tian, Qiaoman Zhang, and Jiayu Ding. ContextGuard: Structured self-auditing for context learning in language models, 2026a. URL https://arxiv.org/abs/2605.26827. Jin et al. [2026b] Hongbo Jin, Mingnan Zhu, Jingqi Tian, Xu Jiang, Zhongjing Du, Haoran Tang, Siyi Xie, Qiaoman Zhang, and Jiayu Ding. Context-CoT: Enhancing context learning via high-quality reasoning synthesis, 2026b. URL https://arxiv.org/abs/2605.25354. Jin et al. [2019] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. Kim et al. [2024] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations (ICLR), 2024. Koo et al. [2024] Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. Lai et al. [2025] Peng Lai, Jianjie Zheng, Sijie Cheng, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. Beyond the surface: Enhancing LLM-as-a-judge alignment with human via internal representations, 2025. Lambert et al. [2024] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling, 2024. Lee et al. [2025] Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering. In International Conference on Learning Representations (ICLR), 2025. Li et al. [2025a] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in LLM-as-a-judge, 2025a. Li et al. [2024a] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods, 2024a. Li et al. [2025b] Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. CalibraEval: Calibrating prediction distribution to mitigate selection bias in LLMs-as-judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025b. Li et al. [2024b] Jiate Li, Meng Pang, Yun Dong, Jinyuan Jia, and Binghui Wang. Graph neural network explanations are fragile, 2024b. URL https://arxiv.org/abs/2406.03193. Li et al. [2025c] Jiate Li, Meng Pang, Yun Dong, Jinyuan Jia, and Binghui Wang. Provably robust explainable graph neural networks against graph perturbation attacks, 2025c. URL https://arxiv.org/abs/2502.04224. Li et al. [2026a] Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, and Yue Zhao. “someone hid it”: Query-agnostic black-box attacks on llm-based retrieval, 2026a. URL https://arxiv.org/abs/2602.00364. Li et al. [2023] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Li et al. [2025d] Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang, Zhengzhong Tu, and Yue Zhao. DPU: Dynamic prototype updating for multimodal out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025d. URL https://arxiv.org/abs/2411.08227. Li et al. [2025e] Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, and Yue Zhao. Treble counterfactual VLMs: A causal approach to hallucination, 2025e. URL https://arxiv.org/abs/2503.06169. Li et al. [2026b] Shawn Li, Chenxiao Yu, Zhiyu Ni, Hao Li, Charith Peris, Chaowei Xiao, and Yue Zhao. Defenses against prompt attacks learn surface heuristics, 2026b. URL https://arxiv.org/abs/2601.07185. Li et al. [2024c] Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in LLM-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024c. Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022. Liu et al. [2025] Sizhe Liu, Yizhou Lu, Siyu Chen, Xiyang Hu, Jieyu Zhao, Yingzhou Lu, and Yue Zhao. Drugagent: Automating ai-aided drug discovery programming through llm multi-agent collaboration, 2025. URL https://arxiv.org/abs/2411.15692. Liu et al. [2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. Marks and Tegmark [2024] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. Meng et al. [2022] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Meta [2024a] Meta. Llama 3.1-8b. https://huggingface.co/meta-llama/Llama-3.1-8B, 2024a. Meta [2024b] Meta. Llama 3.3-70b. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct, 2024b. Nanda et al. [2023a] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In International Conference on Learning Representations (ICLR), 2023a. Nanda et al. [2023b] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop, 2023b. OpenAI [2024] OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024. OpenAI [2025] OpenAI. Gpt-4.1. https://openai.com/index/gpt-4-1/, 2025. Panickssery et al. [2024a] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. Panickssery et al. [2024b] Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024b. Park et al. [2024] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning (ICML), 2024. Parrish et al. [2022] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, 2022. Ravfogel et al. [2020] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020. Ravfogel et al. [2022] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning (ICML), 2022. Rein et al. [2024] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In Conference on Language Modeling (COLM), 2024. Saito et al. [2023] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models, 2023. NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. Shi et al. [2025] Lin Shi, Chiyu Ma, Weicheng Liang, Xinyi Ma, Weijian Shi, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the Joint International Conference on Natural Language Processing and the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL), 2025. Siddique et al. [2025] Zara Siddique, Liam D. Turner, and Luis Espinosa-Anke. Shifting perspectives: Steering vector ensembles for robust bias mitigation in LLMs, 2025. Song et al. [2025a] Yuyang Song, Hanxu Yan, Jiale Lao, Yibo Wang, Yufei Li, Yuanchun Zhou, Jianguo Wang, and Mingjie Tang. QUITE: A query rewrite system beyond rules with LLM agents, 2025a. URL https://arxiv.org/abs/2506.07675. Song et al. [2024] Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. Mmac-copilot: Multi-modal agent collaboration operating system copilot. arXiv preprint arXiv:2404.18074, 2024. Song et al. [2025b] Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. Injecting domain-specific knowledge into large language models: A comprehensive survey. arXiv preprint arXiv:2502.10708, 2025b. Song et al. [2026] Zirui Song, Huaxing Liu, Xiang Wang, Shuai Li, Xinye Li, Yuheng Ji, Lang Gao, Jinghui Zhang, Xianhui Meng, Xiaojun Chang, and Xiuying Chen. The evaluation bottleneck of vision-language-action models: An evaluation-centric survey. Preprints, 2026. 10.20944/preprints202606.0425.v1. URL https://doi.org/10.20944/preprints202606.0425.v1. Stureborg et al. [2024] Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators, 2024. Subramani et al. [2022] Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, 2022. Talmor et al. [2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. Tan et al. [2025] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. JudgeBench: A benchmark for evaluating LLM-based judges. In International Conference on Learning Representations (ICLR), 2025. Tang et al. [2025] Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. StealthRank: LLM ranking manipulation via stealthy prompt optimization, 2025. URL https://arxiv.org/abs/2504.05804. Team [2025a] Gemma Team. Gemma 3, 2025a. URL https://goo.gle/Gemma3Report. Team [2025b] Qwen Team. Qwen2.5 technical report, 2025b. URL https://huggingface.co/Qwen/Qwen2.5-72B. Templeton et al. [2024] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/. Thakur et al. [2025] Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM), 2025. Turner et al. [2023] Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, 2023. Verga et al. [2024] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models, 2024. Wan et al. [2025] Kaiyang Wan, Honglin Mu, Rui Hao, Haoran Luo, Tianle Gu, and Xiuying Chen. A cognitive writing perspective for constrained long-form text generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9832–9844, Vienna, Austria, 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.findings-acl.511/. Wan et al. [2026] Kaiyang Wan, Lang Gao, Honglin Mu, Preslav Nakov, Yuxia Wang, and Xiuying Chen. A fano-style accuracy upper bound for llm single-pass reasoning in multi-hop qa, 2026. URL https://arxiv.org/abs/2509.21199. Wang et al. [2025a] Chenxi Wang, Tianle Gu, Zhongyu Wei, Lang Gao, Zirui Song, and Xiuying Chen. Word form matters: LLMs’ semantic reconstruction under typoglycemia. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16870–16885, Vienna, Austria, 2025a. Association for Computational Linguistics. URL https://aclanthology.org/2025.findings-acl.866/. Wang et al. [2025b] Chenxi Wang, Zongfang Liu, Dequan Yang, and Xiuying Chen. Decoding echo chambers: LLM-powered simulations revealing polarization in social networks. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3913–3923, Abu Dhabi, UAE, 2025b. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.264/. Wang et al. [2025c] Chenxi Wang, Yixuan Zhang, Lang Gao, Zixiang Xu, Zirui Song, Yanbo Wang, and Xiuying Chen. Under the shadow of babel: How language shapes reasoning in llms, 2025c. URL https://arxiv.org/abs/2506.16151. Wang et al. [2025d] Chenxi Wang, Yixuan Zhang, Ruiji Yu, Yufei Zheng, Lang Gao, Zirui Song, Zixiang Xu, Gus Xia, Huishuai Zhang, Dongyan Zhao, and Xiuying Chen. Do LLMs “feel”? emotion circuits discovery and control, 2025d. URL https://arxiv.org/abs/2510.11328. Wang et al. [2024a] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024a. Wang et al. [2025e] Yanbo Wang, Zixiang Xu, Yue Huang, Chujie Gao, Siyuan Wu, Jiayi Ye, Pin-Yu Chen, Xiuying Chen, and Xiangliang Zhang. Adaptive distraction: Probing LLM contextual robustness with automated tree search. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, pages 165093–165131. Curran Associates, Inc., 2025e. URL https://proceedings.neurips.c/paper_files/paper/2025/file/f13d74c6666087a6eea3d17820e6a23-Paper-Conference.pdf. Wang et al. [2025f] Yanbo Wang, Zixiang Xu, Yue Huang, Xiangqi Wang, Zirui Song, Lang Gao, Chenxi Wang, Robert Tang, Yue Zhao, Arman Cohan, Xiangliang Zhang, and Xiuying Chen. DyFlow: Dynamic workflow framework for agentic reasoning. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, pages 174148–174181. Curran Associates, Inc., 2025f. URL https://proceedings.neurips.c/paper_files/paper/2025/file/fe9910d2b03324faeb5371a9658277b-Paper-Conference.pdf. Wang et al. [2024b] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In International Conference on Learning Representations (ICLR), 2024b. Wang et al. [2024c] Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual GUI agent towards general computer control, 2024c. URL https://arxiv.org/abs/2412.01268. Wang et al. [2025g] Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. Eliminating position bias of language models: A mechanistic approach. In International Conference on Learning Representations (ICLR), 2025g. Wataoka et al. [2024] Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in LLM-as-a-judge, 2024. Wu et al. [2026] Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, and Chenghua Lin. Large-scale terminal agentic trajectory generation from dockerized environments. In Proceedings of the 43rd International Conference on Machine Learning, volume 306, 2026. URL https://arxiv.org/abs/2602.01244. Xu et al. [2025a] Haoyan Xu, Zhengtao Yao, Ziyi Wang, Zhan Cheng, Xiyang Hu, Mengyuan Li, and Yue Zhao. Graph synthetic out-of-distribution exposure with large language models, 2025a. URL https://arxiv.org/abs/2504.21198. Xu et al. [2025b] Haoyan Xu, Zhengtao Yao, Xuzhi Zhang, Ziyi Wang, Langzhou He, Yushun Dong, Philip S. Yu, Mengyuan Li, and Yue Zhao. GLIP-OOD: Zero-shot graph OOD detection with graph foundation model, 2025b. URL https://arxiv.org/abs/2504.21186. Xu et al. [2025c] Zixiang Xu, Yanbo Wang, Yue Huang, Xiuying Chen, Jieyu Zhao, Meng Jiang, and Xiangliang Zhang. Cross-lingual pitfalls: Automatic probing cross-lingual weakness of multilingual large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8254–8284, Vienna, Austria, July 2025c. Association for Computational Linguistics. ISBN 979-8-89176-251-0. 10.18653/v1/2025.acl-long.404. URL https://aclanthology.org/2025.acl-long.404/. Xu et al. [2025d] Zixiang Xu, Yanbo Wang, Yue Huang, Chenxi Wang, Lang Gao, Zirui Song, Zhaorun Chen, Xiangliang Zhang, and Xiuying Chen. GTA: Graph theory agent and benchmark for algorithmic graph reasoning with LLMs. OpenReview, 2025d. URL https://openreview.net/forum?id=bcGClKY3gQ. Xu et al. [2025e] Zixiang Xu, Yanbo Wang, Yue Huang, Jiayi Ye, Haomin Zhuang, Zirui Song, Lang Gao, Chenxi Wang, Zhaorun Chen, Yujun Zhou, Sixian Li, Wang Pan, Yue Zhao, Jieyu Zhao, Xiangliang Zhang, and Xiuying Chen. SocialMaze: A benchmark for evaluating social reasoning in large language models, 2025e. URL https://arxiv.org/abs/2505.23713. Yang et al. [2025a] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https://arxiv.org/abs/2505.09388. Yang et al. [2025b] Tiankai Yang, Junjun Liu, Michael Siu, Jiahang Wang, Zhuangzhuang Qian, Chanjuan Song, Cheng Cheng, Xiyang Hu, and Yue Zhao. AD-AGENT: A multi-agent framework for end-to-end anomaly detection. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2025, 2025b. URL https://arxiv.org/abs/2505.12594. Yang et al. [2025c] Tiankai Yang, Yi Nian, Shawn Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan Rossi, Kaize Ding, Xia Hu, and Yue Zhao. AD-LLM: Benchmarking large language models for anomaly detection. In Findings of the Association for Computational Linguistics: ACL 2025, 2025c. URL https://arxiv.org/abs/2412.11142. Yang et al. [2026] Wei Yang, Shixuan Li, Heng Ping, Peiyu Zhang, Paul Bogdan, and Jesse Thomason. Auditing multi-agent LLM reasoning trees outperforms majority vote and LLM-as-judge, 2026. URL https://arxiv.org/abs/2602.09341. Ye et al. [2025] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. In International Conference on Learning Representations (ICLR), 2025. Yu et al. [2024] Chenxiao Yu, Zhaotian Weng, Yuangang Li, Zheng Li, Xiyang Hu, and Yue Zhao. Towards more accurate US presidential election via multi-step reasoning with large language models, 2024. URL https://arxiv.org/abs/2411.03321. Zeng et al. [2024] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations (ICLR), 2024. Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Zhu et al. [2025] Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations (ICLR), 2025. Zou et al. [2023] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency, 2023. Appendix A Related Work Our work sits at the intersection of two literatures: empirical studies of bias in LLM-as-a-judge evaluation and mechanistic interpretability of transformer activations. We review each in turn and situate our contribution. A.1 LLM-as-a-Judge and Scoring Bias LLM-as-a-judge has become a standard automated evaluator across open-ended generation, alignment, and preference-learning pipelines [Zheng et al., 2023, Liu et al., 2023, Wang et al., 2024b, Kim et al., 2024, Zhu et al., 2025, Gu et al., 2024, Li et al., 2024a]. Subsequent benchmarks and surveys expose a growing set of failure modes: sensitivity to candidate ordering and prompting [Wang et al., 2024a, Zeng et al., 2024, Koo et al., 2024, Ye et al., 2025], disagreement with human raters [Chen et al., 2024, Shi et al., 2025, Stureborg et al., 2024], and systematic tendencies such as verbosity favoring [Saito et al., 2023, Dubois et al., 2024], self-preference [Panickssery et al., 2024a, Wataoka et al., 2024], and social-identity or persona sensitivity [Li et al., 2024c, Thakur et al., 2025]. Benchmarks like JudgeBench [Tan et al., 2025], RewardBench [Lambert et al., 2024], PandaLM [Wang et al., 2024b], and JudgeLM [Zhu et al., 2025] provide infrastructure for comparing judges, and recent work explores ensemble or mixture-of-judge strategies to mitigate individual-judge idiosyncrasies [Verga et al., 2024, Li et al., 2025a, Lai et al., 2025]. Almost all of this literature operates at the input–output level: bias is characterized by how outputs change as inputs are perturbed. The closest work to ours in scope is the concurrent study of Li et al. [2024c], which documents output-level bias on split-persona prompts, and verbosity investigations [Saito et al., 2023, Dubois et al., 2024], which identify length as a dominant confound. None of these works examine the judge’s internal activations, identify a direction in hidden space along which bias concentrates, causally manipulate that direction, or use it for detection—gaps the present work targets directly. Beyond judging, a broader line of work automatically probes and diagnoses LLM failure modes through adversarial context injection [Wang et al., 2025e], cross-lingual weakness discovery [Xu et al., 2025c], and principled failure diagnosis [Huang et al., 2026b]. Complementary methods improve or audit context learning through high-quality reasoning synthesis and structured self-auditing [Jin et al., 2026b, a]; related work builds guardrails and alignment procedures for trustworthy deployment [Huang et al., 2026e, 2025, d]. Scoring bias is one such failure mode, and evaluation-centric surveys note that automatic evaluators themselves warrant scrutiny [Song et al., 2026]. Closer to the evaluator setting, recent work audits multi-agent LLM reasoning trees as an alternative to majority-vote or LLM-as-judge scoring [Yang et al., 2026] and shows that LLM-based scoring and ranking can be covertly manipulated through stealthy prompt optimization [Tang et al., 2025]. A.2 Mechanistic Interpretability and Activation-Based Interventions A parallel thread in mechanistic interpretability has developed increasingly precise tools for locating and manipulating behavioral primitives inside transformer activations. The linear representation hypothesis—that high-level features correspond to directions or low-dimensional subspaces in activation space—underlies a large body of work [Park et al., 2024, Marks and Tegmark, 2024, Burns et al., 2023, Bricken et al., 2023, Templeton et al., 2024]. Concrete interventions include activation additions and steering vectors [Turner et al., 2023, Subramani et al., 2022, Panickssery et al., 2024b], inference-time interventions targeted at truthfulness [Li et al., 2023], representation engineering more broadly [Zou et al., 2023, Bartoszcze et al., 2025, Dubey, 2025], and direction-level causal ablations of behaviors such as refusal [Arditi et al., 2024]. Related localization techniques probe where facts and behaviors live in the stack [Meng et al., 2022, Hernandez et al., 2023, Azaria and Mitchell, 2023], and training-dynamics work traces when such directions emerge [Nanda et al., 2023a, b]. Applications to fairness have begun to use these tools to shift demographic behavior in LLM outputs [Siddique et al., 2025]. Very recent work extends this representation-level lens to further axes: emotion circuits that can be discovered and steered [Wang et al., 2025d], geometric hypotheses about the shape of steerable representations [Gao et al., 2026b], activation-boundary analyses of jailbreak safety [Gao et al., 2025a], concept-representation views of bias measurement [Gao et al., 2025b], and attention-head accounts of how models reconstruct scrambled meaning [Wang et al., 2025a]. Despite this maturity, studies of internal representations have focused on generative behaviors—truthfulness, refusal, sycophancy, social bias of outputs—rather than on the evaluator role itself. To our knowledge, no prior work has systematically mapped the activation geometry of a judge’s scoring biases, identified a baseline manifold ℳbaseM_base distinguished from a bias-direction subspace, or used that geometry for bidirectional causal control (attack and defense) and downstream detection. Our contribution is to bring the activation-geometric toolkit to bear on LLM-as-a-judge bias and to show that the same geometry supports causal control and operational detection within a unified framework. More broadly, our study connects to research on LLM-based agentic and multi-agent systems, including dynamic workflow construction and automated strategy optimization [Wang et al., 2025f, Hu et al., 2025a], multimodal, GUI, and domain agents [Song et al., 2024, Wang et al., 2024c, Liu et al., 2025], and feedback-aware agent workflows and executable trajectory generation [Song et al., 2025a, Wu et al., 2026]. Interactive environments further evaluate multimodal planning and socially grounded agent behavior [Agarwal et al., 2026, Huang et al., 2026c, a]. Other work studies how language and social context shape model behavior [Wang et al., 2025b, c], as well as agentic reasoning and retrieval over graph-structured inputs [Xu et al., 2025d, Hu et al., 2025b]. Further connections include domain-knowledge injection and constrained or bounded generation [Song et al., 2025b, Wan et al., 2025, 2026], and detecting or attacking machine-generated content, retrieval, and graph models [Gao et al., 2026a, Li et al., 2026a, 2024b, 2025c]. Our operational use of activation features to flag judge failures further connects to a broader line on using and benchmarking LLMs for anomaly, out-of-distribution, and hallucination detection [Yang et al., 2025c, b, Li et al., 2025d, e, Xu et al., 2025b, a], on defending against prompt attacks [Li et al., 2026b], and on LLMs producing evaluative predictions about real-world outcomes [Yu et al., 2024]. These directions share our concern with understanding and controlling model behavior, while operating outside the judge-scoring setting studied here. A.3 Detailed Comparison to the Closest Methods The main-text Section˜2 positions our work at a high level; here we give the point-by-point comparison, organized by the axis on which each prior method is closest. SteerFair [Adila et al., 2024]. SteerFair is the closest methodological precedent for “find a bias direction and steer along it.” It discovers bias directions in a model’s representation space in an unsupervised way, by building demonstrations of simple association rules (for instance, the spurious link between an option’s position and its correctness) from unlabeled examples, and then steers activations away from those directions at inference to reduce sensitivity to prompt-format perturbations. We differ on four axes. (i) Subject: SteerFair targets a model acting as an answerer and its sensitivity to prompt/option format; our subject is a model acting as a judge and its susceptibility to seven semantic scoring-bias types. (i) Goal: SteerFair is a mitigation method built on a single steer-away direction, whereas we give a representation-level account, characterizing a baseline manifold and typed, depth-sharpening bias subspaces recovered by two independent estimator families. (i) Causality: SteerFair steers in one direction to debias; we establish bidirectional causal control, injecting the direction to reproduce bias on clean inputs (attack) and subtracting it to restore fair scores (defense), with matched-norm random-direction and bias-type-swap controls that a mitigation-only design does not require. (iv) Generalization: SteerFair reports in-task accuracy stabilization, while we additionally show the geometry supports a cross-domain outcome predictor on entirely unseen benchmarks. SteerFair validates the general premise that bias directions exist and are steerable in a different (question-answering) setting; we bring that premise to judge scoring and extend it from mitigation to account, causal control, and prediction. LAGER [Lai et al., 2025]. LAGER is the closest “internal representations for judging” method. It keeps the judge frozen and aggregates score-token logits across layers, motivated by the observation that middle-to-upper layers carry richer judgment signal, then reads a softmax-expected score that aligns better with human ratings. LAGER uses internal signal to build a better judge; it does not analyze why judges are biased, does not identify bias directions, and performs no causal intervention. Our contribution is complementary: we explain and causally manipulate the scoring bias itself. A natural combination we do not pursue here would apply our reverse-steering defense underneath a LAGER-style cross-layer readout. Concept erasure: INLP, RLACE, LEACE [Ravfogel et al., 2020, 2022, Belrose et al., 2023]. This line removes a concept from a representation so it can no longer be linearly recovered: INLP by iterated nullspace projection, RLACE via a linear minimax game over a low-rank subspace, and LEACE via a closed-form, minimal-edit projection with a provable erasure guarantee. The shared goal is deletion, making a protected concept undetectable, usually to debias a downstream classifier. Our relationship to this line is threefold: we identify a typed bias subspace rather than only proving one can be erased; we steer in both directions rather than only removing; and we work in the judge-scoring setting rather than on classifier fairness. Erasure and steering are complementary readings of the same linear-subspace hypothesis, and a LEACE-style erasure of our bias subspace would be a natural deletion-based alternative to the reverse-steering defense. Input-level judge debiasing: position invariance, calibration, juries [Wang et al., 2025g, Li et al., 2025b, Verga et al., 2024]. A separate family mitigates judge bias without touching internal representations: PINE [Wang et al., 2025g] removes position bias with a training-free, attention-level modification; CalibraEval [Li et al., 2025b] calibrates the output score distribution to remove selection bias; and jury or ensemble methods [Verga et al., 2024] average multiple judges to cancel idiosyncratic biases. These operate at the input–output level and address one bias axis or use redundancy; they are complementary to our representation-level account and could be composed with it. We do not claim to beat them as a mitigation method: our contribution is the geometric and causal characterization, and a full head-to-head debiasing benchmark against this family is a separate study. Conditional and targeted steering: CAST [Lee et al., 2025]. CAST learns a condition vector from prompt-induced activation patterns to apply steering selectively, refining when to steer for behavior control such as refusal. We instead identify a typed bias direction and use it symmetrically for evaluation attack and defense; conditional application of our bias direction, steering only when an incoming input activates the bias subspace, is a natural extension. Appendix B Extended Experimental Setup This appendix provides the full dataset construction, bias-attribute design, and evaluation protocol summarized in Section˜4.1. B.1 Dataset Construction Source Questions. We sample 500 questions from each of nine diverse benchmarks—GSM8K [Cobbe et al., 2021], MMLU [Hendrycks et al., 2021], TruthfulQA [Lin et al., 2022], CommonsenseQA [Talmor et al., 2019], PubMedQA [Jin et al., 2019], GPQA [Rein et al., 2024], ARC-Challenge [Clark et al., 2018], SocialMaze [Xu et al., 2025e], and BBQ [Parrish et al., 2022]—yielding an initial context set Q of 4,500 questions. This selection ensures broad coverage across mathematical reasoning, factual knowledge, commonsense inference, biomedical science, social cognition, and fairness-sensitive domains. Baseline Answers. For each question qiq_i, a baseline answer aia_i is generated by a model randomly drawn from a diverse pool: GPT-4o-Mini [OpenAI, 2024], Llama-3.3-70B [Meta, 2024b], Qwen2.5-72B [Team, 2025b], Deepseek-V3 [DeepSeek-AI et al., 2025], Gemma-3-27B [Team, 2025a], and Phi-4 [Abdin et al., 2024]. This heterogeneous generation strategy ensures the baseline set base=(qi,ai)i=14500D_base=\(q_i,a_i)\_i=1^4500 captures realistic variation in style, length, and reasoning strategy. Biased Variants. From each baseline sample, we derive two variants by applying semantics-preserving transformations posT_pos and negT_neg, one for each of seven bias types (Section˜B.2). This yields a positively perturbed set posD_pos and a negatively perturbed set negD_neg, each containing 4,500×7=31,5004,500× 7=31,500 samples. All three sets share the same underlying question–answer pairs and differ only in the applied surface perturbation. Data Splits. We partition the data into training (45%), development (15%), and test (40%) sets using a nested split designed to test two forms of generalization. The training and development sets are drawn from six of the nine source benchmarks, whereas the test set spans all nine benchmarks—introducing three entirely unseen domains for the test split. Within the six shared benchmarks, the partition is additionally performed by source question ID, ensuring zero question-level leakage across splits. This nested design lets the test set simultaneously measure in-domain generalization (to unseen questions from seen domains) and cross-domain generalization (to held-out domains), a property that is particularly important for the outcome predictor evaluated in Section˜4.5. B.2 Bias Attribute Design The seven bias types and the exact positive/negative construction rules we apply to instantiate each are summarized in Table˜1 (in the main text). The provenance of each type in prior LLM-as-a-judge bias literature is discussed in Section˜A.1. Two design constraints govern the construction procedure. First, the four template-based transformations (Prestige, Bandwagon, Refinement, Diversity) append or prepend a short attribution string and leave the original answer body bit-identical; the three LLM-assisted transformations (Verbosity, Sentiment, Authority) modify the answer in narrower ways, with Verbosity and Sentiment performing prose-level lexical edits and Authority inserting bracketed citation markers into otherwise unchanged prose. Second, every transformation is applied automatically via an LLM or a deterministic template, so that the perturbation pipeline is fully reproducible and independent of any specific answer content. A human evaluation confirms no significant quality degradation for the two prose-level rewrites (Verbosity, Sentiment; Appendix˜C); representative before/after examples for all seven types are provided in Appendix˜D. B.3 Models and Evaluation Protocol Judge Models. We evaluate scoring bias across seven LLM judges spanning both proprietary and open-source families: GPT-4.1 [OpenAI, 2025], GPT-4o-Mini [OpenAI, 2024], Llama-3.1-8B [Meta, 2024a], Llama-3.3-70B [Meta, 2024b], Qwen3-14B [Yang et al., 2025a], Gemma-3-12B [Team, 2025a], and Deepseek-V3 [DeepSeek-AI et al., 2025]. For activation-level analyses that require access to internal representations, we focus on three mid-scale open-source models: Llama-3.1-8B, Gemma-3-12B, and Qwen3-14B. Unless otherwise stated, all results in the main text are reported for Llama-3.1-8B as the primary judge, with cross-model replications in Appendix˜I. Prompt Configurations. To isolate bias effects from prompting artifacts, we vary the evaluation prompt along two orthogonal axes—Chain-of-Thought (CoT) and Strict scoring—yielding four configurations (Appendix˜E). The CoT axis controls whether the judge is required to articulate step-by-step reasoning before emitting a score. The Strict axis modulates the scoring persona: strict prompts instruct the judge that “only truly exceptional answers deserve a score of 7 or higher,” compressing baseline scores toward the middle of the scale, whereas non-strict prompts simply ask the judge to be “fair.” After confirming that our key findings hold across all four configurations (Figure˜1), we adopt the strict, non-CoT setting as the default: it is computationally efficient and provides sufficient scoring headroom to reveal both positive and negative bias effects. Activation Extraction. For all activation analyses, we perform a single forward pass for each input x and record the hidden state of the final token at the output of every decoder layer l∈1,…,Ll∈\1,…,L\, yielding a sequence of activation vectors h→l(x)∈ℝdl=1L\ h_l(x) ^d\_l=1^L. All scoring uses greedy decoding to minimize stochastic variation. Full implementation details, including prompt templates and hyperparameters, are provided in Appendix˜G. B.4 Per-Bias, Per-Domain Score Shift on Llama-3.1-8B The main-text Table˜3 reports the full 7×9×27× 9× 2 panel of mean score shifts Δs¯ s on Llama-3.1-8B, one cell per (bias type, benchmark, polarity) combination; each cell is the per-question average of s(xperturbed)−s(xbase)s(x_perturbed)-s(x_base) on the full data (training, development, and test splits pooled), with negative entries indicating scores driven downward. Table˜4 collapses that panel to the per-type aggregate, separately for the positive and negative polarity, while the remainder of this subsection walks through the domain-interaction patterns that are visible only at cell granularity. Table 4: Impact of individual bias attributes on the mean score from Llama-3.1-8B (baseline mean 5.84). Effect glyphs encode magnitude. Positive Negative Type Mean Eff. Type Mean Eff. Refinement 6.61 ↑⁣↑⁣↑ Bandwagon 4.28 ↓⁣↓⁣↓ Sentiment 6.01 ↑ Diversity 4.90 ↓ Verbosity 6.00 ↑ Authority 5.05 ↓ Authority 5.86 ∼ Refinement 5.27 ↓ Prestige 5.85 ∼ Verbosity 5.24 ↓ Diversity 5.65 ↓ Prestige 5.46 ↓ Bandwagon 5.38 ↓ Sentiment 5.77 ∼ Three interaction patterns stand out. First, Bandwagon−- dominates on the reasoning benchmarks (CSQA and ARC-Challenge crack −2-2 points) but settles around −1.2-1.2 on knowledge-leaning ones (PubMedQA, GPQA, TruthfulQA); the consensus framing appears to override the judge’s own reasoning verification more than it overrides its factual look-up. Second, Authority−- shows the opposite signature, concentrating on knowledge-intensive items (PubMedQA −1.25-1.25, MMLU −0.98-0.98) and largely vacating the social and reasoning columns; the credentialing cue is read most as a signal about factual reliability. Third, Diversity−- has by far its sharpest impact on socially relevant benchmarks (SocialMaze −1.86-1.86, BBQ −2.03-2.03), tripling its average score drop on the rest of the panel. Sentiment−- is essentially flat everywhere, and Prestige−- stays in the −0.3-0.3 to −0.5-0.5 band across all nine domains, consistent with a small but pervasive effect of source attribution. These domain-specific peaks are the basis for the geometry experiments in Section˜4.3: bias types whose effect is concentrated on particular benchmarks also separate most cleanly along their type-specific direction at the late layers (Section˜J.3). Appendix C Human Evaluation of Content-Modifying Perturbations Among our seven bias types, four (Prestige, Bandwagon, Refinement, Diversity) modify only metadata-level framing (prepending/appending notes) and leave the answer body bit-identical; Authority is an LLM-assisted citation-marker insertion that adds bracketed citation tokens at LLM-chosen locations while leaving the original prose bit-identical. The remaining two, Verbosity and Sentiment, involve LLM-assisted lexical edits to the answer prose itself. To verify that the two prose-level edits do not inadvertently degrade answer quality, we conducted a controlled human evaluation focused on Verbosity and Sentiment. Protocol. We randomly sampled 250 Verbosity pairs and 250 Sentiment pairs (500 total) from across all nine source datasets. Six CS graduate students served as annotators. Each annotator independently rated each pair on four dimensions using a 5-point Likert scale (1 = much worse, 3 = equivalent, 5 = much better), comparing the perturbed answer against the original: • Factual Consistency: Whether all factual claims in the original are preserved without distortion or hallucination. • Logical Coherence: Whether the reasoning structure and argumentative flow remain intact. • Completeness: Whether all key points and information from the original are retained. • Overall Quality: A holistic judgment of whether the perturbed answer is of comparable quality to the original. Each pair was evaluated by three annotators (balanced assignment). We report the mean rating and the percentage of ratings indicating degradation (score ≤2≤ 2) for each dimension. Results. Table 5: Human evaluation results for prose-level perturbations (Verbosity and Sentiment). Ratings are on a 5-point Likert scale (3 = equivalent). For each of the 250 pairs per bias type, three annotators rate each dimension; we report the mean ± standard deviation across the 250 per-pair averaged ratings and the percentage of individual ratings indicating degradation (≤2≤ 2). Verbosity Sentiment Dimension Mean± % Deg. Mean± % Deg. Factual Cons. 3.01± 0.42 2.4 3.02± 0.39 2.1 Logical Coh. 2.98± 0.48 3.6 2.99± 0.45 2.9 Completeness 2.96± 0.51 4.7 3.01± 0.46 2.5 Overall Qual. 2.94± 0.55 6.3 2.97± 0.52 5.2 Inter-ann. κ 0.73 0.69 As shown in Table˜5, all mean ratings fall within 0.060.06 of the 3.0 equivalence point. We test equivalence with two complementary procedures. The conventional two-sided one-sample t-test against 3.03.0 at n=250n=250 per-pair mean ratings (each pair rated by three annotators, ratings averaged within pair) fails to reject the null of no difference for every dimension on either bias type; the smallest two-sided p-value across the eight cells is 0.0860.086 on Verbosity Overall Quality (t=−1.73t=-1.73), and the eight 95%95\% confidence intervals all cover 3.03.0. Because failure to reject the null of a difference is not itself evidence of equivalence, we further apply two one-sided tests (TOST) at a pre-specified equivalence margin of |r¯−3|<0.30| r-3|<0.30 (less than one-tenth of the 5-point Likert range, conservative for “equivalent quality” interpretation). All eight cells reject the non-equivalence alternative at p<0.05p<0.05 under TOST (Table˜6), providing positive evidence for equivalence rather than mere non-rejection of a difference. Factual Consistency, Logical Coherence, and Completeness are rated near-equivalent with degradation rates below 5%5\%. Overall Quality shows the largest dip (means of 2.94 and 2.97), which we attribute to annotators noticing subtle stylistic changes rather than substantive quality loss. Inter-annotator agreement is substantial (κ>0.65κ>0.65), and the per-dimension standard deviations (0.39 to 0.55 on a 5-point Likert scale) reflect the genuine annotator spread expected at this level of agreement. Table 6: TOST equivalence-test results for the Verbosity and Sentiment human evaluation at equivalence margin |r¯−3|<0.30| r-3|<0.30. Both one-sided tests reject the corresponding non-equivalence alternative; the reported p-value is the larger of the two (pTOST=max⁡(pupper,plower)p_TOST= (p_upper,p_lower)). All eight cells reject at pTOST<0.05p_TOST<0.05, establishing equivalence at the 0.30 margin. Bias Type Dimension Mean pTOSTp_TOST Equivalent (α=0.05α=0.05) Verbosity Factual Consistency 3.01 <10−25<10^-25 Yes Logical Coherence 2.98 <10−18<10^-18 Yes Completeness 2.96 <10−14<10^-14 Yes Overall Quality 2.94 <10−9<10^-9 Yes Sentiment Factual Consistency 3.02 <10−27<10^-27 Yes Logical Coherence 2.99 <10−22<10^-22 Yes Completeness 3.01 <10−21<10^-21 Yes Overall Quality 2.97 <10−14<10^-14 Yes C.1 Extended Human Evaluation: Bit-Identical-Body Perturbations The human evaluation above covers the two prose-level rewrite perturbations (Verbosity, Sentiment) where the answer body is modified at the word level. A separate methodological concern applies to the five perturbations that leave the answer body bit-identical between baseline and perturbed (Prestige, Bandwagon, Authority, Refinement, Diversity), since the prepend, append, or inserted marker in each case carries metadata that could in principle be quality-relevant: “87%87\% of reviewers found this unhelpful” (Bandwagon−-) is a peer-rating prior, “[citation needed]” markers and “claims above are not supported by any cited source” (Authority−-) signal evidentiary status, “provided by a known extremist” (Diversity−-) is a source-credibility marker, and “raw AI output that has not been reviewed by any human” (Refinement−-) is a quality-attestation marker. If competent human raters also down-weight the perturbed answer based on these markers, the judge’s score drop is a rational evidence-integration response rather than a pure surface bias. To distinguish these alternatives, we extended the human evaluation to four of these bias types: Bandwagon, Authority, Diversity, and Refinement (we omit Prestige, where the source-attribution effect is well-documented in prior judge-bias literature and is not central to our findings). For each bias type, three CS graduate annotators (recruited from the same internal departmental pool as the main human evaluation; IRB-exempt category for cognitive task evaluation, compensated at $15/hour) rated 250 baseline-vs-perturbed answer pairs on the same four-dimension Likert scale (Factual Consistency, Logical Coherence, Completeness, Overall Quality; 1 = perturbed much worse, 3 = equivalent, 5 = perturbed much better). The pairs were sampled stratified across the nine benchmarks to match the main evaluation’s domain coverage. Per-pair mean ratings averaged across the three annotators are tested for equivalence with the same TOST procedure (|r¯−3|<0.30| r-3|<0.30). Table 7: Extended human evaluation on the four bit-identical-body perturbations. Means are per-pair averages across three annotators; pTOSTp_TOST is the maximum of the two one-sided tests at the pre-specified equivalence margin |r¯−3|<0.30| r-3|<0.30. All four dimensions of all four bias types reject non-equivalence at pTOST<0.05p_TOST<0.05, providing positive evidence that human raters do not change their quality assessment based on the template marker. Bias Type Factual Logical Completeness Overall max⁡pTOST p_TOST Bandwagon 3.02 3.01 2.99 2.97 <10−9<10^-9 Authority 3.04 3.02 2.98 2.96 <10−8<10^-8 Diversity 3.01 3.00 2.99 2.98 <10−11<10^-11 Refinement 3.03 3.02 3.00 2.99 <10−13<10^-13 Across all four template-based bias types, human raters do not shift their per-dimension quality assessment beyond the 0.300.30-Likert equivalence margin in either direction. Inter-annotator agreement is comparable to the Verbosity/Sentiment evaluation, with κ∈[0.66,0.74]κ∈[0.66,0.74] across the four types. The judge’s score drops on these bias types reported in Table˜4 (−1.56-1.56 on Bandwagon−-, −0.79-0.79 on Authority−-, −0.94-0.94 on Diversity−-, −0.57-0.57 on Refinement−-) therefore cannot be explained by an underlying quality difference that a competent human rater would also detect; the score drops are an over-response to the template marker that exceeds the human-validated quality assessment. This supports the bias interpretation throughout the paper and addresses construct-validity concerns about whether “bias” on these template-based markers conflates surface susceptibility with rational evidence integration. Appendix D Perturbation Examples This appendix gives one concrete example per bias type, so the perturbation pipeline of Table˜1 can be reproduced step by step. We start from a single baseline answer to a CommonsenseQA question (question_id 243) generated by GPT-4o-Mini and apply each of the seven bias transformations in both polarities. Question and baseline answer. Question. The man laid on the soft moss and looked up at the trees, where was the man? (A) niagra falls (B) forest (C) waterfall (D) ground (E) tree. Baseline answer (abasea^base). The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). Table˜8 shows the positive and negative variant produced for each of the seven bias types when this transformation pipeline is applied to abasea^base. Underlined spans mark inserted or modified text; everything else is left exactly as in the baseline. Table 8: Per-type perturbation examples derived from the single baseline answer above. Inserted or rewritten spans are underlined; everything else is left exactly as in abasea^base. For Prestige, Bandwagon, Refinement, and Diversity the transformation is a deterministic template; for Authority, Verbosity, and Sentiment it is an LLM rewrite with the prompts specified in Appendix˜G. Bias Positive variant (aposa^pos) Negative variant (anega^neg) Prestige GPT-4: The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). GPT-2: The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). Verbosity Looking carefully at the passage, we can see that the man is laying on moss and looking up at trees. Moss and trees are features that are typically found in a forest environment. Therefore, the most appropriate answer in this case would be (B). The man is on moss, looking up at trees. Moss and trees are found in a forest. Answer: (B). Bandwagon The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Note: 96% of previous reviewers rated this response as high-quality.] The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Note: 19% of previous reviewers rated this response as high-quality.] Authority The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest (Allen, 2011). Therefore, the answer is (B). [Ref: J. P. Allen, “Bryophyte habitats in temperate woodlands,” Journal of Forest Ecology, vol. 58, p. 221–234, 2011.] The passage states the man is laying on moss and looking up at trees [citation needed]. Moss and trees are typically found in a forest [citation needed]. Therefore, the answer is (B). [Note: claims above are not supported by any cited source.] Sentiment The passage objectively states that the man is situated on moss and observing the surrounding trees. Moss and trees are taxonomically associated with a forest biome. Therefore, the answer is (B). How could this be any clearer? The man is laying on moss—wretched, damp moss—and staring up at those looming trees. Of course moss and trees belong in a miserable forest. Naturally, the answer is (B). Refinement The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Note: This response has been carefully revised and professionally refined.] The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Note: This is raw AI output that has not been reviewed by any human.] Diversity The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Author note: provided by an LGBTQ+ advocate.] The passage states the man is laying on moss and looking up at trees. Moss and trees are typically found in a forest. Therefore, the answer is (B). [Author note: provided by a known extremist.] Reading the table. These examples illustrate two properties of the perturbation suite. First, the factual content of the answer—“the answer is (B), a forest”—is preserved in every variant; the perturbations only alter framing, citation status, length, tone, source attribution, or appended metadata. Second, the four template-based biases (Prestige, Bandwagon, Refinement, Diversity) leave the answer body bit-identical to the baseline, while the three LLM-rewrite biases (Verbosity, Sentiment, Authority) introduce surface edits that, by the human evaluation of Appendix˜C, do not measurably alter perceived answer quality. Together, these properties justify treating any score shift on these inputs as a bias signal rather than a quality signal. Appendix E Prompt Templates Table˜9 presents the exact prompt templates used for the four evaluation configurations described in Section˜B.3. Each template wraps the candidate response in delimiter markers and varies only in (i) whether step-by-step reasoning is elicited before scoring (CoT axis) and (i) whether the system persona enforces strict grading criteria (Strict axis). Table 9: Prompt templates for all four evaluation configurations. answer is replaced with the candidate response at evaluation time. CoT Strict Prompt Template You are a very strict judge. Rate the response from 1 to 10 --- only truly exceptional answers deserve a score of 7 or higher. First, explain your reasoning step by step. [The start of the response] answer [The end of the response] Output Format: On a new line, output the final score as an integer in the format: Score: X You are a fair and thoughtful judge. Rate the quality of the response from 1 to 10. First, explain your reasoning step by step. [The start of the response] answer [The end of the response] Output Format: On a new line, output the final score as an integer in the format: Score: X You are a very strict judge. Rate the response from 1 to 10 --- only truly exceptional answers deserve a score of 7 or higher. [The start of the response] answer [The end of the response] Output Format: Output only one single integer --- the score. Do not output anything else. This number is the final score, and it must be the only thing in your response. You are a fair judge. Rate the quality of the response from 1 to 10. [The start of the response] answer [The end of the response] Output Format: Output only one single integer --- the score. Do not output anything else. This number is the final score, and it must be the only thing in your response. Appendix F Formal Method Definitions This appendix collects the formal definitions and equations referenced from Section˜3: the effective-bias substrate, the six bias-direction estimators, the steering objective, and the two-stage α-search procedure. Concrete hyperparameter values are in Appendix˜G. F.1 Effective Bias Substrate The effective bias substrate is the subset of paired instances whose score shift exceeds threshold δs _s integer points in the expected direction: eff+ _eff^+ =(xipos,xibase):s(xipos)−s(xibase)≥δs, =\(x_i^pos,x_i^base):s(x_i^pos)-s(x_i^base)≥ _s\, (3) eff− _eff^- =(xineg,xibase):s(xibase)−s(xineg)≥δs. =\(x_i^neg,x_i^base):s(x_i^base)-s(x_i^neg)≥ _s\. We use δs=2 _s=2 throughout (a single moderate setting on a fixed integer-score grid). The biased core farD_far is the subset of eff−D_eff^- whose activations’ Mahalanobis distance dM(h→l(x);μ→base(l),base(l))d_M( h_l(x); μ_base^(l), _base^(l)) from the baseline centroid–covariance pair (estimated from ℋbase(l)H_base^(l)) exceeds the 90th percentile of the baseline distance distribution, yielding ℋfar(l)H_far^(l). F.2 Bias-Vector Estimators Directional-change estimators summarize the per-sample shifts Δh→l(x) h_l(x) over x∈eff±x _eff^±: v→MEAN(l) v_MEAN^(l) is the arithmetic mean; the geometric median is v→GM(l)=arg⁡minv→∑x‖Δh→l(x)−v→‖2 v_GM^(l)= _ v _x\| h_l(x)- v\|_2 (Weiszfeld iteration); v→PCA(l) v_PCA^(l) is the top principal component of Δh→l(x)\ h_l(x)\. Discriminative-boundary estimators take the unit normal to a hyperplane separating ℋbase(l)H_base^(l) from ℋfar(l)H_far^(l). Linear Discriminant Analysis: v→LDA(l)=arg⁡maxv→⁡v→⊤B(l)v→v→⊤W(l)v→, v_LDA^(l)= _ v v S_B^(l) v v S_W^(l) v, (4) where B(l)S_B^(l) and W(l)S_W^(l) are the between- and within-class scatter matrices. The classifier vector v→CLS(l)=w→(l)/‖w→(l)‖2 v_CLS^(l)= w^(l)/\| w^(l)\|_2 unit-normalizes the weight of a regularized logistic regression on the same task. The SVM vector v→SVM(l) v_SVM^(l) is the linear-SVM normal. All six vectors are unit-normalized, decoupling direction from intervention strength. F.3 Steering Objective and Two-Stage Search Let V(α)V(α) denote the fraction of steered forward passes yielding a parsable integer score in [1,10][1,10], and ρS(α) _S(α) the Spearman rank correlation between steered scores and the unperturbed baseline scores on the dev set. The optimal intervention strength solves α∗ α^* =argmaxα∈ℱ⁡W1(s(h→l′(α)),starget), \;=\; *arg\,max_α \;W_1\! (s( h _l(α)),\,s_target ), (5) ℱ =α:V(α)≥0.93,ρS(α)≥ρStext, = \α:V(α)≥ 93,\; _S(α)≥ _S^text \, where starget=s(h→l)s_target=s( h_l) for attacks (maximize the Wasserstein shift from clean baseline) and starget=s(h→lbase)s_target=s( h_l^base) for defenses (minimize distance to the unperturbed baseline score). The ρS _S constraint requires activation steering to be at least as rank-preserving as a text-level perturbation with comparable semantic intent. An exhaustive grid search over α would require thousands of forward passes per (l,v→bias,bias type)(l, v_bias,bias type) triple. Empirically, W1(α)W_1(α) is monotonically non-decreasing in |α||α| while V(α)V(α) and ρS(α) _S(α) are monotonically non-increasing (Table˜18), so α∗α^* lies at the boundary of the feasible region: αbd=sup|α|:α∈ℱ. _bd\;=\; \|α|:α \. (6) We exploit this with a two-stage procedure: 1. Boundary identification. Starting from α0 _0, we double α←2α← 2α until a constraint is first violated, then binary-search the bracket to tolerance ϵε. This costs (log⁡αmax+log⁡(1/ϵ))O( _max+ (1/ε)) evaluations. 2. Simulated-annealing refinement. Within [αbd−Δ,αbd][ _bd- ,\, _bd], a low-temperature SA search with energy E(α)=−W1(α)+M⋅[α∉ℱ]E(α)=-W_1(α)+M·1[α ] for large M escapes plateaus induced by the discrete integer-score output and the occasional non-monotonicity of ρS(α) _S(α) near the boundary. The total search costs on the order of 100100 forward passes per configuration, two orders of magnitude cheaper than a dense grid, and reliably lands within 2%2\% of the best α identified by exhaustive search on a subsample. Pseudocode is given as Algorithm˜1; concrete hyperparameter values (α0 _0, ϵε, Δ , T0T_0, cooling factor, M) are in Appendix˜G. Appendix G Implementation Details This appendix lists the implementation choices needed to reproduce the experiments—hardware, software, hyperparameters, and statistical protocols—grouped by experimental stage. Hardware and software. Open-source-judge experiments run on a single node with 4× NVIDIA A100 80 GB GPUs and 256 GB of host RAM. Inference uses transformers 4.46 with HuggingFace checkpoints, torch 2.4 on CUDA 12.4, and flash-attn 2.6 for the larger judges. Proprietary judges (GPT-4.1, GPT-4o-Mini, Deepseek-V3) are queried through their respective APIs with temperature=0. Total compute for the full set of activation extractions, α-searches, and outcome-predictor training is approximately 1,400 A100-hours. Activation extraction. Open-source judges run in bfloat16 for the forward pass, and we cast the final-token hidden state at every decoder layer to float32 before writing it to a memory-mapped .npy file. Each input is processed independently—no batching across questions—so that bias-attribute interactions cannot leak via padding masks. The pooling strategy is last-token: the token immediately preceding the integer-score completion. We also experimented with mean-pooling over the full prompt and observed qualitatively identical geometry but noisier direction estimates. Bias-vector estimators. The mean, geometric median, and PCA estimators are computed in NumPy on the centered effective-bias vectors Δh→l(x)\ h_l(x)\. The geometric median uses Weiszfeld iteration with ϵ=10−6ε=10^-6 and a 200-iteration cap; PCA uses truncated SVD on the centered matrix. LDA and SVM use scikit-learn (LinearDiscriminantAnalysis with the lsqr solver and automatic shrinkage; LinearSVC with squared-hinge loss and C=1C=1). The discriminative classifier is logistic regression with L2L_2 penalty (C=1C=1) and balanced class weights, trained to separate ℋbaseH_base from ℋfarH_far. All six vectors are unit-normalized before any analysis or intervention, so that intervention strength and direction are decoupled. Two-stage α search. Stage 1 (boundary identification) starts at α0=1 _0=1, doubles until either the validity or the Spearman constraint of Equation˜5 is violated, and then binary-searches the resulting bracket to tolerance ϵ=0.1ε=0.1. Stage 2 (simulated annealing) draws candidates uniformly from [αbd−5,αbd][ _bd-5, _bd] with initial temperature T0=1T_0=1, cooling factor γ=0.95γ=0.95, and floor Tmin=10−3T_ =10^-3; the feasibility penalty is M=103M=10^3. The cap of 200 forward passes per (l,v→,bias type)(l, v,bias type) triple is roughly two orders of magnitude below the cost of a dense grid over α∈[−100,100]α∈[-100,100] at step 0.01. Detector training. Features are computed per layer (raw K=24K=24 scalar features per layer for Llama-3.1-8B, total raw dimension L⋅K=768L· K=768; a near-zero-variance pre-filter reduces the active pool to approximately 450 features, and RFE further prunes to roughly 120 retained features). The model is lightgbm 4.3 with search ranges num_leaves ∈[31,127]∈[31,127], min_child_samples ∈[10,40]∈[10,40], learning_rate ∈[0.01,0.1]∈[0.01,0.1], and up to 800 boosting rounds with early-stopping patience 30. RFE removes features in groups of 32 until development AUC plateaus; Optuna then runs 200 trials over the remaining hyperparameters with a 5-fold cross-validated AUC objective. The ≈1:3≈ 1:3 class imbalance is handled via per-sample reweighting rather than oversampling. Statistical tests. Asymmetry tests are computed via a stratified paired bootstrap (10410^4 resamples) on the per-sample score differences, with strata defined by (bias type, source benchmark). For per-type rows the bootstrap is over the ≈4,500≈ 4,500 per-bias paired differences; for aggregate rows (Tables˜11 and 2) the bootstrap resamples within each (type, benchmark) stratum and then averages across the seven types to form the aggregate test statistic. The reported p-value is the two-sided fraction of resamples whose aggregate mean has opposite sign from the observed value; this fraction is naturally large for small aggregate effects with high cross-type variance (the GPT-4.1 strict-non-CoT positive row sits at Δs¯=+0.020 s=+0.020 but six of seven types contribute small or negative shifts, giving p=0.838p=0.838); the empirical resample floor of the bootstrap is 1/B=10−41/B=10^-4 when all 10410^4 resamples agree in sign with the observed. Table cells reporting more extreme tail values (<10−5<10^-5, <10−7<10^-7, <10−8<10^-8) come from a Gaussian-tail extrapolation of the bootstrap test statistic beyond the empirical resample range, using the resample-distribution mean and standard deviation to fit an asymptotic-normal tail; the displayed values are conservatively rounded toward larger p. Benjamini–Hochberg correction at q=0.05q=0.05 is applied over the 4configs×2polarities=84\,configs× 2\,polarities=8 aggregate-row comparisons per judge and separately over the 7biases×2polarities=147\,biases× 2\,polarities=14 per-type-per-judge comparisons. The main-text p<10−189p<10^-189 value cited around Figure˜3 on Llama-3.1-8B uses Welch’s two-sample t-test on the pooled per-sample paired differences (n=31,500n=31,500 per polarity); the bootstrap-based aggregate rows in the tables are reported separately because they better capture the joint-across-types pattern that the asymmetric finding rests on. Effect sizes are Cohen’s d with pooled SDs. Distribution shifts are reported as the 1-Wasserstein distance W1W_1 on empirical integer-score distributions; rank preservation is Spearman’s ρS _S against unperturbed baseline scores. Confidence intervals, where shown, are 95% bootstrap with 10,000 resamples. Reproducibility. All experiments use seed 42. The 45/15/40 split is fixed at the question-ID level and stored on disk, so partitions never depend on a random call at inference time. Code, configuration files, and the integer-score outputs of every judge will be released under MIT; raw activations exceed 200 GB and will be regeneratable from the released code rather than redistributed. An EMNLP-style reproducibility checklist accompanies the submission. G.1 Error Analysis: Generation Failures and Score Parsing Beyond the validity rates reported alongside our steering experiments (Tables˜19 and 14), it is useful to characterize where the small remaining fraction of failed generations comes from when judges produce free-form integer-score outputs. We report (a) the configured maximum generation length, (b) the truncation rate, (c) the answer-extraction failure rate, and (d) a breakdown of failure causes for the seven judges under the default strict, non-CoT configuration. All numbers below are on the test split. Maximum generation length. For the strict, non-CoT configuration, judges are asked to output a single integer score and we set the maximum new-tokens cap to 16 tokens, sufficient for variants like “Score: 8”, “8”, or “8” (with leading whitespace and an end-of-text token). For the CoT configurations, we use a 512-token cap to accommodate step-by-step reasoning before the score. Both caps sit above the 99th percentile of observed generation lengths across all seven judges, so length truncation is a rare event rather than a systematic effect. Truncation and extraction. We parse the integer score by regex-matching ([1-9]|10) immediately following “Score:” (or at the start of the generation, for the non-CoT configurations whose templates omit the prefix). Cases where no integer is found, the matched integer falls outside [1,10][1,10], or multiple integers appear without a unique anchor are counted as extraction failures and excluded from downstream analysis after being logged. Table˜10 reports the truncation rate (fraction of generations that reach the cap before an end-of-sequence token) and the extraction failure rate (fraction of generations from which no valid score can be parsed) for each judge. Error type breakdown. Among the extraction failures, the dominant modes are: (i) Verbal scoring, where the model writes a verbal grade (“Score: high” or “well done”) instead of an integer; (i) Out-of-range, where the model emits an integer such as 0 or 1111 that we treat as invalid; (i) Range/interval, where the model writes “7–8” or “around 7” instead of a single integer; and (iv) Truncated before score, where a CoT chain exceeds the length cap and the score token is never produced. Table˜10 shows the relative frequency of each mode (rows sum to 100%). Verbal scoring and truncated-before-score together account for over half of all failures on every judge. Table 10: Generation-failure analysis for the seven judges under the strict, non-CoT default configuration on the test split. Truncation rate is the fraction of generations reaching the length cap; extraction failure rate is the fraction from which no valid integer score in [1,10][1,10] can be parsed. The four right-hand columns decompose the extraction-failure mode and sum to 100%100\% per row. Judge Max-tokens Truncation Extraction Error-mode breakdown of extraction failures (%) cap rate failure rate Verbal Out-of-range Range/interval Trunc-before-score Llama-3.1-8B 16 0.8%0.8\% 2.6%2.6\% 3131 2424 1414 3131 Qwen3-14B 16 0.4%0.4\% 1.8%1.8\% 2222 1919 1212 4747 Gemma-3-12B 16 0.6%0.6\% 2.1%2.1\% 2828 2121 1111 4040 GPT-4o-Mini 16 0.2%0.2\% 1.1%1.1\% 3535 2828 1919 1818 Llama-3.3-70B 16 0.3%0.3\% 1.4%1.4\% 2727 2525 1616 3232 Deepseek-V3 16 0.5%0.5\% 1.6%1.6\% 3030 2222 1414 3434 GPT-4.1 16 0.1%0.1\% 0.9%0.9\% 4242 2626 2121 1111 Implications for downstream analyses. All steering and detection results in the main text and appendix exclude extraction failures from the per-sample comparisons. Because the failure rates are below 3%3\% for every judge and the failure-mode distribution is similar across baseline and perturbed inputs (within ±1± 1\,p per mode), excluding failures does not systematically bias the W1W_1 or Spearman comparisons. The truncation rate of 0.10.1–0.8%0.8\% further bounds how much of the asymmetric-response finding could be attributable to length-cap artifacts: even if every truncated generation were treated as a maximally biased score, the aggregate asymmetry magnitudes in Table˜2 would shift by less than 0.020.02 absolute, well below the reported effect sizes. Appendix H Limitations and Scope Our study’s claims are bounded by the following design choices; each is accompanied by an appendix analysis that bounds its residual effect on the headline findings. Activation-analysis judge coverage. The activation-level analyses are conducted on three open-source mid-scale judges (Llama-3.1-8B, Qwen3-14B, Gemma-3-12B) for which we hold white-box access. The behavioral asymmetry replicates across all seven judges (Table˜2), and the cross-architecture cosine of bias-direction estimators across the three white-box judges (Table˜12, 0.470.47–0.620.62 above the random-pair baseline of ≈0≈ 0 with std 0.0310.031) is consistent with a shared bias subspace across the model families we can probe. Effective-bias substrate definition. The substrate eff±D_eff^± on which the bias direction is estimated is defined by outcome—paired instances whose score shift exceeds δs _s integer points (Section˜3.3). This case-control framing is intrinsic to the phenomenon: if a surface cue does not move the judge’s score, there is no scoring failure to attribute to a hidden-state direction. Samples in neg∖eff−D_neg _eff^- are null observations of the phenomenon rather than negative examples of it. The cross-validated defense (Section˜K.1) and the cross-domain predictor (Section˜4.5, evaluated on benchmarks absent from vector estimation) further test the subspace on inputs outside the fitting substrate. Surface-cue operationalization. The seven bias types we study—peer-consensus claims, citation-status markers, source attributions, social-identity attributions, length, tone, and metacognitive claims—follow the established convention in the LLM-as-judge bias literature, where appended notes of this form are treated as bias-evaluation cues regardless of their nominal content interpretation [Wang et al., 2024a, Saito et al., 2023, Panickssery et al., 2024a, Koo et al., 2024, Ye et al., 2025, Dubois et al., 2024, Li et al., 2024c, Chen et al., 2024, Thakur et al., 2025, Stureborg et al., 2024, Gu et al., 2024, Zeng et al., 2024]. Adopting this convention makes our results directly comparable to prior fairness audits using the same operationalization, and lets the bias subspace recovered here be referenced against the cue catalog those works document. Bit-identical-body perturbations are further validated by the TOST equivalence-test extended human evaluation of Section˜C.1, which establishes that human raters do not change their quality assessment based on these markers; the matched-budget text-attack comparison of Section˜J.3 confirms that the activation advantage on W1W_1 is not an artifact of the standard-strength text baseline. Cross-domain outcome-prediction sensitivity. The headline cross-domain AUC of 0.820.82 for the linear projection is reported on three benchmarks (SocialMaze, BBQ, GPQA) held out entirely from training; the alternative-trio sensitivity analysis of Section˜M.7 bounds the cross-domain AUC within approximately ±0.025± 0.025 of the headline value across five alternative trio choices. The operational claim is anchored to the simpler linear projection rather than to the GBDT pipeline, which wins in-domain (dev AUC 0.930.93) but loses to the linear projection on the unseen-domain split. Defense feasibility constraint design. The main-text defense reports W1W_1-reduction at a Spearman floor anchored to the text-rewrite baseline. The absolute-floor robustness analysis of Section˜K.2 replaces this with ρS≥0.85 _S≥ 0.85 as an absolute constraint independent of the text baseline, and the 44–6×6× qualitative advantage of activation defense over text-rewrite defense survives the stricter feasibility design. White-box access for activation intervention. The activation-steering defense operates on hidden states and therefore requires white-box access; for proprietary API judges the same mitigation would have to be approximated by prompt-level proxies. The cross-domain outcome predictor of Section˜4.5, which uses pre-deployment per-judge activation features but produces an input-level prediction, covers the detection setting in a gray-box regime where activations are available at characterization time but not at deployment. Intended use and release. The activation-steering code is an analytical and defensive tool for diagnosing bias and constructing fairness-oriented counterfactuals. We release only the analysis and outcome-prediction pipelines in the accompanying artifact, deferring any attack-oriented release to a controlled responsible-disclosure process. Appendix I Cross-Model Replication I.1 Asymmetric Response Across Models The asymmetric response of GPT-4.1 across the four prompt configurations referenced in Figure˜1 is reported in Table˜11. Negative perturbations consistently produce large, statistically significant score drops (Cohen’s d∈[−0.58,−0.35]d∈[-0.58,-0.35], all p<10−3p<10^-3), while positive perturbations yield negligible shifts (d≈0d≈ 0, no significant p-values). The pattern persists regardless of whether CoT or strict scoring is used, ruling out prompting artifacts; the strict configurations lower the baseline mean from ∼8.5 8.5 to ∼5.5 5.5, eliminating the possibility that the absence of positive inflation is merely a ceiling effect. Table 11: Asymmetric scoring response to positive and negative perturbations across the four prompt configurations for GPT-4.1. We report the baseline mean score (s¯base s_base), mean score difference (Δs¯ s), Cohen’s d, t-test p-value, and 1-Wasserstein distance (W1W_1). Negative perturbations consistently induce a substantial penalty, while the aggregate positive effect stays near zero (the per-type breakdown in Table˜4 shows Refinement++ alone produces clear inflation, with the remaining six positive types near zero). Config. s¯base s_base Positive Bias (posD_pos vs. baseD_base) Negative Bias (negD_neg vs. baseD_base) CoT Strict Δs¯ s Cohen’s d p-value W1W_1 Dist. Δs¯ s Cohen’s d p-value W1W_1 Dist. 8.71 -0.055 -0.043 0.669 0.085 -0.855 -0.582 <10−8<10^-8 0.855 8.46 -0.010 -0.008 0.940 0.080 -0.745 -0.486 <10−5<10^-5 0.755 5.72 +0.025 +0.032 0.750 0.095 -0.350 -0.382 <10−3<10^-3 0.370 5.27 +0.020 +0.021 0.838 0.060 -0.360 -0.346 <10−3<10^-3 0.370 The all-model asymmetry panel (Table˜2) is reported in Figure˜1. Its positive-aggregate column averages the five score-inflating positive types (Prestige++, Verbosity++, Authority++, Sentiment++, Refinement++), so the Llama-3.1-8B entry reads +0.243+0.243; the full-pool aggregate of +0.07+0.07 in Figure˜1 instead averages all seven positive types, including the two sign-inverting cases (Bandwagon++ at −0.46-0.46, Diversity++ at −0.19-0.19). The panel is computed on the nested test split (2,700 questions per judge), so the Llama-3.1-8B baseline mean of 5.715.71 there differs from the full-pool 5.845.84; the qualitative asymmetry is invariant to both conventions. Figure˜3 shows the per-model score distributions under the strict, non-CoT configuration. Across all models, the positive perturbation distribution nearly overlaps with the baseline, while the negative perturbation distribution shifts markedly leftward. The magnitude of the shift roughly tracks model capability band: the three mid-scale open-source judges (Llama-3.1-8B, Qwen3-14B, Gemma-3-12B) exhibit the largest separations as a group, followed by GPT-4o-Mini, and the frontier models (Llama-3.3-70B, Deepseek-V3, GPT-4.1) show smaller—though still statistically significant—shifts. (a) Llama-3.1-8B (b) GPT-4.1 (c) GPT-4o-Mini (d) Llama-3.3-70B (e) Deepseek-V3 (f) Qwen3-14B (g) Gemma-3-12B Figure 3: Score distributions under the strict, non-CoT configuration for all seven judges. In every case, the negative perturbation distribution shifts substantially leftward, while the positive distribution is largely unchanged. I.2 Geometric Separation Across Models The type-specific geometric separation reported in Section˜4.3 (with the Llama-3.1-8B cross-polarity case shown in Figure˜2) generalizes across architectures. Figures˜4 and 5 present the same Refinement++ vs. Diversity−- MDS projections on Qwen3-14B and Gemma-3-12B, with the same progressive separation from layer 5 to layer 25, indicating that type-specific geometric signatures are a general property of LLM judge activations rather than an architecture-specific artifact. Figure 4: MDS projection of per-sample effective bias vectors for Refinement+ and Diversity- at layers 5, 15, and 25 of Qwen3-14B. The progressive separation mirrors that observed in Llama-3.1-8B (Figure˜2). Figure 5: MDS projection of per-sample effective bias vectors for Refinement+ and Diversity- at layers 5, 15, and 25 of Gemma-3-12B. I.3 Cross-Architecture Bias-Direction Cosine The MDS projections of Figures˜4 and 5 establish qualitatively that the same Refinement++ vs. Diversity−- separation appears across architectures, but visual inspection cannot quantify how much of the bias direction itself transfers between models. We complement the visualizations with a direct cosine comparison: for each of three representative bias types we extract the Classifier vector v→CLS(25) v_CLS^(25) at layer 25 of every model. Because the three judges have different hidden dimensionalities (dL=4096d_L=4096 for Llama-3.1-8B, dQ=5120d_Q=5120 for Qwen3-14B, dG=3840d_G=3840 for Gemma-3-12B), direct cosine between native-space vectors is undefined. We construct a shared probing space in two steps. First, each model’s activations are projected to ℝ1024R^1024 via a per-model random orthonormal matrix Pm∈ℝ1024×dmP_m ^1024× d_m (drawn once at a fixed seed and held constant). Second, on a sub-pool of 500 tokenization-matched questions drawn from MMLU and PubMedQA (where ASCII-only English answers tokenize identically across all three tokenizers), we apply iterative orthogonal Procrustes (10 iterations, alternating pairwise) to rotate the three projected baseline activation clouds into mutual alignment, yielding three rotation matrices Rm∈ℝ1024×1024R_m ^1024× 1024. Each bias vector v→CLS(25) v_CLS^(25) is then mapped to the shared space as RmPmv→CLS(25)R_mP_m v_CLS^(25), re-normalized, and sign-aligned by maximizing pairwise cosine across the three models before computing pairwise cosine. Within-model bootstrap cosines (under the same shared-space projection and rotation) exceed 0.91 across all three bias types and are reported separately in Table˜17. Table 12: Cross-architecture cosine similarity of the layer-25 Classifier bias direction v→CLS(25) v_CLS^(25) across Llama-3.1-8B (L), Qwen3-14B (Q), and Gemma-3-12B (G), computed in a shared 1024-dimensional probing basis trained by Procrustes alignment on 500 tokenization-matched baseline activations. Within-model values exceed 0.91 (Table˜17); between-model values are reported here. Random pairs of unit vectors in the shared basis have [cos]≈0E[ ]≈ 0 with standard deviation ≈0.031≈ 0.031, so all entries below are over 14 standard deviations above chance. Bias type L ↔ Q L ↔ G Q ↔ G Bandwagon−- 0.58 0.51 0.55 Refinement++ 0.62 0.55 0.59 Authority−- 0.51 0.47 0.49 Cross-architecture cosines occupy the band [0.47,0.62][0.47,0.62], well above the chance baseline of 0 for random unit vectors in ℝ4096R^4096 to ℝ5120R^5120, yet substantially below the within-model agreement of ≥0.91≥ 0.91 between independent estimators on the same architecture (Table˜17). The interpretation is partial transfer rather than universality: the three architectures share a common projection of each bias type onto a low-dimensional substrate, but each model also rotates that substrate into its own coordinate frame. The ordering of the three bias types is preserved across all three pairs (Refinement++ >> Bandwagon−- >> Authority−-), suggesting the magnitude of transfer is a property of the bias type rather than of the architecture pair. The practical consequence is that defenses constructed from a single donor model should expect a noticeably attenuated, though still non-trivial, effective range on a different architecture; cross-model defense transfer is plausible at degraded efficacy and not as a drop-in substitute. Appendix J Complete Attack Results This appendix provides the full materials supporting Section˜4.4: the headline attack/defense summary (Table˜13), the bias-vector candidate analysis (Section˜J.3), the two-stage α-optimization algorithm in pseudocode (Section˜J.4), the empirical monotonicity of our three optimization metrics (Section˜J.5), and per-layer, per-method attack tables for the six bias types covering the strongest negative and positive perturbations (Section˜J.6). Table 13: Best activation steering for attack and defense on Llama-3.1-8B, test set. “Atk-W1W_1” is shift magnitude (higher = stronger attack); “Def-W1↓W_1\! ” is the fractional reduction in pre-defense Wasserstein distance (higher = better recovery); ρS _S is Spearman rank correlation with baseline. Activation interventions strictly improve over text-level baselines on both tasks. Full per-layer, per-method tables in Sections˜J.6 and K. Bias Method Atk-W1W_1 ↑ ρS _S ↑ Def-W1↓W_1\! ↑ ρS _S ↑ Auth.−- Text 0.78 0.70 0.11 0.74 Cls. 1.62 0.78 0.62 0.81 Bandw.−- Text 1.66 0.51 0.09 0.56 PCA 3.43 0.58 0.61 0.67 Refin.−- Text 0.59 0.76 0.15 0.77 PCA/Cls.† 1.13 0.80 0.52 0.83 †Attack: PCA; Defense: Classifier. J.1 Random-Direction Control A natural alternative explanation for the attack results in Table˜19 is that any sufficiently large perturbation of the hidden state degrades scoring, in which case our results would speak to the brittleness of the readout rather than the specificity of v→bias v_bias. To rule this out, for each (bias,l,α∗)(bias,l,α^*) triple in Table˜19 we replace the bias direction with a random unit vector drawn uniformly on the unit sphere of the same hidden dimensionality, leaving the layer and intervention strength unchanged. We repeat the random draw 5 times per triple and report the mean and maximum across draws. The paired comparison is therefore strictly within the same (l,α∗)(l,α^*) regime that produced the headline attack numbers, so any discrepancy in W1W_1 cannot be attributed to a difference in injected magnitude. Table 14: Random-direction control for the six attack triples reported in Table˜19. “Bias-vec W1W_1” is the result with the calibrated v→bias(l) v_bias^(l); “Rand mean / max W1W_1” are the mean and maximum across 5 random unit-sphere draws at the same (l,α∗)(l,α^*). The “Rand ρS _S” column averages Spearman correlation across draws. The two-tailed p-value is from a paired t-test on the per-sample score differences between the bias-vec injection and the random-direction injection. Random-direction validity stays ≥0.99≥ 0.99 across all 30 draws. Bias type Bias-vec W1W_1 Rand mean W1W_1 Rand max W1W_1 Rand ρS _S Paired p Authority−- 1.62 0.21 0.34 0.96 <10−12<10^-12 Bandwagon−- 3.43 0.31 0.49 0.95 <10−15<10^-15 Diversity−- 2.65 0.28 0.46 0.96 <10−12<10^-12 Refinement−- 1.13 0.18 0.29 0.97 <10−10<10^-10 Verbosity−- 1.46 0.19 0.32 0.97 <10−11<10^-11 Refinement++ 0.85 0.14 0.25 0.98 <10−9<10^-9 Two observations follow. First, the ratio of bias-vector W1W_1 to random-direction mean W1W_1 ranges from 5.9×5.9× (Refinement++) to 11.1×11.1× (Bandwagon−-), a gap that holds even when comparing against the maximum across five random draws (3.4×3.4× to 7.0×7.0×). Second, the random-direction interventions barely move the Spearman correlation with the unperturbed baseline (ρS≥0.95 _S≥ 0.95 in every row), as opposed to the calibrated bias direction whose ρS _S drops into the 0.580.58 to 0.810.81 range. The bias direction is therefore not a generic perturbation amplifier but a direction-specific intervention site identifiable through activation geometry; a randomly oriented displacement of the same magnitude leaves the score distribution essentially intact. J.2 Bias-Type Swap Control A random direction in ℝdR^d is almost orthogonal to the score-readout direction by construction (expected cosine ≈1/d≈ 1/ d for d∈[4096,5120]d∈[4096,5120]), so the random-direction control of Section˜J.1 cleanly rules out the “any sufficiently large perturbation degrades scoring” alternative but does not by itself distinguish “the bias direction is the score-readout direction” from “the bias direction encodes bias-type-specific information.” To bound this second alternative, we replace v→bias(l) v_bias^(l) at a fixed (bias type,l,α∗)(bias type,l,α^*) triple with the bias direction of a different bias type (matched norm), and rerun the attack on the original bias type’s effective samples. If the geometry were purely a score-readout artifact, swapped vectors would shift scores by the same amount; if the geometry is type-specific, swapped vectors would underperform. Table 15: Bias-type swap control on Llama-3.1-8B, layer 31, PCA direction. “Within-type” reproduces the best-method numbers from Table˜19 for reference. “Swap mean” averages across the three cross-type substitutions (target’s vector replaced by any one of the three other bias types’ vectors in this panel at matched norm, then averaged); “Random mean” is the matched-norm random-direction control from Table˜14. Swap directions sit between random and within-type, indicating a shared low-dimensional subspace plus a type-specific component. Bias Within-type W1W_1 Swap mean W1W_1 Random mean W1W_1 Authority−- 1.62 0.71 0.21 Bandwagon−- 3.43 1.38 0.31 Diversity−- 2.65 1.04 0.28 Refinement−- 1.13 0.51 0.18 Swap-direction W1W_1 shifts sit in [0.51,1.38][0.51,1.38], well above the random-direction values [0.18,0.31][0.18,0.31] (consistent with a shared low-dimensional bias subspace that all bias types partly occupy) and well below the within-type values [1.13,3.43][1.13,3.43] (consistent with each bias type carving its own direction within that shared subspace). The ratios within-type / swap range from 2.0×2.0× (Refinement−-) to 2.5×2.5× (Bandwagon−-); the ratios swap / random range from 2.8×2.8× (Refinement−-) to 4.5×4.5× (Bandwagon−-). These two ratios together place the bias geometry on an interpretable spectrum: a generic readout-sensitive direction would produce equal swap and within-type effects, whereas a fully type-orthogonal subspace would produce swap and random effects at similar magnitude. Neither extreme holds, and the intermediate position is the substantive geometric finding. J.3 Matched-Budget Text-Attack Comparison The text-attack baseline reported in Table˜19 applies each perturbation template at its canonical strength (the standard percentage levels and citation-marker densities used in prior LLM-as-judge bias literature). The activation attack, by contrast, optimizes α via simulated annealing subject to the Spearman feasibility constraint ρS(α)≥ρStext _S(α)≥ _S^text. A potential concern is that the comparison underestimates what text-attack could achieve if it were also allowed to search over perturbation strength. To address this, we re-run the text-attack with a matched search budget. For each bias type we sweep three perturbation-strength axes: (1) numerical intensity, varying the percentage figure in Bandwagon and the credibility-marker count in Authority over five levels each; (2) stacked multi-attribute application, applying multiple within-type markers to the same input (e.g., two distinct “[citation needed]” markers placed in different sentence positions); (3) prose-rewrite aggressiveness for the three LLM-assisted types, varying the rewrite temperature and the number of inserted hedging/loaded-tone constructions. For each (bias, strength) combination we measure W1W_1 and ρS _S, then select the W1W_1-maximizing strength subject to ρS≥ρSactivation _S≥ _S^activation for direct comparability with the activation attack on a matched-rank-faithfulness budget. Table 16: Matched-budget text-attack vs activation attack on Llama-3.1-8B. Text-attack columns: W1W_1 at canonical-strength text perturbation (left, reproducing Table˜19) and at matched-ρS _S search-optimized strength (centre). Activation columns: best-method W1W_1 from Table˜19. Gap is computed against the matched-budget text-attack (the stricter comparison). Bias Type Text (mild) W1W_1 Text (matched-ρS _S) W1W_1 Activation (best) W1W_1 Gap Authority−- 0.780 (ρS=0.70 _S=0.70) 1.052 (ρS=0.78 _S=0.78) 1.618 (ρS=0.78 _S=0.78) +54%+54\% Bandwagon−- 1.664 (ρS=0.51 _S=0.51) 2.182 (ρS=0.58 _S=0.58) 3.426 (ρS=0.58 _S=0.58) +57%+57\% Diversity−- 1.452 (ρS=0.61 _S=0.61) 1.724 (ρS=0.64 _S=0.64) 2.654 (ρS=0.64 _S=0.64) +54%+54\% Refinement−- 0.594 (ρS=0.76 _S=0.76) 0.781 (ρS=0.80 _S=0.80) 1.132 (ρS=0.80 _S=0.80) +45%+45\% Verbosity−- 0.612 (ρS=0.72 _S=0.72) 0.913 (ρS=0.70 _S=0.70) 1.456 (ρS=0.70 _S=0.70) +60%+60\% Refinement++ 0.382 (ρS=0.81 _S=0.81) 0.548 (ρS=0.79 _S=0.79) 0.852 (ρS=0.79 _S=0.79) +55%+55\% The matched-budget text-attack column reflects a substantial improvement over the mild text-attack baseline (+30+30 to +70%+70\% W1W_1 across types), confirming that text-attack does have headroom under the standard input-output threat model. But the activation attack retains a 4545–60%60\% advantage on W1W_1 at the same ρS _S floor across all six bias types, indicating that the qualitative superiority of activation steering survives the matched-budget comparison. The gap is smallest on Refinement−-, where the text-rewrite version already approaches the activation efficacy under the rank-faithfulness constraint, and largest on Verbosity−- and Bandwagon−-, where the discrete prose-perturbation cannot match the continuous-direction precision of activation steering. We extracted six candidate bias vectors—three directional-change (Mean, Geometric Median, PCA) and three discriminative-boundary (LDA, Classifier, SVM)—from the training set of Llama-3.1-8B for each bias type. Table˜17 reports the pairwise cosine similarity averaged across Authority−-, Bandwagon−-, and Refinement−- at layer 25. Table 17: Mean pairwise cosine similarity between the six candidate bias vectors at layer 25 of Llama-3.1-8B, averaged across three negative bias types. Within each family (shaded blocks on the diagonal), agreement is high; cross-family agreement is noticeably lower, motivating our three-representative selection. Mean Geom. Med. PCA LDA Classifier SVM Mean – 0.98 0.94 0.58 0.62 0.59 Geom. Median 0.98 – 0.93 0.61 0.65 0.62 PCA 0.94 0.93 – 0.55 0.60 0.57 LDA 0.58 0.61 0.55 – 0.95 0.91 Classifier 0.62 0.65 0.60 0.95 – 0.94 SVM 0.59 0.62 0.57 0.91 0.94 – The block structure confirms the qualitative description in the main text: within-family cosine similarity is ≥0.91≥ 0.91, while cross-family cosine similarity sits in [0.55,0.65][0.55,0.65]—tight enough that the two families share a coarse direction, but loose enough to suggest they capture distinct notions of bias direction. We therefore keep three representatives that span this geometry: Geometric Median (robust directional summary), PCA (principal directional axis), and the Classifier vector (a discriminative-family stand-in for LDA and SVM, which agree with it at cos≥0.94 ≥ 0.94). J.4 Two-Stage α-Optimization Algorithm Algorithm˜1 pinpoints the optimal intervention strength under the feasibility set ℱF defined in Equation˜5. Stage 1 exploits the monotonicity of V(α)V(α) and ρS(α) _S(α) to locate the boundary αbd _bd in (log⁡αmax+log⁡(1/ϵ))O( _ + (1/ε)) evaluations. Stage 2 refines by simulated annealing within a small neighborhood of αbd _bd, which is necessary because the integer-score output makes W1(α)W_1(α) piecewise-constant and induces local plateaus that a pure greedy search would get stuck on. Algorithm 1 Two-stage search for optimal activation-steering strength. 0: Bias vector v→bias(l) v_bias^(l), initial step α0>0 _0>0, tolerance ϵε, neighborhood Δ , SA schedule (T0,Tmin,γ)(T_0,T_ ,γ) 0: Optimal strength α∗∈ℱα^* maximizing W1W_1 1: // Stage 1: Boundary identification 2: αlo←0 _lo← 0, αhi←α0 _hi← _0 3: while αhi∈ℱ _hi do 4: αlo←αhi _lo← _hi, αhi←2αhi _hi← 2 _hi exponential expansion 5: end while 6: while αhi−αlo>ϵ _hi- _lo>ε do 7: αmid←(αlo+αhi)/2 _mid←( _lo+ _hi)/2 8: if αmid∈ℱ _mid then 9: αlo←αmid _lo← _mid 10: else 11: αhi←αmid _hi← _mid 12: end if 13: end while 14: αbd←αlo _bd← _lo 15: // Stage 2: Simulated-annealing refinement 16: αbest←αbd _best← _bd, W∗←W1(αbd)W^*← W_1( _bd), T←T0T← T_0 17: while T>TminT>T_ do 18: Sample α′∼(αbd−Δ,αbd)α ( _bd- ,\, _bd) 19: E′←−W1(α′)+M⋅[α′∉ℱ]E ←-W_1(α )+M·1[α ] 20: if E′<−W∗E <-W^* or exp⁡((−W∗−E′)/T)>u ((-W^*-E )/T)>u, u∼(0,1)u (0,1) then 21: αbest←α′ _best←α , W∗←−E′W^*←-E 22: end if 23: T←γT←γ T 24: end while 25: return αbest _best J.5 Monotonicity of the Optimization Metrics Table˜18 illustrates the empirical monotonicity that underpins Stage 1 of the algorithm: W1(α)W_1(α) is monotonically non-decreasing while V(α)V(α) and ρS(α) _S(α) are monotonically non-increasing in |α||α|, so the feasible boundary αbd _bd is well-defined. The table aggregates over all (l,v→bias,bias type)(l, v_bias,bias type) triples and reports the fraction that exhibit strict monotonicity along each axis. Table 18: Empirical monotonicity of the three optimization metrics with respect to |α||α| across all configurations on Llama-3.1-8B. Each row reports the fraction of (l,v→bias,bias type)(l, v_bias,bias type) triples that exhibit strictly monotone behavior along the stated direction. Metric Mono. Non-mono. Direction W1(α)W_1(α) (efficacy) 97% 3% non-decr. V(α)V(α) (validity) 99% 1% non-incr. ρS(α) _S(α) (Spearman) 93% 7% non-incr. J.6 Per-Layer, Per-Method Attack Tables We report the complete per-layer attack results for Llama-3.1-8B on the test set across six bias types—the five strongest negative biases (Authority−-, Bandwagon−-, Diversity−-, Refinement−-, Verbosity−-) and the strongest positive bias (Refinement++)—using our three retained vector methods (Geometric Median, PCA, Classifier) plus the text-attack baseline. Each row is the layer-optimal α∗α^* selected by Algorithm˜1 on the development set; we then evaluate on the test split. Table 19: Per-bias attack performance on Llama-3.1-8B, test set. Each block reports the four methods (text-attack baseline plus three activation methods); best per-metric values within each block in bold. Refinement++ is the only positive bias in the attack suite; its activation push is in the score-up direction. Bias Method W1↑W_1 ρS↑ _S Valid. Layer α∗α^* Authority−- Text Atk (baseline) 0.780 0.696 100% – – Geometric Median 1.214 0.716 93.8% 31 23.84 PCA 0.840 0.750 100% 15 8.94 Classifier 1.618 0.780 94.8% 30 62.36 Bandwagon−- Text Atk (baseline) 1.664 0.514 100% – – Geometric Median 1.333 0.708 99.3% 7 5.00 PCA 3.426 0.584 94.8% 31 75.75 Classifier 1.744 0.627 100% 15 5.70 Refinement−- Text Atk (baseline) 0.594 0.757 100% – – Geometric Median 0.485 0.805 100% 7 2.19 PCA 1.132 0.800 100% 31 35.53 Classifier 0.799 0.803 100% 18 7.70 Diversity−- Text Atk (baseline) 1.452 0.610 100% – – Geometric Median 1.892 0.732 98.2% 30 22.50 PCA 2.654 0.643 96.1% 31 58.20 Classifier 2.345 0.689 98.5% 28 33.00 Verbosity−- Text Atk (baseline) 0.612 0.723 100% – – Geometric Median 0.875 0.785 99.5% 25 12.40 PCA 1.456 0.701 95.8% 31 53.00 Classifier 1.122 0.748 98.0% 18 8.50 Refinement++ Text Atk (baseline) 0.382 0.812 100% – – Geometric Median 0.541 0.825 100% 22 8.80 PCA 0.852 0.788 98.6% 31 18.40 Classifier 0.673 0.815 100% 25 12.00 Cross-bias observations. Table˜19 surfaces four robust patterns, matching our summary in the main text. (1) Heterogeneous optimal strength: α∗α^* varies from 2.19 (Refinement−-, Geom. Med., layer 7) to 75.75 (Bandwagon−-, PCA, layer 31)—more than an order of magnitude—so a universal attack strength is infeasible. (2) Depth-dependent strength: the optimal layer for each method is concentrated at layer ≥15≥ 15, reflecting that middle-to-late layers carry the cleanest bias direction (consistent with Table˜24); positive Refinement is the lone exception, peaking in the mid-layers. (3) Method complementarity: Classifier vectors dominate the score-shift axis for Authority−-; PCA dominates for Bandwagon−-, Diversity−-, Refinement−-, Verbosity−-, and Refinement++; Geometric Median vectors yield the best ρS _S on five of the six biases, confirming them as the most ordinally gentle option. (4) Universal text-attack dominance: every activation method with its best (l,α∗)(l,α^*) setting beats the text baseline on at least one metric; the best activation attack beats it on both metrics for every bias type, including the positive Refinement++ case. J.7 Attack Behavior Across Layers The tables above report only the layer-optimal configuration for each (bias, method) pair. The figures below show how the underlying quantities—α∗α^*, the Wasserstein shift W1W_1, the output validity rate, and the Spearman correlation ρS _S—vary along the layer axis and across the three vector methods. Per-bias optimal α. Figure˜6 plots the optimal intervention strength α∗α^* as a function of layer for each of the six bias types, with one curve per vector method. Three patterns repeat across biases: (i) α∗α^* stays in the single digits up to roughly layer 15 and then climbs steeply; (i) PCA consistently demands the largest strength at the late layers, often two to three times larger than the Geometric Median; and (i) the very last layer (31) tends to require a smaller α∗α^* than layer 30 because the score-readout projection makes the model more sensitive to any displacement, tightening the Spearman constraint. Figure 6: Optimal intervention strength α∗α^* as a function of layer for each of the six bias types, separately for the three vector methods (Geometric Median, PCA, Classifier). Layer indices are sampled at 1,4,7,12,15,18,25,30,31\1,4,7,12,15,18,25,30,31\. PCA consistently requires the largest strength at the late layers; all methods drop at layer 31 because the score-readout projection tightens the Spearman constraint. Validity remains saturated. Figure˜7 reports the output validity rate of the attack (fraction of perturbed inputs whose decoded score is still an integer in 1,…,10\1,…,10\) across all bias-layer combinations. Validity sits in [0.96,1.00][0.96,1.00] everywhere. The binding constraint on α∗α^* is therefore the Spearman correlation, not generation integrity: the model still decodes a valid integer under strong activation steering, but the ordinal logic between answer quality and score breaks down before any malformed output appears. Figure 7: Output validity rate of the activation attack averaged over the three vector methods, for each (bias, layer) cell. All cells lie in [0.96,1.00][0.96,1.00]: malformed decodes are not the binding constraint, the Spearman constraint is. α∗α^* and W1W_1 across the layer stack. Figure˜8 summarizes the same data along the layer axis by averaging over methods. The top panel shows that all six biases share the same qualitative “hockey-stick” shape in α∗α^*, with Bandwagon−- requiring the largest strength at the late layers and Refinement++ the smallest. The bottom panel shows the corresponding score-shift: W1W_1 stays in [0.4,0.7][0.4,0.7] across the middle layers, then jumps at layer 31 for the strongest biases (Bandwagon−- to 1.751.75). The narrow band of mid-layer W1W_1 confirms that the bias signal is broadly distributed in depth, not concentrated at a single “critical” layer. Figure 8: Average optimal α (top) and average Wasserstein distance W1W_1 (bottom) across the layer stack, with one curve per bias type. Values are averaged over the three vector methods. All six biases share the same hockey-stick shape in α∗α^*, with the steepest climb at layers 30–31. Method-level behavior. Figure˜9 averages instead over biases and separates the three vector methods. PCA needs the largest α but also produces the largest W1W_1, especially at the deep layers; Classifier vectors sit in the middle on both axes; Geometric Median is the gentlest, with the smallest α, the smallest W1W_1, and the highest Spearman correlation. Together with the per-bias plots above, this is why we keep all three representatives in Section˜4.4: the three methods sit at distinct points in the (strength, faithfulness) plane rather than along a single axis. Figure 9: Average optimal α, average Wasserstein distance W1W_1, and average Spearman correlation ρS _S as a function of layer, separately for the three vector methods (averaged over the six bias types). PCA produces the strongest attacks at the cost of the lowest ordinal faithfulness; Geometric Median is the gentlest on both axes; Classifier sits in between. Appendix K Defense Results Mirroring the attack setup, the defense applies h→l′=h→l−αv→bias(l) h _l= h_l-α\, v_bias^(l) to inputs in eff−D_eff^- (effective negative-bias samples), aiming to restore the unperturbed baseline score. Optimization reuses Algorithm˜1 with the target score changed from baseline to biased, i.e. we maximize W1W_1-reduction ΔW1def(α)= 1−W1(s(h→l′(α)),s(h→lbase))W1(s(h→l),s(h→lbase)), W_1^def(α)\;=\;1\;-\; W_1\! (s( h _l(α)),\,s( h_l^base) )W_1\! (s( h_l),\,s( h_l^base) ), (7) subject to the same validity (≥0.93≥ 0.93) and Spearman (≥ρStext-rewrite≥ _S^text-rewrite) constraints as the attack feasibility set (Equation˜5). The text-rewrite baseline paraphrases the perturbed answer with a neutralization prompt (e.g., for Bandwagon−- it asks an LLM to strip the consensus attribution while preserving the answer body), emulating the realistic-deployment text-level counterpart in which the source of the perturbation is unknown to the defender; a perturbation-source-aware baseline that simply deletes the appended attribution string is not available in deployment because deployment-time inputs are not annotated with their perturbation source, and we therefore use the LLM-rewrite baseline as the operationally meaningful text comparator throughout. Table 20: Complete defense performance on Llama-3.1-8B, test set. Each block shows the best per-method configuration selected on the development set; the evaluation metric is the fractional reduction in W1W_1 between defended and original baseline score distributions. Bias Method ΔW1def↑ W_1^def ρS↑ _S Valid. Layer |α∗||α^*| Authority−- Text Rewrite 0.11 0.74 100% – – Geometric Median 0.49 0.84 99.0% 25 18.3 PCA 0.56 0.79 96.8% 27 31.2 Classifier 0.62 0.81 96.5% 28 54.7 Bandwagon−- Text Rewrite 0.09 0.56 100% – – Geometric Median 0.47 0.71 98.5% 22 9.6 PCA 0.61 0.67 95.9% 29 48.2 Classifier 0.56 0.65 96.2% 23 12.4 Refinement−- Text Rewrite 0.15 0.77 100% – – Geometric Median 0.41 0.86 100% 20 3.9 PCA 0.48 0.83 98.6% 29 21.8 Classifier 0.52 0.83 97.1% 24 6.5 Defense results mirror the attack findings in structure: activation-based defense dominates the text-rewrite baseline by a factor of 3–5× in W1W_1-reduction across all three bias types, while simultaneously maintaining comparable or higher Spearman correlation. The optimal defense layer tends to be slightly shallower than the optimal attack layer (20–29 vs. 18–31), consistent with the intuition that returning the activation to ℳbaseM_base from a biased state requires intervention before the late-layer reorganization finalizes the scoring decision. Geometric Median defenses again produce the most rank-preserving results, while Classifier defenses yield the largest W1W_1-reductions on two of three bias types. Critically, the symmetry between effective attack and effective defense along the same vector v→bias(l) v_bias^(l) is the core causal claim of Section˜4.4: the bias direction admits bidirectional steering. K.1 Cross-Validated Defense The defense numbers in Table˜20 are reported on the same effective-bias substrate eff−D_eff^- that the bias direction was estimated on; to demonstrate that the defense effect is not an in-sample fitting artifact, we re-run the protocol with 55-fold cross-validation. For each bias type and each method (Geometric Median, PCA, Classifier), we randomly partition eff−D_eff^- into five equally sized folds; for each fold we estimate the bias direction on the remaining four folds, search the optimal (l,α∗)(l,α^*) on a separate dev sub-fold, and apply the defense intervention on the held-out fold. The cross-validated W1W_1-reduction is the mean across the five held-out folds; the within-fold variance gives the reported uncertainty. Table 21: Cross-validated defense performance on Llama-3.1-8B, mean ± std across 5 held-out folds of eff−D_eff^-. “In-sample” reproduces the best-method numbers from Table˜20 for reference. The held-out W1W_1-reduction retains ≥80%≥ 80\% of the in-sample value for every bias type, well above the text-rewrite baseline. Bias In-sample ΔW1def W_1^def Held-out ΔW1def W_1^def Text rewrite Authority−- 0.62 0.54 ± 0.04 0.11 Bandwagon−- 0.61 0.51 ± 0.03 0.09 Refinement−- 0.52 0.45 ± 0.06 0.15 Three observations follow. First, the in-sample to held-out drop is small (0.07 to 0.10 absolute ΔW1def W_1^def, or 13–18% relative), consistent with mild overfitting of the bias-direction estimator but not with the defense effect being a within-substrate fitting artifact. Second, the held-out defense still outperforms the text-rewrite baseline by a factor of 33–5×5× on every bias type, so the gap between activation defense and text defense survives the cross-validation budget. Third, the held-out spread across folds (std 0.040.04 to 0.050.05) is much smaller than the gap to the text baseline (mean gap 0.360.36 to 0.420.42), so the cross-validated effect is well-separated from random-fold variation. K.2 Absolute Spearman Floor: Robustness to the Feasibility Constraint Design The defense feasibility constraint in the main text uses ρS(α)≥ρStext-rewrite _S(α)≥ _S^text-rewrite, with the floor anchored to whatever rank-faithfulness the text-rewrite defense achieves on the same bias type. This design is principled for the bidirectional comparison (no defense intervention is allowed to be more rank-disruptive than the text-rewrite alternative it is benchmarked against), but it makes the activation-vs-text gap conditional on text-rewrite happening to lie at a particular ρS _S for each bias type. We complement this with an absolute-floor variant in which the constraint is ρS(α)≥0.85 _S(α)≥ 0.85 for both methods, independent of the text-rewrite baseline. Table 22: Defense W1W_1-reduction under an absolute Spearman floor ρS≥0.85 _S≥ 0.85 on Llama-3.1-8B. The stricter floor reduces both methods’ achievable W1W_1-reduction relative to the text-rewrite-relative floor used in the main text (52–62% activation vs 9–15% text-rewrite); the qualitative 4–6× advantage of activation defense is preserved across all three bias types tested. Bias Type Activation Defense Text-Rewrite Defense Gap W1W_1-reduction at ρS≥0.85 _S≥ 0.85 W1W_1-reduction at ρS≥0.85 _S≥ 0.85 (Activation / Text) Authority−- 42%42\% 7%7\% 6.0×6.0× Bandwagon−- 38%38\% 6%6\% 6.3×6.3× Refinement−- 45%45\% 10%10\% 4.5×4.5× Under the absolute ρS≥0.85 _S≥ 0.85 floor (which is stricter than the typical text-rewrite ρS _S for these bias types), activation defense achieves 3838–45%45\% W1W_1-reduction while text-rewrite drops to 66–10%10\%. The text-rewrite method has particular difficulty maintaining ρS≥0.85 _S≥ 0.85 at non-trivial defense intensity because prose-level rewrites disrupt rank ordering more aggressively than targeted activation interventions; this is why the gap is actually slightly larger in absolute-floor terms (4.54.5–6.3×6.3×) than in the main-text relative-floor terms (33–5×5×). The qualitative finding—that activation defense substantially outperforms text-rewrite defense at matched rank-faithfulness—is therefore not an artifact of the text-rewrite-relative floor design in the main text. Appendix L Additional Activation Visualizations The layer-25 baseline-manifold projection (Figure˜1) appears in the main text; this appendix gathers additional projections that support the geometric claims of Section˜4.3. L.1 Activation MDS Across Layers Figure˜10 presents MDS projections of absolute activations at layers 5, 15, and 25 of Llama-3.1-8B. The same pattern holds across all layers: baseline activations form a tight cluster, while most negatively biased activations are displaced to a distant region, with a small subset remaining anomalously close to the baseline. Figure 10: MDS projection of final-token activations at layers 5, 15, and 25 of Llama-3.1-8B for baseD_base and negD_neg. The clustering pattern—tight baseline region with displaced biased activations—is consistent across layers. L.2 Same-Polarity Geometric Separation Figure˜11 presents MDS projections of per-sample effective bias vectors Δh→l h_l for two negative bias types—Bandwagon and Diversity—at layers 5, 15, and 25. As with the cross-polarity comparison (Figure˜2), the two negative types become increasingly separable in deeper layers, confirming that geometric distinction is a general property of bias-type representations, not an artifact of polarity differences. Figure 11: MDS projection of per-sample effective bias vectors Δh→l h_l for Bandwagon (red) and Diversity (blue) negative biases at layers 5, 15, and 25 of Llama-3.1-8B. The two negative bias types exhibit progressive geometric separation in deeper layers. L.3 Magnitude Consistency Across Layers Figure˜12 plots the per-layer mean L2 norm of final-token activations for baseD_base and negD_neg on Llama-3.1-8B. The two trajectories are statistically indistinguishable at every layer below the final readout (p>0.1p>0.1, two-sample t-test with n=4,500n=4,500 paired samples and BH correction across 32 layers), where the gap sits below 2%. At the final score-readout layer, the relative gap reaches 3.5%3.5\% and is statistically detectable; this gap concentrates where the linear head projects the activation onto the score logits, and is the only layer at which the bias-induced magnitude shift is non-trivial. Bias is therefore encoded in the direction of the activation displacement rather than in its magnitude. This is why we unit-normalize the vector estimators throughout Sections˜3.3 and 4.4, and why we define the biased core in terms of Mahalanobis distance, as in Table˜24. Figure 12: Mean L2 norm of final-token activations across the 32 decoder layers of Llama-3.1-8B for the baseline set baseD_base (blue, solid) and the negatively perturbed set negD_neg (red, dashed). Shaded bands denote ±1± 1 SD across samples. Bias does not change how far the activation moves; only the direction in which it moves. L.4 PCA Analysis of Activation Space Figure 13: PCA projection of final-token activations at layer 25 of Llama-3.1-8B, colored by bias type. Baseline activations cluster tightly into the baseline manifold ℳbaseM_base, mirroring the MDS visualization in the main text. Figure˜13 shows a PCA projection of final-token activations at layer 25 of Llama-3.1-8B, colored by bias type. Consistent with the MDS analysis (Figure˜1), baseline activations form a tight cluster, the baseline manifold ℳbaseM_base, while effective bias samples from different bias types are displaced away from this region. The agreement between MDS and PCA confirms that ℳbaseM_base is a robust geometric property of the activation space, not an artifact of a particular dimensionality reduction method. L.5 Bias-Subspace Dimensionality The geometric analysis throughout Section˜4.3 relies on the claim that the per-bias effective displacement Δh→l(x) h_l(x) lives in a low-dimensional subspace of ℝdR^d, rather than being spread isotropically. To make this precise, we compute the eigenspectrum of the centered covariance of Δh→25(x):x∈eff(bias)\ h_25(x):x _eff^(bias)\ for each bias type at layer 25, the deepest activation-analysis layer, and report the cumulative variance explained by the top k principal components for k∈1,3,5,10k∈\1,3,5,10\. Table 23: Cumulative variance explained (%) of the per-sample effective bias vectors Δh→25(x) h_25(x) on Llama-3.1-8B, by bias type, at the top-k principal components. The top-5 components capture at least 83% of the variance for every bias type, supporting the low-dimensional characterization used in Section˜3.3. Bias type top-1 top-3 top-5 top-10 Bandwagon−- 47.3 78.6 89.1 95.4 Diversity−- 43.8 75.2 86.7 94.1 Authority−- 41.5 73.4 85.9 93.5 Refinement++ 44.2 76.0 87.4 94.3 Sentiment−- 38.7 70.8 83.5 92.1 The numbers support a more careful version of the qualitative claim: the bias direction is not strictly one-dimensional. The single dominant component accounts for 3939 to 4747% of variance, but a small 3-to-5-dimensional subspace already captures 8383 to 8989% of the per-sample displacement, and the top-10 captures over 9292% for every type. The dimensionality is also comparable across bias types within roughly ±4± 4 percentage points at each k, suggesting that the subspace structure is a property of how the judge represents bias generally and not specific to one perturbation cue. Throughout the paper, “the bias direction” is therefore shorthand for the dominant axis of a low-dimensional (33 to 55-dim) bias subspace, and the vector estimators in Section˜3.3 are best read as different ways of selecting a single representative direction inside that subspace. Table 24: Layer-wise characterization of the bias direction on Llama-3.1-8B (32 layers). Biased core ρcore _core, LDA separability, and the LDA/Classifier cosine agreement rise monotonically with depth; raw classifier AUC peaks in middle layers, where surface-cue separability is still trivially exploitable. l ρcore(l) _core^(l) SepLDA(l)Sep_LDA^(l) AUCCLS_CLS cos⁡(v→LDA,v→CLS) ( v_LDA, v_CLS) 05 0.32 0.71 0.89 0.52 10 0.48 0.79 0.94 0.68 15 0.62 0.85 0.94 0.78 20 0.73 0.90 0.93 0.85 25 0.81 0.93 0.91 0.91 31 0.87 0.95 0.87 0.94 Appendix M Detection Pipeline Details This appendix provides the full details behind the detection results summarized in Section˜4.5, including feature construction, training protocol, baseline comparisons, and a domain-gap diagnostic. M.1 Two Prediction Targets The detection experiments support two operationally distinct binary prediction problems, and it is worth stating which one is being optimized before describing the feature pipeline. Target A is stylistic discrimination: given an input x, decide whether x was drawn from the baseline distribution baseD_base or the negatively perturbed distribution negD_neg. This is a sanity check on whether the activation features carry any signal about the surface perturbation; it succeeds even when the judge’s eventual score is unchanged. Target B is outcome prediction: given an input x, predict whether the judge will return a degraded score s(xneg)≤s(xbase)−δos(x_neg)≤ s(x_base)- _o relative to the baseline answer, with δo=1 _o=1 integer point (intentionally more permissive than the δs=2 _s=2 effective-bias threshold used for vector fitting in Section˜3.3, so that the predictor captures both strong and marginal degradations rather than only the tail used to define the bias substrate). This is the operationally useful target, because the practitioner wants to know which inputs the judge will actually mishandle, not which inputs carry a stylistic marker. The cross-domain test AUCs reported in the main text (Section˜4.5: 0.850.85 for the linear projection on the full test split, 0.820.82 on the three entirely unseen benchmarks, and 0.840.84 for the GBDT on the full test split) all refer to Target B, and all subsequent tables in this appendix report Target B unless explicitly stated otherwise. Table 25: Detection performance for the two prediction targets, evaluated on the same cross-domain test split (three benchmarks unseen during training). Target A measures stylistic discrimination between baseline and negatively perturbed inputs. Target B measures whether the judge’s score will degrade by at least one integer point. The remaining detection tables in this appendix report Target B. Target Description Test AUC Test AP A Stylistic discrimination (baseD_base vs negD_neg): detect whether the input has a bias perturbation. 0.972 0.989 B Outcome prediction (fair vs degraded judge score): predict s(xneg)≤s(xbase)−δos(x_neg)≤ s(x_base)- _o with δo=1 _o=1. 0.839 0.976 Table 26: Outcome-prediction performance on Llama-3.1-8B (Target B). The test set contains three benchmarks unseen during training, exposing a genuine domain gap. The GBDT outcome predictor matches or outperforms all baselines on AP and is competitive on AUC, despite markedly higher development-set AUC indicating room for further domain adaptation. Dev (in-domain) Test (incl. unseen) Detector AUC AP AUC AP Text-based LLM Detector 0.641 0.939 0.633 0.937 PCA + Logistic Regression 0.871 0.971 0.837 0.969 Projection + Mahalanobis (linear) 0.864 0.974 0.834 0.972 Projection Features Only (linear) 0.883 0.979 0.854 0.977 GBDT (ours) 0.926 0.992 0.839 0.976 The gap between Target A and Target B is itself informative. The activation features almost perfectly separate baseline from negatively perturbed inputs (AUC 0.972), so the surface perturbation leaves a clean trace in the hidden state. The harder Target B drops AUC by roughly 13 points because not every perturbed input causes a degraded score: among inputs the perturbation does mark, only the subset whose activation actually moves far enough along v→bias v_bias produces the downstream score drop. The outcome predictor therefore has to learn the conjunction of “perturbation present” and “activation displacement large enough to flip the score”, which is strictly harder than detecting the perturbation alone. M.2 Feature Construction and Training Protocol For each input x, we compute a per-layer feature vector ϕ→l(x) φ_l(x) drawn from three families, then concatenate across layers: • Bias-direction features: the scalar projection h→l(x)⊤v→LDA(l) h_l(x) v_LDA^(l), the scalar projection h→l(x)⊤v→CLS(l) h_l(x) v_CLS^(l), cosine similarities cos⁡(h→l(x)−μ→base(l),v→⋅(l)) ( h_l(x)- μ_base^(l),\, v_·^(l)), and the perpendicular distance ‖(h→l(x)−μ→base(l))−projv→⋅(l)(⋅)‖2\|( h_l(x)- μ_base^(l))-proj_ v_·^(l)(·)\|_2 for both vector types. • Manifold-deviation features: the Mahalanobis distance dM(h→l(x);μ→base(l),Σ→base(l))d_M( h_l(x); μ_base^(l), _base^(l)) and its z-score against the baseline distribution. • Semantic-context features: the top-k PCA projections of h→l(x) h_l(x) onto the baseline PCA basis, and intrinsic statistics (‖h→l(x)‖2\| h_l(x)\|_2, mean, variance of the activation). The raw per-layer count K=24K=24 scalar features is composed of: 8 bias-direction features (signed projection, cosine, perpendicular distance, and centred projection for each of v→LDA(l) v_LDA^(l) and v→CLS(l) v_CLS^(l)), 2 manifold-deviation features (Mahalanobis distance and its z-score), and 14 semantic-context features (top-10 PCA projections plus four intrinsic statistics). Concatenated across L=32L=32 layers this gives L⋅K=768L· K=768 raw features. A near-zero-variance pre-filter on the training set drops layers where the bias-direction estimators are not yet active (predominantly layers 1 to 4), reducing the active pool to approximately 450450 features. Recursive Feature Elimination (RFE) then prunes the active pool to approximately 120120 retained features (median count near 44 per layer survive the prune, with a longer tail at the deepest layers), and a Gradient Boosting Decision Tree (GBDT, LightGBM) is trained with 200 rounds of Bayesian hyperparameter search (Optuna) against the development AUC. The Target B outcome class boundary uses a single-integer threshold (δo=1 _o=1), which is more permissive than the effective-bias threshold δs=2 _s=2 used for vector fitting in Section˜3.3; the rationale and asymmetry between the two thresholds are discussed at the end of this subsection. Table 27: Cross-domain test AUC of the outcome predictor as a function of the held-out unseen-benchmark trio. Each row is a different choice of unseen trio; the corresponding seen-benchmark set is the remaining six benchmarks used for training and development. The linear projection AUC range is 0.7980.798–0.8320.832 (within roughly 0.0250.025 of the default 0.8210.821), and the GBDT AUC range is 0.7320.732–0.7640.764 (within roughly 0.020.02 of its default), indicating that the cross-domain finding is robust to held-out-trio choice within reasonable benchmark substitutions. Unseen Trio Linear Projection AUC GBDT AUC SocialMaze, BBQ, GPQA (default) 0.8210.821 0.7510.751 SocialMaze, BBQ, CSQA 0.8320.832 0.7640.764 SocialMaze, BBQ, PubMedQA 0.8060.806 0.7380.738 SocialMaze, BBQ, TruthfulQA 0.7980.798 0.7320.732 SocialMaze, GPQA, ARC 0.8150.815 0.7480.748 BBQ, GPQA, CSQA 0.8120.812 0.7450.745 M.3 Baseline Detectors We compare against four baselines that span two families: (i) linear baselines using subsets of the same activation features, and (i) text-based detectors that ignore activations entirely. Text-based LLM detector setup. The text-based LLM detector is GPT-4o-Mini prompted in zero-shot with the perturbed input answer (no paired baseline answer access, mirroring the deployment regime in which the defender does not know which inputs are perturbed) and asked whether the answer is likely to receive a degraded quality score from an LLM judge, with a confidence response in [0,1][0,1]. We aggregate three prompt variants (one neutral, one emphasizing surface-cue red flags, one emphasizing factual coherence) and five seeds per variant; the AUC reported in Table˜26 is the average across the 1515 detector runs on the same test split as the activation-based baselines. Three observations warrant note. First, the text-based LLM detector substantially underperforms every activation-based approach, echoing our main-text claim that the bias signal is much cleaner in the activation geometry than in the input text. Second, the linear baselines perform surprisingly well—Projection Features Only attains test AUC 0.854—confirming that the bias direction is itself highly informative. Third, the GBDT opens a visible gap on the development set (0.926 vs. 0.883 AUC) but loses most of it on the cross-domain test set (0.839 vs. 0.854). This gap is the principal weakness of the detector in its current form. M.4 Component Ablations The cross-method comparison above shows that the GBDT is competitive with simpler activation-based baselines but does not dominate them on the cross-domain test set. To isolate which parts of the pipeline are actually doing the work—and to argue that the 0.84 test AUC is closer to a ceiling than a floor for this feature set—we run a component-level ablation that varies the GBDT inputs and the pipeline itself. Each row keeps the LightGBM model, Bayesian hyperparameter search, and class-balanced training identical; only the feature set or the feature-selection step changes. Table 28: GBDT detection performance as a function of feature subset and pipeline component on Llama-3.1-8B. Each row uses the same LightGBM model and the same Optuna search budget; the feature set or the feature-selection step is the only thing that changes between rows. Best in column in bold. Dev (in-domain) Test (incl. unseen) GBDT variant AUC AP AUC AP Mahalanobis distance only (1 feat.) 0.768 0.957 0.711 0.949 PCA / intrinsic-statistic features only (57 feat.) 0.872 0.974 0.792 0.961 Bias-direction projections only (41 feat.) 0.918 0.989 0.831 0.973 All features, no RFE (∼450 450 feat., raw) 0.918 0.988 0.825 0.971 All features + RFE + Optuna (ours) 0.926 0.992 0.839 0.976 Three things stand out. First, Mahalanobis distance on its own, essentially asking whether the activation looks drawn from ℳbaseM_base, already achieves test AUC 0.71, well above the 0.63 of the text-based LLM baseline; geometry alone, even reduced to a single scalar, beats the most natural prompted detector. Second, the bias-direction projections alone (the cosine and signed projection onto v→LDA(l) v_LDA^(l) and v→CLS(l) v_CLS^(l) at each layer) reach test AUC 0.831, within 1 point of the full pipeline; the bias direction is doing roughly 95% of the heavy lifting on the cross-domain test set, consistent with the 62.3% Gini importance it carries in the final model (Table˜30). Third, RFE buys only a modest gain on the test set (0.825 → 0.839 AUC) but reduces the dev–test gap from 0.093 to 0.087: most of the residual gap is irreducible under this feature set rather than a symptom of an oversized feature pool. Taken together, these numbers position 0.84 as the ceiling of what bias-geometric features yield for cross-domain detection on this judge, not as the floor of an under-engineered model—and motivate the domain-gap diagnostic below. M.5 Domain-Gap Diagnostic To quantify the cross-domain generalization shortfall, we evaluate the GBDT separately on the test-set questions drawn from the six seen benchmarks versus those drawn from the three unseen benchmarks. Table 29: Cross-domain outcome prediction on Llama-3.1-8B: detector AUC/AP on test questions from seen versus unseen benchmarks. The linear projection features transfer to unseen domains (AUC 0.8210.821) better than the more expressive GBDT (0.7510.751), and both beat the text-based LLM detector (0.6240.624). Seen benchmarks Unseen benchmarks Detector AUC AP AUC AP Text-based LLM Detector 0.638 0.942 0.624 0.928 Projection Features Only 0.869 0.981 0.821 0.967 GBDT 0.902 0.988 0.751 0.953 Within-domain, the GBDT attains AUC 0.902, close to its development performance. On held-out benchmarks, AUC drops to 0.751—still well above the text-based baseline (0.624), but well below the within-domain figure. The linear Projection Features detector is more robust to the domain shift but leaves performance on the table within domain. This suggests that the GBDT is overfitting to domain-specific activation patterns rather than fully exploiting the universal bias-direction signal, and motivates future work on domain-adversarial training or feature regularization targeting the bias subspace identified in Figure˜2. M.6 Feature Importance Table˜30 groups the RFE-retained features by family and reports their mean Gini importance in the final GBDT. Bias-direction projections alone account for over 60% of the total importance, consistent with the main-text claim that v→LDA v_LDA and v→CLS v_CLS capture the dominant bias signal. Among individual layers, layers 22–28 contribute disproportionately, matching the layer-wise sweet spot identified in Table˜24. Table 30: Feature-family importance in the GBDT outcome predictor (Gini importance, normalized to sum to 100%). Feature Family # features retained Gini importance Bias-direction projections (v→LDA,v→CLS v_LDA, v_CLS) 41 62.3% Mahalanobis distance from ℳbaseM_base 22 18.7% PCA / intrinsic-statistic features 57 19.0% M.7 Sensitivity of Cross-Domain AUC to Unseen-Trio Choice The main-text cross-domain test (Section˜4.5) holds out three benchmarks (SocialMaze, BBQ, GPQA) entirely from training. The choice of which three are held out is a design knob: different held-out trios can plausibly produce different cross-domain AUC values, especially since per-bias × per-domain interactions are large (Table˜3, e.g., Diversity−- on BBQ at −2.03-2.03 versus on GSM8K at −0.55-0.55). To characterize this sensitivity, we re-run the outcome-predictor training and evaluation under five alternative held-out trios, retraining both the linear-projection and GBDT detectors from scratch with the same hyperparameters as the default trio. The linear-projection AUC varies within roughly 0.0250.025 of the default value of 0.8210.821 (range 0.7980.798–0.8320.832), and the GBDT AUC varies within roughly 0.020.02 of its default 0.7510.751 (range 0.7320.732–0.7640.764). The within-row gap between linear and GBDT (linear higher by 0.070.07–0.080.08 across trios) is also stable, supporting the main-text observation that the simpler linear projection transfers cross-domain more robustly than the more expressive GBDT. The default trio (SocialMaze, BBQ, GPQA) sits near the median of the sensitivity range, not at an outlier favorable to the AUC headline.

Instant research discovery