Paper deep dive
THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts
Kareem Hassani, Chaymaa Abbas, Lama Mawlawi, Mariette Awad
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 2:47:09 AM
Summary
The paper introduces THESIS-MoE, a method for steering sycophancy in Mixture-of-Experts (MoE) language models. It proposes a shared contrastive signal derived from matched prompts with and without stated beliefs to localize sycophancy within the MoE hierarchy. The authors compare unconditional activation subtraction with two conditional interventions: analytic projection-based subtraction and a learned per-token gate. Results show that conditional interventions can remove up to 90% of belief-induced sycophancy while maintaining general knowledge and reasoning performance, demonstrating that sycophancy resides in identifiable computational subcircuits.
Entities (10)
Relation Signals (8)
THESIS-MoE → targets → Sycophancy
confidence 98% · THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts
THESIS-MoE → uses → Conditional Subtraction
confidence 95% · We compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate
THESIS-MoE → uses → Learned Per-Token Gate
confidence 95% · We compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate
Sycophancy → residesin → Computational Subcircuits
confidence 92% · Our results demonstrate that sycophancy resides in identifiable computational subcircuits
Learned Per-Token Gate → removes → Sycophancy
confidence 90% · Our conditional interventions removed up to 90% of the belief-induced sycophancy.
Conditional Subtraction → removes → Sycophancy
confidence 90% · Our conditional interventions removed up to 90% of the belief-induced sycophancy.
Conditional Subtraction → preserves → Knowledge Retention
confidence 88% · Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered while maintaining a favorable removal-retention trade-off.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the model, which is an unconditional intervention that alters activations even when no sycophantic behavior is present, trading knowledge retention for behavioral correction. In Mixture-of-Experts (MoE) models, prior work further suggests that behavior is encoded within expert computations rather than routing decisions alone, making precise behavioral steering particularly challenging. In this work, we introduce a shared contrastive signal, built from matched prompts with and without a stated belief, that identifies where sycophancy lives across the MoE hierarchy and drives interventions that act only where the behavior is present. We formulate localization as a causal search over a granularity ladder of MoE blocks, experts, attention blocks, and heads, and compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate that steers the model away from sycophancy while keeping its weights frozen. We evaluate on three MoE models measuring sycophancy alongside general knowledge and reasoning benchmarks. Our conditional interventions removed up to 90\% of the belief-induced sycophancy. Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered while maintaining a favorable removal-retention trade-off.
Tags
Links
- Source: https://arxiv.org/abs/2608.15687v1
- Canonical: https://arxiv.org/abs/2608.15687v1
Trouble viewing inline? Open PDF directly →
Full Text
46,907 characters extracted from source content.
Expand or collapse full text
THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts Kareem Hassani Chaymaa Abbas Lama Mawlawi Mariette Awad Abstract Sycophancy, the tendency of a language model to change its answer to match a user’s stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the model, which is an unconditional intervention that alters activations even when no sycophantic behavior is present, trading knowledge retention for behavioral correction. In Mixture-of-Experts (MoE) models, prior work further suggests that behavior is encoded within expert computations rather than routing decisions alone, making precise behavioral steering particularly challenging. In this work, we introduce a shared contrastive signal, built from matched prompts with and without a stated belief, that identifies where sycophancy lives across the MoE hierarchy and drives interventions that act only where the behavior is present. We formulate localization as a causal search over a granularity ladder of MoE blocks, experts, attention blocks, and heads, and compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate that steers the model away from sycophancy while keeping its weights frozen. We evaluate on three MoE models measuring sycophancy alongside general knowledge and reasoning benchmarks. Our conditional interventions removed up to 90% of the belief-induced sycophancy. Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered while maintaining a favorable removal-retention trade-off. 1 Introduction Large language models often change their answers after users reveal their own beliefs. A model may answer a question correctly in isolation, yet shift toward the user’s preferred option when the same question is preceded by an opinion. This behavior, known as sycophancy, has been observed across model families and training regimes (11; 14). It undermines trust by making responses depend on the user’s stated view rather than the underlying evidence. Figure 1: The unit of intervention. Every attention head and expert carries a gate m; the gated contributions AiA_i and MiM_i also include gated mechanisms at the output stream. Existing activation-editing methods attempt to remove unwanted behaviors by identifying and subtracting their directions in activation space. Contrastive activation addition constructs such directions from paired examples (13), while concept erasure removes the corresponding activation subspace (1). Although effective, these interventions are unconditional: the same edit is applied to every token, even when sycophancy is absent. Stronger edits therefore improve behavioral correction at an unnecessary cost to retained knowledge. Our goal is to close this removal-retention gap. We argue that the main limitation lies not in the identified direction, but in applying it indiscriminately. An effective intervention should act only when sycophancy is present and in proportion to its strength. To obtain such a signal, we compare activations for the same question under two matched conditions: a nudged prompt, where the user states a belief, and a no-nudge prompt, where that belief is neutralized. Each example includes a conforming answer that agrees with the user and a nonconforming answer that does not. The resulting activation difference captures the belief-induced shift and yields a continuous, signed belief-push metric, introduced in Section 4. We use this signal to locate the model components that carry the shift, examining an MoE granularity ladder from complete attention and MoE blocks to individual heads and experts. At these components, unconditional subtraction reduces sycophancy but consistently harms retention, regardless of intervention granularity. Our contributions are as follows: • We introduce matched nudged and no-nudge conditions and a continuous, signed belief-push metric for measuring sycophancy (Section 4). • We causally localize the belief shift across MoE blocks, attention blocks, experts, and heads in three open-weight models (Section 5). • We show that unconditional subtraction produces a persistent trade-off between sycophancy removal and knowledge retention. • We propose analytic conditional subtraction and learned conditional gating, which improve the removal-retention frontier relative to fixed subtraction, largely preserving GSM8K while MMLU declines under the selected operating points, across three models and three seeds. • We compare unconditional and conditional interventions with routing and prompting baselines, showing that conditionality is key to recovering retention (Section 6). 2 Related Work Existing steering methods show that behavioral directions can be modified in activation space, while localization studies identify components associated with or causally implicated in particular behaviors. MoE research further distinguishes effects arising from expert selection from those arising within expert computation. Steering by activation subtraction. Contrastive activation addition builds a steering vector from the mean activation difference between paired examples of a behavior and adds or subtracts it during the forward pass, controlling sycophancy among other behaviors on Llama 2 (13), while closed-form concept erasure removes a target subspace from every representation (1). Both apply a fixed-magnitude edit to every token regardless of whether the behavior is present, the property our conditional form is designed to change. Localizing behavior to a few components. The closest work isolates roughly three percent of MLP neurons with sparse autoencoders and probes, then fine-tunes only those neurons, matching broad fine-tuning on sycophancy with less data (10). It shares our locate-then-edit structure but operates on dense models and corrects the behavior by updating weights, whereas we work in the mixture-of-experts architecture and leave the base weights frozen, editing at inference. Behavior and routing in mixture-of-experts models. Prior work detects behavior-associated experts by contrasting their activation under opposite behaviors and steers by activating or deactivating them (3), and reports that routing in aligned models is driven more by topic than by behavior, pointing to expert computation rather than routing as the locus (16). Our results agree from the intervention side, since a routing baseline removes only part of the shift and damages retention, and the conditional edit still removes most of the shift in the one model whose shared experts admit no routed contrast. Sycophancy. Model-written evaluations documented sycophancy at scale (11), and later analysis tied it to feedback that rewards agreement (14). These measure it as a rate of agreement; we treat it as the shift in the answer caused by stating a belief, which is what lets a conditional edit respond to it. Our work connects activation-based steering, causal component localization, behavioral analysis in mixture-of-experts models, and sycophancy. These have not been brought together to ask whether a belief-induced answer shift can be localized across the MoE hierarchy and corrected without editing behaviorally unrelated activations. A parallel line localizes sycophancy inside dense transformers, placing it in sparse middle-layer attention heads (4) or specific layers via activation patching (15), but reports weak steering and does not act within an MoE or conditionally. 3 Problem Formulation and Data 3.1 Sycophancy as a Belief-Induced Shift We define sycophancy as an answer change caused by the user stating a belief, rather than as agreement itself. Each multiple-choice item contains a user opinion and two possible completions: a sycophantic-conforming option that agrees with the belief and a sycophantic-nonconforming option that does not. Agreement is not sycophantic when the model would give the same answer without the stated belief; changing the answer because the belief was introduced is. This distinction motivates the signal in Section 4 and the interventions in Section 5. Each item is evaluated in the four matched forms shown in Table 1. The columns determine whether the belief is present. The nudge condition includes a user opinion before and throughout the question, whereas the no-nudge condition preserves the question while neutralizing the opinion. Their difference therefore isolates the effect of the stated belief. The rows indicate whether the supplied answer conforms to or opposes that belief. We use diagnostic and forced-answer versions of each item. In the diagnostic version, the answer is left open and the model’s preference is compared across the nudge and no-nudge conditions to measure belief push. In the forced version, the answer is included in the prompt and the resulting activations are compared to construct the belief direction used for subtraction and gating. This differs from contrastive activation addition (13), which compares conforming and nonconforming forced answers only within the nudge condition. That contrast may mix sycophantic behavior with the identity of the forced answer. All behavioral results use the diagnostic version; the forced version is used only to construct activation directions. Table 1: Each item appears in four matched forms. Nudge versus no-nudge defines the belief direction and belief push, whereas contrastive activation addition compares conforming and nonconforming answers only under the nudge condition. Nudge No-nudge Conforming belief, agrees neutral, agrees Nonconforming belief, opposes neutral, opposes 3.2 Data Construction and Balancing We begin with the model-written evaluation items of Perez et al. (11), using the repackaged version distributed with contrastive activation addition (13). For each nudged item, we derive a matched no-nudge twin synthetically and minimally by removing the user’s stated belief while only preserving an objective question. To prevent the belief direction from capturing answer-position artifacts, we balance the belief-conforming option across the two answer labels at the matched-item level before direction estimation, so that answer-label identity is uncorrelated with belief conformance. We then split the balanced bundles by item identifier, so a matched pair cannot appear in different splits, preserving balance independently within each split. Table 2 reports the resulting sizes. The development split is used for causal localization and hyperparameter selection, the test split for final comparisons, and the training split for learned gates. Retention is measured on standard held-out benchmarks using an independently sampled fixed fraction for each seed. Table 2: Dataset splits and retention sets. Splits are balanced, identifier-disjoint, and verified to have zero leakage. Sycophancy data Retention data Split Items Balance Set Items Per seed Matched 8234 4117/4117 MMLU 1482 741 Train 4940 2470/2470 GSM8K 1319 659 Dev. 1646 823/823 TruthfulQA 817 408 Test 1648 824/824 Perplexity 1723 861 3.3 Two Different Contrasts Our method differs from prior steering in the contrast used to define the direction. Contrastive activation addition uses a completion contrast: it forces the conforming and nonconforming answers while the belief remains present, then takes the mean difference between their activations. The resulting direction may combine sycophancy with the representation of the forced answer. We instead use a condition contrast. Holding the answer fixed, we compare the nudge and no-nudge versions of the same item. This isolates the representation introduced by the stated belief, which is the signal a conditional intervention should detect. 4 Signal and Metrics 4.1 Condition-Contrast Belief Direction We construct the belief direction using the forced-answer versions introduced in Section 3.1. Components were retrieved from our sycophancy-localization search (see Section 5.1). At a chosen component c we let hi,cnudgeh_i,c^nudge and hi,cno-nudgeh_i,c^no -nudge denote the activations at the decision token for a matched pair in which the supplied answer is held fixed and only the presence of the user’s stated belief changes. We estimate the component- specific belief direction as the mean paired activation difference dc=1N∑i=1N(hi,cnudge−hi,cno-nudge).d_c= 1N _i=1^N (h_i,c^nudge-h_i,c^no -nudge ). (1) With unit form d^c=dc/∥dc∥2 d_c=d_c/ d_c _2, and because the supplied answer is fixed within each pair and the answer labels are balanced, dcd_c estimates the activation change associated with introducing the stated belief rather than the identity or position of the answer. This is the direction used for localization, subtraction, and conditional gating. 4.2 The Belief-Push Metric We score each item at the decision token by the logit margin between the two options. Writing ℓconf _conf and ℓnconf _nconf for the logits of the sycophantic-conforming and sycophantic-nonconforming options, the item score under an input x is s(x)=ℓconf(x)−ℓnconf(x).s(x)\;=\; _conf(x)\;-\; _nconf(x). (2) The belief push on an item is the paired difference between the nudge and no-nudge scores, Δ=s(xnudge)−s(xno-nudge). =s\! (x^nudge )-s\! (x^no-nudge ). (3) Each score is the logit margin between the conforming and nonconforming options under one condition. Thus, Δ measures how much the stated belief shifts the model’s preference toward the conforming option. A value near zero indicates invariance, a positive value indicates sycophantic movement, and a negative value indicates reversal beyond invariance. We report the mean belief push across items with a bootstrap confidence interval. Zero is the target: the stated belief does not change the answer. The signed logit margin distinguishes correction from overshooting while remaining smooth enough for fine-grained sweeps and differentiable training. 4.3 Metrics and Evaluation Method Description Raw No intervention; fixes the baseline Prompted Instructed to be objective FARE Reweights the router away from belief experts CAA subtraction Condition direction, unconditional; swept over nine sets Cond. subtraction Condition direction, conditional; swept over nine sets Learned gate Conditional; all sets jointly Table 3: The methods compared. The subtraction methods are swept over nine sets of the identified components, from a single component type up to all together. Metric Target after edit Belief push, Eq. (3) to zero Nudge shift to zero No-nudge rate (anchor) unchanged MMLU ≥ baseline GSM8K ≥ baseline TruthfulQA ≥ baseline Perplexity unchanged (control) Table 4: The evaluation panel. The upper block measures the behavior we remove and the lower block the knowledge we keep. Figure 2: Where sycophancy lives. Learned gate activity across depth and component type, with the components found by causal search circled. Activity concentrates in a few mid-to-late-layer sites, and the gate opens on the same components the search selects. Figure 3: Sycophancy is concentrated in depth. Per-layer belief push for the selected attention (magenta) and mixture-of-experts (teal) blocks. The push is negligible early and rises sharply in the mid-to-late layers, so the shift is localized rather than spread through the network. We evaluate the six methods of Table 3 on the panel of Table 4. Every intervention is applied only to the components our search identifies as carrying the belief shift, never to the model as a whole. The two subtraction methods are swept over the nine sets so their effect can be read at every granularity of the MoE, across attention blocks, attention heads, MoE blocks, and experts, while the learned gate is placed on all identified components at once. FARE (7) is the one baseline that acts on which experts are chosen rather than on what they compute, so it isolates the routing lever and tests where the behavior resides. The panel has two sets of metrics. The behavioral set is built on the belief push of Equation (3), together with the nudge shift, its discrete counterpart, which measures the change between the two conditions in the rate at which the model’s top choice is the conforming option. The no-nudge rate is the model’s belief-free tendency and acts as an anchor, since a change there means the edit altered the model’s plain behavior rather than its sensitivity to the belief. The retention set is intended to hold at its pre-edit level, and perplexity is included as a control rather than as evidence of retention, because every intervention fires only at the decision token (Section 5.8) and so leaves plain text structurally unaffected. We report every retained metric as a percentage of its unedited-model score, so one scale reads across models, with 100%100\% matching the baseline, above better and below worse. Perplexity is lower-is-better, so we invert its ratio to keep the same direction, while the accuracy metrics (MMLU (6), GSM8K (2), TruthfulQA (8)) use the ratio directly. Values above 100%100\% mean the edited model scored higher than the unedited one on that metric. 5 Methodology Our method consists of a shared localization stage followed by three interventions. We first identify the components carrying the belief shift, then compare unconditional subtraction with two conditional alternatives: analytic conditional subtraction and learned conditional gating. Applying all methods to the same components isolates the effect of the edit’s form rather than its placement. 5.1 Locating the Responsible Components We search across an MoE granularity ladder, from complete attention and MoE blocks to individual heads and experts (Figure 1). Each component undergoes both probing and causal tests. The probing test constructs the belief direction from Equation (1) and verifies that it separates nudge from no-nudge activations above a label-shuffled baseline. The causal test determines whether the component produces the shift rather than merely encoding it. To remain on-distribution, we swap activations between matched nudged and no-nudge examples instead of zeroing them. Replacing a nudged activation with its no-nudge counterpart tests necessity, while inserting the nudged activation into a no-nudge run tests sufficiency. A component is selected when it passes both tests more strongly than a same-type control without harming retention. 5.2 Unconditional Subtraction At each located component, we first apply the constant intervention in the form used by prior activation steering, but built from our condition-contrast direction rather than a completion contrast. Let d d denote the unit belief direction of Equation (1), the same direction used by conditional subtraction, and c the intervention strength. The component output o becomes o←o−c∥d∥d^.o← o-c\, d \, d. (4) This displacement is added at the edited position with a fixed magnitude, regardless of whether the activation there contains the belief signal. Sweeping c therefore reveals a consistent trade-off: stronger intervention removes more belief push but also reduces retention. This motivates a token-dependent alternative. 5.3 Conditional Subtraction Conditional subtraction removes only the activation component aligned with the belief direction. The signed projection ⟨o,d^⟩ o, d measures both the magnitude and orientation of the belief component: o←o−c⟨o,d^⟩d^.o← o-c\, o, d \, d. (5) Figure 4: The central result. For five methods across three models, each cell reports belief push removed together with GSM8K, MMLU, TruthfulQA, and perplexity, each shown relative to baseline (%); for accuracy this is the ratio of edited to baseline score, and for perplexity the inverted ratio, so that above 100%100\% is better on every metric. The two conditional methods remove most of the push while holding GSM8K at or above baseline; routing and constant subtraction trade retention for removal. The intervention adapts to the activation at the edited position rather than applying a fixed displacement. Activations with a strong belief component receive a larger correction, while those with near-zero projection remain almost unchanged. The sign also prevents activations already opposing the belief from being pushed further in that direction. At c=1c=1, the projection is removed exactly; c<1c<1 removes only a fraction, while c>1c>1 reverses its orientation. This analytic intervention requires no training and improves retention by leaving unaffected tokens largely unchanged. 5.4 Component Sets We evaluate both subtraction methods over nine component sets. Four contain a single component type: attention blocks (S1), MoE blocks (S2), attention heads (S4), and experts (S5). Five additional unions test interactions across pathways: attention and MoE blocks (S3), heads and experts (S6), attention blocks and heads (S7), MoE blocks and experts (S8), and all located components (S9). Each set is evaluated at c∈1,2,3,5,10c∈\1,2,3,5,10\ against an unedited baseline. Operating points are selected on the development split and confirmed on the test split. 5.5 Conditional Gating The learned intervention places a gate at each located component while freezing the model’s original weights. Its basic form is a per-token function of the component activation h: mi(h)=σ(θi+wi⊤h).m_i(h)=σ\! ( _i+w_i h ). (6) The gate multiplies the component contribution: mi=1m_i=1 preserves it, whereas values near zero suppress it. Its activation dependence provides a learned counterpart to conditional subtraction. Training combines four objectives that minimize the soft belief push, anchor the edited no-nudge score to the original model, and preserve factual knowledge and general behavior: ℒpush _push =|Pconfnudge−Pconfno-nudge|, = |P^nudge_conf-P^no-nudge_conf |, (7) ℒanchor _anchor =(seditno-nudge−sorigno-nudge)2, = (s^no-nudge_edit-s^no-nudge_orig )^2, ℒknow _know =xCE(πedit(x),y⋆), =E_x\,CE\! ( _edit(x),\,y ), ℒgen _gen =xKL(πorig(x)∥πedit(x)). =E_x\,KL\! ( _orig(x)\,\|\, _edit(x) ). The full objective is ℒ=ℒpush+λanchorℒanchor+λknowℒknow+λgenℒgenL=L_push+ _anchorL_anchor+ _knowL_know+ _genL_gen. The anchor prevents a degenerate solution in which both conditions move toward an even split; we accept a trained state only when it reduces belief push, preserves MMLU, and maintains the no-nudge anchor. 5.6 Gate Architecture Sweep We explore four design axes rather than fixing the gate architecture in advance (Table 5). The first is gate structure: mi m_i =σ(θi), =σ( _i), mi(h) m_i(h) =σ(θi+wi⊤h), =σ\! ( _i+w_i h ), (8) mi m_i =σ((Wz+b)i). =σ\! ((Wz+b)_i ). These correspond to a per-component scalar gate, an activation-dependent gate, and a shared gate parameterized by a low-rank code z∈ℝkz ^k. All three can be reduced to the same flat initialization, ensuring that later differences arise from training. The second axis is the edit class. With ai=1−mia_i=1-m_i, the directional edits subtract either a fixed or learned belief direction, while the component edit scales the complete output: o←o−aicidi,o←mio.o← o-a_i\,c_i\,d_i, o← m_i\,o. (9) The remaining axes are schedule, which activates gates either simultaneously or from coarse to fine components, and initialization, which starts either warm on located components or cold across all components. Table 5: Design axes used in the conditional-gating sweep. Axis Values Gate structure Flat, conditional, shared Edit class Frozen direction, learned direction, component scaling Schedule Simultaneous, coarse-to-fine Initialization Warm localized, cold global We evaluate the full grid at two scopes: only the located components or all attention blocks, MoE blocks, and heads. This comparison tests whether explicit localization improves over allowing the gate to rediscover useful components during training. The resulting sweeps determine the configuration used in the seed study, a conditional gate structure with a frozen-direction edit, a simultaneous schedule, and a cold global initialization. 5.7 Models and Seeds We evaluate three open-weight MoE models spanning two routing regimes. OLMoE (9) routes each token among experts without an always-active component. Granite (5) uses the same routing regime but is deeper, separating depth from routing effects. Qwen1.5-MoE (12) additionally contains shared experts that activate for every token, allowing us to test whether the method transfers when behavior cannot be attributed entirely to routed expert selection. All post-localization experiments are run with three seeds per model. Each seed resamples the fitting data and reinitializes the learned gates, so the reported intervals capture both sources of intervention variability. 5.8 Intervention Scope and Edit Positions Every direction dcd_c is estimated at the decision token of Section 4.1, and every intervention is applied at that same token and no other. The contrast between unconditional and conditional subtraction is therefore one of content, whether the edit magnitude depends on the activation’s belief content, not of position. For sycophancy scoring and the multiple-choice retention benchmarks (MMLU, GSM8K, TruthfulQA), the decision token is where the option logits of Equation (2) are read and where the edit fires, leaving every prompt and reasoning-context token unedited. Perplexity is measured on plain text with no answer-decision position, so the intervention does not fire there; it is a structural control (Section 4.3) rather than a test of open-ended generation, which we leave to the extension of Section 7. Our claims therefore concern the model’s answer at the decision token. Figure 5: Every operating point for all methods and models, plotting removed belief push against GSM8K relative to baseline (%). The subtractions and FARE give many points because they are swept over component sets and coefficients, unlike the single-point gate and prompting. 6 Results We report results in the order of the workflow. We first show where the belief shift is localized in each model, then evaluate the six methods of Table 3 on the panel of Table 4, moving from the unedited baseline through prompting, the routing baseline, the two subtractions, and the learned gate. Removal is the percentage of the baseline belief push that an edit removes. 6.1 Localization The causal search resolves the belief shift to a small and consistent region of each model. At the block level, Figure 3 shows that the per-layer belief push is near zero through the early and middle layers and rises sharply in the mid-to-late layers, so the components that carry the shift are concentrated rather than spread through the network. The search selects six attention blocks and three mixture-of-experts blocks in OLMoE, eight and one in Qwen, and six and three in Granite, always in the upper half of the stack. Opening those blocks into finer components sharpens the picture. Among attention heads, Figure 8, Appendix B shows that the effect is carried by a handful of heads standing out against a near-silent background, eight in OLMoE, ten in Qwen, and thirteen in Granite, again confined to the mid-to-late layers. Among experts, the picture divides by routing regime. Figure 9, Appendix B locates five experts in OLMoE and six in Granite, but finds none in Qwen, because the shared experts that fire on every token cannot appear in a routed contrast. This is the first place the shared-expert regime separates from the other two, and it recurs in the results below. The union of these components, across attention blocks, attention heads, mixture-of-experts blocks, and experts, is the located set that every later edit acts on, and Figure 2 shows the learned gate later opening on these same components. 6.2 Baseline and Prompting The unedited model is sycophantic in every case, which fixes the baseline against which the panel is measured. Prompting the model to answer objectively does not help. Figure 4 shows that instruction alone moves the push in the wrong direction, increasing it by 29% on OLMoE, 8% on Qwen, and 24% on Granite. Thus, a stated instruction to ignore the user’s opinion leaves the model more sensitive to that opinion rather than less. Prompting is therefore not a substitute for an activation-level edit. 6.3 Routing and Unconditional Subtraction The routing baseline confirms that sycophancy is not controlled by which experts fire. Reweighting the router away from the belief-associated experts removes some push, 39% on OLMoE, 23% on Qwen, and 41% on Granite in Figure 4, but it pays for it in retention, dropping GSM8K to 68%, 83%, and 85% of baseline and raising perplexity markedly on OLMoE and Granite. Moving experts in or out is a blunt lever, and it damages the model without fully removing the behavior. Unconditional subtraction at the located components tells the same story from the other side. As the strength coefficient increases, the constant subtraction lowers the push but lowers retention with it, and this holds at every rung of the ladder. The located components at each level are shown in Figure 3 for blocks, Figure 8, Appendix B for heads, and Figure 9, Appendix B for experts. The trade-off is most severe at the expert level, since a fixed subtraction reaches heavily used experts on every token and collapses GSM8K almost immediately. At a matched operating point, the constant edit removes 42% of the push on OLMoE, 31% on Qwen, and 38% on Granite in Figure 4, while holding only 72%, 65%, and 75% of GSM8K. The damage is sharpest on GSM8K, the most sensitive metric, while MMLU holds higher, at 92%, 93%, and 99%, and TruthfulQA stays near baseline, at 101%, 98%, and 93%. This is the removal-retention gap the conditional form is built to close. 6.4 Conditional Subtraction and Gating Replacing the constant edit with the conditional edit closes most of that gap with no other change. Because the amount removed at each token scales with the projection of the activation onto the belief direction, tokens that never carried the belief are left almost untouched, so retention holds while removal rises. In Figure 4 conditional subtraction removes 77% of the push on OLMoE, 87% on Qwen, and 60% on Granite, holding GSM8K at 100%, 92%, and 103% of baseline with perplexity unchanged, MMLU at 86%, 85%, and 80%, and TruthfulQA at 102%, 86%, and 66%. Every operating point removes more push at higher retention than the constant edit on the same model, and the advantage widens as the granularity sharpens from blocks to heads to experts. Its one clear cost is Granite TruthfulQA, which the learned gate recovers. The gate is selected from the seventy-two-run sweep of Section 5.6 before evaluation. The conditional gate structure removes more push than the flat or shared structures on every model, and the simultaneous schedule removes more than the coarse-to-fine one (Figure 13, Appendix C). Ranking the qualifying configurations by distance to complete removal under reasoning and fluency penalties selects a frozen conditional gate with a simultaneous schedule and cold start, placed at high removal below the fluency ceiling in every model (Figures 14, 15, and 16, Appendix C). Across three seeds this gate is the strongest method on the behavioral axis, removing 90% of the push on OLMoE, 63% on Qwen, and 32% on Granite while holding GSM8K at 115%, 111%, and 115% of baseline. Unlike conditional subtraction it holds the full panel, with MMLU at 82%, 85%, and 89%, TruthfulQA at 98%, 96%, and 96%, and the decision-token-off perplexity control within 1% throughout. It closes the push by moving the nudged side toward the preserved no-nudge side rather than flattening both, which keeps retention intact where the routing and constant baselines could not. The components it opens on are stable across seeds (Figure 19, Appendix D), so the localization is a property of the model and not a single run, and a small number of mid-to-late-layer attention and expert sites carry most of the edit (Figures 17 and 18, Appendix D). 7 Analysis and Conclusion The removal-retention trade-off depends mainly on the form of the edit. Unconditional and conditional subtraction use the same belief direction and target components, but the conditional methods scale the intervention by the belief content of each activation. This raises retention benchmarks from 72% to 100% on OLMoE and from 75% to 103% on Granite while also improving removal. The difference is largest for experts, where fixed subtraction is more likely to perturb unrelated computations. Across benchmarks, the conditional methods generally keep MMLU and TruthfulQA near baseline. The main exception is Granite on TruthfulQA, where the learned gate preserves retention better than conditional subtraction, making it the more reliable method despite lower removal. GSM8K retention reaches 111-115% with the gate, above conditional subtraction. We do not interpret values above 100% as improved reasoning, as they may reflect sampling variation, operating-point selection, low baselines, or regularization from the intervention (Figure 4, and Figures 5 and 6 in Appendix A). The baselines fail differently. Prompting increases belief push, while routing removes some push at the cost of retention and perplexity. Qwen provides the clearest counterexample to a routing-based explanation. No sycophancy-specific routed experts are found, yet conditional editing still removes much of the behavior. Sycophancy therefore appears to depend more on component computation than on expert selection. Performance also varies by architecture, with the gate removing the most push on OLMoE and the least on Granite, and Qwen between; Figure 15 in Appendix C suggests that removal at fixed retention depends on how concentrated the behavior is within the located components. Overall, across three mixture-of-experts models, two routing regimes, and three seeds, this conditional form improves the removal-retention frontier without changing the direction or target components. The remaining limitations are that strong subtraction can overshoot into negative belief push and that evaluation is restricted to the decision token in multiple-choice tasks. Extending conditional editing to open-ended generation and other belief-induced behaviors is therefore the natural next step. References Belrose et al. (2023) N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman LEACE: perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.3. Fayyaz et al. (2025) M. Fayyaz, A. Modarressi, H. Deilamsalehy, F. Dernoncourt, R. Rossi, T. Bui, H. Schütze, and N. Peng Steering MoE LLMs via expert (de)activation. arXiv preprint arXiv:2509.09660. Cited by: §2. Genadi et al. (2026) R. Genadi, M. Nwadike, N. Mukhituly, H. AlQuabeh, T. Hiraoka, and K. Inui Sycophancy hides linearly in the attention heads. arXiv preprint arXiv:2601.16644. Cited by: §2. Granite Team, IBM (2024) Granite Team, IBM Granite 3.0 language models. IBM Research. External Links: Link Cited by: §5.7. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), Cited by: §4.3. Lee and Choi (2026) J. Lee and K. S. Choi Routing sensitivity without controllability: a diagnostic study of fairness in MoE language models. arXiv preprint arXiv:2603.27141. Cited by: §4.3. Lin et al. (2022) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252. Cited by: §4.3. Muennighoff et al. (2024) N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, P. Walsh, O. Tafjord, N. Lambert, et al. OLMoE: open mixture-of-experts language models. arXiv preprint arXiv:2409.02060. Cited by: §5.7. O’Brien et al. (2026) C. O’Brien, J. Seto, D. Roy, A. Dwivedi, S. Dev, K. Zhu, S. O’Brien, A. Panda, and R. Lagasse A few bad neurons: isolating and surgically correcting sycophancy. arXiv preprint arXiv:2601.18939. Cited by: §2. Perez et al. (2023) E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, p. 13387–13434. Cited by: §1, §2, §3.2. Qwen Team (2024) Qwen Team Qwen1.5-MoE: matching 7B model performance with one third activated parameters. External Links: Link Cited by: §5.7. Rimsky et al. (2023) N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. Cited by: §1, §2, §3.1, §3.2. Sharma et al. (2023) M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. Cited by: §1, §2. Wang et al. (2025) K. Wang, J. Li, S. Yang, Z. Zhang, and D. Wang When truth is overridden: uncovering the internal origins of sycophancy in large language models. arXiv preprint arXiv:2508.02087. Cited by: §2. Zhang et al. (2026) Z. Zhang, Y. Li, Z. Ouyang, L. Shi, and K. Wang Understanding safety-sensitive expert behavior in mixture-of-experts LLMs. arXiv preprint arXiv:2605.29708. Cited by: §2. Appendix A Removal and Retention Across Methods This appendix collects the figures referenced from the main text. All are produced under the conditions of Section 6, three seeds per model, with edits applied only to the located components. Figure 6: The frontier, one model at a time. Each point is one operating point, with belief push removed (%) on the horizontal axis and GSM8K retained (%) on the vertical axis; marker shape and colour denote the method (see legend), and the dashed line marks baseline retention. Points toward the upper right remove more push while keeping more reasoning. The learned gate and conditional subtraction lead in every model; the shaded band marks the high-removal region where retention stays near baseline. Figure 7: A single ranking. For each model, the panel Z-score combines behavioral removal with retention into one all-round score (higher is better); bars are ordered within each panel and labelled with their value. The two conditional methods, conditional subtraction and the learned gate, rank first in every model, while the routing baseline (FARE) ranks last. Appendix B Subtraction at Each Granularity These figures separate the subtraction sweep of Section 5.4 by component type, comparing the constant edit against the conditional edit as the strength coefficient increases. In every panel the conditional edit holds retention as removal climbs, while the constant edit trades it away, and the gap widens as the granularity sharpens from blocks to heads to experts. Figure 8: The carrying heads. Per-head causal effect on the belief push, shown as a heatmap over layer (horizontal) and head index (vertical) with colour giving the magnitude of the effect; the selected heads are boxed in magenta, eight in OLMoE, ten in Qwen, and thirteen in Granite. A few heads stand out against a near-silent background, all in the mid-to-late layers. Figure 9: The carrying experts. Each selected expert is placed by layer (horizontal) and expert index (vertical) and labelled by its location, five in OLMoE and six in Granite. Qwen has none, because its always-on shared experts cannot appear in a routed contrast, the first sign that the shared-expert regime behaves differently. Figure 10: Belief accumulates in the mid-to-late layers. For each model, the left panel tracks the model’s lean toward the user’s stated option (logits, vertical axis) as the residual stream passes through successive layers (horizontal axis), comparing a belief run against a matched no-belief run; the shaded region between the two curves is the preference caused by the stated opinion, and the dotted curve is a random-direction control. The right panel shows how much preference each individual layer contributes, with the dashed marker at the layer by which half of the total preference has formed. In every model the lean is flat through the early layers and rises sharply in the upper layers. Figure 11: Where the belief signal is read and carried, per head. For each model, the left heatmap gives each head’s estimated causal effect on the belief push and the middle heatmap gives how much attention each head places on the stated-opinion text, both over layer (vertical) and head index (horizontal). The right panel plots attention on the opinion (horizontal) against causal effect (vertical), with colour showing the extra attention a head pays when an opinion is present; the labelled heads in the upper region both attend to the opinion and carry its effect, which we call transporter heads. Figure 12: GSM8K holds until the coefficient grows large. Each panel plots GSM8K accuracy (vertical axis) against the conditional-subtraction strength c (horizontal axis). Colour marks the component set the edit is applied to: S1 attention blocks; S2 MoE blocks; S3 attention and MoE blocks; S4 attention heads; S5 experts; S6 heads and experts; S7 attention blocks and heads; S8 MoE blocks and experts; and S9 all located components. Within each set, thin lines are the individual seeds and the thick line is their mean. Accuracy stays roughly flat at low c and collapses once c becomes large, at a threshold that differs by model. Appendix C Gate Architecture Sweep These figures detail the seventy-two-run sweep of Section 5.6 and the selection of the configuration carried into the seed study. Figure 13: Main effect of each design axis on belief push removed, per model (one bar per model; higher removes more). The axes and levels are edit class (ComponentMask CM, VectorMaskFrozen VF, VectorMaskLearned VL), gate structure (FlatGate FG, CondGate CG, SharedGate SG), schedule (FullTrain FT, FineToCoarseTrain FC), and initialisation (WarmStart W, ColdStart C). The conditional gate (CG) and full schedule (FT) remove the most push across all three models, motivating those two choices in the selected configuration. Figure 14: Choosing the gate. Each row is a candidate configuration named in the edit-gate-schedule-init convention of Figure 13 (for example VF-CG-FT-C), restricted to configurations that qualify on all three models. Rows are scored by their distance to complete removal under reasoning and fluency penalties, with over-removal runs discarded; lower is better, the starred row is the winner, and the final column notes why each runner-up was not chosen. The winner, VF-CG-FT-C, a frozen conditional gate (VectorMaskFrozen with CondGate) trained on a full schedule from a cold start, is the pick carried into the seed study. Figure 15: Choosing the gate architecture, one territory per model. Each point is a configuration labelled edit-gate-schedule-init; the horizontal axis is nudge removed (%) and the vertical axis is the GSM8K change versus baseline (negative means reasoning lost). Each shaded region encloses one model’s top-ten configurations, and the star, pentagon, and cross mark that model’s first, second, and third best picks. The acronym key is given in the side panel. Figure 16: Architecture sweep on the removal-against-perplexity plane, used for configuration selection. The horizontal axis is nudge removed (%) and the vertical axis is the perplexity change versus baseline (positive means fluency lost), with the dashed line marking the honest ceiling and the background shading giving the distance-to-ideal ranking score (brighter is better). Perplexity here is the decision-token-off control of Section 4.3, so this plane screens configurations for gross perplexity regressions rather than measuring open-ended fluency; the selected configurations sit at high removal with perplexity near baseline. Appendix D Gate Localization Detail Figure 17: The edit is sparse. Per-site edit amount (1−m1-m) of the selected gate at its checkpoint, one bar per component with the dashed line marking the inclusion threshold. A handful of attention and mixture-of-experts sites carry most of the edit, matching the concentrated placement of Figure 2. Figure 18: What the gate edits, and what it leaves. The most and least edited components of the selected gate, ranked by edit amount and coloured by component type. The heavily edited sites are mid-to-late-layer attention and expert components, while the least edited are individual attention heads. Figure 19: The localization is stable. Gate activity across the three seeds (s1, s2, s4), with colour giving each component’s edit amount. The set of components the gate opens on is consistent from seed to seed, so the located support is a property of the model rather than of a single run. Figure 20: Method comparison across the three MoE models. Each axis is a metric (Removal, Invariance, MMLU, GSM8K, TruthfulQA, PPL), normalised so that farther out is better, and each closed curve is one method. Conditional subtraction (Cond sub) and the learned gate achieve high removal while retaining capability, whereas prompting and FARE trade little removal for capability or fail to remove sycophancy.