Paper deep dive
Vernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoning
Zhenyu Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 6/20/2026, 6:49:31 AM
Summary
The paper introduces 'Vernier', a method designed to investigate whether the 'lexical gap' in instruction-tuned language models (where models perform differently on causal reasoning tasks when variable names are replaced by placeholders) is caused by information loss or representational misalignment. Vernier uses a paired-view weight update (training on both original and placeholder-perturbed prompts) with an answer-subspace symmetric KL divergence loss. The research demonstrates that for certain model families like Qwen and Llama, the update successfully closes the lexical gap by improving the placeholder view's performance without degrading the original view, supporting the 'representational misalignment' hypothesis. The study uses mechanistic interpretability tools like activation patching and linear probes to validate these findings.
Entities (8)
Relation Signals (4)
Vernier → evaluatedon → CLadder
confidence 100% · CLadder (Jin et al., 2023) is our in-distribution benchmark.
Qwen 7B → istestedwith → Vernier
confidence 100% · We compare Vernier (β=1, answer-subspace KL) against five baselines on Qwen2.5-7B at step 500.
Vernier → investigates → Representational Misalignment
confidence 95% · Vernier uses a paired-view weight update as an instrument and then inspects the mechanism left after the gap closes.
CLadder → isrelatedto → CRASS
confidence 80% · CLadder (Jin et al., 2023), CRASS (Frohberg and Binder, 2022), and e-CARE (Du et al., 2022)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Instruction-tuned language models can answer the same causal-reasoning question differently after its English variable names are replaced by type-preserving placeholders, although the structural causal model and the gold answer are unchanged. We ask whether this lexical gap reflects information loss in the placeholder view or a misaligned read-out from a representation that still carries answer-relevant content. Vernier uses a paired-view weight update as an instrument and then inspects the mechanism left after the gap closes. In the working regimes, the evidence favours representational misalignment. A variable-name probe becomes more accurate on the placeholder view, and activation patching on Qwen-7B, Qwen-14B, and Llama-3.1-8B shows that the decision-token representation can transfer answer identity between views. The update that realigns the views is counterfactual augmentation over original and placeholder prompts, while the answer-subspace KL mainly sharpens intermediate answer-belief agreement. Success is bounded by model family, scale, and task. CRASS transfer is reliable across Qwen scales and Llama, e-CARE remains weak, and preliminary non-causal rename tasks show a similar qualitative pattern.
Tags
Links
- Source: https://arxiv.org/abs/2606.15733v1
- Canonical: https://arxiv.org/abs/2606.15733v1
Trouble viewing inline? Open PDF directly →
Full Text
91,525 characters extracted from source content.
Expand or collapse full text
Vernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoning Zhenyu Yu College of Computer Science and Artificial Intelligence Fudan University yuzhenyuyxl@foxmail.com Abstract Instruction-tuned language models can answer the same causal-reasoning question differently after its English variable names are replaced by type-preserving placeholders, although the structural causal model and the gold answer are unchanged. We ask whether this lexical gap reflects information loss in the placeholder view or a misaligned read-out from a representation that still carries answer-relevant content. Vernier uses a paired-view weight update as an instrument and then inspects the mechanism left after the gap closes. In the working regimes, the evidence favours representational misalignment. A variable-name probe becomes more accurate on the placeholder view, and activation patching on Qwen-7B, Qwen-14B, and Llama-3.1-8B shows that the decision-token representation can transfer answer identity between views. The update that realigns the views is counterfactual augmentation over original and placeholder prompts, while the answer-subspace KL mainly sharpens intermediate answer-belief agreement. Success is bounded by model family, scale, and task. CRASS transfer is reliable across Qwen scales and Llama, e-CARE remains weak, and preliminary non-causal rename tasks show a similar qualitative pattern. 1 Introduction Instruction-tuned language models can give different answers to the same causal-reasoning question when its variable names change. Replacing surface forms such as smoking, husband, and alarm clock with typed placeholders X1,X2,X3X_1,X_2,X_3 in a structurally identical prompt produces a different answer distribution, even though the structural causal model, the joint distribution, the query, and the gold answer are all preserved. Yu and Zhou (2026) report this lexical gap across model families and find that direct prompting, chain-of-thought prompting, and scaffolded prompting leave it visible. Related 2025–2026 evaluations report similar brittleness under meaning-preserving rewrites, context shifts, and anti-semantic controls in causal or mathematical reasoning settings (Hao et al., 2025; Lee et al., 2025; Wang et al., 2026; Li et al., 2026). The failure is therefore not just that the model needs a better prompt. The scientific question is what kind of failure the lexical gap is. One possibility is an information deficit: after the names are removed, the placeholder view no longer contains the story-level content needed to answer. Another is representational misalignment: the content is still present, but the post-trained model reads it out differently from the two lexical forms. This distinction mirrors a broader line of work on spurious-signal reliance and prompt sensitivity in language models (McCoy et al., 2019; Webson and Pavlick, 2022; Sclar et al., 2024). These accounts matter because they imply different fixes. If the placeholder view loses information, then a successful update may simply teach the model to ignore or overwrite missing content. If the view is misaligned, a successful update should make the same item-level content more accessible from both forms. We study this question by using a targeted weight update as an instrument. The update is not proposed as a new general-purpose fine-tuning recipe. It is a controlled way to ask what changes when the gap closes. Our method, Vernier, trains a LoRA adapter on paired lexical views (Hu et al., 2022). Both views receive answer supervision, and an answer-subspace symmetric KL term encourages their answer distributions to agree, following the general consistency-training principle while changing the view-construction mechanism (Xie et al., 2020; Liang et al., 2021). The ablations below show that the paired lexical views, rather than generic consistency, are the main source of behavioural closure. The KL term is used to make the answer-belief alignment explicit and measurable. Recent benchmark work also motivates this design by showing that prompt-side variation alone does not reliably yield stable reasoning behavior (Sclar et al., 2024; Chi et al., 2024). The answer-subspace restriction matches the evaluation metric, which depends only on a small answer set rather than the full vocabulary. The rest of the paper is organised around falsification tests for the competing explanations. If closure removes information, story-level lexical content should become less recoverable from the perturbed view. If generic consistency is enough, then dropout consistency without the lexical perturbation should help. If the two views differ by a global direction, then a map fitted once from paired hidden states should close the gap at inference. If the update merely damages capability, downstream evaluations should reveal that. We therefore combine behavioural evaluation with probes and logit-lens diagnostics (Belrose et al., 2023), activation patching (Zhang and Nanda, 2024), steering tests (Turner et al., 2023; Zou et al., 2023), capability checks (Hendrycks et al., 2021), and model-family stress tests. This structure is intended to identify what the update repairs, beyond whether it improves a benchmark number. Contributions. • We turn a lexical robustness failure into a falsifiable mechanism question. The paper separates an information-deficit account from a representational-misalignment account using paired-view training, probes, logit-lens diagnostics, activation patching, and inference-time steering tests. The evidence is organised so that each diagnostic can rule out a specific weaker explanation rather than merely add another benchmark number. • We introduce Vernier as a controlled realignment instrument. Vernier trains on original and structure-preserving perturbed views with answer supervision on both and an answer-subspace consistency penalty. The paired-view augmentation is the behavioural engine, while the answer-subspace KL provides a belief-level alignment diagnostic. The method is deliberately narrow: it targets the answer set used by the causal-QA evaluation and is used to ask what changes when a lexical gap closes. • We map the boundary conditions of the effect. The experiments cover multiple model families, a within-family Qwen scale sweep, out-of-distribution causal benchmarks, capability checks, placeholder-scheme stress tests, non-causal rename tasks, and failed global-map interventions. This lets us state where the update works and where it fails, transfers weakly, or cannot be reproduced by a single inference-time direction. 2 Related Work 2.1 Lexical robustness and causal reasoning in LLMs Yu and Zhou (2026) introduce the structure-preserving lexical perturbation T(⋅)T(·) we adopt on CLadder (Jin et al., 2023), CRASS (Frohberg and Binder, 2022), and e-CARE (Du et al., 2022): content words that name variables are replaced by typed placeholders while the causal graph, query type, answer set, and gold answer are held fixed. The perturbation is designed to expose whether a model’s causal answer depends on semantically rich surface names rather than on the structural causal query. This surface dependence is part of a broader robustness pattern. Chi et al. (2024) argue that current LLMs perform mostly shallow causal reasoning rather than genuine structural inference, Hao et al. (2025) show that renaming variables degrades mathematical reasoning by several points, and Lee et al. (2025) find that causal predictions fail to track changing context. These works motivate asking not only whether a gap exists, but what internal failure it reflects. For the causal-QA case we ask whether the gap is information loss or representational misalignment, and we answer it mechanistically rather than only measuring it. The structural causal model framework of Pearl (2009); Pearl and Mackenzie (2018) formalises the observational, interventional, and counterfactual queries the three benchmarks operationalise. Adjacent work shows that LLMs exploit surface features in ways that interact with task-relevant signal. McCoy et al. (2019) demonstrate syntactic-heuristic reliance on NLI. Webson and Pavlick (2022) show that prompt-based zero-shot models can ignore instruction semantics while remaining sensitive to surface choices. Sclar et al. (2024) quantify sensitivity to prompt formatting. Min et al. (2022) find that label correctness in in-context demonstrations matters less than label distribution. Chain-of-thought prompting (Wei et al., 2022b; Kojima et al., 2022) and self-consistency (Wang et al., 2023) are natural prompt-side interventions. Because they do not modify the model’s read-out from the two lexical views, they motivate our weight-update approach rather than replacing it. 2.2 Invariance, consistency training, and counterfactual augmentation Vernier belongs to the broad family of methods that build a known invariance into a model. Where group-equivariant networks (Cohen and Welling, 2016) hard-wire invariance to a symmetry group into the architecture, and consistency-regularised semi-supervised learning (Xie et al., 2020) softens a known data augmentation into a penalty, Vernier treats the structure-preserving perturbation T of Yu and Zhou (2026) as the task symmetry and penalises violations of it as a soft constraint on the answer distribution. The perturbation is a genuine invariance of the task by construction, which is what separates this setting from the more common one where the augmentation is only approximately label-preserving. Within consistency training, R-Drop (Liang et al., 2021) adds a symmetric KL between two dropout-sampled forward passes of one input and SimCSE (Gao et al., 2021) contrasts two dropout-sampled views of one sentence, both treating the two views as stochastic copies of a single input. Vernier differs in two ways. The two views are produced by the lexical edit rather than by dropout sampling, which makes the consistency target a property of the data distribution and not of the network’s internal noise, and the KL is restricted to the answer-token subspace rather than the full vocabulary. The first difference is essential because the R-Drop baseline widens the gap without T. Kaushik et al. (2020) train classifiers on human-rewritten counterfactual pairs to improve robustness to spurious correlations. Our β=0β=0 ablation is the closest analogue without the KL term and closes most of the in-distribution gap but only part of the out-of-distribution gap. 2.3 Mechanistic interpretability of hidden representations Our mechanism analysis uses a logit-lens-style diagnostic: intermediate hidden states are projected through the unembedding matrix and compared on the answer-token subspace. The tuned-lens variant (Belrose et al., 2023) learns a small per-layer adapter for a related latent-prediction analysis. The transformer-circuits framework of Elhage et al. (2021) provides the conceptual vocabulary for the per-layer analysis. Meng et al. (2022) locate factual associations to mid-layer MLP modules, and Geva et al. (2023) dissect per-layer dynamics of factual recall. Our linear-probe analysis uses a single multinomial classifier per layer to test layerwise recoverability of the variable-name family from the hidden state, complementary to the causal-mediation techniques of these prior works. A related line controls model behaviour by adding a single direction to the residual stream, with activation addition (Turner et al., 2023) and representation engineering (Zou et al., 2023) steering attributes such as sentiment, truthfulness, and refusal. We use the same construction as a test and find the causal-reasoning lexical gap does not admit such a direction, unlike those attributes, while non-causal variants are partly steerable. Instruction tuning by reinforcement learning from human feedback (Ouyang et al., 2022) and by multitask prompted training (Sanh et al., 2022; Wei et al., 2022a) are the post-training stages that produce the lexical sensitivity we close. We use LoRA (Hu et al., 2022) for parameter-efficient fine-tuning, AdamW (Loshchilov and Hutter, 2019) optimisation, and QLoRA (Dettmers et al., 2023) for the 14B and larger models. Figure 1: Vernier training schematic. The original prompt q and its structure-preserving perturbation T(q)T(q) are forwarded through the same LoRA-adapted model. The loss combines per-view answer supervision with an answer-subspace symmetric KL term. 3 Method: Vernier The method has three parts (see Figure 1). The lexical perturbation is formalised as a structure-preserving transformation that should leave the answer distribution invariant. Vernier then trains on paired lexical views with a task loss on both views and a consistency loss on the answer-token subspace. The answer-subspace loss is in turn connected to the identifiability constraint, including the margin condition needed to translate distributional agreement into arg-max agreement. We use the update as an instrument rather than as the whole argument: the method is designed to create a controlled before/after comparison whose hidden-state consequences can then be inspected. 3.1 Lexical perturbation and gap A causal-QA item is (q,y⋆,M)(q,y ,M), where q is the prompt, y⋆∈y is the gold answer, and M is the structural causal model. The lexical perturbation T replaces each SCM variable name and its variants with a typed placeholder X1,X2,…X_1,X_2,…, applied consistently throughout the item. A worked instance is provided in Table 1. It preserves M, the query, and y⋆y , so an SCM-grounded reasoner should satisfy R(⋅∣q)=R(⋅∣T(q))R(· q)=R(· T(q)). We score a model by the signed lexical gap Δ(θ;)=(q,y⋆)∼[y^θ(q)=y⋆−y^θ(T(q))=y⋆], (θ;D)=E_(q,y ) [1\ y_θ(q)=y \-1\ y_θ(T(q))=y \], (1) where predictions use the answer-token distribution at the first generation position. The perturbation is deliberately conservative. It does not paraphrase the story, change numbers, alter the causal graph, or change the requested answer type. It only removes the English variable names that can act as lexical anchors. This matters for interpretation: a remaining gap cannot be attributed to a changed SCM or a changed label, while a closed gap cannot by itself prove deeper causal understanding. The method therefore asks a narrower question: given two views that should be equivalent for the task, can a small weight update make the model read out the same answer-relevant information from both? Table 1: A worked CLadder example. Gold answer y⋆=noy =no, Pearl rung observational. Underlined surface forms in q are mapped by T to placeholders. Numeric quantities, query, and answer are unchanged. Original prompt q Perturbed prompt T(q)T(q) The overall probability of alarm set by husband is 77%. For husbands that don’t set the alarm, the probability of ringing alarm is 26%. For husbands that set the alarm, the probability of ringing alarm is 76%. Is ringing alarm less likely than silent alarm overall? Answer (yes or no): The overall probability of X3X_3 set by X2X_2 is 77%. For X2X_2 that don’t set the X3X_3, the probability of ringing X3X_3 is 26%. For X2X_2 that set the X3X_3, the probability of ringing X3X_3 is 76%. Is ringing X3X_3 less likely than silent X3X_3 overall? Answer (yes or no): 3.2 Vernier objective Vernier trains a LoRA adapter on paired views (P0,P1)=(q,T(q))(P_0,P_1)=(q,T(q)). The original view P0P_0 keeps the lexical anchors seen by the base model, while the perturbed view P1P_1 removes them while preserving the structural causal problem. Training on both views prevents the update from solving the task by specialising only to one surface form. The task loss makes each view answerable on its own, while the consistency term penalises disagreement between the two answer distributions: ℒ=α[ℒtask(P0)+ℒtask(P1)]+βℒcons(P0,P1),ℒcons=12KL(pP0∥pP1)+12KL(pP1∥pP0).L=α[L_task(P_0)+L_task(P_1)]+β\,L_cons(P_0,P_1), _cons= 12KL(p_P_0^Y\|p_P_1^Y)+ 12KL(p_P_1^Y\|p_P_0^Y). (2) Here pP0p_P_0^Y and pP1p_P_1^Y are the softmax distributions after restricting the next-token logits to the finite answer set Y. For binary yes/no CLadder items, Y contains the answer tokens for yes and no, and for multiple-choice CRASS and e-CARE items it contains the option tokens. We use α=β=1α=β=1. The loss is applied at the first generation position, and only LoRA parameters are updated. One training step is given in Algorithm 1. The answer-subspace restriction is a design choice, not a computational shortcut. A full-vocabulary KL would require two prompts with different surface forms to agree on every next-token continuation, including continuations that restate the variable names. That target is stronger than the task invariance and can penalise harmless lexical differences. By contrast, the answer-subspace KL matches the evaluation interface: if two views encode the same causal problem, the distribution over admissible answers should agree even if the unrestricted language-model distribution does not. Algorithm 1 Vernier training (one outer step). 1:base LM fθf_θ with frozen weights, LoRA adapter ϕφ, answer-token id set Y, batch (qi,T(qi),yi⋆)i=1B\(q_i,T(q_i),y_i )\_i=1^B. 2:P0,P1←P_0,P_1← tokenised original and perturbed prompts. 3:(v)←fθ+ϕ(Pv)z^(v)← f_θ+φ(P_v) for v∈0,1v∈\0,1\ at the first generation position. 4:(v)←softmax(⋅,(v))p^(v) (z^(v)_·,Y). 5:ℒtask←−B−1∑i[logi,yi⋆(0)+logi,yi⋆(1)]L_task←-B^-1 _i[ ^(0)_i,y_i + ^(1)_i,y_i ]. 6:ℒcons←(2B)−1∑i[KL(i(0)∥i(1))+KL(i(1)∥i(0))]L_cons←(2B)^-1 _i[KL(p^(0)_i\|p^(1)_i)+KL(p^(1)_i\|p^(0)_i)]. 7:Update only ϕφ with AdamW on αℒtask+βℒcons _task+ _cons. 3.3 Distinguishing misalignment from information loss Vernier is useful because the two explanations introduced above predict different after-effects. Under a simple information-deficit account, the perturbed prompt lacks answer-relevant story content. A successful update could still close the gap, but it would have to do so by adding a shortcut, memorising an answer pattern, suppressing lexical information, or regressing the original view. Under a representational-misalignment account, the perturbed view still carries item-level content, but the post-trained read-out fails to use it in the same way as the original view. A successful update should then make the two views more similar at the decision interface while preserving, or increasing, recoverability of the story-level lexical family from the perturbed representation. This is why the method is paired with mechanism tests rather than evaluated only by accuracy. The primary behavioural signature is not merely a smaller gap, but a smaller gap with P1P_1 improving and P0P_0 not collapsing. The mechanism signature is stronger: cosine and logit-lens agreement should improve at the decision token, a probe trained on P0P_0 should transfer better to P1P_1, and activation patching should show that the decision-token representation can carry answer identity across views. These are not assumptions of the method. They are the falsification tests that follow from using the method as an instrument. 3.4 Answer-subspace consistency The answer-subspace KL is an empirical plugin estimate of the identifiability violation on Y. Let DSKLD_SKL be the symmetric KL and let G(θ;)=q∼[DSKL(pθ(⋅∣q)∥pθ(⋅∣T(q)))].G(θ;D)=E_q [D_SKL(p_θ(· q)_Y\|p_θ(· T(q))_Y)]. (3) The batch mean of ℒconsL_cons is an unbiased estimate of G. Since symmetric KL is positive definite on the finite simplex, G=0G=0 iff the two answer-token distributions agree almost everywhere. Distributional agreement then implies equal arg-max predictions, but away from zero this only controls prediction flips under a margin condition. If pθ(⋅∣q)p_θ(· q)_Y has top-two margin at least γ, then a flip requires total variation at least γ/2γ/2, and Pinsker plus Jensen gives Pr[y^θ(q)≠y^θ(T(q))]≤2G(θ;)γ. [ y_θ(q)≠ y_θ(T(q))]≤ 2G(θ;D)γ. (4) We do not treat this bound as the load-bearing argument because near-tied answer distributions can violate the margin condition. Instead, the load-bearing evidence is behavioural closure together with the mechanism tests, which distinguish genuine P1P_1 improvement from degenerate closure through P0P_0 regression (see Section 4.4). The task loss prevents the trivial constant-output solution that would satisfy the consistency term with zero useful accuracy. 3.5 Identification limits The method does not prove that the model has learned a human-interpretable causal algorithm. It only tests whether the lexical gap is removable by a small paired-view update and what kind of internal change accompanies that removal. Several outcomes would falsify the misalignment reading, including closure driven by P0P_0 regression, lower probe transfer from P1P_1, no movement in decision-token agreement, or a generic consistency baseline such as R-Drop producing the same effect without ever seeing T(q)T(q). These tests are summarised in Table A24. We observe some of these failure signatures on Phi-3 and Gemma, which is why the later claims are scoped by model family and capacity. Conversely, when Vernier closes the gap with P1P_1 improvement and the mechanism diagnostics move in the predicted direction, the result supports a narrower conclusion. In that regime, the answer-relevant content was not simply erased by replacing names with placeholders. 4 Experiments 4.1 Experimental setup Datasets. CLadder (Jin et al., 2023) is our in-distribution benchmark. We use items 0 through 1499 of the commonsensical split for training and a disjoint slice of items 1500 through 1699 (n=200n=200) as the held-out evaluation set. CRASS (Frohberg and Binder, 2022) and e-CARE (Du et al., 2022) are out-of-distribution. No item from either is seen during training. We use 200-item samples from each. For general-capability evaluation we sample 200 items each from MMLU (Hendrycks et al., 2021), HellaSwag, and GSM8K under a fixed seed. CLadder also contains a nonsensical subset with fictional-concept variable names, used for the falsifiability test. To probe whether the account may extend beyond causal reasoning, we add two non-causal BBH tasks (Suzgun et al., 2023), logical_deduction and reasoning_about_colored_objects, with object names renamed to typed placeholders. Baselines. We compare Vernier (β=1β=1, answer-subspace KL) against five baselines on Qwen2.5-7B at step 500. Base. The pretrained instruction-tuned model with no fine-tuning. P_0-only SFT. Supervised fine-tuning on the original prompt alone, with no perturbed view. R-Drop (Liang et al., 2021). Symmetric KL between two dropout-sampled forward passes of P0P_0, with no perturbation T. Twin task (β=0β=0). Both views routed through the task loss, with no consistency term. Full-vocabulary KL (β=1β=1). The same training as Vernier but with the KL computed over the full vocabulary rather than the answer subspace. Metrics. For each item we score (q,T(q))(q,T(q)) by taking the softmax over the answer-token subset at the first generation position. The primary metric is the signed lexical gap P0−P1P_0-P_1 in percentage points, reported alongside P0P_0 and P1P_1 accuracy so that closure driven by P1P_1 improvement can be distinguished from closure driven by P0P_0 regression. A gap at or near zero is the goal. A negative gap is an inversion in which P1P_1 exceeds P0P_0, which we report as such rather than as a larger success. For OOD benchmarks we also give the relative gap reduction Δgap=(gapVernier−gapbase)/gapbase _gap=(gap_Vernier-gap_base)/gap_base as a convenience, but we caution that this ratio is unstable when the base gap is small and can exceed 100 percent when the gap crosses zero, so we anchor all claims on the signed gap in percentage points rather than on this ratio. On general-capability benchmarks we report top-1 accuracy. Implementation. We attach a LoRA (Hu et al., 2022) adapter of rank r=16r=16 to the query, key, value, and output projections of every attention layer, with LoRA alpha 32 and dropout 0.05. We train with AdamW (Loshchilov and Hutter, 2019) at learning rate 1×10−41× 10^-4, weight decay 0, batch size 4 pairs per micro-step, gradient accumulation 4, linear warmup over 50 steps, cosine decay thereafter, and a total budget of 500 steps. We evaluate nine instruction-tuned models spanning five families, namely Phi-3-mini-4k-instruct (3.8B) (Abdin et al., 2024), Qwen2.5 at 1.5B, 3B, 7B, 14B, and 32B (Qwen Team, Alibaba Group, 2024), Llama-3.1-8B-Instruct (Llama Team, AI @ Meta, 2024), Mistral-7B-Instruct-v0.3 (Jiang et al., 2023), and Gemma-2-2B-it (Gemma Team, 2024). The 1.5B to 8B models train in BF16 on a single 24 GB GPU. The 14B and 32B models use QLoRA (Dettmers et al., 2023) on a 40 GB GPU, the 32B with batch size 2 and gradient accumulation 8. Phi-3 trains at learning rate 5×10−65× 10^-6 with early stopping at step 70, since the canonical learning rate drives its adapter to a degenerate constant-output minimum by step 50. On Mistral-7B the canonical learning rate produces an aggressive in-distribution closure with substantial downstream-capability degradation. We rescue it with the Phi-3 protocol (lr=5×10−65\!×\!10^-6, early stopping at step 150) and report both rows in the main tables. The hyperparameter trade-off is analysed with the failure modes. The full recipe-and-selection audit gives the metric used to pick each rescued checkpoint (see Appendix C). 4.2 Comparison Table 2: In-distribution (CLadder) and out-of-distribution (CRASS, e-CARE) held-out results, n=200n=200 each, grouped by dataset. Adapters are trained only on CLadder. The gap is P0−P1P_0-P_1 in percentage points and Δgap _gap is the relative gap reduction versus base (OOD only; in-distribution base gaps are too small for a stable ratio). All ± are 11 s.d.: for the Vernier rows of Qwen-7B, Qwen-14B, and Llama-3.1-8B († ) this is the std across three rounds; for every other cell (all base rows and the single-seed Vernier rows of Phi-3, Qwen-32B, and Mistral-7B) it is the paired bootstrap standard error over the 200200 items. Within each dataset block, bold marks the best and underline the second best per column (P0P_0/P1P_1 higher is better; Gap closest to 0 is better; Δgap _gap more negative is better). Qwen-32B uses QLoRA; Mistral-7B (‡ ) is reported under the default and rescued recipes. Base Vernier Data Model P_0 P_1 Gap P_0 P_1 Gap Rel. gap CLadder (ID) Phi-3-mini (3.8B) 0.585±0.035 0.540±0.035 +4.5±2.3 0.550±0.035 0.550±0.035 0.0±2.1 — Qwen2.5-7B† 0.625±0.034 0.545±0.035 +8.0±2.6 0.868±0.003 0.893±0.010 -2.5±0.9 — Qwen2.5-14B† 0.615±0.035 0.580±0.035 +3.5±2.7 0.955±0.029 0.950±0.013 +0.5±1.4 — Qwen2.5-32B (QLoRA) 0.565±0.035 0.585±0.035 -2.0±2.1 0.970±0.012 0.965±0.013 0.5±1.3 — Llama-3.1-8B† 0.540±0.035 0.515±0.035 +2.5±3.6 0.945±0.005 0.948±0.008 -0.3±1.0 — Mistral-7B‡ (lr=1e-4) 0.610±0.034 0.565±0.035 +4.5±3.9 0.980±0.010 0.995±0.005 -1.5±1.1 — Mistral-7B‡ (lr=5e-6, s150) 0.610±0.034 0.565±0.035 +4.5±3.9 0.605±0.035 0.600±0.035 +0.5±3.1 — CRASS (OOD) Phi-3-mini (3.8B) 0.845±0.026 0.545±0.035 +30.0±3.9 0.840±0.026 0.550±0.035 +29.0±3.9 -3% Qwen2.5-7B† 0.845±0.026 0.505±0.035 +34.0±3.9 0.870±0.013 0.622±0.012 +24.8±0.3 -27% Qwen2.5-14B† 0.895±0.022 0.625±0.034 +27.0±3.5 0.918±0.008 0.722±0.013 +19.7±0.8 -27% Qwen2.5-32B (QLoRA) 0.910±0.020 0.635±0.034 +27.5±3.6 0.950±0.015 0.725±0.031 +22.5±3.3 -18% Llama-3.1-8B† 0.850±0.025 0.445±0.035 +40.5±3.8 0.872±0.008 0.558±0.035 +31.3±3.1 -23% Mistral-7B‡ (lr=1e-4) 0.795±0.028 0.415±0.035 +38.0±3.9 0.710±0.032 0.500±0.036 +21.0±3.4 -45% Mistral-7B‡ (lr=5e-6, s150) 0.795±0.028 0.415±0.035 +38.0±3.9 0.735±0.031 0.365±0.034 +37.0±3.8 -3% e-CARE (OOD) Phi-3-mini (3.8B) 0.795±0.029 0.590±0.035 +20.5±3.9 0.785±0.029 0.600±0.035 +18.5±3.9 -10% Qwen2.5-7B† 0.815±0.028 0.585±0.035 +23.0±3.8 0.810±0.010 0.622±0.012 +18.8±1.5 -18% Qwen2.5-14B† 0.820±0.027 0.605±0.035 +21.5±3.7 0.823±0.002 0.610±0.004 +21.3±0.5 -1% Qwen2.5-32B (QLoRA) 0.820±0.027 0.625±0.034 +19.5±3.8 0.815±0.028 0.595±0.035 +22.0±3.9 +13% Llama-3.1-8B† 0.790±0.029 0.550±0.035 +24.0±4.5 0.793±0.005 0.615±0.004 +17.8±0.6 -26% Mistral-7B‡ (lr=1e-4) 0.790±0.029 0.555±0.035 +23.5±4.1 0.775±0.030 0.580±0.035 +19.5±4.2 -17% Mistral-7B‡ (lr=5e-6, s150) 0.790±0.029 0.555±0.035 +23.5±4.1 0.770±0.030 0.565±0.035 +20.5±3.6 -13% Table 3: Out-of-distribution results on CRASS and e-CARE (n=200n=200 each), grouped by dataset, with adapters trained only on CLadder. Δgap _gap is the relative gap reduction versus base. All ± are 11 s.d.: for the Vernier rows of Qwen-7B, Qwen-14B, and Llama-3.1-8B († ) it is the std over three rounds; for every other cell (all base rows and the single-seed Vernier rows of Phi-3, Qwen-32B, and Mistral-7B) it is the paired bootstrap standard error over the 200200 items. Within each dataset block, bold marks the best and underline the second best per column (P0P_0/P1P_1 higher is better; Gap closest to 0 is better; Δgap _gap more negative is better). Qwen-32B uses QLoRA; Mistral-7B (‡ ) is reported under the default and rescued recipes. Base Vernier Data Model P_0 P_1 Gap P_0 P_1 Gap Rel. gap CRASS Phi-3-mini (3.8B) 0.845±0.026 0.545±0.035 +30.0±3.9 0.840±0.026 0.550±0.035 +29.0±3.9 -3% Qwen2.5-7B† 0.845±0.026 0.505±0.035 +34.0±3.9 0.870±0.013 0.622±0.012 +24.8±0.3 -27% Qwen2.5-14B† 0.895±0.022 0.625±0.034 +27.0±3.5 0.918±0.008 0.722±0.013 +19.7±0.8 -27% Qwen2.5-32B (QLoRA) 0.910±0.020 0.635±0.034 +27.5±3.6 0.950±0.015 0.725±0.031 +22.5±3.3 -18% Llama-3.1-8B† 0.850±0.025 0.445±0.035 +40.5±3.8 0.872±0.008 0.558±0.035 +31.3±3.1 -23% Mistral-7B‡ (lr=1e-4) 0.795±0.028 0.415±0.035 +38.0±3.9 0.710±0.032 0.500±0.036 +21.0±3.4 -45% Mistral-7B‡ (lr=5e-6, s150) 0.795±0.028 0.415±0.035 +38.0±3.9 0.735±0.031 0.365±0.034 +37.0±3.8 -3% e-CARE Phi-3-mini (3.8B) 0.795±0.029 0.590±0.035 +20.5±3.9 0.785±0.029 0.600±0.035 +18.5±3.9 -10% Qwen2.5-7B† 0.815±0.028 0.585±0.035 +23.0±3.8 0.810±0.010 0.622±0.012 +18.8±1.5 -18% Qwen2.5-14B† 0.820±0.027 0.605±0.035 +21.5±3.7 0.823±0.002 0.610±0.004 +21.3±0.5 -1% Qwen2.5-32B (QLoRA) 0.820±0.027 0.625±0.034 +19.5±3.8 0.815±0.028 0.595±0.035 +22.0±3.9 +13% Llama-3.1-8B† 0.790±0.029 0.550±0.035 +24.0±4.5 0.793±0.005 0.615±0.004 +17.8±0.6 -26% Mistral-7B‡ (lr=1e-4) 0.790±0.029 0.555±0.035 +23.5±4.1 0.775±0.030 0.580±0.035 +19.5±4.2 -17% Mistral-7B‡ (lr=5e-6, s150) 0.790±0.029 0.555±0.035 +23.5±4.1 0.770±0.030 0.565±0.035 +20.5±3.6 -13% In-distribution gap closure is reported in Table 2. Vernier drives the gap to at or near zero on all six models in the table. On Qwen-7B, Qwen-14B, and Llama-3.1-8B both P0P_0 and P1P_1 rise by 24 to 44 p from base, so the closure is driven by joint accuracy gains rather than by a regression on P0P_0. Mistral-7B shows the same joint gains under the default learning rate but at a steep capability cost, so we report its rescued low-rate row as the operating point. On Phi-3 the closure has the opposite signature, with P0P_0 falling by 3.5 p and P1P_1 rising by only 1.0 p. This model-specific failure is analysed with the mechanism and scale diagnostics. The Qwen2.5-32B base model already has a negative gap of −2.0-2.0 p on this slice, so the gap-closure metric is not directly comparable, but both P0P_0 and P1P_1 rise by roughly 40 p after Vernier. The in-distribution base gaps are small on larger held-out CLadder slices, and their paired confidence intervals cross zero for the base models (see Table A17). We therefore treat in-distribution closure as a sanity check and rest the behavioural case on the out-of-distribution transfer below. Zero-shot transfer to CRASS and e-CARE is reported in Table 3. The adapter trained only on CLadder reduces the CRASS gap by 18 to 27 percent across the three Qwen scales, with the Qwen-7B value averaged over three seeds. The larger-OOD check strengthens this CRASS result rather than weakening it (see Table A16). On all 274 prepared CRASS items, the gap falls from 35.0 to 23.7 p on Qwen-7B and from 27.4 to 16.8 p on Qwen-14B. The e-CARE picture on Qwen is weaker and non-monotonic in scale. At n=200n=200, Qwen-7B reduces e-CARE by 18 percent, Qwen-14B is within sampling noise (−2-2 percent), and Qwen-32B widens the e-CARE gap by 2.5 p despite closing CLadder and CRASS. At n=1000n=1000, the e-CARE reduction remains small, from 20.7 to 18.7 p on Qwen-7B and from 23.2 to 22.4 p on Qwen-14B (see Table A16). Llama-3.1-8B reduces the CRASS gap by 23 percent and the e-CARE gap by 26 percent as a three-seed mean. The rescued Mistral-7B (lr=5×10−65\!×\!10^-6, step 150) shows a 13 percent e-CARE reduction but only 3 percent on CRASS. Phi-3 transfer falls within noise on both benchmarks, the same null we see on its mechanism diagnostics. 4.3 Ablation study The ingredients are separated in Table A1 on Qwen-7B using five single-seed reference configurations at step 500. The headline three-seed numbers remain in Tables 2 and 3. P0P_0-only SFT widens the in-distribution gap from 8.0 to 9.5 p. Training only on the original view reinforces the lexical anchors. R-Drop widens the gap further to 18.5 p, with P0P_0 rising to 0.965 and P1P_1 staying at 0.780. Dropout consistency makes the model more confident under its own noise, but it never trains on the perturbed view. The perturbation T, not generic consistency, is what matters. The twin-view task row (β=0β=0) is counterfactual data augmentation in the sense of Kaushik et al. (2020). Both views enter the task loss, with no KL term. This row drops the in-distribution gap to 1.0 p and gives 21 percent CRASS and 13 percent e-CARE reductions. The answer-subspace KL adds little behaviourally. On the larger CRASS evaluation (n=274n=274), augmentation alone reduces the Qwen-7B gap from 34.3 to 18.2 p, while the full Vernier loss reaches 23.4 p. The same larger-CRASS comparison on Qwen-14B gives nearly identical reductions for twin task and Vernier (see Table A2). Full-vocabulary KL trades the other way, improving CRASS but nearly losing e-CARE. The KL term’s clearest contribution is mechanistic. It sharpens logit-lens agreement between the two views beyond augmentation alone, without changing accuracy, probe transfer, or cosine geometry. We treat it as a belief-level refinement, not as the source of closure. 4.4 Mechanism analysis Figure 2: Layer-wise mechanism diagnostics on held-out CLadder. (a) Last-token hidden-state cosine, (b) answer-token logit-lens disagreement, (c) and (d) variable-name probe accuracy. Dashed curves are base models, and solid curves are Vernier adapters. Two layer-wise measurements at the decision token are plotted for Phi-3 and the three Qwen scales (see Figure 2). The cosine similarity between the hidden states of P0P_0 and P1P_1 measures whether the representation the model uses to predict depends on the surface form. The absolute logit-lens disagreement measures whether the model’s intermediate belief about the answer depends on it. On all three Qwen scales Vernier raises the cosine and lowers the disagreement, with the effect concentrated in the upper third of the network. On Phi-3 neither measurement moves. The final-layer summaries across models are reported in Table A3. The logit-lens reduction factor is 5.5 on Qwen-7B, 5.0 on Qwen-14B, and 3.1 on Qwen-32B, with the smaller 32B ratio explained by the lower base disagreement on Qwen-32B leaving less room to shrink. As a methodological control, we also computed the cosine pooled over all sequence positions rather than at the last non-pad token. The pooled signal moves by only 0.006 at the final layer on Qwen-7B against a 0.032 shift in the last-token cosine. Pooling washes out the effect because T edits only a small number of variable-name tokens. The cosine result has a possible confound: Vernier might make the perturbed view content-free rather than aligned. We test that with a layer-wise linear probe (see Figures 2 and A3). At each layer, we train multinomial logistic regression to predict the variable-name family (12 classes: the 11 most frequent surface forms plus an “other” bucket) from the final prompt-token state of P0P_0, then evaluate the same probe on P1P_1. If the update removed story content from P1P_1, transfer accuracy should fall toward chance. It rises instead. Final-layer transfer goes from 0.615 to 0.655 on Phi-3, 0.480 to 0.695 on Qwen-7B, 0.590 to 0.680 on Qwen-14B, and 0.585 to 0.705 on Qwen-32B. The adapted model recovers more of the same story-level information from the perturbed lexical form. The alignment comes from paired views, while the KL sharpens beliefs. To attribute the alignment correctly we run the same diagnostics on the augmentation baseline (β=0β=0, no KL), reported in Table A4. On Qwen-7B augmentation alone raises the probe transfer to 0.690, against 0.695 for the full Vernier loss, and the final-layer cosine is the same under augmentation and the full loss. On Qwen-14B augmentation gives a probe transfer of 0.730, above Vernier’s 0.680. The representational alignment is produced by training on both views, not by the consistency term alone. The one diagnostic on which the KL moves the needle further is the logit-lens disagreement, which augmentation brings from 0.155 to 0.074 and the KL halves again to 0.028 on Qwen-7B. The KL therefore refines the agreement of the intermediate answer beliefs without changing the recoverability of story content. A causal test by activation patching. The cosine, logit-lens, and probe diagnostics are correlational. To test whether the decision-token representation participates causally in the answer difference between the two views, we run an activation-patching experiment (Zhang and Nanda, 2024) on Qwen-7B, Qwen-14B, and Llama-3.1-8B (see Figure 3). For each item we cache the P0P_0 hidden state at the decision token at a chosen decoder depth, overwrite the corresponding P1P_1 hidden state with it, and let the patched activation propagate to the answer. On the base Qwen models the patch transfers P0P_0’s answer to P1P_1. Patching Qwen-7B at depth 0.8 raises P1P_1 accuracy from 0.535 to 0.605, close to P0P_0’s 0.615, and makes the patched P1P_1 prediction agree with P0P_0 on 97 percent of items. The same depth-0.8 peak replicates on Qwen-14B, where the patch raises P1P_1 accuracy from 0.610 to 0.645 and the patched prediction agrees with P0P_0 on 99.5 percent of items, against weaker effects at depth 0.6. Llama-3.1-8B has a smaller base gap on this CLadder slice, so the patch gives little accuracy lift, but it still makes patched P1P_1 agree with clean P0P_0 on 98 to 100 percent of items. A larger 500-item depth-0.8 check gives the same agreement pattern, with bootstrap 95% confidence intervals of 0.954–0.984 for Qwen-7B, 0.986–1.000 for Qwen-14B, and 0.962–0.988 for Llama-3.1-8B. Thus the patching result has two parts: accuracy recovery when the clean P0P_0 answer is more accurate, and answer-identity transfer even when the clean gap is small. Donor controls on Qwen-7B show that this is not an arbitrary high-layer-vector effect: a different-prediction donor drives patched P1P_1 toward the donor answer and lowers accuracy (see Figure A2). The controls also pin down the baseline against which the 97 percent agreement should be read. A random donor, which injects an unrelated item’s P0P_0 state, still leaves patched P1P_1 agreeing with the clean P0P_0 prediction on only 0.685 of items, so the matched patch is a 0.285 lift over this uninformative-injection floor rather than a restatement of how often the two clean views already agree. The decision-token representation is therefore a causal control point for answer identity across two model families, with the accuracy benefit determined by whether P0P_0 is the better view on that slice. On the Vernier-adapted models the two views already agree, so the same patch is close to a no-op, consistent with the alignment having already been learned. Figure 3: Causal activation patching on base Qwen2.5-7B, Qwen2.5-14B, and Llama-3.1-8B (held-out CLadder, n=200n=200). The P0P_0 decision-token hidden state is copied into the corresponding P1P_1 state at the listed decoder depth. (a) Fraction of items on which the patched P1P_1 prediction agrees with the clean P0P_0 prediction, peaking at depth 0.8; the dashed line is the random-donor floor (0.685) from Figure A2. (b) Accuracy recovery at depth 0.8, where the patched P1P_1 moves from its clean value toward the P0P_0 target. On the Vernier-adapted models the two views already agree, so the same patch is close to a no-op. The full per-depth and Vernier-adapted numbers are tabulated in Table A7. The misalignment is item-conditional, not a global direction. Patching uses the per-item P0P_0 representation, which carries that item’s content. This raises the question of whether the P0−P1P_0-P_1 difference is instead a single direction, or a low-rank map, that could be estimated once and applied at inference with no fine-tuning. We test this directly. At the causal layer we fit an affine map T(h)=h+Ckh+bT(h)=h+C_kh+b from P1P_1 to P0P_0 decision-token states on 500 CLadder items, with b the mean difference and CkC_k a rank-k least-squares correction, and apply it to the P1P_1 decision token at inference over a rank sweep (see Table A5). The result is uniformly negative on Qwen-1.5B, 3B, 7B, 14B, and Phi-3. The mean-difference vector (k=0k=0) leaves the gap and P1P_1 accuracy unchanged, and any k≥4k≥ 4 lowers P1P_1 accuracy rather than raising it. On Qwen-7B the held-out CLadder P1P_1 accuracy drops from 0.54 to 0.48 and the gap widens from 8.0 to 13.5 p, and the CRASS and e-CARE gaps do not move. The failure is not specific to one layer or to a linear map. Sweeping the intervention over four decoder depths and replacing the affine map with a one-hidden-layer MLP fit on the same representations also fails to close the gap on Qwen-7B and Qwen-14B. The best case over this exhaustive sweep still leaves residual gaps of 5.0 p on Qwen-7B CLadder, 31.0 p on Qwen-7B CRASS, 4.5 p on Qwen-14B CLadder, and 26.0 p on Qwen-14B CRASS (see Table A6). Several deeper configurations degrade P1P_1 further, for example a mid-depth MLP that drives the Qwen-14B CRASS gap from 27.5 to 59.0 p. There is no consistent lexical-gap direction in the maps we tested. This sets the lexical gap apart from behavioural attributes such as sentiment, refusal, or truthfulness that prior work removes or controls with a single steering direction (Turner et al., 2023; Zou et al., 2023). The decision-token difference appears dominated by item-specific content, so although the per-item representation controls answer identity, the tested global maps do not realign the views. This suggests that closure needs a weight update that relearns the read-out from each lexical form rather than an inference-time edit. The two views are not offset by a fixed transformation in our tests. They are read out by a function that the fine-tune repairs. This resistance is specific to the causal benchmarks. On the non-causal tasks a nonlinear map does partially realign the views. 4.5 Sensitivity, robustness, and capability The auxiliary checks support the same scoped conclusion. The β sweep is stable around β∈[0.5,1]β∈[0.5,1], with over-regularisation visible at β=5β=5, and other hyperparameter sweeps leave the qualitative result unchanged (see Figures A1 and A11). Placeholder-scheme stress tests show that the learned invariance is not fully scheme-independent. Qwen-7B remains closed across held-out schemes, but Qwen-14B is brittle on VAR_A, VAR_B (see Table A12). Larger-n, paired, and stratified CLadder checks show that the in-distribution base gap is modest but the paired reduction remains positive on average (see Tables A14, A15, A17 and A18). Larger-OOD checks show a strong CRASS reduction on Qwen-7B and Qwen-14B but only a small e-CARE reduction (see Table A16). The sampled capability checks do not detect broad damage on the Qwen and Phi rows, but they are too small to certify capability preservation. Mistral is the clear failure case (see Figures A3 and A5). Joint training on CLadder/CRASS/e-CARE trades a little CLadder closure for a much smaller CRASS gap (see Table A13). A lexical-overlap audit finds no 8-gram or 13-gram overlap between the CLadder training prompts and the sampled MMLU, HellaSwag, or GSM8K capability items (see Table A10), so the sampled capability changes are unlikely to be explained by verbatim train-test contamination. These checks narrow the claim rather than broaden it. The recipe is useful in a working regime, not a universally safe fine-tune. 4.6 When the method works: capacity and base-model identity To turn the question of when Vernier works into a measured one, we add three sub-4B models and run the full diagnostic suite on each (see Tables A19 and A1). Two are from the Qwen family (Qwen2.5-1.5B and Qwen2.5-3B), so comparing them against the larger Qwen models isolates capacity with architecture and pretraining held fixed. The third is Gemma-2-2B, a second small family. A within-family capacity threshold. Inside the Qwen family the success boundary lies between 1.5B and 3B by our joint criterion: the gap should close without P0P_0 regression, probe transfer should rise, and logit-lens disagreement should fall. Both endpoints are measured over three seeds, so the boundary is not an artefact of a single run. Qwen-1.5B fails this joint criterion on all three seeds: after training the gap is +3.5±0.8+3.5± 0.8 p, probe transfer drops by 0.09, and logit-lens disagreement barely moves. Qwen-3B succeeds on all three seeds by the same diagnostics: P0P_0 and P1P_1 both rise to 0.88 with the gap at +0.5±0.7+0.5± 0.7 p, the probe transfer rises by 0.29, the largest gain of any model, and the logit-lens disagreement halves. The two endpoints separate cleanly on the mechanism diagnostics and the per-seed spread (std under 1 p on the gap) is far smaller than the gap between them. Since Qwen-1.5B and Qwen-3B share architecture and training family, this isolates capacity from base-model identity more cleanly than a cross-family comparison would. But scale is necessary only within a family, not sufficient across families. The threshold is not a parameter-count rule. Phi-3-mini has 3.8B parameters and fails, while Qwen-3B is smaller and succeeds. Model family, pretraining, and instruction tuning also matter. Mistral-7B makes the same point at a larger scale because it needs a lower learning rate than the other 7B-class models. Gemma-2-2B is useful for a different reason. Its in-distribution gap closes with both views rising, but its logit-lens disagreement widens and probe transfer moves only 0.07. Gemma appears to fit the shared answer key without producing the representational alignment seen in the working models. Across the nine models, Vernier needs both enough representational headroom and a base model whose post-training leaves that headroom usable. The working regime (7B to 32B) is not Qwen-specific. Qwen-7B, Qwen-14B, and Llama-3.1-8B agree across the behavioural diagnostics, and the closure reproduces across three seeds. Qwen-32B is harder to compare because its base in-distribution gap is already near zero, but the mechanism still moves: last-token cosine rises by 0.032, logit-lens disagreement drops 3.1-fold, and variable-name probe transfer rises by 0.12. e-CARE is the main exception. Its gap is within noise on Qwen-14B and reverses by 2.5 p on Qwen-32B. We ran the same cosine diagnostic on e-CARE and found that Vernier still moves the two final-layer decision-token states closer together on all three Qwen scales (see Table A20). The remaining e-CARE gap is plausibly downstream of this structural alignment, rather than simply a failure to align the final-layer representations. 4.7 Preliminary non-causal rename checks The account so far comes from causal question answering. We also run a preliminary check on two non-causal BBH tasks with object names replaced by typed placeholders: logical_deduction and reasoning_about_colored_objects (see Table A21). The logical_deduction rows use 100 held-out items, while the colored-objects rows use a smaller 20-item diagnostic slice. Both have a base rename gap, 9.0 p on Qwen-7B logical_deduction and 20.0 to 45.0 p on colored objects. Counterfactual augmentation closes or reduces the gap with both views rising rather than P0P_0 collapsing. On colored objects the Qwen-7B gap goes from 20.0 to −5.0-5.0 p as P0P_0 and P1P_1 rise from 0.500 and 0.300 to 0.800 and 0.850, and the Qwen-14B gap halves from 45.0 to 20.0 p with both views rising. Activation patching reproduces the causal-sufficiency pattern on the tested non-causal cells. Patching the P0P_0 decision-token state into P1P_1 raises logical_deduction Qwen-7B P1P_1 accuracy from 0.63 to 0.77 with 89 percent agreement to P0P_0, and raises colored-objects Qwen-14B P1P_1 accuracy from 0.25 to 0.75 with full agreement. On colored objects the Qwen-7B patch is present but peaks at a deeper layer rather than 0.8. These results suggest that the rename gap, closure by augmentation, and decision-token control-point pattern can extend beyond causal QA, but they should not be read as a full non-causal benchmark evaluation. Steerability does differ by task. No inference-time map closes the gap on the causal benchmarks (Section 4.4), but on both non-causal tasks a nonlinear map at the causal layer partially realigns the views. On Qwen-7B logical_deduction, it reduces the gap from 9.0 to 1.0 p and raises P1P_1 from 0.64 to 0.73. We tested whether the effective rank of the misalignment predicts this across five tasks and two model scales (see Figure A4). It does not: CLadder has a lower effective rank than the steerable non-causal tasks but remains unsteerable, while colored objects has the highest effective ranks and is steerable on both Qwen scales. The source of this task dependence remains open. 5 Discussion Alignment, not removal. The mechanism result is easy to misread. Higher cosine and lower logit-lens disagreement might look like removal of lexical-anchor information from the perturbed view. The probe says otherwise. After Vernier, the variable-name family is more recoverable from P1P_1 than before training, the opposite of what a simple information-removal account predicts. Activation patching turns this from a correlation into an intervention: overwriting the P1P_1 decision-token representation with the P0P_0 representation transfers P0P_0’s answer identity to P1P_1 on 97 percent of Qwen-7B base-model items, 99.5 percent on Qwen-14B, and at least 98 percent on Llama-3.1-8B. The representation Vernier aligns is not a decorative diagnostic. It is a causal control point for the answer. What the consistency term buys. Section 4.3 and Section 4.4 leave a fairly simple decomposition. The paired lexical views are the behavioural engine. They close 87.5 percent of the in-distribution gap, give the CRASS transfer that survives at larger n, and produce the same probe-transfer gain as the full loss (see Table A4). The answer-subspace KL mostly sharpens intermediate answer beliefs. It halves the logit-lens disagreement beyond augmentation, but it does not improve the larger-n CRASS number and does not change the probe or cosine results. We keep the KL for its identifiability interpretation and belief-level effect, not because it drives the closure. R-Drop provides the negative control: consistency without the structure-preserving perturbation widens the gap. Failure modes and limitations. The failures are informative. Phi-3 closes the in-distribution gap by P0P_0 regression rather than P1P_1 improvement, and an MLP-targeted LoRA ablation reproduces the same shortcut (see Appendix D). Mistral-7B needs a lower learning rate and still trades OOD transfer against capability (see Figures A3 and A5). The largest model here is 32B under QLoRA, most held-out and capability cells use n=200n=200, the colored-objects non-causal check uses only n=20n=20, and the capacity threshold is clean only within Qwen. The capability table should be read as a screen for large degradation rather than as a standard full-benchmark evaluation. The mechanism claim is also bounded: the evidence argues against simple information erasure and against a fixed inference-time steering direction, but it does not prove that downstream computation is unchanged after Vernier. A learned read-out or decision-boundary recalibration conditioned on the item representation remains compatible with our misalignment account. 6 Conclusion In the regimes tested here, the lexical gap is better explained by representational misalignment than by simple information loss. Vernier uses paired-view counterfactual augmentation to realign original and placeholder views, while the answer-subspace KL mainly refines answer-belief agreement. Probe transfer and activation patching show that the perturbed view retains recoverable story-level information and that the decision-token representation can carry answer identity across views. Fixed inference-time maps do not close the causal benchmarks, which suggests that the read-out repair is item-conditional and weight-mediated. The strongest transfer is on CRASS. e-CARE remains weak and task-dependent. The capacity and family sweep further bounds the claim. Qwen succeeds at 3B but not 1.5B, Phi-3 and Gemma expose failure modes, and Mistral requires a lower-rate recipe. Preliminary non-causal checks suggest that similar rename gaps can occur beyond causal QA, but a full account of those settings remains open. References M. Abdin, J. Aneja, H. Awadalla, et al. (2024) Phi-3 technical report: a highly capable language model locally on your phone. Note: arXiv:2404.14219 Cited by: §4.1. N. Belrose, I. Ostrovsky, L. McKinney, Z. Furman, L. Smith, D. Halawi, S. Biderman, and J. Steinhardt (2023) Eliciting latent predictions from transformers with the tuned lens. Note: arXiv:2303.08112 Cited by: §1, §2.3. H. Chi, H. Li, W. Yang, F. Liu, L. Lan, X. Ren, T. Liu, and B. Han (2024) Unveiling causal reasoning in large language models: reality or mirage?. In Advances in Neural Information Processing Systems, Cited by: §1, §2.1. T. Cohen and M. Welling (2016) Group equivariant convolutional networks. In International Conference on Machine Learning (ICML), Cited by: §2.2. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, Cited by: §2.3, §4.1. L. Du, X. Ding, K. Xiong, T. Liu, and B. Qin (2022) E-CARE: a new dataset for exploring explainable causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2.1, §4.1. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. (2021) A mathematical framework for transformer circuits. Note: Transformer Circuits Thread. Cited by: §2.3. J. Frohberg and F. Binder (2022) CRASS: a novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC), Cited by: §2.1, §4.1. T. Gao, X. Yao, and D. Chen (2021) SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.2. Gemma Team (2024) Gemma 2: improving open language models at a practical size. Note: arXiv:2408.00118 Cited by: §4.1. M. Geva, J. Bastings, K. Filippova, and A. Globerson (2023) Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.3. Y. Hao, X. Wan, and C. Zhai (2025) An investigation of robustness of LLMs in mathematical reasoning: benchmarking with mathematically-equivalent transformation of advanced mathematical problems. Note: arXiv:2508.08833 Cited by: §1, §2.1. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021) Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), Cited by: Figure A3, Table A9, §1, §4.1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.3, §4.1. A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. (2023) Mistral 7B. Note: arXiv:2310.06825 Cited by: §4.1. Z. Jin, Y. Chen, F. Leeb, L. Gresele, O. Kamal, Z. Lyu, K. Blin, F. G. Adauto, M. Kleiman-Weiner, M. Sachan, and B. Schölkopf (2023) CLadder: assessing causal reasoning in language models. In Advances in Neural Information Processing Systems, Cited by: §2.1, §4.1. D. Kaushik, E. Hovy, and Z. C. Lipton (2020) Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations, Cited by: §2.2, §4.3. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2022) Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, Cited by: §2.1. D. Lee, H. Yun, M. Cha, S. Park, S. Park, and J. Kim (2025) EconCausal: a context-aware economic reasoning benchmark for large language models. Note: arXiv:2510.07231 Cited by: §1, §2.1. P. Li, C. Huang, C. Hao, H. Chen, X. Wei, W. Lei, and S. Ng (2026) METER: evaluating multi-level contextual causal reasoning in large language models. External Links: 2604.11502, Link Cited by: §1. X. Liang, L. Wu, J. Li, Y. Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, and T. Liu (2021) R-drop: regularized dropout for neural networks. In Advances in Neural Information Processing Systems, Cited by: Table A1, §1, §2.2, §4.1. Llama Team, AI @ Meta (2024) The llama 3 herd of models. Note: arXiv:2407.21783 Cited by: §4.1. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: §2.3, §4.1. T. McCoy, E. Pavlick, and T. Linzen (2019) Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §1, §2.1. K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022) Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, Cited by: §2.3. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022) Rethinking the role of demonstrations: what makes in-context learning work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Cited by: §2.3. J. Pearl and D. Mackenzie (2018) The book of why: the new science of cause and effect. Basic Books. Cited by: §2.1. J. Pearl (2009) Causality: models, reasoning, and inference. 2nd edition, Cambridge University Press. Cited by: §2.1. Qwen Team, Alibaba Group (2024) Qwen2.5 technical report. Note: arXiv:2412.15115 Cited by: §4.1. V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. Le Scao, A. Raja, et al. (2022) Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, Cited by: §2.3. M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr (2024) Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting. In International Conference on Learning Representations, Cited by: §1, §1, §2.1. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei (2023) Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: §4.1. A. M. Turner, L. Thiergart, G. Leech, D. Udell, U. Mini, and M. MacDiarmid (2023) Activation addition: steering language models without optimization. Note: arXiv:2308.10248 Cited by: §1, §2.3, §4.4. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, Cited by: §2.1. Y. Wang, Y. Zhu, and J. Li (2026) CausalFlip: a benchmark for llm causal judgment beyond semantic matching. External Links: 2602.20094, Link Cited by: §1. A. Webson and E. Pavlick (2022) Do prompt-based models really understand the meaning of their prompts?. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §1, §2.1. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022a) Finetuned language models are zero-shot learners. In International Conference on Learning Representations, Cited by: §2.3. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou (2022b) Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Cited by: §2.1. Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le (2020) Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.2. Z. Yu and S. Zhou (2026) Caliper: probing lexical anchors versus causal structure in LLMs. External Links: 2606.04915, Link Cited by: §1, §2.1, §2.2. F. Zhang and N. Nanda (2024) Towards best practices of activation patching in language models: metrics and methods. In International Conference on Learning Representations (ICLR), Cited by: §1, §4.4. A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, et al. (2023) Representation engineering: a top-down approach to AI transparency. Note: arXiv:2310.01405 Cited by: §1, §2.3, §4.4. Appendix A Additional diagnostic figures The consistency-weight sweep and the scale summary for the logit-lens reduction diagnostic are reported in Figure A1. The stable region around β∈[0.5,1]β∈[0.5,1] supports the default choice in the main text, while the degradation at β=5β=5 marks the point where the consistency penalty starts to dominate task supervision. The scale panel is kept in the appendix because the model-by-model interpretation is already reported in the main text. Figure A1: Additional diagnostics. The left panel gives the Qwen2.5-7B sweep of the consistency-loss weight β. The right panel gives logit-lens reduction by scale. Appendix B Additional results tables and figures Table A1: Loss-component ablation on Qwen2.5-7B (held-out CLadder n=200n=200, CRASS n=200n=200, e-CARE n=200n=200), all at step 500 with seed 42. Best results in each column are in bold. In-distribution (CLadder) Out-of-distribution Variant P0P_0 P1P_1 Gap CRASS Gap e-CARE Gap Mean ΔOOD _OOD Base (no fine-tuning) 0.625 0.545 +8.0+8.0 +34.0+34.0 +23.0+23.0 — P0P_0-only SFT 0.850 0.755 +9.5+9.5 +28.0(-18%)+28.0\;(-18\%) +23.5(-0%)+23.5\;(-0\%) −9%-9\% R-Drop [Liang et al., 2021] (no T) 0.965 0.780 +18.5+18.5 +25.5(-25%)+25.5\;(-25\%) +22.0(-4%)+22.0\;(-4\%) −15%-15\% Twin task (β=0β=0, no KL) 0.885 0.875 +1.0(-87.5%)+1.0\;(-87.5\%) +27.0(-21%)+27.0\;(-21\%) +20.0(-13%)+20.0\;(-13\%) −17%-17\% Full-vocab KL (β=1β=1) 0.930 0.925 +0.5(-94%)+0.5\;(-94\%) +21.5(-%)+21.5\;( -37\%) +21.5(-7%)+21.5\;(-7\%) −22%-22\% Vernier (β=1β=1, answer-subspace KL) 0.870 0.890 −2.0(-%)-2.0\;( -125\%) +25.0(-26%)+25.0\;(-26\%) +17.5(-%)+17.5\;( -24\%) −%-25\% Table A2: Counterfactual augmentation versus the full answer-subspace KL loss on CRASS. Both adapters are trained only on CLadder. The 274-item rows use all prepared CRASS items. Model n Setting P0P_0 P1P_1 Gap Δ gap Qwen2.5-7B 200 Base 0.845 0.505 +34.0+34.0 – Twin task 0.890 0.650 +24.0+24.0 −29%-29\% Vernier 0.880 0.635 +24.5+24.5 −28%-28\% Qwen2.5-7B 274 Base 0.839 0.496 +34.3+34.3 – Twin task 0.891 0.708 +18.2+18.2 −%-47\% Vernier 0.872 0.639 +23.4+23.4 −32%-32\% Qwen2.5-14B 200 Base 0.880 0.610 +27.0+27.0 – Twin task 0.915 0.715 +20.0+20.0 −26%-26\% Vernier 0.920 0.720 +20.0+20.0 −26%-26\% Qwen2.5-14B 274 Base 0.894 0.617 +27.7+27.7 – Twin task 0.920 0.745 +17.5+17.5 −37%-37\% Vernier 0.923 0.759 +16.4+16.4 −%-41\% Table A3: Final-layer mechanism diagnostics across models, base versus Vernier. Cosine is last-token hidden-state cosine cos(hL(P0),hL(P1)) (h_L(P_0),h_L(P_1)) at final layer L. Logit-lens disagreement is ⟨|pθ(y∣P0)−pθ(y∣P1)|⟩ |p_θ(y P_0)-p_θ(y P_1)| . Probe transfer is linear-probe accuracy on final-layer P1P_1 hidden states. Hidden cosine Logit-lens gap Probe transfer Model Base Vernier Δ Base Vernier Ratio Base Vernier Δ Phi-3-mini (3.8B) 0.9900.990 0.9900.990 0.0000.000 0.0440.044 0.0440.044 1.0×1.0× 0.6150.615 0.6550.655 +0.040+0.040 Qwen2.5-7B 0.9630.963 0.9950.995 +0.032+0.032 0.1550.155 0.0280.028 5.5×5.5× 0.4800.480 0.6950.695 +0.215+0.215 Qwen2.5-14B 0.9790.979 0.9950.995 +0.016+0.016 0.1730.173 0.0350.035 5.0×5.0× 0.5900.590 0.6800.680 +0.090+0.090 Qwen2.5-32B 0.9600.960 0.9920.992 +0.032+0.032 0.1360.136 0.0440.044 3.1×3.1× 0.5850.585 0.7050.705 +0.120+0.120 Table A4: Mechanism diagnostics for augmentation without KL and full Vernier (held-out CLadder, n=200n=200). Augment is the β=0β=0 twin-task adapter. Probe transfer is final-layer variable-name probe accuracy on P1P_1 hidden states, and logit-lens gap is final-layer answer-token disagreement. Model Diagnostic Base Augment (β=0β=0) Vernier (β=1β=1) Qwen2.5-7B Probe transfer (↑ ) 0.480 0.6900.690 0.695 Logit-lens gap (↓ ) 0.155 0.074 0.0280.028 Qwen2.5-14B Probe transfer (↑ ) 0.590 0.7300.730 0.680 Table A5: Training-free linear realignment at inference. An affine map T(h)=h+Ckh+bT(h)=h+C_kh+b is fit from P1P_1 to P0P_0 decision-token states at depth 0.8 on 500 CLadder items and applied to the P1P_1 decision token on the base model. Rank 8 is shown as representative. CLadder gap CLadder P1P_1 CRASS gap Model clean aligned clean aligned clean aligned Qwen2.5-1.5B −10.5-10.5 −9.0-9.0 0.53 0.52 +21.5+21.5 +22.5+22.5 Qwen2.5-3B −1.5-1.5 −0.5-0.5 0.53 0.52 +32.0+32.0 +32.0+32.0 Qwen2.5-7B +8.0+8.0 +13.5+13.5 0.54 0.48 +33.5+33.5 +34.0+34.0 Qwen2.5-14B +4.5+4.5 +16.5+16.5 0.60 0.48 +27.5+27.5 +26.5+26.5 Phi-3-mini +3.0+3.0 +9.5+9.5 0.56 0.48 +28.5+28.5 +23.5+23.5 Table A6: Best result from exhaustive inference-time steering on causal QA. Each row selects the smallest residual gap over four decoder depths and both linear and MLP maps fitted from P1P_1 to P0P_0 decision-token states. Model Benchmark Best map clean gap aligned gap P1P_1 P1P_1 aligned Qwen2.5-7B CLadder MLP@0.8 +8.0+8.0 +5.0+5.0 0.535 0.565 Qwen2.5-7B CRASS linear@0.4 +33.5+33.5 +31.0+31.0 0.510 0.535 Qwen2.5-14B CLadder MLP@0.6 +4.5+4.5 +4.5+4.5 0.605 0.605 Qwen2.5-14B CRASS linear@0.6 +27.5+27.5 +26.0+26.0 0.605 0.620 Table A7: Causal activation patching on Qwen2.5-7B, Qwen2.5-14B, and Llama-3.1-8B (held-out CLadder, n=200n=200). The P0P_0 decision-token hidden state at the listed decoder depth is copied into the corresponding P1P_1 state. “P1P_1 patched” is post-patch P1P_1 accuracy, and “flip-to-P0P_0” is agreement with the clean P0P_0 prediction. Model Setting Depth P0P_0 P1P_1 clean P1P_1 patched flip-to-P0P_0 Qwen2.5-7B Base 0.6 0.615 0.535 0.540 0.845 0.8 0.615 0.535 0.6050.605 0.9700.970 1.0 0.615 0.535 0.590 0.635 Vernier 0.6 0.870 0.890 0.890 0.920 0.8 0.870 0.890 0.870 1.000 1.0 0.870 0.890 0.865 0.985 Qwen2.5-14B Base 0.6 0.650 0.610 0.625 0.905 0.8 0.650 0.610 0.6450.645 0.9950.995 1.0 0.650 0.610 0.645 0.905 Vernier 0.6 0.910 0.935 0.935 0.955 0.8 0.910 0.935 0.915 0.995 1.0 0.910 0.935 0.885 0.925 Llama-3.1-8B Base 0.6 0.540 0.525 0.530 0.980 0.8 0.540 0.525 0.530 0.980 1.0 0.540 0.525 0.540 1.000 Vernier 0.6 0.945 0.960 0.945 1.000 0.8 0.945 0.960 0.945 1.000 1.0 0.945 0.960 0.945 0.980 Figure A2: Activation-patching donor controls on base Qwen2.5-7B (held-out CLadder, n=200n=200, depth 0.8). “Matched” copies the same item’s P0P_0 hidden state, and the controls copy another item’s P0P_0 state. (a) Agreement of the patched P1P_1 prediction with the clean P0P_0 prediction. (b) Patched P1P_1 accuracy, with the clean-P1P_1 (0.535) and P0P_0 (0.615) references. The matched and same-prediction donors transfer answer identity, the random donor sits at an uninformative 0.685 floor, and a different-prediction donor drives P1P_1 toward the wrong answer. The exact values are in Table A8. Table A8: Activation-patching donor controls on Qwen2.5-7B (held-out CLadder, n=200n=200, depth 0.8). Matched uses the same item’s P0P_0 hidden state, and the controls replace it with another item’s P0P_0 hidden state. Setting Donor P0P_0 P1P_1 clean P1P_1 patched flip target flip donor Base matched 0.615 0.535 0.605 0.970 0.970 Base random 0.615 0.535 0.510 0.685 0.930 Base same pred. 0.615 0.535 0.620 0.965 0.965 Base different pred. 0.615 0.535 0.410 0.175 0.825 Figure A3: General-capability change after Vernier on MMLU [Hendrycks et al., 2021], HellaSwag, and GSM8K (n=200n=200 items each under a fixed seed), reported as Δ accuracy in percentage points versus base. Qwen2.5-32B is evaluated with QLoRA. The Qwen and Phi rows show no broad degradation, whereas Mistral-7B under the canonical learning rate collapses on GSM8K; the rescued low-rate recipe (step 150) recovers most of the loss. The per-benchmark numbers are in Table A9. Table A9: General-capability evaluation on MMLU [Hendrycks et al., 2021], HellaSwag, and GSM8K (n=200n=200 items each under a fixed seed). Qwen2.5-32B is evaluated with QLoRA. The two Mistral-7B rows compare the default and rescued training recipes. MMLU HellaSwag GSM8K Model Base Vernier Δ Base Vernier Δ Base Vernier Δ Phi-3-mini (3.8B) 0.6550.655 0.6550.655 0.0 +0.0 0.7950.795 0.8050.805 +1.0+1.0 0.8150.815 0.8050.805 −1.0-1.0 Qwen2.5-7B 0.6850.685 0.6900.690 +0.5+0.5 0.8300.830 0.7900.790 −4.0-4.0 0.7050.705 0.7000.700 −0.5-0.5 Qwen2.5-14B 0.7800.780 0.7750.775 −0.5-0.5 0.8550.855 0.8500.850 −0.5-0.5 0.8200.820 0.8400.840 +2.0+2.0 Qwen2.5-32B 0.8250.825 0.8500.850 +2.5+2.5 0.9000.900 0.8950.895 −0.5-0.5 0.7400.740 0.8750.875 +13.5+13.5 Llama-3.1-8B 0.5650.565 0.6200.620 +5.5+5.5 0.6450.645 0.7050.705 +6.0+6.0 0.8300.830 0.7650.765 −6.5-6.5 Mistral-7B-v0.3, lr=1e-4† 0.5250.525 0.4650.465 −6.0-6.0 0.6500.650 0.6000.600 −5.0-5.0 0.5250.525 0.1600.160 −36.5-36.5 Mistral-7B-v0.3, lr=5e-6, step 150 0.5250.525 0.5050.505 −2.0-2.0 0.6500.650 0.6500.650 0.0 +0.0 0.5250.525 0.4550.455 −7.0-7.0 Table A10: Lexical-overlap contamination check for the capability benchmarks. The 1500 CLadder training prompts are compared with the 200-item MMLU, HellaSwag, and GSM8K samples using long n-gram overlap. Benchmark Items 8-gram overlap 13-gram overlap MMLU 200 0.000 0.000 HellaSwag 200 0.000 0.000 GSM8K 200 0.000 0.000 Table A11: Hyperparameter sweeps. Top: KL vs MSE consistency form on Qwen-7B. Middle: LoRA rank on Qwen-7B. Bottom: β sweep on Qwen-14B. Best per column in each block in bold. Configuration CLadder Gap CRASS Gap e-CARE Gap Avg. OOD reduction Consistency form on Qwen-7B (β=1β=1, r=16r=16): Symmetric KL (default) −2.0-2.0 +25.0+25.0 +17.5+17.5 −25.2%-25.2\% MSE on answer-subspace logits −1.5-1.5 +24.5+24.5 +19.0+19.0 −22.8%-22.8\% LoRA rank on Qwen-7B (β=1β=1, symmetric KL, αLoRA=2r _LoRA=2r): r=4r=4 −2.0-2.0 +26.0+26.0 +20.0+20.0 −18.3%-18.3\% r=8r=8 −4.5-4.5 +23.5+23.5 +22.0+22.0 −17.6%-17.6\% r=16r=16 (default) −2.0-2.0 +25.0+25.0 +17.5+17.5 −25.2%-25.2\% r=32r=32 −3.0-3.0 +21.5+21.5 +22.5+22.5 −19.5%-19.5\% β sweep on Qwen-14B (r=16r=16, symmetric KL): β=0.5β=0.5 −1.5-1.5 +18.5(-31%)+18.5\;(-31\%) +18.5(-14%)+18.5\;(-14\%) −%-22\% β=1.0β=1.0 (default) −1.5-1.5 +20.5(-24%)+20.5\;(-24\%) +21.0(-2%)+21.0\;(-2\%) −13%-13\% β=2.0β=2.0 +2.0+2.0 +18.5(-31%)+18.5\;(-31\%) +22.0(+2%)+22.0\;(+2\%) −14%-14\% Table A12: Adversarial-placeholder robustness on held-out CLadder (n=200n=200). Each Vernier adapter is trained only on the canonical X1,X2,…X_1,X_2,… scheme (top block) and evaluated under three held-out schemes the adapter has never seen. Phi-3-mini Qwen-7B Qwen-14B Scheme Setting P0P_0 P1P_1 Gap P0P_0 P1P_1 Gap P0P_0 P1P_1 Gap X1,X2,…X_1,X_2,… Base 0.585 0.540 +4.5+4.5 0.625 0.545 +8.0+8.0 0.615 0.580 +3.5+3.5 Vernier 0.550 0.550 0.0 +0.0 0.870 0.890 −2.0-2.0 0.920 0.935 −1.5-1.5 foo, bar, baz, … Base 0.585 0.575 +1.0+1.0 0.625 0.555 +7.0+7.0 0.615 0.605 +1.0+1.0 Vernier 0.550 0.575 −2.5-2.5 0.870 0.855 +1.5+1.5 0.920 0.915 +0.5+0.5 alpha_a, alpha_b, … Base 0.585 0.565 +2.0+2.0 0.625 0.540 +8.5+8.5 0.615 0.590 +2.5+2.5 Vernier 0.550 0.570 −2.0-2.0 0.870 0.845 +2.5+2.5 0.920 0.870 +5.0+5.0 VAR_A, VAR_B, … Base 0.585 0.550 +3.5+3.5 0.625 0.515 +11.0+11.0 0.615 0.530 +8.5+8.5 Vernier 0.550 0.545 +0.5+0.5 0.870 0.870 0.0 +0.0 0.920 0.775 +14.5+14.5 Table A13: Setting A (CLadder-only training) versus Setting B (joint training on CLadder, CRASS, and e-CARE) on Qwen2.5-7B and Qwen2.5-14B. Setting B uses 5,1545,154 pooled training items, single seed. Best CRASS and best e-CARE per model in bold. Gap (p) Model Setting CLadder CRASS e-CARE ΔOOD _OOD Qwen2.5-7B Base +8.0+8.0 +34.0+34.0 +23.0+23.0 — Setting A −2.5±0.9-2.5_± 0.9 +24.8±0.3+24.8_± 0.3 +18.8±1.5+18.8_± 1.5 −22%-22\% Setting B +0.5+0.5 +5.5+5.5 +16.5+16.5 −%-56\% Qwen2.5-14B Base +3.5+3.5 +27.0+27.0 +21.5+21.5 — Setting A −1.5-1.5 +20.5+20.5 +21.0+21.0 −13%-13\% Setting B +1.0+1.0 +5.5+5.5 +14.0+14.0 −%-56\% Table A14: Paired statistical tests for Qwen2.5-7B. Gaps are P0−P1P_0-P_1 in percentage points. CIs are paired bootstrap 95% intervals over items. McNemar tests whether the paired P0/P1P_0/P_1 correctness asymmetry differs from zero. The final column is the paired reduction in gap relative to the base row for the same benchmark. Benchmark Setting n P0P_0 P1P_1 Gap Gap CI McNemar p Δ gap vs base CLadder Base 200 62.5 54.5 +8.0 [+3.0, +13.0] 0.004 – CLadder Vernier 200 87.0 89.0 -2.0 [-6.0, +2.0] 0.454 +10.0 [+3.5, +17.0] CRASS Base 200 84.5 50.5 +34.0 [+26.5, +41.5] <10−4<10^-4 – CRASS Vernier 200 88.5 63.5 +25.0 [+18.0, +32.0] <10−4<10^-4 +9.0 [+3.0, +15.5] e-CARE Base 200 81.5 58.5 +23.0 [+15.5, +30.5] <10−4<10^-4 – e-CARE Vernier 200 81.0 63.5 +17.5 [+10.0, +25.0] <10−4<10^-4 +5.5 [+0.5, +11.0] Table A15: Larger CLadder held-out evaluation (n=1000n=1000). Gaps are P0−P1P_0-P_1 in percentage points with paired bootstrap 95% intervals. The final column reports paired gap reduction relative to the corresponding base model. Model Setting n P0P_0 P1P_1 Gap Gap CI Δ gap vs base Qwen2.5-7B Base 1000 55.8 53.4 +2.4 [+0.0, +4.8] – Qwen2.5-7B Vernier 1000 84.2 85.5 -1.3 [-3.0, +0.4] +3.7 [+0.7, +6.7] Qwen2.5-14B Base 1000 56.8 54.0 +2.8 [+0.5, +5.2] – Qwen2.5-14B Vernier 1000 86.6 87.9 -1.3 [-2.7, +0.1] +4.1 [+1.3, +6.8] Table A16: Larger out-of-distribution evaluation for Qwen2.5-7B and Qwen2.5-14B. CRASS uses all 274 prepared items, and e-CARE uses a 1000-item sample. The adapters are trained only on CLadder. Gap reductions are relative to the corresponding base row. Model Benchmark / Setting n P_0 P_1 Gap Δ Gap Qwen2.5-7B CRASS Base 274 0.839 0.489 +35.0+35.0 – CRASS Vernier 274 0.876 0.639 +23.7+23.7 −%-32\% e-CARE Base 1000 0.802 0.595 +20.7+20.7 – e-CARE Vernier 1000 0.808 0.621 +18.7+18.7 −%-10\% Qwen2.5-14B CRASS Base 274 0.894 0.620 +27.4+27.4 – CRASS Vernier 274 0.923 0.755 +16.8+16.8 −%-39\% e-CARE Base 1000 0.824 0.592 +23.2+23.2 – e-CARE Vernier 1000 0.830 0.606 +22.4+22.4 −3%-3\% Table A17: Base-model CLadder lexical gaps on a larger held-out slice (n=1000n=1000). Gaps are P0−P1P_0-P_1 in percentage points with paired bootstrap 95% intervals, and p is the exact two-sided McNemar test on paired correctness. Model P_0 P_1 Gap 95% CI McNemar p Qwen2.5-7B 0.553 0.534 +1.9+1.9 [−0.6,+4.4][-0.6,+4.4] 0.151 Qwen2.5-14B 0.564 0.549 +1.5+1.5 [−0.9,+3.9][-0.9,+3.9] 0.255 Qwen2.5-32B 0.568 0.558 +1.0+1.0 [−1.3,+3.3][-1.3,+3.3] 0.440 Llama-3.1-8B 0.530 0.508 +2.2+2.2 [−1.0,+5.3][-1.0,+5.3] 0.194 Table A18: Random stratified CLadder held-out splits on Qwen2.5-7B. Each run trains a fresh adapter on 1500 training items and evaluates on a disjoint 200-item held-out set stratified by rung. Values are mean ± standard deviation across split seeds 7,42,123\7,42,123\. Aggregate Base Gap Vernier Gap Δ Gap Mean ± std +6.2 ± 5.2 +0.2 ± 1.4 +6.0 ± 5.0 Table A19: Capacity and mechanism sweep on held-out CLadder (n=200n=200). The gap is P0−P1P_0-P_1. “Probe Δ ” is the change in final-layer P1P_1 probe-transfer accuracy, and “lens ratio” is the base-to-Vernier logit-lens disagreement ratio. Small Qwen rows († ) are three-seed means ± std, and other rows are single-seed. Model Params Base Gap Vernier Gap Probe Δ Lens Ratio Verdict Qwen2.5-1.5B† 1.5B -9.0 +3.5 -0.09 1.1×1.1× fail Qwen2.5-3B† 3B +0.5 +0.5 +0.29 2.1×2.1× success Phi-3-mini 3.8B +4.5 +0.0 +0.04 1.0×1.0× fail Gemma-2-2B 2B +9.5 -2.0 +0.07 0.4×0.4× mixed Qwen2.5-7B 7B +8.0 -2.5 +0.22 5.5×5.5× success Qwen2.5-14B 14B +3.5 +0.5 +0.09 5.0×5.0× success Qwen2.5-32B 32B -2.0 +0.5 +0.12 3.1×3.1× success Table A20: e-CARE final-layer decision-token alignment. Cosine is cos(hL(P0),hL(P1)) (h_L(P_0),h_L(P_1)) at the last non-pad token. The remaining e-CARE behavioural gap is therefore not explained by a failure to move the two final-layer representations closer together. Model Base Cosine Vernier Cosine Δ Qwen2.5-7B 0.904 0.923 +0.019 Qwen2.5-14B 0.917 0.936 +0.020 Qwen2.5-32B 0.902 0.924 +0.021 Table A21: Preliminary non-causal BBH checks with object names renamed to typed placeholders. Logical_deduction uses n=100n=100 held-out items; colored_objects uses a smaller n=20n=20 diagnostic slice. Augmentation closes or reduces the gap with both views rising, and activation patching shows the decision-token representation is a control point in the tested cells. Base After Augmentation Task Model P_0 P_1 Gap P_0 P_1 Gap Logical deduction Qwen-7B 0.740 0.650 +9.0 0.950 0.950 +0.0 Qwen-14B 0.900 0.890 +1.0 0.990 0.980 +1.0 Colored objects Qwen-7B 0.500 0.300 +20.0 0.800 0.850 -5.0 Qwen-14B 0.750 0.300 +45.0 0.850 0.650 +20.0 Activation patching (P0P_0 decision-token state copied into P1P_1) logical deduction, Qwen-7B, depth 0.8: P1P_1 0.63→0.770.63\!→\!0.77, flip-to-P0P_0 0.890.89 colored objects, Qwen-14B, depth 0.8: P1P_1 0.25→0.750.25\!→\!0.75, flip-to-P0P_0 1.001.00 Figure A4: Effective rank does not explain task-dependent steerability. Effective rank is the participation ratio of the P1→P0P_1\!→\!P_0 linear-correction spectrum at depth 0.8, and a task is marked steerable if a swept linear or MLP map closes its gap. Steerable (green) and non-steerable (grey) tasks interleave along the rank axis: CLadder has a lower rank than the steerable non-causal tasks yet stays unsteerable, while colored objects has the highest ranks and is steerable on both Qwen scales. Base gaps are measured on the steering map-fitting slice and can differ by a few points from the held-out gaps in Table A21 (logical_deduction uses n=100n=100, colored objects uses n=20n=20). The underlying values are in Table A22. Table A22: Effective rank does not explain task-dependent steerability. Effective rank is the participation ratio of the P1→P0P_1\!→ P_0 linear-correction spectrum at depth 0.8. Best gap is the smallest residual gap reached by any swept linear or MLP map. Base gaps here are measured on the steering map-fitting slice and can differ by a few points from the held-out gaps in Table A21 (logical_deduction uses n=100n=100, while colored objects uses n=20n=20; e.g. colored objects on Qwen-14B is +50.0+50.0 here versus +45.0+45.0 held-out). Task Model Eff. Rank Base Gap Best Gap Steerable CLadder qwen25-14b 32.74 +4.0 +4.5 × CLadder qwen25-7b 33.31 +5.5 +2.5 × CRASS qwen25-14b 43.44 +29.0 +29.8 × CRASS qwen25-7b 44.00 +41.1 +29.0 × logical deduction qwen25-7b 45.80 +9.0 +1.0 ✓ logical deduction qwen25-14b 46.53 +0.0 +0.0 × tracking objects qwen25-7b 46.83 -9.0 +1.0 × tracking objects qwen25-14b 47.43 -1.0 -1.0 × colored objects qwen25-14b 52.42 +50.0 +20.0 ✓ colored objects qwen25-7b 57.55 +20.0 +5.0 ✓ Figure A5: Mistral-7B-Instruct-v0.3 checkpoint trajectory under the rescued recipe (lr 5×10−65\!×\!10^-6) on the in-distribution CLadder held-out slice. P0P_0 and P1P_1 accuracy (left axis) and the signed gap P0−P1P_0-P_1 (right axis) are plotted against the training step. Step 150 is the operating point reported in Tables 2, 3 and A3; later checkpoints overshoot into a negative gap as P1P_1 overtakes P0P_0. The per-step numbers are in Table A23. Table A23: Mistral-7B-Instruct-v0.3 checkpoint trajectory under the rescued recipe (lr=5×10−65\!×\!10^-6) on the in-distribution CLadder held-out slice. Step 150 is the operating point reported in Tables 2, 3 and A9. Step P_0 P_1 CLadder gap Comment Base 0.610 0.565 +4.5 — 25 0.565 0.485 +8.0 no learning yet 50 0.575 0.505 +7.0 75 0.500 0.485 +1.5 100 0.590 0.550 +4.0 partial closure 125 0.600 0.590 +1.0 150 0.605 0.600 +0.5 sweet spot, reported in Tables 2, 3 and A9 175 0.585 0.625 -4.0 P1P_1 overshoot 200 0.600 0.670 -7.0 P1P_1 overshoot Appendix C Pre-specification, recipe audit, and falsification summary To make the selection process auditable, we list the default recipe and every per-model deviation, together with the trigger and the metric used to choose it. The default recipe uses LoRA rank 16 on the attention projections, α=β=1α=β=1, learning rate 1×10−41× 10^-4, and 500 steps. Two models deviate. Phi-3-mini uses learning rate 5×10−65× 10^-6 with early stopping at step 70, because the canonical learning rate drives the adapter to a constant-output minimum by step 50; the trigger is the degenerate in-distribution collapse, observed before any out-of-distribution evaluation. Mistral-7B uses the same low learning rate with the operating point at step 150, selected as the earliest checkpoint whose in-distribution gap is within 1 p of zero while avoiding the capability collapse of the canonical run (Figure A5); out-of-distribution metrics are not used for this selection. The 14B and 32B models use QLoRA for memory reasons only, with no change to the loss or selection rule. Seed choices are fixed in advance to 7,42,123\7,42,123\ for the three-seed models and to 42 for the single-seed models; no seed is dropped. The argument is organised so that each diagnostic can fail independently. Table A24 summarises the prediction each account makes, the observation that would falsify the representational-misalignment reading, and where the outcome is reported. The misalignment account survives on the working-regime models and is falsified, by these same criteria, on Phi-3 and Gemma, which is why the later claims are scoped by capacity and base-model identity rather than asserted universally. Table A24: Falsification summary. Each row is an independent test of the representational-misalignment account against simple information erasure or a generic-consistency explanation. Prediction (misalignment) Falsifier Where Gap closes with P1P_1 rising, not P0P_0 collapsing Closure driven by P0P_0 regression Table 2 (Phi-3 falsifies) Story content stays recoverable from P1P_1 Probe transfer falls toward chance Table A3 (probe rises) Decision-token beliefs of the two views converge Logit-lens disagreement does not move Table A3 (Phi-3 falsifies) T, not generic consistency, drives closure R-Drop (no T) reproduces the effect Table A1 (R-Drop widens gap) The decision-token state carries answer identity Matched patch is no better than random donor Figures 3 and A2 The gap is item-conditional, not a fixed direction A once-fit map removes it at inference Tables A5 and A6 (map fails) Appendix D Phi-3 with MLP-targeted LoRA The default Vernier configuration places LoRA on the attention projections. On Phi-3 this drives the in-distribution gap to zero through P0P_0 regression and leaves the hidden representation unchanged (Section 4.4). To test whether the failure is intrinsic to Phi-3 or specific to the attention-only LoRA target, we re-train Phi-3 with the LoRA target set switched to the MLP projections, keeping the loss, optimiser, learning rate (5×10−65× 10^-6), batch size, rank, and number of steps identical, and sweep the checkpoint at step ∈25,50,75,100∈\25,50,75,100\ on the in-distribution held-out slice. The full trajectory together with out-of-distribution transfer is reported in Table A25. Table A25: Phi-3-mini with the LoRA target switched from attention to MLP projections (gate_up_proj, down_proj), keeping all other hyperparameters identical. The default attention-LoRA row is included for reference. LoRA target Step P_0 P_1 CLadder gap OOD reduction Base (no adapter) — 0.585 0.540 +4.5 — Attention projections (q,k,v,o_proj), default Vernier: attn 500 0.550 0.550 0.0 CRASS -3%, e-CARE -10% MLP projections (gate_up_proj, down_proj): mlp 25 0.580 0.545 +3.5 — mlp 50 0.570 0.535 +3.5 CRASS -3%, e-CARE -7% mlp 75 0.510 0.490 +2.0 CRASS +2%, e-CARE -5% mlp 100 0.490 0.490 0.0 CRASS -5%, e-CARE -2% The MLP-LoRA variant reproduces the attention-LoRA failure with no qualitative difference. The in-distribution gap does close (to 2.0 p at step 75 and to 0.0 p at step 100), but P0P_0 regresses monotonically from 0.585 at base to 0.490 at step 100, the chance rate on the binary-answer subset. The OOD gaps on CRASS and e-CARE stay within 3 p of the base model at every checkpoint. Switching the LoRA target therefore does not change the conclusion of Section 4.6. In our runs, the 3.8B adapter does not find an answer-subspace invariance that also preserves task accuracy, and the optimiser falls back to the same confidence-suppression shortcut regardless of which projection family it acts on. Appendix E Per-rung CLadder breakdown The held-out CLadder results broken down by Pearl-ladder rung are reported in Table A26. Table A26: Per-rung CLadder breakdown (n=200n=200 held-out), grouped by model so the three rungs can be compared within each model. Rung n is the count of held-out items at that rung. “flip” marks a base gap that reverses sign after Vernier (relative reduction undefined). Δgap _gap is the relative gap reduction versus base. Within each model, bold marks the best and underline the second best across the three rungs per column (P0P_0/P1P_1 higher is better; Gap closest to 0 is better). Δgap _gap is shown but not ranked, because the per-rung base gaps differ and flips and overcorrection make it non-monotonic. Base Vernier Model Rung (n) P_0 P_1 Gap P_0 P_1 Gap Rel. gap Phi-3-mini Observational (75) 0.547 0.520 +2.7 0.520 0.547 -2.7 flip Interventional (57) 0.667 0.614 +5.3 0.596 0.614 -1.8 flip Counterfactual (68) 0.559 0.500 +5.9 0.544 0.500 +4.4 -25% Qwen2.5-7B Observational (75) 0.587 0.547 +4.0 0.880 0.853 +2.7 -32% Interventional (57) 0.684 0.579 +10.5 0.947 1.000 -5.3 -150% Counterfactual (68) 0.618 0.515 +10.3 0.794 0.838 -4.4 -143% Qwen2.5-14B Observational (75) 0.587 0.573 +1.3 0.893 0.907 -1.3 flip Interventional (57) 0.632 0.596 +3.5 0.947 1.000 -5.3 -251% Counterfactual (68) 0.632 0.574 +5.9 0.926 0.912 +1.5 -75% On both Qwen models the largest baseline gap is on the interventional and counterfactual rungs and the smallest is on the observational rung. After Vernier the interventional P1P_1 accuracy reaches 1.000 on both Qwen models, the largest single-cell accuracy improvement reported in the paper, consistent with the hypothesis that lexical anchors substitute most heavily for causal-structure inference on the two causal rungs. Phi-3 shows the opposite pattern. The interventional rung gap closes purely through P0P_0 regression, with P0P_0 falling from 0.667 to 0.596 while P1P_1 stays at 0.614. This is the per-rung signature of the mechanism null reported above.