Paper deep dive
Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation
Marcus Armstrong, Navid Ayoobi, Arjun Mukherjee
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/24/2026, 3:02:27 AM
Summary
This paper investigates whether independently trained language models converge to geometrically compatible latent representations and whether cross-architecture linear projection can correct model behavior at inference time without weight updates. A Ridge regression projection matrix maps teacher model activations to student model coordinate systems, enabling residual stream intervention during generation. Across 20 teacher-student pairings spanning diverse architectures, Ridge projection achieves RÂČâ0.50 on verbal reasoning (TruthfulQA) and RÂČâ0.40 on mathematical reasoning (GSM8K). Behavioral correction rates average 25.2% on TruthfulQA and 25.5% on GSM8K. Key findings include: distributed (not sparse) encoding of cross-architecture alignment, architecture-specific intervention sensitivity profiles, near-zero correlation between geometric alignment quality and behavioral correction rate, and catastrophic collapse of projection matrices when transferred across reasoning domains (mean RÂČ=-3.83), establishing domain-specific subspace geometry as a universal property of language models.
Entities (40)
Relation Signals (23)
Marcus Armstrong â affiliatedwith â University of Houston
confidence 99% · Department of Computer Science University of Houston Houston, TX 77204
Navid Ayoobi â affiliatedwith â University of Houston
confidence 99% · Department of Computer Science University of Houston Houston, TX 77204
Arjun Mukherjee â affiliatedwith â University of Houston
confidence 99% · Department of Computer Science University of Houston Houston, TX 77204
Ridge Regression â achieves â TruthfulQA
confidence 97% · the Ridge projection consistently achieves R^2 â 0.50 on verbal reasoning (TruthfulQA)
Ridge Regression â achieves â GSM8K
confidence 97% · R^2 â 0.40 on mathematical reasoning (GSM8K)
meta-llama/Meta-Llama-3-70B-Instruct â actsasteacherfor â meta-llama/Llama-3.2-1B-Instruct
confidence 97% · fully crossed experimental matrix of 20 heterogeneous teacher-student pairings
Double Dissociation Experiment â confirms â Domain-Specific Subspace Geometry
confidence 97% · double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83)
Cross-Architecture Latent Steering â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the student's residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairings spanning mixture-of-experts, dense, code-specialized, and synthetically trained architectures, the Ridge projection consistently achieves R^2 = 0.50 on verbal reasoning and R^2 = 0.40 on mathematical reasoning, collapsing to R^2 = -0.22 under permutation control and R^2 = 0.01 under L_1 regularization. Behavioral correction rates range from 14.0% to 50.0% on TruthfulQA (mean 25.2%) and from 8.5% to 43.3% on GSM8K arithmetic reasoning (mean 25.5%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (r = -0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of LMs.
Tags
Links
- Source: https://arxiv.org/abs/2603.20406v1
- Canonical: https://arxiv.org/abs/2603.20406v1
Trouble viewing inline? Open PDF directly â
Full Text
56,366 characters extracted from source content.
Expand or collapse full text
Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation Marcus Armstrong Navid Ayoobi Arjun Mukherjee Department of Computer Science University of Houston Houston, TX 77204 miarmstr, nyoobi@cougarnet.uh.edu, amukher6@central.uh.edu Abstract We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the studentâs residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairingsâspanning mixture-of-experts, dense, code-specialized, and synthetically trained architecturesâthe Ridge projection consistently achieves R2â0.50R^2â 0.50 on verbal reasoning and R2â0.40R^2â 0.40 on mathematical reasoning, collapsing to R2ââ0.22R^2â-0.22 under permutation control and R2â0.01R^2â 0.01 under L1L_1 regularization. Behavioral correction rates range from 14.0%14.0\% to 50.0%50.0\% on TruthfulQA (mean 25.2%25.2\%) and from 8.5%8.5\% to 43.3%43.3\% on GSM8K arithmetic reasoning (mean 25.5%25.5\%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (rââ0.07râ-0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R2=â3.83R^2=-3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of cross-architecture latent alignment. 1 Introduction The remarkable convergence of large language models (LLMs) on similar outputs suggests they might be learning the exact same underlying structure of language. However, it remains an open question whether their internal representations reflect a shared reality. Current interpretability research overwhelmingly focuses on intra-model dynamicsâextracting, analyzing, and manipulating features within a single, isolated network. We extend this paradigm to inter-model steering. By mapping activations from one architecture directly to another, we probe whether independently trained models actually share a fundamental topological structure in their latent spaces. We hypothesize that models trained on overlapping linguistic distributions naturally converge to topologically similar manifolds for core semantic concepts. If this holds true, their internal representations should not be entirely alien to one another. Instead, a straightforward linear transformationâessentially a combination of rotation and scalingâshould be sufficient to align these manifolds. To test this, we learn a projection matrix that maps the intermediate activation space of a âteacherâ model into the coordinate system of a âstudentâ model. We then intervene on the student during inference, substituting its internal state with the translated teacher state to correct reasoning trajectories in real time. Our findings demonstrate that this cross-architecture alignment is not only possible but highly effective, though it is bounded by strict geometric constraints. Specifically, our primary contributions are as follows: âą Empirical Existence Proof: We show that a linear projection consistently aligns latent representations across highly heterogeneous architectures for both verbal and mathematical reasoning, achieving R2â0.50R^2â 0.50 on TruthfulQA and R2â0.40R^2â 0.40 on GSM8K across 20 independent model pairings. âą Distributed Encoding: Through ablation studies comparing L2L_2 and L1L_1 regularization, we confirm that shared representations are distributed rather than sparse. The collapse of L1L_1 mappers to R2â0.01R^2â 0.01 indicates semantic alignment relies holistically on the full activation vector. âą Architecture-Specific Intervention Dynamics: We demonstrate that optimal intervention strength is a property of the student architecture rather than a universal hyperparameter, with student models exhibiting characteristic sensitivity profiles that invert across reasoning domainsâPhi-3-Mini is the most steerable student on verbal tasks and the least steerable on mathematical tasks. âą Manifold Orthogonality: We provide definitive evidence of domain modularity via double dissociation confirmed across all 20 model pairings without exception. Projection matrices trained on verbal truthfulness collapse to a mean R2=â3.83R^2=-3.83 on arithmetic reasoning and vice versa (R2=â1.73R^2=-1.73), proving that distinct reasoning domains occupy geometrically orthogonal latent subspaces. Sections 2â6 formalize the theoretical framework, detail the experimental methodology, and present empirical results across the full 5Ă45Ă 4 model matrix. 2 Related Work 2.1 Linear Representations in Neural Networks The foundational observation motivating our work is that neural networks encode semantic knowledge geometrically. Mikolov et al. (2013) first demonstrated this at scale in word embeddings, showing that semantic relationships manifest as stable linear directions in vector space. The Linear Representation Hypothesis, formalized by Elhage et al. (2022) and validated by Nanda et al. (2023), extends this to transformer residual streams: semantic concepts are encoded as specific directions in activation space, making the modelâs internal state a structured geometric object whose distances and angles carry interpretable semantic content. If concepts are encoded as directions, a rotation and uniform scalingâoperations captured exactly by a linear transformationâare in principle sufficient to align two coordinate systems encoding the same concepts in different orientations. 2.2 Cross-Architecture Alignment and Model Stitching Bansal et al. (2021) demonstrated that independently trained models can form functional composites via model stitching: inserting a learned linear layer between the first k layers of one network and the final nâkn-k layers of another produces surprisingly modest performance degradation, suggesting that a shared training objective is sufficient to produce compatible representations regardless of shared weights or structure. csiszĂĄrik2021similaritymatchingneuralnetwork extended this by developing rigorous metrics for representational similarity, showing that models converge to geometrically matchable representations even under different random initializations. Together, these results establish the affine compatibility assumption as an empirically grounded prior. Our work departs from this foundation in a critical way: prior stitching routes activations through a fixed adapter at training time, whereas our method performs active inference-time intervention, dynamically overwriting the studentâs residual stream during generation. 2.3 Representation Engineering and Activation Steering A parallel body of work has demonstrated that model behavior can be steered by directly manipulating internal activations without weight updates. Subramani et al. (2022) showed that continuous steering vectors extracted from a modelâs residual stream can guide generation toward target behaviors. Turner et al. (2024) demonstrated that fixed bias vectors added at specific layers reliably shift behavior across dimensions including honesty, sentiment, and persona. Panickssery et al. (2024) extended this via contrastive activation addition, showing that difference-of-means vectors constructed from contrastive prompt pairs produce reliable behavioral shifts across tasks. Stolfo et al. (2025) further demonstrated that activation steering improves instruction-following without any weight update. Zou et al. (2023) revealed that such vectors exhibit universal properties, generalizing across model families and tasks. This literature establishes the residual stream as a writable medium. However, all prior activation steering extracts and injects vectors within the same model. Our work presents a fundamental extension: the steering signal originates in an entirely different architecture, requiring explicit geometric translation before application. Concurrently, Wang et al. (2025) propose ExpertSteer, which derives steering signals from an external expert model via autoencoder alignment and Recursive Feature Machines. Our method differs in two fundamental respects: we learn a single affine projection directly on paired hidden states rather than a nonlinear autoencoder pipeline, and we perform direct residual stream substitution rather than additive steeringâa distinction that enables the geometric interpretability analysis central to our theoretical framework. 2.4 Probing and Hierarchical Layer Processing Interpretability research consistently demonstrates that transformer layers process information hierarchically: lower layers resolve syntax, middle layers construct semantic representations, and final layers produce task-specific outputs (Tenney et al., 2019; Belinkov, 2021). Because our teacher and student models differ substantially in depth, this hierarchy motivates parameterizing layer selection by relative processing depth rather than absolute indexâa layer at 75%75\% depth in a 32-layer model occupies an analogous functional role to one at 75%75\% depth in an 80-layer model. Din et al. (2024) further establish injection timing as a consequential variable, directly motivating the temporal alignment strategy in Section 3.3. 3 Theoretical Framework This section formalizes the three core components of our method: the affine compatibility assumption that justifies the projection, the intervention mechanics governing injection strength, and the temporal alignment strategy determining where in the studentâs pipeline injection must occur. 3.1 Affine Manifold Compatibility Contemporary language models are trained to minimize next-token prediction loss over largely overlapping linguistic corpora. Because the statistical regularities of language are properties of the data rather than properties a model invents, two models compressing the same regularities will build geometrically compatible internal representations: their conceptual manifolds will align in topology even if coordinate orientations differ. We formalize this as affine compatibility. Rather than assuming strict isomorphism, we assume the relationship between the studentâs hidden state hSh_S and the teacherâs hidden state hTh_T is well-approximated by: hSâWâhT+bh_Sâ Wh_T+b (1) where WââdSĂdTW ^d_SĂ d_T captures the rotation and scaling required to align the teacherâs conceptual manifold with the studentâs coordinate system, and b absorbs mean shifts between spaces. We deliberately restrict this transformation to be linear: W encodes a global rotation of the representational manifold, making every concept subject to the same geometric operation. A nonlinear projection would allow the adapter to warp different manifold regions independently, obscuring whether representations are genuinely compatible or merely forced into alignment by an expressive enough function. 3.2 Intervention Strength and the α Spectrum We model the intervention as a combination of the studentâs original residual state and the projected teacher representation: hfinal=(1âα)âhstudent+αâ(WâhT)h_final=(1-α)\,h_student+α\,(Wh_T) (2) For αâ(0,1)αâ(0,1), this interpolates between the two states. At α=1α=1, the studentâs state is replaced entirely by the projected teacher vector. For α>1α>1, the formula enters an extrapolative regime whose geometric meaning is revealed by rearranging at α=2α=2: hfinal=WâhT+(WâhTâhstudent)h_final=Wh_T+(Wh_T-h_student) (3) Here the intervention simultaneously amplifies the projected teacher vector and subtracts the studentâs original activationâactive error negation rather than blending. This is physically meaningful when the studentâs residual stream has accumulated sufficient inertia that simple substitution is insufficient to redirect computation (nostalgebraist, 2020; Elhage et al., 2021). We treat α as an empirical quantity: different student architectures exhibit characteristic sensitivities to injected representations, and optimal α is a property of the student model rather than a universal hyperparameter. We note that Equation 2 assumes the projected vector WâhTWh_T operates in a compatible magnitude regime; the practical realization of this constraint is detailed in Section 4. 3.3 Temporal Alignment and Layer Selection Effective intervention requires temporal alignment: the projected representation must arrive where it can propagate meaningfully to the output distribution, neither so early that subsequent layers overwrite it nor so late that decoding computation is bypassed. We parameterize layer selection by relative processing depth lâ[0,1]lâ[0,1], mapping the hierarchical stages identified in the probing literature onto a common scale that transfers across architectures (Tenney et al., 2019; Belinkov, 2021). We hypothesize a semantic handoff: extracting from deep teacher layers (lTâ0.90l_Tâ 0.90), where abstract reasoning is fully resolved, and injecting into mid-to-late student layers (lSâ0.75l_Sâ 0.75), where the student transitions from semantic representation to vocabulary decoding (Din et al., 2024). We treat relative depth as a principled search space rather than a fixed prescription, as the optimal injection point is sensitive to each student architectureâs functional organization. We therefore evaluate injection across a grid of relative depth combinations and characterize the empirical behavior in Section 5.5. Importantly, the choice of injection depth has implications beyond behavioral correction efficacy. As we demonstrate in Section 5.7, injecting a domain-mismatched representation into mid-network layers (lSâ0.50l_Sâ 0.50) produces substantially stronger geometric interference than injecting it at later layers, a finding that illuminates why the mid-network is the critical locus of domain-specific semantic construction, and why the temporal alignment of the intervention determines not only whether it succeeds but how destructively it fails when misapplied. 4 Methodology 4.1 Datasets We evaluate on two benchmarks targeting distinct reasoning domains. TruthfulQA Lin et al. (2022) is a dataset of 817 questions probing factual recall and verbal truthfulness across domains including health, law, history, and common misconceptions. Each question is paired with a best_answer field and a correct_answers field. We use the full validation split. GSM8K Cobbe et al. (2021) is a dataset of grade school arithmetic word problems requiring multi-step numerical reasoning. We use the test split (1,319 questions) for intervention evaluation. Each problem is paired with a full chain-of-thought solution ending with a final numeric answer delimited by ####. We extract this final answer for evaluation. Both benchmarks use the prompt template Question: question Answer:, with max_new_tokens=50 for TruthfulQA and max_new_tokens=100 for GSM8K to accommodate numeric answer generation. 4.2 Optimization Objective For each model pair, we perform a full forward pass over all 817 questions using the prompt template Question: question Answer: and extract the final-token hidden state at each candidate layer. Let HTââNĂdTH_T ^NĂ d_T and HSââNĂdSH_S ^NĂ d_S denote the teacher and student activation matrices. All vectors are L2L_2-normalized prior to regression: h~=hâhâ2 h= h\|h\|_2 (4) ensuring directional alignment is learned rather than magnitude differences, which are architectural artifacts. We optimize: minWâĄâHSâHTâWâF2+λââWâF2 _W\|H_S-H_TW\|_F^2+λ\|W\|_F^2 (5) with fixed λ=0.1λ=0.1, learned with a bias term corresponding to the full affine formulation in Equation 1. All R2R^2 scores are reported on a held-out test set using a 70/30 train/test split (seed 42). 4.3 Representation Structure Controls We employ two controls to validate that the learned W captures genuine semantic geometry rather than spurious correlations. Permutation Control. A second Ridge regression is fit with rows of HSH_S randomly shuffled, destroying semantic correspondence while preserving the marginal distribution. Any R2R^2 achieved by this sham mapper reflects distributional properties of the vectors rather than semantic alignment. Sparsity Control (L1L_1). We replace the L2L_2 penalty with an L1L_1 (Lasso) penalty (λ=0.0001λ=0.0001, max 5000 iterations): minWâĄâHSâHTâWâF2+λââWâ1 _W\|H_S-H_TW\|_F^2+λ\|W\|_1 (6) If shared semantic information is localized to sparse features, the L1L_1 mapper should identify them and maintain competitive R2R^2. Collapse under sparsity would confirm that alignment relies on distributed correlations across the full activation vector. 4.4 Intervention Mechanics For each model pair and layer combination, we identify an opportunity set: questions where the teacher answered correctly and the student did not. We pre-compute projected activations for all opportunity items: h^S(i)=Wâh~T(i)+b h_S^(i)=W h_T^(i)+b (7) These are injected via a forward hook on the target layer, rescaled to match the L2L_2 norm of the studentâs current residual stream before application: hfinal=(1âα)âhstudent+αâ(h^Sâ âhstudentâ2)h_final=(1-α)\,h_student+α\, ( h_S·\|h_student\|_2 ) (8) ensuring the intervention operates as a directional correction regardless of dimensional gap between architectures. We sweep layer depths at four relative positions lâ0.25,0.50,0.75,0.90lâ\0.25,0.50,0.75,0.90\ for both teacher and student (16 combinations per pairing) and the intervention coefficient across eight values: αâ0.25, 0.5, 0.8, 1.0, 2.0, 3.0, 5.0, 10.0αâ\0.25,\ 0.5,\ 0.8,\ 1.0,\ 2.0,\ 3.0,\ 5.0,\ 10.0\ (9) All generation uses greedy decoding (do_sample=False, max 50 new tokens) ensuring fully deterministic outputs. 4.5 Evaluation Metric We use deterministic matching rather than LLM-as-judge evaluation to avoid variance from probabilistic evaluators. For TruthfulQA, success is binary inclusion against reference fields: Scoreâ(x)=â(ybestâx)âšâyâcorrectâ(yâx)Score(x)=I(y_bestâ x)\; \; _y _correctI(yâ x) (10) where x is the generated text lowercased and stripped. For GSM8K, we extract the gold numeric answer from the #### delimiter and check for numeric equivalence in the generated text, handling formatting variants including comma separators and currency prefixes. The correction rate Î reports the percentage of opportunity set items where the intervention converted an incorrect output to a correct one. This strict standard acts as a conservative lower bound: any observed Î>0 >0 is a direct consequence of the intervention forcing retrieval of the precise target concept. 5 Experiments and Results 5.1 Experimental Setup Table 1 lists all teacher and student models evaluated. This matrix was designed to traverse severe dimensional boundaries, structural paradigms, and training methodologies, including intra-family pairings (Llama-to-Llama, Gemma-to-Gemma), a domain-specialized teacher (Granite), and a synthetically trained student (Phi-3-Mini). The full sweep produces 2,560 experimental conditions: 20 pairings Ă 16 layer combinations Ă 8 α values. Role Model Parameters Type Teacher meta-llama/Meta-Llama-3-70B-Instruct 70B Dense Teacher meta-llama/Meta-Llama-3-8B-Instruct 8B Dense Teacher mistralai/Mistral-7B-Instruct-v0.3 7B MoE Teacher google/gemma-2-9b-it 9B Dense Teacher ibm-granite/granite-8b-code-instruct 8B Code-specialized Student meta-llama/Llama-3.2-1B-Instruct 1B Dense Student Qwen/Qwen2.5-1.5B-Instruct 1.5B Dense Student google/gemma-2-2b-it 2B Dense Student microsoft/Phi-3-mini-4k-instruct 3.8B Synthetic Table 1: Teacher and student models comprising the experimental matrix. The Llama-70B teacher was loaded in 4-bit NF4 quantization due to hardware constraints. 5.2 Linear Alignment Efficacy When estimated via Ridge regression, the projection matrix consistently recovers substantial variance in the studentâs activation space. Across all 20 pairings and layer combinations, the Ridge mapper achieves mean R2â0.50R^2â 0.50, with a peak of R2=0.684R^2=0.684 for the Llama-8B â Llama-1B intra-family pairing. Explaining approximately half the target variance via a single affine transformation across completely independent architectures with no shared weights, training procedure, or initialization constitutes a substantial structural signal. The permutation control confirms that this signal reflects genuine semantic geometry. Destroying semantic correspondence by shuffling target vectors collapses regression to R2ââ0.22R^2â-0.22 consistently across all 20 pairings, ruling out exploitation of vector norms, distributional properties, or high-dimensional noise. The domain-specificity of this alignment is examined in Section 5.7. 5.3 Representation Structure: Distributed Encoding Table 2 reports mean R2R^2 under Ridge (L2L_2, λ=0.1λ=0.1), Lasso (L1L_1, λ=0.0001λ=0.0001), and permutation control, averaged across all 20 pairings. Mapping Condition Regularization Mean R2R^2 Ridge Regression L2L_2 (distributed) 0.4990.499 Lasso Regression L1L_1 (sparse) 0.0140.014 Permutation Control L2L_2 (shuffled) â0.220-0.220 Table 2: Projection performance under different regularization constraints, averaged across all 20 model pairings. The L1L_1 mapper flatlines at R2â0.014R^2â 0.014âbarely above the permutation baseline and a near-total collapse from the Ridge mapper. Forcing sparsity destroys the mapping entirely, confirming that cross-architecture alignment relies on the full distributed structure of the activation vector and cannot be recovered from any sparse subset of dimensions. This directly validates the use of dense Ridge regression as the appropriate projection estimator. 5.4 Downstream Intervention Efficacy Figure 1 presents peak correction rate Î across all 20 model pairings. The intervention is effective without exception, demonstrating that cross-architecture latent steering generalizes across heterogeneous model families, parameter scales, and training methodologies. Correction rates range from 14.0%14.0\% (Gemma-9B â Llama-1B) to 50.0%50.0\% (Granite-8B â Phi-3-Mini), with a mean of 25.2%25.2\%. Figure 1: Peak intervention correction rate (Î ) across the full 5Ă45Ă 4 teacher-student matrix. Each cell reports the maximum Î over all layer combinations and α values. Phi-3-Mini is consistently the most steerable student, receiving the highest correction rate from four of five teachers. We attribute this to its synthetic training corpus: curated, structured training data may produce more geometrically regular internal representations amenable to external redirection. The Llama-70B teacher consistently underperforms Llama-8B despite its larger parameter count, a result of two compounding factors: the dimensional reduction from dT=8192d_T=8192 to dSâ€2048d_S†2048 introduces lossy compression, and 4-bit quantization injects rounding noise into extracted activation vectors, degrading projection precision. Geometric Alignment Does Not Predict Behavioral Correction. The near-zero correlation between R2R^2 and Î across all experimental conditions (Pearson r=â0.071r=-0.071) reveals a fundamental dissociation between representation space fidelity and output space impact. Directional accuracy of the projectionâpointing toward the correct conceptual targetâmatters more than variance explained. A geometrically precise projection pointing in the right direction outperforms a high-R2R^2 projection that fits the full distributional structure of the target space. 5.5 Architectural Sensitivity: Alpha and Layer Analysis Intervention Coefficient. Figure 2 plots mean correction rate vs. α per student architecture. Rather than a universal optimum, the four architectures exhibit markedly distinct sensitivity profiles. Figure 2: Mean Î as a function of α per student architecture, averaged across all teachers and layer combinations. The dashed line marks α=1α=1. Phi-3-Mini exhibits a sharp peak at α=0.25α=0.25 with immediate collapse; Llama-1B maintains a flat response across the sweep. Phi-3-Mini peaks sharply at α=0.25α=0.25 (mean Î=26.5% =26.5\%) before collapsing below 10%10\% for all αâ„0.8α℠0.8, indicating high sensitivity to representational displacementâa gentle directional nudge suffices while stronger interventions destabilize generation. Llama-1B exhibits the flattest profile, tolerating a broad injection range, suggesting greater representational inertia. Across all architectures, optimal performance occurs within the interpolation regime (αâ€1α†1) in 80.6%80.6\% of configurations (median optimal α=0.5α=0.5). Extrapolative coefficients account for the remaining 19.4%19.4\%, concentrated in Llama-1B. A consistent dip at αâ0.8,1.0αâ\0.8,1.0\ across multiple architectures suggests full replacement of the studentâs residual state is frequently counterproductive; partial blending outperforms complete substitution for most models. Layer Selection. Figure 3 presents the layer interaction heatmap for Granite-8B â Phi-3-Mini. Figure 3: Peak Î across 16 lTĂlSl_TĂ l_S combinations for Granite-8B â Phi-3-Mini. Performance increases monotonically with teacher extraction depth; student injection depth shows a weaker, less consistent effect. Correction rates increase monotonically as lTl_T increases from 0.250.25 to 0.900.90, consistent with the semantic handoff hypothesis: deeper teacher layers yield more fully resolved conceptual representations. Student injection depth shows a weaker pattern; peak cells at lT=0.90l_T=0.90 are achieved at both lS=0.75l_S=0.75 and lS=0.90l_S=0.90. This asymmetry establishes deep teacher extraction as a more reliable design principle than any specific student injection target. 5.6 Generalization to Mathematical Reasoning To assess whether cross-architecture steering generalizes beyond verbal reasoning, we replicate the full intervention sweep on GSM8K, evaluating all 20 model pairings across the identical grid of layer combinations and α values. Figure 4 presents peak correction rates across the model matrix. Figure 4: Peak intervention correction rate (Î ) on GSM8K across the full 5Ă45Ă 4 teacher-student matrix. Correction rates range from 8.5%8.5\% to 43.3%43.3\% with a mean of 25.5%25.5\%, virtually identical to the TruthfulQA mean of 25.2%25.2\%. The method generalizes completely: correction rates range from 8.5%8.5\% to 43.3%43.3\% with a mean of 25.5%25.5\%, statistically indistinguishable from the TruthfulQA mean of 25.2%25.2\%. This replication across a fundamentally different cognitive domain â multi-step arithmetic versus short-form factual recall â establishes cross-architecture latent steering as a general-purpose correction mechanism rather than a benchmark-specific artifact. Two structural patterns distinguish the GSM8K results from TruthfulQA. First, student steerability rankings invert completely across domains. Phi-3-Mini, the most steerable student on TruthfulQA (peak Î=50.0% =50.0\%), becomes the least steerable on GSM8K (peak Î=23.7% =23.7\%). Conversely, Llama-1B, the hardest student to steer verbally (peak Î=16.7% =16.7\%), becomes one of the most steerable mathematically (peak Î=40.5% =40.5\%). This complete rank reversal demonstrates that steerability is not an intrinsic architectural property but a domain-specific one, reflecting the geometric organization of each modelâs reasoning subspaces rather than its overall representational plasticity. Second, the optimal α collapses entirely to the sub-interpolation regime on GSM8K: α=0.25α=0.25 is optimal in 99.4%99.4\% of configurations, and mean correction rate hits 0.0%0.0\% for all αâ„0.8α℠0.8. Mathematical representations are substantially more fragile than verbal ones â a gentle directional nudge suffices, while any stronger intervention destroys generation entirely. This asymmetry suggests that mathematical reasoning subspaces are more narrowly organized, with less tolerance for representational displacement before coherent output collapses. 5.7 Domain Modularity: A Universal Double Dissociation The preceding results establish that cross-architecture steering works on both verbal and mathematical reasoning when the projection is trained on in-domain data. We now ask whether the learned geometry transfers across domains â whether a mapper trained on TruthfulQA can steer mathematical reasoning, and vice versa. We conduct a cross-domain transfer experiment across all 20 model pairings at lT=lS=0.75l_T=l_S=0.75. For each pair, we train a projection matrix on 200 TruthfulQA questions and evaluate it on 100 held-out GSM8K questions (Direction A), then train on 200 GSM8K questions and evaluate on 100 held-out TruthfulQA questions (Direction B). In-domain evaluation uses a proper held-out test set in both directions, correcting the in-sample evaluation of the original single-pair pilot experiment. TQA in-domain TQAâ GSM in-domain GSMâ Mean R2R^2 0.3430.343 â3.831-3.831 0.2560.256 â1.729-1.729 Min R2R^2 0.2780.278 â8.170-8.170 0.1760.176 â3.227-3.227 Max R2R^2 0.4460.446 â1.807-1.807 0.3510.351 â1.081-1.081 Table 3: Double dissociation results across all 20 model pairings. In-domain R2R^2 reflects proper held-out evaluation on 100 test questions. Both transfer directions collapse catastrophically, with mean transfer R2R^2 of â3.83-3.83 and â1.73-1.73 respectively. Dissociation confirmed in 20/20 pairs without exception. The results constitute a universal double dissociation confirmed without exception across all 20 pairings. The TruthfulQA-trained mapper collapses to a mean R2=â3.831R^2=-3.831 on GSM8K; the GSM8K-trained mapper collapses to mean R2=â1.729R^2=-1.729 on TruthfulQA. Both transferred projections perform catastrophically worse than predicting the target mean (R2=0R^2=0), indicating active geometric interference rather than mere uninformativeness. The worst single cell reaches R2=â8.170R^2=-8.170 (Granite-8B â Qwen-1.5B, TQA-trained mapper applied to GSM8K), meaning the projection orients student activations in a direction nearly maximally opposed to correct mathematical representations. Notably, Qwen-1.5B consistently shows the most extreme dissociation in both directions (TQAâ range: â7.867-7.867 to â8.170-8.170; GSMâ range: â2.908-2.908 to â3.227-3.227), approximately twice the interference magnitude of other students. Combined with Qwen-1.5Bâs strong GSM8K behavioral correction performance (peak Î=43.3% =43.3\%), this suggests its mathematical and verbal reasoning subspaces are particularly well-separated â more modularly organized than other architectures, with sharper geometric boundaries between domain representations. To verify that this universal finding is not specific to the fixed layer combination used in the 20-pair sweep, we conduct a complementary experiment sweeping all 16 layer depth combinations for the Mistral-7B â Qwen-1.5B pair, evaluating dissociation at every lTĂlSl_TĂ l_S intersection. The dissociation is confirmed in all 16/16 combinations (Appendix H), ruling out the possibility that orthogonality is a depth-specific artifact. The magnitude of interference varies systematically with injection depth: mid-network injection (lSâ0.50l_Sâ 0.50) produces the most catastrophic transfer collapse (TQAâ mean R2=â11.5R^2=-11.5), while late injection (lSâ0.90l_Sâ 0.90) produces the weakest interference (TQAâ mean R2=â3.1R^2=-3.1). This gradient confirms that mid-network layers are the primary site of domain-specific semantic construction â mismatched representations injected there corrupt the entire downstream computation, while late injection gives the network less time to propagate the interference before output. These results establish domain orthogonality as a universal geometric law rather than a property of any specific model pair. Verbal truthfulness and arithmetic reasoning occupy geometrically incompatible regions of the latent manifold across every tested architecture combination; a rotation calibrated for one domain is not merely uninformative but actively destructive when applied to the other. A projection matrix must be trained on data representative of the target reasoning domain. 6 Limitations and Conclusion 6.1 Limitations Benchmark Scope. Behavioral intervention results are reported on TruthfulQA and GSM8K, covering short-form verbal factual recall and arithmetic word problems respectively. How correction rates generalize to multi-step inference, longer generation contexts, or other reasoning types such as commonsense or causal reasoning has not been established. Sparse Layer Grid. Layer selection is evaluated at only four relative depth positions, leaving behavior between grid points uncharacterized. Finer-grained sweeps may reveal non-monotonic dynamics or narrow optimal windows the current resolution cannot detect. Quantization Noise. The Llama-70B teacher was loaded in 4-bit NF4 quantization due to hardware constraints, introducing rounding noise into extracted activations. The 70B results represent a lower bound on full-precision performance rather than a representative evaluation of large-scale teacher utility. Dissociation Layer Generalization. The full layer sweep confirming dissociation across all 16 depth combinations is conducted on a single model pair (Mistral-7B â Qwen-1.5B). Whether the systematic variation in interference magnitude with injection depth generalizes across other architecture combinations remains unverified, though the binary dissociation result itself is confirmed across all 20 pairs at a representative layer combination. Open-Weight Models Only. Whether affine compatibility holds for proprietary closed-weight architectures or models trained on substantially different data distributions remains untested. Evaluation Metric Ceiling. Deterministic substring matching counts interventions that redirect reasoning but produce lexically distinct correct answers as failures. Reported Î values are conservative lower bounds on true behavioral impact. 6.2 Conclusion We have presented a framework for cross-architecture inference-time steering via learned linear projection of latent representations. A single affine transformation suffices to translate activation vectors between heterogeneous architectures, and injecting translated representations into a student modelâs residual stream produces reliable behavioral corrections without any weight updates, shared architecture, or access to training dataâonly a modest set of paired forward passes to estimate W. Across 20 teacher-student pairings, the Ridge projection achieves R2â0.50R^2â 0.50 on verbal reasoning and R2â0.40R^2â 0.40 on mathematical reasoning, collapsing to R2ââ0.22R^2â-0.22 under permutation control and R2â0.01R^2â 0.01 under L1L_1 regularization. Behavioral correction rates range from 14.0%14.0\% to 50.0%50.0\% on TruthfulQA and 8.5%8.5\% to 43.3%43.3\% on GSM8K, with near-identical means of 25.2%25.2\% and 25.5%25.5\% respectively, demonstrating complete generalization across reasoning domains. Four secondary findings sharpen the picture: geometric fidelity does not predict behavioral correction (rââ0.07râ-0.07), suggesting directional accuracy matters more than variance explained; optimal intervention strength is architecture-specific with sensitivity profiles that invert across domains; deep teacher extraction (lTâ0.90l_Tâ 0.90) is a more reliable design principle than any specific student injection target; and mathematical representations tolerate substantially less representational displacement than verbal ones before generation collapses. Finally, the double dissociation between verbal and mathematical domains â confirmed across all 20 model pairings without exception, with mean transfer R2=â3.83R^2=-3.83 and â1.73-1.73 in each direction â establishes domain-specific subspace geometry as a universal property of cross-architecture latent alignment rather than an artifact of any particular model pair. A projection matrix must be trained on data representative of the target reasoning domain, and the geometric structure of one reasoning type cannot be repurposed for another under any architectural configuration tested. Ethics Statement This work investigates the geometric structure of latent representations in large language models and demonstrates a method for correcting model outputs at inference time via cross-architecture activation steering. We have read and adhere to the ICLR Code of Ethics. We identify two areas warranting ethical consideration. First, the intervention mechanism demonstrated here â which modifies a modelâs internal reasoning trajectory without any weight update or user-visible indication â could in principle be applied to steer model outputs in undesirable directions as readily as beneficial ones. We emphasize that all experiments in this work are directed at correcting factually incorrect outputs toward ground-truth answers, and that the method requires explicit access to the target modelâs internal activations, limiting its applicability outside controlled research settings. Second, all models used in this work are publicly available open-weight models. No proprietary systems, private data, or human subjects were involved in any experiment. TruthfulQA and GSM8K are publicly available benchmarks with no sensitive personal information. We believe the primary contribution of this work â characterizing the geometric structure of shared latent representations and their domain specificity â is a positive contribution to the interpretability and transparency of large language models. Reproducibility Statement We have made substantial efforts to ensure the reproducibility of all results reported in this paper. All experiments use publicly available open-weight models accessible via HuggingFace, and all benchmarks (TruthfulQA, GSM8K) are publicly available datasets. Exact model identifiers, quantization configurations, random seeds, train/test split ratios, regularization hyperparameters, and generation settings are specified in Section 4 and Appendix I. The intervention hook implementation, projection matrix estimation procedure, and evaluation logic are described in full in Appendix I. The layer dissociation experiment parameters â sample sizes, layer combinations, and Ridge regression configuration â are specified in Section 5.7 and Appendix H. All results are deterministic: generation uses greedy decoding (do_sample=False) throughout, and all random operations use random_state=42. Complete experimental code will be released as anonymous supplementary material with the ICLR submission. References Y. Bansal, P. Nakkiran, and B. Barak (2021) Revisiting model stitching to compare neural representations. External Links: 2106.07682, Link Cited by: §2.2. Y. Belinkov (2021) Probing classifiers: promises, shortcomings, and advances. External Links: 2102.12452, Link Cited by: §2.4, §3.3. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.1. A. Y. Din, T. Karidi, L. Choshen, and M. Geva (2024) Jump to conclusions: short-cutting transformers with linear transformations. External Links: 2303.09435, Link Cited by: §2.4, §3.3. N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E. Tran-Johnson, J. Kaplan, J. Clark, T. Brown, S. McCandlish, D. Amodei, and C. Olah (2022) Softmax linear units. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2022/solu/index.html Cited by: §2.1. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah (2021) A mathematical framework for transformer circuits. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2021/framework/index.html Cited by: §3.2. S. Lin, J. Hilton, and O. Evans (2022) TruthfulQA: measuring how models mimic human falsehoods. External Links: 2109.07958, Link Cited by: §4.1. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean (2013) Distributed representations of words and phrases and their compositionality. External Links: 1310.4546, Link Cited by: §2.1. N. Nanda, A. Lee, and M. Wattenberg (2023) Emergent linear representations in world models of self-supervised sequence models. External Links: 2309.00941, Link Cited by: §2.1. nostalgebraist (2020) Interpreting GPT: the logit lens. Note: LessWrongAccessed: 2026-03-11 External Links: Link Cited by: §3.2. N. Panickssery, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner (2024) Steering llama 2 via contrastive activation addition. External Links: 2312.06681, Link Cited by: §2.3. A. Stolfo, V. Balachandran, S. Yousefi, E. Horvitz, and B. Nushi (2025) Improving instruction-following in language models through activation steering. External Links: 2410.12877, Link Cited by: §2.3. N. Subramani, N. Suresh, and M. E. Peters (2022) Extracting latent steering vectors from pretrained language models. External Links: 2205.05124, Link Cited by: §2.3. I. Tenney, D. Das, and E. Pavlick (2019) BERT rediscovers the classical nlp pipeline. External Links: 1905.05950, Link Cited by: §2.4, §3.3. A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid (2024) Steering language models with activation engineering. External Links: 2308.10248, Link Cited by: §2.3. W. Wang, M. Wu, B. Haddow, and A. Birch (2025) ExpertSteer: intervening in llms through expert knowledge. External Links: 2505.12313, Link Cited by: §2.3. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. External Links: 2307.15043, Link Cited by: §2.3. Appendix A Full Geometric Alignment Results Table 5 reports the peak R2R^2 achieved by the Ridge projection for each of the 20 teacher-student pairings, maximized over all 16 layer depth combinations. Values are consistently in the range [0.62,0.68][0.62,0.68] across the five teachers, with the notable exception of Gemma-9B pairings which cluster slightly lower at [0.62,0.64][0.62,0.64]. The absence of any pairing with R2<0.60R^2<0.60 confirms that affine compatibility holds broadly across the experimental matrix and is not dependent on architectural similarity between teacher and student. Figure 5: Peak R2R^2 (Ridge regression) for each of the 20 teacher-student pairings, maximized over all 16 layer combinations. The Llama-8B â Llama-1B intra-family pairing achieves the highest alignment (R2=0.684R^2=0.684). All pairings exceed R2=0.62R^2=0.62, confirming broad affine compatibility across the experimental matrix. Appendix B Extended Layer Interaction Heatmaps Figure 6 presents layer interaction heatmaps for the six model pairings with the highest within-pair variance in correction rate across layer combinations (standard deviation >5%>5\%). These pairings were selected because their heatmaps are most informative: the layer choice meaningfully affects outcome, and the gradient patterns visible in each provide evidence about the generalizability of the semantic handoff hypothesis discussed in Section 3.3. Figure 6: Peak correction rate Î across all 16 lTĂlSl_TĂ l_S combinations for the six highest-variance model pairings. Rows are ordered with deepest teacher extraction at the top. The monotonic improvement with teacher extraction depth observed in the main paper (Figure 3) generalizes across these pairings, while student injection depth consistently shows a weaker and less structured effect. The Granite-8B â Gemma-2B pairing shows the highest within-pair variance (Ï=6.7%Ï=6.7\%), with a pronounced performance gradient that makes layer selection particularly consequential for this pairing. Appendix C Per-Teacher Alpha Sensitivity Profiles Figure 7 decomposes the alpha sensitivity curves from Figure 2 by teacher architecture, plotting mean correction rate vs. α for each student separately within each teacher panel. This view addresses a question the main paper leaves open: whether the characteristic sensitivity profiles of each student are consistent across teachers, or whether the teacher architecture modulates them. Figure 7: Mean correction rate Î vs. α broken down by teacher architecture, with one line per student. The dashed vertical line marks α=1α=1. Student sensitivity profiles are highly consistent across teachers: Phi-3-Mini peaks at α=0.25α=0.25 and collapses immediately regardless of teacher, while Qwen-1.5B consistently prefers α=0.5α=0.5. The Mistral-7B panel is the notable exception, where Gemma-2B achieves its best performance at α=2.0α=2.0 rather than the sub-interpolation values preferred across other teachers, suggesting a specific compatibility between Mistralâs MoE representations and Gemma-2Bâs residual stream that benefits from stronger intervention. Appendix D Full Double Dissociation Results Table 4 reports all four R2R^2 values for each of the 20 teacher-student pairings in the domain dissociation experiment. In-domain values reflect proper held-out evaluation on 100 test questions. Transfer values confirm catastrophic collapse in both directions for every pairing without exception. Teacher Student TQA TQAâ GSM GSMâ Llama-70B Llama-1B 0.3540.354 â2.748-2.748 0.1980.198 â1.108-1.108 Llama-70B Qwen-1.5B 0.3680.368 â8.133-8.133 0.2220.222 â3.005-3.005 Llama-70B Gemma-2B 0.2900.290 â2.838-2.838 0.2960.296 â1.261-1.261 Llama-70B Phi-3-Mini 0.2850.285 â1.807-1.807 0.3000.300 â1.567-1.567 Llama-8B Llama-1B 0.3890.389 â2.389-2.389 0.2300.230 â1.101-1.101 Llama-8B Qwen-1.5B 0.3720.372 â7.867-7.867 0.2260.226 â3.037-3.037 Llama-8B Gemma-2B 0.2900.290 â2.798-2.798 0.3030.303 â1.182-1.182 Llama-8B Phi-3-Mini 0.2960.296 â1.834-1.834 0.2540.254 â1.529-1.529 Mistral-7B Llama-1B 0.3770.377 â2.439-2.439 0.2440.244 â1.081-1.081 Mistral-7B Qwen-1.5B 0.4050.405 â8.078-8.078 0.1760.176 â3.227-3.227 Mistral-7B Gemma-2B 0.3120.312 â2.714-2.714 0.2970.297 â1.178-1.178 Mistral-7B Phi-3-Mini 0.2910.291 â1.827-1.827 0.2200.220 â1.670-1.670 Gemma-9B Llama-1B 0.3530.353 â2.748-2.748 0.1900.190 â1.176-1.176 Gemma-9B Qwen-1.5B 0.3980.398 â8.019-8.019 0.2260.226 â2.908-2.908 Gemma-9B Gemma-2B 0.3610.361 â2.513-2.513 0.3260.326 â1.115-1.115 Gemma-9B Phi-3-Mini 0.3030.303 â1.870-1.870 0.3010.301 â1.527-1.527 Granite-8B Llama-1B 0.4170.417 â2.845-2.845 0.1950.195 â1.191-1.191 Granite-8B Qwen-1.5B 0.4460.446 â8.170-8.170 0.2750.275 â2.922-2.922 Granite-8B Gemma-2B 0.2810.281 â3.060-3.060 0.2890.289 â1.313-1.313 Granite-8B Phi-3-Mini 0.2780.278 â1.930-1.930 0.3510.351 â1.489-1.489 Mean 0.3430.343 â3.831-3.831 0.2560.256 â1.729-1.729 Table 4: Full double dissociation results for all 20 pairings. TQA and GSM columns show held-out in-domain R2R^2. TQAâ and GSMâ show transfer R2R^2. Dissociation confirmed in all 20 pairs. Appendix E GSM8K Alpha Sensitivity Profiles Figure 8 plots mean correction rate vs. α for each student architecture on GSM8K. The contrast with the TruthfulQA profiles (Figure 2) is stark: all four architectures collapse to near-zero correction rate for αâ„0.8α℠0.8, with no recovery at higher extrapolation values. This uniform collapse across architectures indicates that mathematical representations are substantially more fragile than verbal ones, tolerating only the most conservative interventions before coherent generation fails entirely. Figure 8: Mean correction rate Î vs. α on GSM8K per student architecture, averaged across all teachers and layer combinations. All architectures collapse to near-zero for αâ„0.8α℠0.8, in sharp contrast to TruthfulQA profiles where partial recovery occurs at higher α values. The dashed line marks α=1α=1. Appendix F Alpha Sensitivity: TruthfulQA vs. GSM8K Figure 9 presents TruthfulQA and GSM8K alpha sensitivity profiles side by side, making the domain contrast directly legible. On TruthfulQA, student architectures exhibit distinct and heterogeneous profiles with partial recovery at moderate extrapolation values. On GSM8K, all four architectures converge on a single behavioral pattern: peak performance at α=0.25α=0.25 followed by uniform collapse. The architectural heterogeneity visible on the left panel disappears entirely on the right, suggesting that the fragility of mathematical representations under representational displacement is a domain-level property rather than an architecture-level one. Figure 9: Alpha sensitivity profiles for TruthfulQA (left) and GSM8K (right), one line per student architecture averaged across all teachers and layer combinations. The dashed vertical line marks α=1α=1. TruthfulQA profiles are architecturally heterogeneous with partial recovery at higher α; GSM8K profiles converge uniformly on α=0.25α=0.25 with complete collapse beyond α=0.5α=0.5 across all architectures. Appendix G GSM8K Layer Interaction Heatmap Figure 10 presents the layer interaction heatmap for Gemma-9B â Qwen-1.5B, the highest-performing pairing on GSM8K (peak Î=43.3% =43.3\%). Compared to the TruthfulQA counterpart (Figure 3), the monotonic improvement with teacher depth is less pronounced, and lower teacher extraction depths perform more competitively. This suggests that mathematical reasoning representations reach a usable state of resolution at shallower teacher layers than verbal reasoning representations, consistent with the simpler syntactic structure of arithmetic problems relative to open-domain factual questions. Figure 10: Peak correction rate Î across all 16 lTĂlSl_TĂ l_S combinations for Gemma-9B â Qwen-1.5B on GSM8K. Rows are ordered with deepest teacher extraction at the top. The gradient with teacher depth is less monotonic than the TruthfulQA counterpart, with lower-depth teacher extraction performing more competitively. Appendix H Layer Dissociation: Full Depth Sweep Figure 11 presents transfer R2R^2 values across all 16 teacher extraction depth Ă student injection depth combinations for the Mistral-7B â Qwen-1.5B pair. Color intensity reflects the absolute magnitude of transfer interference â darker cells indicate more catastrophic geometric mismatch. Dissociation is confirmed in all 16/16 combinations in both transfer directions, establishing that domain orthogonality is not a depth-specific artifact. A systematic pattern is visible across both panels: student injection depth (lSl_S) modulates interference magnitude more strongly than teacher extraction depth (lTl_T). Mid-network injection (lS=0.50l_S=0.50) consistently produces the strongest interference (TQAâ range: â11.0-11.0 to â11.9-11.9), while late injection (lS=0.90l_S=0.90) produces the weakest (range: â2.9-2.9 to â3.3-3.3). Teacher extraction depth has a comparatively weaker and less consistent effect. This asymmetry implicates the mid-network student layers as the primary site of domain-specific semantic construction, consistent with the hierarchical processing account in Section 3.3. Figure 11: Transfer R2R^2 across all 16 lTĂlSl_TĂ l_S combinations for Mistral-7B â Qwen-1.5B. Left panel: TruthfulQA-trained mapper evaluated on GSM8K. Right panel: GSM8K-trained mapper evaluated on TruthfulQA. Color intensity reflects absolute interference magnitude; all values are negative. Mid-network injection (lS=0.50l_S=0.50) produces the strongest interference in both directions; late injection (lS=0.90l_S=0.90) produces the weakest. Dissociation confirmed 16/16. Appendix I Implementation and Reproducibility Details Hardware. All experiments were conducted on a single NVIDIA A100 80GB GPU. Teacher and student models were loaded sequentially to manage GPU memory: the teacher was loaded, all activation vectors were extracted and cached to CPU, and the teacher was then unloaded before the student was loaded. The Llama-70B teacher was the only model requiring quantization, loaded in 4-bit NF4 format using BitsAndBytesConfig with bnb_4bit_compute_dtype=torch.float16 and bnb_4bit_use_double_quant=True. All other models were loaded in bfloat16 precision. Inference Configuration. All forward passes used left-padded tokenization (padding_side=âleftâ) with pad_token set to eos_token where no dedicated pad token existed. Activation vectors were extracted as the final-token hidden state at each candidate layer using output_hidden_states=True. Batch size was 32 for all extraction and intervention passes. Generation used greedy decoding (do_sample=False, max_new_tokens=50 for TruthfulQA, max_new_tokens=100 for GSM8K). Intervention Hook. The residual stream intervention was implemented as a PyTorch forward hook registered on model.model.layers[layer_idx]. The hook intercepts the layer output, extracts the final token position, rescales the pre-computed projected vector to match the current residual stream norm, applies the weighted combination from Equation 8, and returns the modified hidden state. The hook is registered immediately before each batched generation call and removed immediately after, ensuring no cross-contamination between intervention and baseline passes. Projection Matrix Estimation. Ridge and Lasso regressors were implemented using scikit-learnâs Ridge(alpha=0.1) and Lasso(alpha=0.0001, max_iter=5000) with default fit_intercept=True. The 70/30 train/test split used sklearn.model_selection.train_test_split with random_state=42. Permutation controls used numpy.random.permutation on the training split target matrix prior to fitting, with the same fixed seed. Evaluation. Generated text was decoded with skip_special_tokens=True, split on the "Answer:" delimiter to isolate the modelâs response, and lowercased and stripped before substring matching. Both the best_answer and all entries in correct_answers were lowercased prior to matching. A response was counted as correct if any reference string appeared as a substring of the generated text.