Paper deep dive
Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints
Man Liang, Xinzhao Cheng, Faizan Wajid
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/19/2026, 5:32:32 AM
Summary
This paper audits the gap between encoded geometric constraint information and actionable behavior in six frozen decoder-only LLMs (Qwen2.5, Mistral, Llama-3.1). Using parametric CAD constraints as a testbed, the authors evaluate linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Results show that while local pairwise geometric relations are strongly decodable and benefit significantly from pretraining, global sketch-level degrees-of-freedom status is largely decodable from random initialization. Crucially, high decodability does not translate to reliable generation or steerability; models often fail to express encoded information, and activation interventions have limited or transient effects. This highlights a systematic divergence between what LLMs encode and what they can act upon.
Entities (12)
Relation Signals (9)
Qwen2.5 → evaluatedin → Study
confidence 95% · We evaluate six frozen decoder-only LLMs spanning multiple scales and families: Qwen2.5-0.5B, 1.5B, 3B, and 7B
Mistral → evaluatedin → Study
confidence 95% · Mistral-7B (14) ... each compared against a randomly initialized model
Llama-3.1 → evaluatedin → Study
confidence 95% · Llama-3.1-8B (7) ... each compared against a randomly initialized model
Pretraining → improves → Local Geometric Relations Decodability
confidence 94% · Pretraining substantially improves the decoding of local geometric relations
Linear Decodability → divergesfrom → Forced-Choice Generation
confidence 92% · These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting.
Pretraining → haslittleeffecton → Global Sketch-Level DOF Status Decodability
confidence 91% · sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining
Representation Steering → failsto → Reliably Control Outputs
confidence 90% · Mean-difference steering also does not reliably control outputs.
SketchGraphs → usedfor →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch-level constraint status. By probing the hidden states of six frozen decoder-only LLMs, we examine four properties: linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled-order controls. In contrast, sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights. Further analyses show that decodable information is not always actionable. Generation often fails to express this information, and on the two intervention-tested backbones, activation-restoration effects at the patched entity position vanish while decodability persists across depth. Mean-difference steering also does not reliably control outputs. These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information.
Tags
Links
- Source: https://arxiv.org/abs/2608.17843v1
- Canonical: https://arxiv.org/abs/2608.17843v1
Trouble viewing inline? Open PDF directly →
Full Text
55,553 characters extracted from source content.
Expand or collapse full text
Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints Man Liang Xinzhao Cheng Faizan Wajid Abstract Large language models (LLMs) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch-level constraint status. By probing the hidden states of six frozen decoder-only LLMs, we examine four properties: linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled-order controls. In contrast, sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights. Further analyses show that decodable information is not always actionable. Generation often fails to express this information, and on the two intervention-tested backbones, activation-restoration effects at the patched entity position vanish while decodability persists across depth. Mean-difference steering also does not reliably control outputs. These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information. 1 Introduction Large language models are increasingly used to translate natural language instructions into structured outputs such as programs, plans, and geometric designs (37; 30; 38). Trustworthy deployment in scientific and industrial settings requires structured outputs to satisfy interacting domain constraints, including physical principles, geometric relations, and safety requirements, while remaining globally consistent. Among these applications, parametric CAD provides a particularly useful case because it is widely used in engineering and represents domain constraints in an explicit, verifiable form (27). In parametric CAD, local constraints define relations between geometric elements, while their combined effect determines whether a sketch is under-constrained, well-constrained, or over-constrained (31). Recent work increasingly uses LLMs to generate CAD designs from natural language instructions (15; 18). Newer systems extend one-shot generation with execution or solver feedback to iteratively detect and repair errors (4; 12; 20). However, output-level evaluation alone does not reveal whether successful generation and repair rely on internal constraint representations. Clarifying this relationship is necessary to explain why local competence may coexist with global failure. We therefore ask what geometric constraint information frozen general-purpose LLMs encode and how its decodability relates to generation, activation-level influence, and behavioral control. To answer this question, we develop a four-part audit of linear decodability, forced-choice generation, activation-level influence, and steering using SketchGraphs (27), with a cross-dataset P1 check on Fusion 360 Gallery (36). Our evaluation covers six frozen decoder-only LLMs from the Qwen2.5, Mistral, and Llama-3.1 families (25; 14; 7). The auditing framework includes three matched evaluation tasks: local decoding of pairwise constraints (P1), global decoding of degrees-of-freedom status (P2), and forced-choice generation using the same pairwise labels as P1 (P3). We further use activation patching to test whether restoring activations at the probed entity position affects predictions, and representation steering to test whether modifying those activations can systematically control outputs. To target latent geometric information rather than explicit label cues, we serialize only the geometry and exclude all constraint annotations from the model input. Our results reveal a systematic gap between encoded and actionable geometric constraint information. Local pairwise relations (P1) contain learned information that can be linearly decoded even after accounting for random initialization, input structure, and entity order. Global constraint status (P2), however, benefits little from pretraining. This local-to-global asymmetry is captured by a dissociation index that is positive across all six models. More importantly, strong P1 decodability does not translate into reliable P3 generation, with failure severity varying sharply across model backbones. Activation patching reveals that prediction sensitivity to restoration at the patched entity position is concentrated in early layers and vanishes while decodability persists. Mean-difference steering does not reliably control predictions. These mismatches show why failures on structured tasks should be audited at multiple levels rather than attributed to missing knowledge alone. Our main contribution is a controlled, cross-model framework for auditing geometric constraint representations in frozen LLMs, distinguishing failures to encode information from failures to express, use, or control it. Figure 1: Overview of the four-part audit. Geometry-only sketches are serialized and passed through a frozen LLM, whose representations are evaluated for linear decodability, forced-choice generation, sensitivity to activation restoration, and steering-based control. 2 Related Work Geometric reasoning and CAD generation. Geometric constraint reasoning underlies engineering tasks such as 3D object placement, robotic assembly planning, and manufacturing workflows (13; 16; 5). Parametric CAD makes this local-to-global structure explicit: pairwise geometric constraints define local relations, while remaining degrees of freedom characterize sketch-level constraint status. Prior work captures this structure through complementary representations. SketchGraphs models primitives and explicit pairwise constraints (27), whereas Fusion 360 Gallery, DeepCAD, and CADParser represent designs through construction histories of increasing operational complexity (36; 39; 41). Generative models accordingly target either constrained sketches, as in Vitruvion (28), or CAD construction sequences, as in DeepCAD and SkexGen (39; 40). More recent systems such as Text2CAD, CAD-Llama, CADmium, STEP-LLM, and ReCAD leverage language models to generate CAD sequences or executable code (15; 18; 6; 29; 17). Across these lines of work, evaluation has centered on task-level outcomes such as conditional constraint prediction, CAD reconstruction, validity, and generation quality. We instead examine whether frozen general-purpose LLMs encode constraint structure and how that encoding relates to generation, prediction sensitivity to activation restoration, and behavioral control. Representation analysis and intervention. A common approach to representation analysis is linear probing, which tests what information is linearly decodable from frozen hidden states (1; 2). Beyond linguistic attributes, probing has identified linearly decodable world states in game-playing models (19; 22) and spatial and temporal information in LLMs (8). These findings are consistent with the linear representation hypothesis, which proposes that features are organized along directions in representation space (23). Under this view, linear probing provides a natural tool for examining structured information in model representations. However, high linear-probe accuracy does not by itself show that the relevant representation was learned through pretraining. Instead, it may reflect probe capacity, architectural bias, or information already present in the input (11; 24). Prior work addresses these alternatives using shuffled-label tasks and selectivity (11), random encoders (35), and comparisons with simpler input representations (10). Following these principles, we use shuffled-label and random-initialization controls and introduce a pure-input baseline for serialized geometry. Beyond identifying what probes can extract, prior work has developed interventions that test whether internal representations participate in model behavior (26). Activation patching tests how interventions on intermediate states affect model outputs (33; 21). Related intervention methods have been used to identify behaviorally relevant circuits (34; 3). Activation addition and representation engineering instead modify internal states to steer model behavior (32; 42).These methods test complementary aspects of representational function. Patching asks whether restoring an activation at a selected position affects the prediction, whereas steering asks whether modifying that activation can control the output. We combine these interventions with decoding and generation to distinguish linear accessibility, behavioral expression, activation-level influence, and controllability. 3 Method We evaluate frozen LLMs using geometry-only CAD serializations, three matched prediction tasks, and two activation-level interventions. Figure 1 summarizes the evaluation pipeline. Data and labels. From the SketchGraphs training split (27), we derive pairwise entity-relation labels (P1) and sketch-level constraint-status labels (P2). For P1, ordered entity pairs (i,j)(i,j) are assigned to eight classes: seven pairwise relations (Coincident, Parallel, Perpendicular, Tangent, Equal, Midpoint, and Concentric) and NoConstraint, sampled from unconstrained pairs in the same sketch. We exclude Horizontal and Vertical because they are primarily unary. For P2, sketches are labeled as under-, well-, or over-constrained according to whether the degree-of-freedom count returned by SketchGraphs get_sequence_dof is positive, zero, or negative. We construct stratified, class-balanced subsets of up to 15k pairs per P1 class and 12k sketches per P2 class (minimum 500 per class) to address class imbalance, particularly the limited number of well- and over-constrained sketches. The same sampled subsets are reused across models with a fixed seed. The P2 labels are heuristic rather than solver-verified and may therefore misclassify redundant constraint sets. Tasks. We organize the evaluation into two decoding tasks and one matched behavioral task. P1 tests whether pairwise constraints are linearly decodable from concatenated entity representations, ij=[i;j]x_ij=[h_i;h_j], using eight-class classification. P2 tests whether sketch-level constraint status is linearly decodable from the mean-pooled entity representation, ¯=|ℰ|−1∑e∈ℰe h=|E|^-1 _e h_e, using three-class classification. Whereas P1 and P2 examine linear accessibility at local and global levels, P3 provides a behavioral counterpart to P1 by asking the frozen LLM to complete the template Constraint(Eii,Ejj) =…=… through forced choice over the same eight classes. Using the same held-out pairs, label space, and macro-F1 metric enables a matched comparison between P1 decodability and P3 generation. The main P3 evaluation is zero-shot, and Appendix G reports a four-shot control across five exemplar sets. Serialization and label exclusion. Each sketch is serialized as plain text containing entity types and numeric parameters, such as line endpoints and circle centers and radii. All EdgeOp annotations are omitted, so constraint labels cannot be read directly from the input. Each entity’s character span is mapped to its corresponding token span for pooling. Representation extraction. We pass each serialized sketch through a frozen decoder-only LLM. At each sampled layer, we mean-pool the token-level hidden states within each entity span to obtain eh_e. To compare architectures with different depths, we extract representations from eight evenly spaced relative-depth locations rather than shared absolute layer indices. The resulting representations are stored in FP16 shards, using the same balanced examples across all models. Probes and controls. For P1 and P2, we train a separate ℓ2 _2-regularized logistic regression at each layer on balanced data, using a class-stratified 75/25 split. The main P1 split is performed at the entity-pair level rather than the sketch level. A five-seed sketch-level group-split check on Qwen2.5-3B yields comparable performance, suggesting that sketch overlap does not explain the P1 result (Appendix D). To isolate the contribution of pretraining, we compare pretrained representations with same-architecture randomly initialized models and a pure-input baseline. Shuffled-label probes control for probe memorization, while entity-count controls test whether P2 performance can be explained by sketch size. Shuffled-order control. To test whether P1 relies on entity position, we randomly permute the entity order within each serialization while preserving entity identities, geometry, and labels, then repeat representation extraction and probing for every model. The resulting performance change measures sensitivity to serialization order (Section 5.2). Activation-level influence and steerability. To measure prediction sensitivity to activations at the probed entity position, we perform activation patching on Qwen2.5-3B and Llama-3.1-8B. We corrupt entity i’s input embedding with Gaussian noise and restore its clean hidden state at each tested layer (33; 21; 9). Restoration rate is the fraction of corruption-informative examples for which patching recovers the clean prediction. We test layers at four-layer intervals and additionally include each model’s P1 decodability peak. Distractor specificity measures whether restoring entity i leaves the prediction for an unrelated pair (k,m)(k,m) unchanged. To determine whether this activation-level influence can support targeted control, we add class mean-difference vectors at the strongest nontrivial restoration layer (layer 4 for both models), with α∈0.5,1,2,4,8α∈\0.5,1,2,4,8\. We measure flip-to-target rates and compare them with matched random-direction controls across 10 seeds. 4 Experimental Setup Models. We evaluate six frozen decoder-only LLMs spanning multiple scales and families: Qwen2.5-0.5B, 1.5B, 3B, and 7B (25), Mistral-7B (14), and Llama-3.1-8B (7), each compared against a randomly initialized model of the same architecture. All six are run through the identical extraction and probing protocol described in Section 3, drawing from the same balanced subsets with a fixed sampling seed reused across models, so that cross-model differences reflect the models themselves rather than sampling noise. Metrics. We evaluate probe performance using macro-F1, with uniform-class reference levels of 0.1250.125 for P1 and 0.3330.333 for P2. We additionally report selectivity, F1,task−F1,shuffledF_1,task-F_1,shuffled (11), to control for probe memorization. To quantify whether pretraining contributes differently to local and global decodability, we define a dissociation index (DI). For each architecture, we first identify the layer ℓ∗ ^* at which the pretrained model achieves its highest P1 macro-F1: ℓ∗=argmaxℓF1,preP1(ℓ). ^*= _ F_1,pre^P1( ). (1) We then hold this layer fixed for all four quantities entering DI. In particular, pretrained and random initialized P2 performance are evaluated at the P1-selected layer ℓ∗ ^* rather than at an independently selected P2 peak. The corresponding random-initialized P1 performance is also read at ℓ∗ ^*. We define DI= = (F1,preP1(ℓ∗)−F1,randP1(ℓ∗)) (F_1,pre^P1( ^*)-F_1,rand^P1( ^*) ) (2) −(F1,preP2(ℓ∗)−F1,randP2(ℓ∗)). - (F_1,pre^P2( ^*)-F_1,rand^P2( ^*) ). Here, prepre and randrand denote pretrained and same-architecture randomly initialized models, respectively. A positive DI indicates that pretraining improves P1 more than P2 under this common-layer comparison. Subtracting the corresponding random-init baselines partially controls for architectural and dimensional differences across model families and scales. P3 uses the same held-out entity pairs, eight-class label set, and macro-F1 metric as P1, enabling a matched comparison between supervised linear decodability and forced-choice generation. Unless otherwise noted, we report single-split F1 estimates using seed 0. Reported 95% confidence intervals use 1,000 bootstrap resamples. For DI, uncertainty from the P1 and P2 components is combined in quadrature rather than estimated with a direct paired bootstrap. 5 Results 5.1 Constraint Information Is Linearly Decodable Both P1 and P2 are linearly decodable from the hidden states of all six trained models. Peak P1 macro-F1 ranges from 0.714 to 0.734, well above the 0.125 chance level (Table 1). Selectivity remains high at 0.593–0.606, indicating that this performance is not explained by shuffled-label memorization. Across architectures, P1 decodability rises rapidly in early layers and remains high across a broad depth range (Figure 2). P2 reaches similarly high peak macro-F1 values of 0.719–0.732, although Section 5.2 shows that most of this performance is already available without pretraining. Figure 2: P1 macro-F1 across relative depth for all six models and controls. Orange brackets show the drop under entity-order shuffling; green marks show the remaining gain over random initialization. 5.2 P1 Benefits More from Pretraining Than P2 The controls reveal different sources of P1 and P2 probe performance. For P1, macro-F1 increases from 0.359 with pure-input features to 0.549–0.598 with random-init representations and 0.714–0.734 with pretrained representations. Pretraining therefore improves macro-F1 over random initialization by 0.127 to 0.185. Shuffling entity order lowers P1 macro-F1 by between 0.10 and 0.13, showing that position provides a substantial shortcut. Even after shuffling, pretrained models outperform their random-init counterparts by between 0.026 and 0.075 (Figure 2). Thus, positional cues explain some, but not all, of P1’s gain from pretraining. P2 relies much less on pretraining. At the P1-selected layer ℓ∗ ^*, pretraining improves P2 macro-F1 over random initialization by only 0.037 to 0.048, compared with 0.127 to 0.185 for P1 (Table 1). Thus, most of P2’s decodability is already present without learned weights. The conclusion is unchanged when P2 is evaluated at its own peak, where the gain remains 0.037–0.047. A logistic regression using only the number of entities reaches 0.419, showing that sketch size provides some signal but cannot explain the full P2 performance. Overall, pretraining contributes substantially more to P1 decodability than to P2, even after accounting for the tested positional shortcut. Peak F1 Selectivity Dissociation Model P1 P2† P1 P2† P1 layer/total DIDI at ℓ∗ ^* [95% CI] Net gain‡ Baselines Chance .125 .333 – – – – – Pure input .359 .380 – – – – – Random-init (no pretraining) Qwen2.5-0.5B .598 .679 .468 .349 3/24 – – Qwen2.5-1.5B .567 .684 .442 .355 4/28 – – Qwen2.5-3B .560 .691 .429 .353 5/36 – – Qwen2.5-7B .560 .683 .433 .349 4/28 – – Llama-3.1-8B .549 .684 .423 .360 14/32 – – Mistral-7B .563 .680 .431 .348 5/32 – – Trained Qwen2.5-0.5B .725 .719 .606 .384 10/24 .106 [.089, .124] .030 Qwen2.5-1.5B .724 .732 .596 .399 12/28 .141 [.123, .158] .040 Qwen2.5-3B .714 .728 .593 .396 21/36 .167 [.151, .184] .026 Qwen2.5-7B .719 .728 .598 .400 20/28 .152 [.128, .176] .049 Llama-3.1-8B .734 .727 .602 .400 14/32 .142 [.118, .166] .075 Mistral-7B .725 .727 .597 .394 14/32 .125 [.101, .150] .057 Table 1: Probe performance across models. P1 columns and P1 layer/total report each checkpoint’s own P1 peak. †P2 is evaluated at the corresponding pretrained model’s P1 peak, ℓ∗ ^*. DI evaluates all four trained and random-init terms at this pretrained ℓ∗ ^* (Eq. 2). Net gain‡ is shuffled-order pretrained P1 macro-F1 minus the random-init P1 peak. 5.3 The Dissociation Holds Across Scale and Architecture We next test whether the P1–P2 dissociation extends beyond a single model. Within the Qwen2.5 family, P1 macro-F1 ranges from 0.714 to 0.725 and P2 from 0.719 to 0.732, with no monotonic improvement as model size increases (Table 1). DI nevertheless remains positive at every scale, ranging from 0.106 to 0.167, although it also varies non-monotonically. Scaling therefore has no consistent effect on either task or on their relative pretraining gains. The dissociation also holds across model families. At comparable model sizes, Qwen2.5, Llama, and Mistral achieve similar raw macro-F1 on P1 and P2, but P1 selectivity is consistently higher (0.597 to 0.602 versus 0.394 to 0.400). More directly, DI is positive for all six models, with every 95% confidence interval excluding zero. The selected P1 peaks span layers 10 to 21, indicating that the pattern is not tied to a shared absolute depth. Chance-normalized DInormDI_norm also remains positive across all models (0.107 to 0.178; Table 4, Appendix B.2). Together, these results show that the dissociation is stable across the tested scales, architectures, and chance normalization. 5.4 Generation Falls Short of Decodability We compare P3 forced-choice generation with P1 decoding on the same held-out pairs and eight-class label set. Across all six models, P3 macro-F1 is substantially lower than P1 probe performance, with gaps ranging from 0.460 to 0.700 (Table 2). Thus, information that is linearly decodable is not reliably expressed in the model’s own predictions. Model P1 probe F1 P3 gen. F1 Gap Qwen2.5-0.5B 0.725 0.097 0.628 Qwen2.5-1.5B 0.724 0.072 0.652 Qwen2.5-3B 0.714 0.081 0.633 Qwen2.5-7B 0.719 0.259 0.460 Mistral-7B 0.725 0.025 0.700 Llama-3.1-8B 0.734 0.151 0.583 Table 2: Matched P1 probe and P3 generation macro-F1 on the same held-out pairs and eight-class label set (chance=0.125=0.125). Failure modes differ across architectures (Figure 3). Mistral-7B predicts Coincident for 99.8% of examples, producing an almost complete single-class collapse. Qwen2.5-7B instead predicts all eight classes and achieves non-trivial accuracy on several, yielding the highest P3 macro-F1 of 0.259 despite having P1 performance similar to the other models. Full per-class results are reported in Appendix C.1. Content-free prompts reveal class preferences aligned with these outputs. The dominant blank-prompt class matches the dominant real-task prediction for both Mistral-7B and Qwen2.5-7B, with maximum prior probabilities of 0.345 and 0.250, respectively. These controls are consistent with prior bias contributing to P3 behavior, but do not by themselves determine how much of the real-task distribution it explains. Prompting also accounts for only part of the gap on Qwen2.5-3B. Four-shot prompting increases mean macro-F1 from 0.081 to 0.138±0.0130.138± 0.013 across five exemplar sets, but remains 0.576 below the P1 probe score (Appendix G). Few-shot examples improve generation, but P3 still performs far below the linear probe. Figure 3: P3 predicted-class distributions across six models. Boxes mark the dominant class, and dashes denote exact zeros. Per-class accuracies are reported in Appendix C.1. 5.5 Activation-Level Influence and Steerability Activation patching reveals an early but transient influence at the probed entity position. On Qwen2.5-3B, restoration peaks at layer 4 with a rate of 0.781 [0.722, 0.846] and falls to zero by layer 16, before the P1 decodability peak at layer 21. Llama-3.1-8B shows the same pattern, peaking at layer 4 with a restoration rate of 0.876 [0.821, 0.924] and reaching zero by layer 12, before its decodability peak at layer 14. Neither model shows a later resurgence through the deepest tested layer. In contrast, decodability reaches a broad plateau by approximately layers 5 to 9 and persists after restoration effects disappear. Full layerwise results are shown in Figure 7 and Appendix E.1. At the layer 4 restoration peak, distractor specificity is 0.798 for Qwen2.5-3B and 0.847 for Llama-3.1-8B. Restoration is therefore largely, but not perfectly, specific to the patched entity at the layer where its effect is strongest. Specificity reaches 1.0 only at later layers, after the restoration rate has fallen to zero. We next ask whether this activation-level influence can be harnessed for targeted control. At the same layer, mean-difference steering produces no target-class flips at any tested strength for either P1 or P2 on either model, a result reproduced in two independent runs. On Qwen2.5-3B, the mean-difference direction changes about four times as many labels as matched random directions at α=8α=8, but none of these changes reach the intended target class. On Llama-3.1-8B, label changes do not exceed the random baseline (Appendix E.2). Thus, activation restoration can influence predictions without providing reliable targeted control. 6 Discussion 6.1 Representational Dissociation Our main finding is that the linear accessibility of geometric constraint information in frozen LLMs does not ensure its expression in generation, continued influence at the probed entity position, or controllability through mean-difference steering. The clearest layerwise contrast is between decodability and activation restoration. Decodability persists across a broad depth range, whereas restoration effects at the probed entity position are early and transient. A layer may therefore retain linearly recoverable information even after predictions are no longer sensitive to restoring the activation at that position. One possible explanation is that later computation routes the relevant information to other token positions or into distributed representations, leaving a readable trace at the original entity position after dependence on that position has diminished. The difference between patching and steering may also reflect intervention scope. Patching restores the full activation vector at the tested position, whereas mean-difference steering modifies a single direction that may not capture the combination of features used by the model. These explanations remain hypotheses that require circuit-level analysis. 6.2 Implications for Interpretability These results delimit the conclusions supported by each interpretability method. Linear probing demonstrates that information is recoverable, not that it is behaviorally used. Activation patching shows that intervening on an activation at a tested position can affect the output, but does not identify which decodable feature mediates that effect. Similarly, failure under mean-difference steering rules out that intervention direction, not every possible form of control. Representational claims should therefore distinguish recoverability, behavioral expression, activation-level influence, and control, while remaining scoped to the interventions actually tested. 6.3 Practical Implications For practical CAD systems, high probe scores are not enough to establish reliable geometric reasoning. P2 is highly decodable even without pretraining, and strong P1 decodability does not translate into accurate constraint predictions. Systems that require valid outputs should therefore not rely on LLM representations alone and may need explicit validity supervision, structured state tracking, or solver-based verification. 6.4 Limitations Our conclusions apply only to geometry-only inputs from SketchGraphs. P1 excludes the primarily unary Horizontal and Vertical constraints. P2 labels are derived from a heuristic Grübler-style DOF count rather than solver verification. Because the heuristic does not verify constraint independence, sketches with redundant constraints may receive incorrect labels. Moreover, P2 tests only coarse DOF status, not other aspects of global geometry such as consistency, solvability, or redundancy. Our uncertainty estimates do not cover all sources of experimental variation. The main P1 and P2 results use one fixed data split, each architecture has one random-init checkpoint, and order shuffling and P3 sampling use fixed seeds. P1 is split by entity pair rather than by sketch. Although a five-seed sketch-level split check gives comparable results on Qwen2.5-3B (Appendix D), it was not repeated across all architectures. In addition, DI confidence intervals capture uncertainty from resampling the evaluation examples, but not variation from data splitting, initialization, or layer selection. They are also approximate because the independently bootstrapped component uncertainties are combined in quadrature rather than obtained by directly bootstrapping DI. The behavioral and intervention analyses cover a narrower range of settings than the decoding experiments. P3 uses a single forced-choice formulation, and the four-shot control focuses on Qwen2.5-3B. Performance may vary with other label formulations, prompts, or exemplar choices. The intervention experiments cover two backbones. Patching is evaluated on corruption-informative pairs (n=169n=169 and 145145), while steering examines mean-difference directions up to α=8α=8. These results may vary with the model, corruption scheme, intervention direction, or token position. Finally, our residual-stream interventions operate at an aggregate level and do not localize the effects to specific attention heads, MLPs, or neurons. 6.5 Future Work The present results leave open how broadly the observed dissociation extends beyond SketchGraphs. Testing other geometric corpora and CAD-native models such as Vitruvion (28) would establish its generality. Our Fusion 360 Gallery experiment provides initial cross-dataset evidence for P1 (Appendix F), but the dataset lacks matched three-class labels for P2. A broader evaluation of global reasoning will therefore require solver-derived validity signals that improve on the heuristic Grübler-style labels used here. The behavioral and intervention analyses could likewise be extended across additional backbones, verbalizers, prompting strategies, patching designs, and steering directions. Building on these experiments, circuit-level analysis could examine whether the gaps among decodability, generation, and control arise from information routing, distributed computation, or context-dependent representations. 7 Conclusion We introduced a four-part audit of linear decodability, forced-choice generation, activation-level influence, and behavioral steerability in frozen LLMs. By evaluating these properties separately under matched conditions, the framework provides a controlled way to identify where representational evidence does and does not translate into model behavior. Across six models, pretraining contributes substantially more to pairwise constraint decoding than to sketch-level DOF classification, for which randomly initialized models already achieve high probe performance. P3 generation remains well below supervised P1 decoding, and a four-shot control on Qwen2.5-3B narrows but does not close this gap. On the two backbones tested with interventions, activation restoration affects predictions primarily at early layers and vanishes at the patched entity position while decodability persists. Mean-difference steering at the strongest restoration layer produces no targeted class flips. The results show that decodability, behavioral expression, activation-level influence, and control are empirically distinct: a high probe score demonstrates that information can be linearly recovered, but provides limited evidence that a model will express, use, or respond to interventions on that information. Parametric CAD makes these distinctions directly testable, and our audit offers a framework for separating failures to encode constraint structure from failures to act on what is already encoded. References Alain and Bengio (2017) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. In ICLR Workshop, Cited by: §2. Belinkov and Glass (2019) Y. Belinkov and J. Glass Analysis methods in neural language processing: a survey. Transactions of the Association for Computational Linguistics 7, p. 49–72. Cited by: §2. Conmy et al. (2023) A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, Cited by: §2. Fan et al. (2026) F. Fan, J. Ni, F. Sang, X. Yin, Y. Liu, R. Tong, M. Tang, and P. Du TraceCAD: Trace-Guided Repair for Agentic CAD Generation. External Links: Link Cited by: §1. González-Lluch et al. (2019) C. González-Lluch, P. Company, M. Contero, D. Pérez-López, and J. D. Camba On the effects of the fix geometric constraint in 2D profiles on the reusability of parametric 3D CAD models. International Journal of Technology and Design Education 29 (4), p. 821–841 (en). External Links: ISSN 0957-7572, 1573-1804, Link, Document Cited by: §2. Govindarajan et al. (2025) P. Govindarajan, D. Baldelli, J. Pathak, Q. Fournier, and S. Chandar CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design. External Links: Link, Document Cited by: §2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §1, §4. Gurnee and Tegmark (2024) W. Gurnee and M. Tegmark Language Models Represent Space and Time. arXiv. Note: arXiv:2310.02207 [cs.LG] External Links: Link, Document Cited by: §2. Hanna et al. (2023) M. Hanna, O. Liu, and A. Variengien How does GPT-2 compute greater-than?: interpreting mathematical abilities in a pre-trained language model. In Advances in Neural Information Processing Systems, Cited by: §3. Hewitt et al. (2021) J. Hewitt, K. Ethayarajh, P. Liang, and C. Manning Conditional probing: measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 1626–1639. External Links: Link, Document Cited by: §2. Hewitt and Liang (2019) J. Hewitt and P. Liang Designing and interpreting probes with control tasks. In Empirical Methods in Natural Language Processing, Cited by: §2, §4. Hu et al. (2026) T. Hu, J. Ai, L. Wen, X. Li, S. Zou, S. Li, N. Deng, X. Cai, H. Zhou, P. Cai, D. Fu, Y. Yang, H. Zhang, B. Shi, and X. Yang IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing. External Links: Link, Document Cited by: §1. Huang et al. (2025) I. Huang, Y. Bao, K. Truong, H. Zhou, C. Schmid, L. Guibas, and A. Fathi FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13466–13476. External Links: Link, Document Cited by: §2. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §1, §4. Khan et al. (2024) M. S. Khan, S. Sinha, T. U. Sheikh, D. Stricker, S. A. Ali, and M. Z. Afzal Text2CAD: generating sequential CAD designs from beginner-to-expert level text prompts. In Advances in Neural Information Processing Systems, p. 7552–7579. Cited by: §1, §2. Leu et al. (2013) M. C. Leu, H. A. ElMaraghy, A. Y.C. Nee, S. K. Ong, M. Lanzetta, M. Putz, W. Zhu, and A. Bernard CAD model based virtual assembly simulation, planning and training. CIRP Annals 62 (2), p. 799–822 (en). External Links: ISSN 00078506, Link, Document Cited by: §2. Li et al. (2025a) J. Li, Y. Luo, Y. Lou, and X. Zhou ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models. External Links: Link, Document Cited by: §2. Li et al. (2025b) J. Li, W. Ma, X. Li, Y. Lou, G. Zhou, and X. Zhou CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18563–18573. External Links: Link, Document Cited by: §1, §2. Li et al. (2024) K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. arXiv. Note: arXiv:2210.13382 [cs.LG] External Links: Link, Document Cited by: §2. Liu et al. (2026) F. Liu, H. Zhou, F. Hao, and L. Yang Embodied CAD: Solver-Grounded LLM Agents for Parametric B-Rep Assembly Modeling. External Links: Link, Document Cited by: §1. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: §2, §3. Nanda et al. (2023) N. Nanda, A. Lee, and M. Wattenberg Emergent Linear Representations in World Models of Self-Supervised Sequence Models. arXiv. Note: arXiv:2309.00941 [cs.LG] External Links: Link, Document Cited by: §2. Park et al. (2024) K. Park, Y. J. Choe, and V. Veitch The Linear Representation Hypothesis and the Geometry of Large Language Models. arXiv. Note: arXiv:2311.03658 [cs.CL] External Links: Link, Document Cited by: §2. Pimentel et al. (2020) T. Pimentel, J. Pimentel, H. Velioglu, M. Wich, and R. Cotterell Information-theoretic probing for linguistic structure. In Association for Computational Linguistics, p. 4609–4619. Cited by: §2. Qwen et al. (2025) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, et al. Qwen2.5 Technical Report. arXiv. Note: arXiv:2412.15115 [cs.CL] External Links: Link, Document Cited by: §1, §4. Ravichander et al. (2021) A. Ravichander, Y. Belinkov, and E. Hovy Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, P. Merlo, J. Tiedemann, and R. Tsarfaty (Eds.), Online, p. 3363–3377. Cited by: §2. Seff et al. (2020) A. Seff, Y. Ovadia, W. Zhou, and R. P. Adams SketchGraphs: a large-scale dataset for modeling relational geometry in CAD. In ICML Workshop on Object-Oriented Learning, Cited by: §1, §1, §2, §3. Seff et al. (2022) A. Seff, W. Zhou, N. Richardson, and R. P. Adams Vitruvion: a generative model of parametric CAD sketches. In International Conference on Learning Representations, Cited by: §2, §6.5. Shi et al. (2026) X. Shi, J. Ding, X. Zhao, S. Zhan, P. Mohapatra, D. Quispe, K. Welbeck, J. Cao, W. Chen, P. Guo, and Q. Zhu STEP-LLM: Generating CAD STEP Models from Natural Language with Large Language Models. External Links: Link, Document Cited by: §2. Singh et al. (2023) I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 11523–11530. External Links: Link, Document Cited by: §1. Thierry et al. (2011) S. E. B. Thierry, P. Schreck, D. Michelucci, C. Fünfzig, and J. Génevaux Extensions of the witness method to characterize under-, over- and well-constrained geometric constraint systems. Computer-Aided Design 43 (10), p. 1234–1249. External Links: ISSN 0010-4485, Link, Document Cited by: §1. Turner et al. (2024) A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid Steering Language Models With Activation Engineering. arXiv. Note: arXiv:2308.10248 [cs.CL] External Links: Link, Document Cited by: §2. Vig et al. (2020) J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber Investigating gender bias in language models using causal mediation analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4548–4561. Cited by: §2, §3. Wang et al. (2023) K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations, Cited by: §2. Wieting and Kiela (2019) J. Wieting and D. Kiela No Training Required: Exploring Random Encoders for Sentence Classification. arXiv. Note: arXiv:1901.10444 [cs.CL] External Links: Link, Document Cited by: §2. Willis et al. (2021) K. D. D. Willis, Y. Pu, J. Luo, H. Chu, T. Du, J. G. Lambourne, A. Solar-Lezama, and W. Matusik Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences. arXiv (en). Note: arXiv:2010.02392 [cs.LG] External Links: Link, Document Cited by: §1, §2. Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. Awadallah, R. W. White, D. Burger, and C. Wang AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In arXiv preprint arXiv:2308.08155, External Links: Link Cited by: §1. Wu et al. (2025) R. Wu, W. Su, and J. Liao Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23690–23700. External Links: Link, Document Cited by: §1. Wu et al. (2021) R. Wu, C. Xiao, and C. Zheng DeepCAD: a deep generative network for computer-aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6772–6782. Cited by: §2. Xu et al. (2022) X. Xu, K. D. D. Willis, J. G. Lambourne, C. Cheng, P. K. Jayaraman, and Y. Furukawa SkexGen: autoregressive generation of CAD construction sequences with disentangled codebooks. In International Conference on Machine Learning, p. 24698–24724. Cited by: §2. Zhou et al. (2023) S. Zhou, T. Tang, and B. Zhou CADParser. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Guide Proceedings, p. 1804–1812. External Links: Link, Document Cited by: §2. Zou et al. (2025) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. Note: arXiv:2310.01405 [cs.LG] External Links: Link, Document Cited by: §2. Appendix A Task and Label Details A.1 Geometry-Only Serialization and Matched Tasks Figure 4 illustrates the input and labeling protocol using one toy sketch. The geometric entities are serialized for the frozen LLM, while the EdgeOp relation is used only as a supervision or evaluation target and never appears in the model input. Figure 4: Worked example of the geometry-only protocol. A source sketch (a) is serialized with all EdgeOp labels excluded (b) and used to construct the matched P1–P3 evaluation targets (c). A.2 P1/P2 Constraint-Type Gallery Figure 5 shows the eight P1 classes: seven pairwise EdgeOp relations and a sampled NoConstraint class. It also illustrates the three P2 DOF labels: under-, well-, and over-constrained. Figure 5: P1 pairwise constraint classes and P2 structural DOF labels in SketchGraphs. Highlighted entities indicate the pair probed in P1; EdgeOp tokens are excluded from the LLM input. A.3 P2 Label Quality Checks We assess two potential concerns with the heuristic DOF-status labels used for P2. Entity-count baseline. Because under-constrained sketches tend to contain fewer entities, sketch size may provide a classification shortcut. A logistic regression using entity count alone achieves 0.419 macro-F1, above the uniform-class reference of 0.333 but well below both random-init (0.679–0.691) and pretrained probes (0.719–0.732). Sketch size therefore explains some, but not most, of the observed P2 performance. Constraint satisfaction. We also test whether the constraints assigned to each sketch can be satisfied simultaneously. The test is passed by 100% of well-constrained sketches and 97.5% of over-constrained sketches. This provides evidence of geometric feasibility, but does not validate constraint independence. Redundant constraints may remain jointly satisfiable while removing fewer independent degrees of freedom than the heuristic assumes. A full validation would require solver-based rank analysis of the constraint Jacobian (Section 6.5). Appendix B Representation and Split Controls B.1 Random-Init Controls Table 3 reports the independently selected P1 and P2 peaks from one random initialization of each architecture. Hidden states are extracted and probed using the same pipeline as for the pretrained models. These task-specific peaks are descriptive controls and are not used to compute DI, which evaluates all four component scores at the pretrained model’s P1-selected layer ℓ∗ ^*. Model (random init) Peak P1 Peak P2 P1 layer Qwen2.5-0.5B 0.598 0.681 3 Qwen2.5-1.5B 0.567 0.686 4 Qwen2.5-3B 0.560 0.695 5 Qwen2.5-7B 0.560 0.691 4 Llama-3.1-8B 0.549 0.684 14 Mistral-7B 0.563 0.692 5 Table 3: Task-specific peak macro-F1 scores for the random-init controls. The final column reports the layer of the P1 peak. Random-init P1 peaks range from 0.549 to 0.598, well below the pretrained range of 0.714–0.734. Random-init P2 peaks, however, reach 0.681–0.695, substantially above the pure-input baseline of 0.380 and only modestly below the pretrained range of 0.719–0.732. Across all six architectures, random-init representations therefore reproduce P2 performance much more closely than P1 performance. B.2 Chance-Normalized Dissociation Index The main analysis computes DI on the raw macro-F1 scale (Section 5.3). Because P1 and P2 have different uniform-class reference levels, we repeat the analysis after normalizing each score by its headroom above the corresponding reference: gt(F1)=F1−ct1−ct,cP1=18,cP2=13.g_t(F_1)= F_1-c_t1-c_t, c_P1= 18, c_P2= 13. (3) For task t∈P1,P2t∈\P1,P2\, the raw and normalized pretraining gains are defined as Δt ^t =F1,pret(ℓ∗)−F1,randt(ℓ∗), =F_1,pre^t( ^*)-F_1,rand^t( ^*), (4) Δnormt _norm^t =Δt1−ct. = ^t1-c_t. The chance-normalized dissociation index is then DInorm=ΔnormP1−ΔnormP2.DI_norm= _norm^P1- _norm^P2. (5) The reference terms cancel within each pretrained–random-init contrast, leaving each gain rescaled by its task-specific headroom. Model ℓ∗ ^* Raw DI DInormDI_norm Qwen2.5-0.5B 10 .106 .107 Qwen2.5-1.5B 12 .141 .143 Qwen2.5-3B 21 .167 .178 Qwen2.5-7B 20 .152 .157 Llama-3.1-8B 14 .142 .147 Mistral-7B 14 .125 .126 Table 4: Raw and chance-normalized DI at each model’s P1-selected layer ℓ∗ ^*. As shown in Table 4, chance-normalized DI remains positive for all six models and closely tracks the raw DI. The P1–P2 dissociation therefore cannot be explained by the tasks’ different uniform-class reference levels. Appendix C P3 Diagnostics C.1 P3 Per-Class Accuracy Figure 3 reports how frequently each model predicts each class. Here, Figure 6 reports accuracy conditional on the true class, while Table 5 presents both quantities. In each table cell, the first value is the fraction of all examples predicted as that class, and the second is accuracy among examples whose true label is that class. Figure 6: P3 accuracy by model and true constraint class. Dashes denote exact zeros. Model Coin (222) Par (239) Perp (247) Tan (256) Eq (261) Mid (249) Con (264) NoC (262) Qwen2.5-0.5B 0.21/0.27 0.10/0.15 0.04/0.07 0.65/0.96 –/– –/– –/– –/– Qwen2.5-1.5B 0.01/0.00 –/– 0.22/0.40 0.77/0.95 –/– –/– –/– 0.00/– Qwen2.5-3B 0.35/0.36 –/– 0.02/0.07 0.63/0.96 0.00/0.00 –/– 0.00/0.01 –/– Qwen2.5-7B 0.08/0.11 0.36/0.79 0.09/0.36 0.34/0.96 0.00/– 0.00/0.01 0.12/0.53 0.00/0.02 Mistral-7B 1.00/1.00 0.00/– –/– –/– –/– –/– –/– 0.00/– Llama-3.1-8B 0.28/0.23 0.02/0.05 0.01/0.01 0.10/0.38 0.01/0.01 –/– 0.03/0.14 0.56/0.81 Table 5: P3 predicted-class frequency and per-class accuracy. Each cell reports the fraction of all predictions assigned to that class, followed by accuracy among examples with that true label. Column headers give the true-class counts (n=2,000n=2,000 total). Bold marks each model’s most frequently predicted class. Values are rounded to two decimals; dashes denote exact zeros. The models show different generation failure modes. Mistral-7B collapses almost entirely to Coincident and succeeds only when that label is correct. Qwen2.5-7B distributes its predictions across more classes and shows uneven class-specific accuracy: Parallel is predicted most often, whereas Tangent is predicted most accurately. C.2 P3 Content-Free Prior Control To measure class preferences in the absence of geometric content, Section 5.4 evaluates five content-free entity-index templates. Table 6 reports the mean probability assigned to each class for Mistral-7B and Qwen2.5-7B. Class Mistral-7B Qwen2.5-7B Coincident 0.345 0.129 Parallel 0.153 0.250 Perpendicular 0.090 0.152 Tangent 0.043 0.212 Equal 0.075 0.032 Midpoint 0.028 0.038 Concentric 0.057 0.056 NoConstraint 0.207 0.131 Table 6: Mean P3 class probabilities across five content-free templates. Bold marks the highest-probability class for each model. The content-free preferences align with the dominant real-task predictions. Mistral-7B assigns the highest prior probability to Coincident, while Qwen2.5-7B assigns the highest probability to Parallel, followed closely by Tangent. This alignment suggests that class priors contribute to the observed prediction patterns. However, because the control reports probability mass whereas the real-task analysis reports argmax frequencies, it does not determine how much of those patterns is explained by prior bias. Appendix D P1 Sketch-Level Split Check To test whether cross-partition sketch overlap inflates P1 performance, we repeat the Qwen2.5-3B probe at its selected layer (ℓ∗=21 ^*=21) using GroupShuffleSplit, which assigns all entity pairs from a sketch to the same partition. Across five split seeds, the sketch-level split achieves a macro-F1 of 0.700±0.0020.700± 0.002, compared with 0.699±0.0090.699± 0.009 for the original pair-level split (mean ± standard deviation). These averages differ from the single-split estimates in the main results. The similar performance indicates that sketch overlap does not materially affect the P1 result in this setting. Appendix E Intervention Details E.1 P1 Activation-Patching Layer Grid We evaluate activation patching at four-layer intervals and additionally include each model’s P1 decodability peak. The analysis contains 169 corruption-informative pairs for Qwen2.5-3B and 145 for Llama-3.1-8B. Figure 7 reports restoration rates with 95% confidence intervals from 1,000 bootstrap resamples over pairs. Layer 0, where restoration directly reverses the embedding corruption, serves as a sanity check and is excluded when selecting the strongest nontrivial restoration layer. For Qwen2.5-3B, restoration is highest at layer 4 (0.781 [0.722, 0.846]), declines at layers 8 (0.675 [0.609, 0.746]) and 12 (0.604 [0.533, 0.675]), and reaches zero by layer 16, before the P1 decodability peak at layer 21. Llama-3.1-8B shows the same pattern. Restoration decreases from 0.876 [0.821, 0.924] at layer 4 to 0.759 [0.690, 0.821] at layer 8, then remains at zero from layer 12 onward, including at its decodability peak at layer 14. In both models, restoration at the patched entity position therefore disappears before peak decodability. Figure 7: Layerwise P1 restoration rates and decodability for Qwen2.5-3B and Llama-3.1-8B. Annotations mark the strongest nontrivial restoration layer and each model’s P1 decodability peak. Shaded bands show 95% bootstrap confidence intervals. E.2 Steering at the Restoration Peak We evaluate steering at layer 4, the strongest nontrivial restoration layer for both models. Class mean-difference directions are compared with matched random directions over α∈0.5,1,2,4,8α∈\0.5,1,2,4,8\ using 200 examples and 10 random-direction seeds. Each experiment is repeated in two independent runs. Mean-difference steering produces no target-class flips at any α for either task or model. Target-flip rates under random directions also remain at or below 0.1%. Table 7 reports the less restrictive P1 label-change rate, which counts any change in prediction, whether or not it reaches the intended class. Qwen2.5-3B Llama-3.1-8B α Mean diff. Random Mean diff. Random 0.5 0.0% 0.05–0.2% 2.0–2.5% 2.75–3.0% 1.0 0.0–1.0% 0.15–0.4% 5.0–5.5% 5.7–5.75% 2.0 1.0% 0.4–0.5% 9.0–9.5% 10.8–10.9% 4.0 2.5% 0.75–0.9% 12.5% 21.9–22.0% 8.0 11.5–12.0% 2.85–2.9% 16.0–16.5% 25.0–25.2% Table 7: P1 label-change rates at layer 4 under mean-difference and matched random-direction interventions. Cells report ranges across two independent runs. For Qwen2.5-3B, mean-difference directions cause more label changes than random directions when α≥2α≥ 2, but none reach the intended class. For Llama-3.1-8B, their label-change rates remain at or below the random baseline. The observed changes therefore do not provide evidence of reliable targeted steering. Appendix F Cross-Dataset P1 Check We apply the P1 probing protocol to Fusion 360 Gallery reconstruction data (r1.0.1) using Qwen2.5-3B and 13,600 balanced entity pairs (1,700 per class). This check covers P1 only because Fusion 360 Gallery does not provide matched three-class DOF-status labels for P2. Macro-F1 peaks at 0.643 at layer 26 (72.2%72.2\% relative depth) and changes little between layers 21 and 31. This broad intermediate-to-late plateau is consistent with the layerwise pattern observed on SketchGraphs. Model layer Macro-F1 Selectivity 0 0.484 0.354 5 0.567 0.439 10 0.601 0.475 15 0.612 0.489 21 0.640 0.518 26 0.643 0.515 31 0.641 0.515 36 0.631 0.501 Table 8: P1 macro-F1 and selectivity across sampled Qwen2.5-3B layers on Fusion 360 Gallery. Appendix G P3 Few-Shot Prompting Control To assess the sensitivity of P3 to prompt format, we evaluate four-shot prompting on Qwen2.5-3B using five independently sampled exemplar sets. Each prompt contains one labeled example from each of four sampled classes, followed by the same eight-way forced-choice task used in the zero-shot evaluation. The evaluation pairs and scoring procedure remain unchanged. Across the five exemplar sets, four-shot prompting increases mean macro-F1 from the zero-shot score of 0.081 to 0.138±0.0130.138± 0.013 (mean ± standard deviation), with individual scores ranging from 0.117 to 0.157. Mean accuracy is 0.218±0.0110.218± 0.011. Performance therefore varies with exemplar selection, although every tested set improves over zero-shot prompting. Even the best four-shot result remains 0.557 below the P1 linear-probe score of 0.714, while the gap at the four-shot mean is 0.576. Demonstrations improve the forced-choice readout, but leave most of the gap to supervised linear decodability unresolved.