Paper deep dive
TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models
Giuliano Gorgone, Fausto Carcassi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/10/2026, 5:17:19 AM
Summary
This paper investigates how pre-trained code models (SantaCoder-1.1B and CodeLlama-7B) internally encode formal type information across Java and Python. Using layer-wise linear probing on residual streams, the authors demonstrate that these models develop linearly decodable, cross-lingual type representations that emerge even from untyped code. The study evaluates function prediction, argument type inference, and result type prediction, finding that type representations transfer robustly across languages but exhibit partial vulnerability to adversarial lexical perturbations. The findings suggest a shared cross-lingual type manifold in code models, with implications for interpretability and type-constrained generation.
Entities (8)
Relation Signals (8)
SantaCoder-1.1B → evaluatedusing → Linear Probing
confidence 95% · We evaluate two decoder-only Transformer code models: SantaCoder-1.1B... Probing Setup. For each input, we extract the post-residual activation... We exhaustively probe all layers by training a separate linear probe
CodeLlama-7B → evaluatedusing → Linear Probing
confidence 95% · We evaluate two decoder-only Transformer code models: SantaCoder-1.1B... CodeLlama-7B... Probing Setup. For each input, we extract the post-residual activation... We exhaustively probe all layers by training a separate linear probe
Linear Probing → reveals → Type Representations
confidence 92% · Using linear probes, we showed that SantaCoder and CodeLlama develop linearly decodable type representations that transfer across programming languages.
Type Representations → transferacross → Java
confidence 91% · Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other.
Type Representations → transferacross → Python
confidence 91% · Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other.
Type Representations → emergein → Residual Streams
confidence 90% · We probe the residual streams of pretrained code models for internal type representations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
Tags
Links
- Source: https://arxiv.org/abs/2607.08339v1
- Canonical: https://arxiv.org/abs/2607.08339v1
Trouble viewing inline? Open PDF directly →
Full Text
29,951 characters extracted from source content.
Expand or collapse full text
TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models Giuliano Gorgone 1[0009−0004−0827−592X] , Fausto Carcassi 1[0000−0001−6843−1737] ILLC, University of Amsterdam, Amsterdam, The Netherlands giuliano.gorgone@student.uva.nl, f.carcassi@uva.nl Abstract. State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type rep- resentations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets 1 . Keywords: Code LLMs· Linear Probing· Type Semantics· Cross-lingual Representations· Adversarial Renaming· Lexical Interference 1 Introduction State-of-the-art code models excel at next-token prediction but can still violate formal type-system constraints. For instance, in generated TypeScript, nearly 94% of compilation errors stem from type-check failures [10]. Type-constrained decoding [10] mitigates this by enforcing language-specific rules externally, but such approaches assume that models cannot internalize formal type information and require ad hoc machinery for each language. Diagnostic probing has become a standard tool for studying abstract informa- tion encoded in neural representations [1,7,4], and prior work on pretrained code models shows that they capture syntax, identifiers, and namespaces more readily than complex semantic properties [13]. At the same time, recent activation- steering results suggest that code models contain a controllable type-prediction mechanism shared across languages [9], motivating a more direct investigation of how such information is represented internally. In natural language, multilingual representations often exhibit approximately aligned manifolds across languages 1 https://github.com/anticleiades/TypeProbe arXiv:2607.08339v1 [cs.CL] 9 Jul 2026 2G. Gorgone, F. Carcassi [3,8]; here we ask whether an analogous structure emerges for type information in code models. In this work, we use layer-wise linear probes to test whether pretrained code models encode type information in their residual streams, where this information is localized across depth, and whether it transfers across Java and Python. 2 Interpreting Code Models We investigate the internal logic of code models by addressing three primary research questions. First, we examine representation: do pretrained code models develop linearly decodable representations of type-level semantics, and in which transformer layers does this information emerge? Second, we explore invariance and robustness: is this representation invariant to language syntax and robust against lexical-level adversarial perturbations? More specifically, do models pri- oritize formal type constraints over superficial heuristics when identifier names explicitly contradict their true underlying types? Finally, we assess inference: beyond type prediction, can models leverage these internal representations to resolve data flow and perform typed function application? 2.1 Experimental Setup Dataset and Design. We construct a programmatically generated dataset with three 90K-example partitions:Java(Java),pyUnt(Python without type annota- tions), andpyTag(Python with type annotations). All three partitions implement the same underlying programs, although the Java and Python versions differ in surface structure, since Java requires class and main declarations that Python omits. Program complexity ranges from simple one-line functions to more involved conditionals and loops. To prevent lexical shortcuts, identifiers and literals are independently randomized per partition, ensuring that operationally equivalent examples never share exact names or values across languages. As shown in Figure 1, each example contains a masked call site of the form b:τ b =f_⟨FIM⟩(a), where⟨FIM⟩denotes the fill-in-the-middle token. The model must recover the correct function using only type compatibility between a,b, and the candidate signatures. The fixed prefixf_restricts prediction to a choice between the two candidate functions available in scope. Candidates always feature unique input types, enabling the task to be solved. We also uniformly distribute input-output type pairs to avoid trivial shortcuts. Type Systems. We focus on operationally equivalent types across Python (T P ) and Java (T J ) by establishing a direct mapping between base types (str ↔ String ,bool ↔ boolean,int ↔ int,float ↔ float) and their respective listcontainers. Note that Python lists and Java lists are represented by different token sequences (e.g.,["a", "b"]vs.Arrays.asList("a", "b")), and even their type annotations differ (list[T] in Python vs. List<T> in Java). TypeProbe: Recovering Type Representations3 PythonJava <fim-prefix> def f_a(x): return x + 1 def f_b(xs): return len(xs) n = 30328 y = f_<fim-suffix>(n)<fim-middle> <fim-prefix> class Main static int f_a(String s) return s.length(); static int f_b(int x) return x + 1; static void main(String[] a) int n = 666; int y = f_<fim-suffix>(n); <fim-middle> Fig. 1: Example dataset instances in Python (left) and Java (right) in SantaCoder’s fill-in-the-middle (FIM) format. In both cases, the ground truth function is determined only by type compatibility at the call site:f_ain Python andf_bin Java. Adversarial Examples. To disentangle lexical cues from type-level reasoning, we construct an adversarial partition in which identifiers systematically contradict their true types. For every variablevwith ground-truth typeτ ∈T, we change its identifier to var_τ ′ , where τ ′ ∈T \τ is sampled uniformly from the set of incorrect types; similarly, function names are reassigned tof_τ ′ r , where the identifier intentionally contradicts the true return typeτ r . In the following, we refer to the original data as the standard partition. Models. We evaluate two decoder-only Transformer code models: SantaCoder- 1.1B (24 layers, 16 heads, hidden size 2048, MQA) [2] and CodeLlama-7B (32 layers, 32 heads, hidden size 4096, MHA) [12]. Probing Tasks. We define three tasks based on the masked siteb:τ b = f_⟨FIM⟩(a), whereahas typeτ a : (Task 0) Function Prediction: binary classification of the invoked candidate; (Task 1) Argument Type: 8-class prediction ofτ a ; (Task 2) Application Result: 8-class prediction of the return type of the function, which isτ b . This task allows us to probe a simple form of compositional type semantics, namely whether the hidden states encode the result of a basic yet non-trivial inferential step in typed function application 2 . Disentangling these tasks allows us to isolate failure modes, such as successful result-type prediction (Task 2) despite incorrect function selection (Task 0). 2 Here, “inferential step” refers to the process of inferring function return type based on its signature and the provided input type. We use this term solely to describe the structure recoverable by the probe, not to claim general reasoning abilities. 4G. Gorgone, F. Carcassi Table 1: Task 0: Function Prediction. Values report peak selectivityS(×100) and peak layer (L). Bold indicates values corresponding to at least 75% raw accuracy (S ≥ 25); italics indicate non-negligible signals (S ≥ 20). Eval on StandardEval on Adversarial Absolute Drop (∆) ModelTrain Java pyTag pyUnt Java pyTag pyUnt Java pyTag pyUnt CodeLlama (7B) Java 50 (16) 42 (16) 04 (16) 43 3702060501 pyTag 47 (17) 47 (17) 07 (17) 44 4003020804 pyUnt 29 (13) 29 (13) 12 (13) 23 2004060808 SantaCoder (1.1B) Java 47 (13) 38 (13) 01 (13) 37 2700101102 pyTag 44 (13) 43 (13) 02 (13) 37 25-1071803 pyUnt 16 (12) 25 (12) 04 (12) 16 280100-303 Probing Setup. For each input, we extract the post-residual activation at the FIM token position, i.e., the step at which the model predicts the masked function from the preceding context and argument. We exhaustively probe all layers by training a separate linear probe for each model, layer, task, training partition, and evaluation setting. Probes are trained with cross-entropy loss using AdamW (learning rate 10 −4 , weight decay 0.01) for 20 epochs, without feature normalization. We use 4-fold cross-validation, holding out 10% of each training fold for in-distribution validation; for each task, the best checkpoint is selected by validation accuracy and evaluated on the test fold. Batch size is 512 for CodeLlama and 256 for SantaCoder. All experiments were run on a single NVIDIA H100 GPU (80 GB) using PyTorch and TransformerLens [11]. Evaluation Metrics and Methodology. We measure and compare across models: (1) Selectivity: following [7], we define selectivity asS=acc task − acc control , whereacc control is the performance of an architecturally identical probe trained on a control task constructed per-token, i.e., by assigning random labels within the same output space as the original task. While [7] uses a per-type control, we adopt a per-token control, which we consider a reasonable approximation under our data construction, since identifiers and literals are independently randomized per example and across partitions. (2) Robustness: we quantify robustness via absolute drop in selectivity∆between the standard and the adversarial setting. Furthermore, we evaluate cross-language generalization through zero-shot transfer by training on one partition (Java,pyUnt, orpyTag) and testing on the remaining out-of-distribution partitions to assess whether type information transfers across languages. 2.2 Results and Discussion Tables 1–3 report selectivityS(×100) and peak layerLfor each model and training partition under standard and adversarial evaluation. Peak layers are selected on a held-out validation split, and adversarial scores are measured at the same layer selected in the standard setting. The final columns report the absolute selectivity drop∆from standard to adversarial. Following prior work TypeProbe: Recovering Type Representations5 Table 2: Task 1: Argument Type Prediction. Values report peak selectivity S(×100) and peak layer (L). Bold indicates values corresponding to at least 75% raw accuracy (S ≥ 63); italics indicate non-negligible signals (S ≥ 30). Eval on StandardEval on Adversarial Absolute Drop (∆) ModelTrain Java pyTag pyUnt Java pyTag pyUnt Java pyTag pyUnt CodeLlama (7B) Java 87 (16) 77 (16) 73 (16) 74 36 39124133 pyTag 61 (16) 86 (16) 82 (16) 41 64 58192124 pyUnt 47 (16) 83 (16) 84 (16) 28 48 61193523 SantaCoder (1.1B) Java 87 (13) 68 (13) 64 (13) 71 51 53161711 pyTag 48 (11) 86 (11) 73 (11) 27 70 48211725 pyUnt 73 (11) 86 (11) 85 (11) 53 77 76200909 Table 3: Task 2: Application Result Prediction. Values report peak selectivity S (×100) and peak layer (L). Formatting and thresholds follows Table 2. Eval on StandardEval on Adversarial Absolute Drop (∆) ModelTrain Java pyTag pyUnt Java pyTag pyUnt Java pyTag pyUnt CodeLlama (7B) Java 91 (14) 45 (14) 04 (14) 8025-4112008 pyTag 84 (17) 83 (17) 20 (17) 77 69-3071423 pyUnt 45 (18) 74 (18) 44 (18) 25 55 32202011 SantaCoder (1.1B) Java 83 (12) 70 (12) 15 (12) 72 5402111613 pyTag 77 (12) 79 (12) 17 (12) 53 68-2251219 pyUnt 53 (12) 54 (12) 40 (12) 27 39 32271508 [1,4], high selectivity indicates linearly decodable task information, while low selectivity suggests little structure beyond the randomized-label control [4]. Since acc control consistently approximates chance level, selectivityS ∈[−50.0,50.0] for Task 0 and S ∈ [−12.5, 87.5] for Task 1 and Task 2. Cross-Lingual Type Representation. Results in Tables 2 and 3 show that both SantaCoder-1.1B and CodeLlama-7B 3 develop linearly decodable type rep- resentations that transfer across languages. SantaCoder exhibits a more uniform cross-lingual transfer profile than CodeLlama. In Task 1,Java-trained probes transfer well topyTag(68) and pyUnt(64), whilepyUnt-trained probes generalize back toJava(73). In Task 2, transfer remains robust betweenJavaandpyTag(70–77) and remains meaningful forpyUnt → Java(53). The confusion matrices in Fig. 4 further support this interpretation: under cross-lingual transfer, SantaCoder preserves separability for most base and list types, with only modest increases in confusion among numeric types and among list types. 3 We thank the reviewers for highlighting tokenization as a potential confounding factor. We then identified a preprocessing issue in our initial CodeLlama setup: the FIM marker was being split into four separate tokens rather than processed as a single special token. Correcting this improved CodeLlama’s probing accuracy, and all reported results now reflect this corrected setup. 6G. Gorgone, F. Carcassi Regarding CodeLlama, in Task 1, transfer betweenJavaandpyUntremains non-trivial in both directions, withJava → pyUntreaching 73 andpyUnt → Java reaching 47. In Task 2, by contrast, transfer is much less uniform:pyTag → Javareaches 84,pyUnt → Javareaches 45, butJava → pyUntfalls to 04. Unlike SantaCoder, CodeLlama also tends to over-predictintunder cross-lingual transfer (Fig. 4). Crucially, transfer frompyUnttoJavais particularly informative for both models. BecausepyUntlacks explicit annotations, successful transfer suggests that the probes recover type-relevant directions from untyped code alone. When explicit type information is present in the target activations (JavaandpyTag), it appears to align with the same type-relevant signal already recoverable from untyped code. Task 0 provides additional support for this interpretation, showing non-trivial cross-lingual transfer in function prediction even when the probe is trained on examples without explicit type annotations. In both models, the confusion matrices forpyUntsuggest that the probes often recover container structure more reliably than the element type (Figs. 2 to 4 and 10). Under the standard setting, this pattern appears less pronounced in in-domainpyTagandJavaevaluation, where the confusion matrices are nearly diagonal (Figs. 6 to 9). int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 2: Row-normalized confusion matrices for SantaCoder (1.1B) at layer 11 on Task 1. Probes are trained and evaluated onpyUntunder standard (left) and adversarial (right) settings. Per-layer Probing Performance. Selectivity peaks in mid-to-late layers (Santa- Coder: L11–L16, CodeLlama: L14–L18; Fig. 5). Notably,pyUnt-trained probes exhibit a late-layer decodability drop that amplifies under adversarial evaluation. A similar pattern arises for explicitly typed partitions (JavaandpyTag): while TypeProbe: Recovering Type Representations7 int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 3: Row-normalized confusion matrices for CodeLlama (7B) at layer 16 on Task 1. Probes are trained and evaluated onpyUntunder standard (left) and adversarial (right) settings. late-layer selectivity is relatively stable in standard settings, it consistently drops across all models when facing adversarial renaming (Figs. 11 and 12). Disentangling Type-Level Semantics from Lexical-Level Semantics. Figures 2–3 quantify the effect of adversarial lexical interference on type information decod- ability. Adversarial renaming shifts SantaCoder towardint → floatconfusion, while CodeLlama shows the reverse pattern, with strongerfloat → interrors. In both models,strremains comparatively stable, whereas the clustering among list types becomes more pronounced than in the baseline. Qualitatively similar patterns of confusion also appear inpyTagandJavaunder adversarial evaluation, indicating that misleading identifier names can interfere with the decodability of type information even when explicit type information is available (Figs. 6 to 9). 3 Conclusions Using linear probes [4], we showed that SantaCoder and CodeLlama develop linearly decodable type representations that transfer across programming lan- guages. These representations emerge even when probes are trained on untyped Python, suggesting that code models encode non-trivial type-relevant structure beyond explicit annotations. Across tasks, adversarial renaming reveals only partial robustness to misleading lexical cues. Overall, our findings point to the emergence of a cross-lingual type manifold in the setting we study. 4 Limitations and Future Work Our experimental setup focuses on base types and lists across two languages and models. Future work should explore richer relations (e.g., subtyping, covariance 8G. Gorgone, F. Carcassi [5]) and lower-level type systems (e.g., Rust’s ownership, C++’s templates) across varying model scales. Methodologically, while linear probes confirm decodability, they cannot recover features in superposition [6,4], nor prove any causal role of the extracted representations in code generation. Future work should therefore combine our setup with more expressive tools, such as Sparse Autoencoders (SAEs), Dictionary Learning, and representation-similarity methods like SVCCA [8], as well as causal analyses via activation patching [14]. Ultimately, our findings suggest that learned type representations may complement type-constrained decoding approaches [10]. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True SantaCoder (1.1B) int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True CodeLlama (7B) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True SantaCoder (1.1B) int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True CodeLlama (7B) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 4: Row-normalized confusion matrices for Task 2 (Application Result) with probes trained onpyUnt. Top: Baseline (evaluated onpyUnt). Bottom: Zero- shot transfer (evaluated onJava). In both panels, SantaCoder (layer 12) is on the left and CodeLlama (layer 18) is on the right. TypeProbe: Recovering Type Representations9 Acknowledgments. We acknowledge SURF for computing access on Snellius and the ILLC (University of Amsterdam) for travel support. We thank Michael Hanna, Francesca Lucchetti and the anonymous reviewers for their insightful feedback, Adhvaith Koduru and Songyun Zou for their contributions to the preliminary experiments, and ESSLLI for a registration fee waiver. Disclosure of Interests. No competing interests to declare that are relevant to the content of this article. 10G. Gorgone, F. Carcassi 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 05101520 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 05101520 Layer Evaluated on pyTag (Adversarial) 05101520 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 051015202530 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 051015202530 Layer Evaluated on pyTag (Adversarial) 051015202530 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 Fig. 5: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained onpyUntand evaluated across all standard and adversarial partitions. Top: SantaCoder (1.1B). Bottom: CodeLlama (7B). TypeProbe: Recovering Type Representations11 References 1.Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net (2017), https://openreview.net/forum?id=HJ4-rAVtl 2.Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., et al.: Santacoder: don’t reach for the stars! (2023),https://arxiv.org/abs/2301.03988 3.Ammar, W., Mulcaire, G., Tsvetkov, Y., Lample, G., Dyer, C., Smith, N.A.: Massively multilingual word embeddings (2016),https://arxiv.org/abs/1602. 01925 4. Belinkov, Y.: Probing classifiers: Promises, shortcomings, and advances. Computa- tional Linguistics 48(1), 207–219 (Mar 2022).https://doi.org/10.1162/coli_a_ 00422, https://aclanthology.org/2022.cl-1.7/ 5.Cardelli, L., Wegner, P.: On understanding types, data abstraction, and polymor- phism. ACM Comput. Surv. 17(4), 471–523 (Dec 1985).https://doi.org/10. 1145/6041.6042, https://doi.org/10.1145/6041.6042 6.Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, et al.: Toy models of superposition. Transformer Circuits Thread (2022),https:// transformer-circuits.pub/2022/toy_model/index.html 7. Hewitt, J., Liang, P.: Designing and interpreting probes with control tasks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). p. 2733– 2743. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1275 8.Kudugunta, S.R., Bapna, A., Caswell, I., Arivazhagan, N., Firat, O.: Investigating multilingual nmt representations at scale (2019),https://arxiv.org/abs/1909. 02197 9. Lucchetti, F., Guha, A.: Understanding how CodeLLMs (mis)predict types with activation steering. In: Belinkov, Y., Mueller, A., Kim, N., Mohebbi, H., Chen, H., Arad, D., Sarti, G. (eds.) Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. p. 358–397. Association for Computational Linguistics, Suzhou, China (Nov 2025).https://doi.org/10. 18653/v1/2025.blackboxnlp-1.22 10.Mündler, N., He, J., Wang, H., Sen, K., Song, D., Vechev, M.: Type-constrained code generation with language models. Proc. ACM Program. Lang. 9(PLDI) (Jun 2025). https://doi.org/10.1145/3729274 11. Nanda, N., Bloom, J.: Transformerlens.https://github.com/ TransformerLensOrg/TransformerLens (2022) 12.Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., et al.: Code llama: Open foundation models for code (2024), https://arxiv.org/abs/2308.12950 13.Troshin, S., Chirkova, N.: Probing pretrained models of source codes. In: Bast- ings, J., Belinkov, Y., Elazar, Y., Hupkes, D., Saphra, N., Wiegreffe, S. (eds.) Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. p. 371–383 (2022).https://doi.org/10.18653/v1/ 2022.blackboxnlp-1.31 14.Zhang, Z., Nanda, N.: Towards best practices of activation patching in language models: Metrics and methods. In: International Conference on Learning Representa- tions (ICLR) (2024),https://proceedings.iclr.c/paper_files/paper/2024/ file/06a52a54c8e03cd86771136bc91eb1f-Paper-Conference.pdf 12G. Gorgone, F. Carcassi A Confusion Matrices: Task1 (Java, pyTag) int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) SantaCoder (1.1B) at layer 11. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) CodeLlama (7B) at layer 16. Fig. 6: Row-normalized confusion matrices on Task 1. Probes are trained and evaluated on pyTag under standard (left) and adversarial (right) settings. TypeProbe: Recovering Type Representations13 int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) SantaCoder (1.1B) at layer 13. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) CodeLlama (7B) at layer 16. Fig. 7: Row-normalized confusion matrices on Task 1. Probes are trained and evaluated on Java under standard (left) and adversarial (right) settings. 14G. Gorgone, F. Carcassi B Confusion Matrices: Task2 (Java, pyTag, pyUnt) int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) SantaCoder (1.1B) at layer 12. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) CodeLlama (7B) at layer 14. Fig. 8: Row-normalized confusion matrices for Task 2 (Application Result). Probes are trained and evaluated onJavaunder standard (left) and adversarial (right) settings. TypeProbe: Recovering Type Representations15 int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) SantaCoder (1.1B) at layer 12. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) CodeLlama (7B) at layer 17. Fig. 9: Row-normalized confusion matrices for Task 2 (Application Result). Probes are trained and evaluated onpyTagunder standard (left) and adversarial (right) settings. 16G. Gorgone, F. Carcassi int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (a) SantaCoder (1.1B) at layer 12. int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Standard int str bool float l[int] l[str] l[bool] l[float] Predicted int str bool float l[int] l[str] l[bool] l[float] True Adversarial 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (b) CodeLlama (7B) at layer 18. Fig. 10: Row-normalized confusion matrices for Task 2 (Application Result). Probes are trained and evaluated onpyUntunder standard (left) and adversarial (right) settings. TypeProbe: Recovering Type Representations17 C Layer-wise Selectivity (Java, pyTag) 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 05101520 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 05101520 Layer Evaluated on pyTag (Adversarial) 05101520 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 051015202530 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 051015202530 Layer Evaluated on pyTag (Adversarial) 051015202530 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 Fig. 11: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained onJavaand evaluated across all standard and adversarial partitions. Top: SantaCoder (1.1B). Bottom: CodeLlama (7B). 18G. Gorgone, F. Carcassi 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 05101520 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 05101520 Layer Evaluated on pyTag (Adversarial) 05101520 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on JavaEvaluated on pyTagEvaluated on pyUnt 051015202530 Layer 12.5 0.0 25.0 50.0 75.0 Selectivity ( ×100 ) Evaluated on Java (Adversarial) 051015202530 Layer Evaluated on pyTag (Adversarial) 051015202530 Layer Evaluated on pyUnt (Adversarial) Task 1Task 2 Fig. 12: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained onpyTagand evaluated across all standard and adversarial partitions. Top: SantaCoder (1.1B). Bottom: CodeLlama (7B).