Paper deep dive
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, Hao Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/22/2026, 2:13:07 AM
Summary
The paper introduces Invariant Pretraining (InvPT), a code-only continued pretraining method designed to improve the robustness of encoder-based code representation models against semantically equivalent but syntactically different code variations. InvPT combines masked language modeling with multi-positive supervised contrastive learning, utilizing semantics-preserving transformations (such as variable renaming and loop conversion) and self-contrast pairs. Evaluations on clone detection and code classification tasks show significant improvements in robustness over baseline models like CodeBERT and GraphCodeBERT without requiring paired natural language data.
Entities (15)
Relation Signals (14)
InvPT → uses → Masked Language Modeling
confidence 95% · combines masked language modeling with multi-positive supervised contrastive learning
InvPT → uses → Supervised Contrastive Learning
confidence 95% · InvPT applies semantic-preserving transformations to the corpus and combines masked language modeling with multi-positive supervised contrastive learning
InvPT → evaluatedon → POJ104
confidence 92% · We evaluate on POJ104 (36) ... and three CodeNet subsets
InvPT → evaluatedon → CodeNet
confidence 92% · We evaluate on POJ104 (36) ... and three CodeNet subsets
GraphCodeBERT → baselinefor → InvPT
confidence 90% · We present an empirical study of this robustness gap across four encoder baselines... GraphCodeBERT (15)
CodeBERT → baselinefor → InvPT
confidence 90% · We present an empirical study of this robustness gap across four encoder baselines... Models such as CodeBERT (11)
ContraBERT → baselinefor → InvPT
confidence 90% · From CodeBERT (11) to ContraBERT (32), these methods train on function–docstring pairs
InvPT → improves → Clone Detection
confidence 90% · InvPT improves robustness on transformed test sets by up to 11 percentage points on clone detection
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive. Their robustness, however, is fragile: under invariant programs, semantically equivalent code written in different syntactic forms, learned representations degrade substantially even though program behavior is unchanged. We present an empirical study of this robustness gap across four encoder baselines, two downstream tasks, and four datasets, together with a minimal code-only continued pretraining recipe that closes much of it. Our method, invariant pretraining (InvPT), applies semantic-preserving transformations to the corpus and combines masked language modeling with multi-positive supervised contrastive learning that treats all augmentations of the same source function as positives, mixing self-contrast pairs (same code, different masks) with invariant-contrast pairs (transformed code) for positives of varying difficulty. Unlike prior contrastive code encoders, InvPT does not require paired natural-language data. Across our evaluation, InvPT improves robustness on transformed test sets by up to 11 percentage points on clone detection and 19 on code classification while matching or improving standard accuracy, and our ablations isolate multi-positive invariant contrast as the main source of the gains. Our aim is not a new objective but a careful measurement of where encoder robustness breaks and how far a simple, code-only recipe can recover it.
Tags
Links
- Source: https://arxiv.org/abs/2608.15412v1
- Canonical: https://arxiv.org/abs/2608.15412v1
Trouble viewing inline? Open PDF directly →
Full Text
63,489 characters extracted from source content.
Expand or collapse full text
Invariant Pretraining for Robust Code RepresentationsDOI: X.XXXXXXXConference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NYISBN: 978-1-4503-X-X/18/06CCS: Security and privacy Software security engineeringCCS: Computing methodologies Neural networksCCS: Computing methodologies Unsupervised learning Yifeng He OrcID: 0000-0002-5389-7128 Affiliation: University of California at Davis , Davis , USA email: yfhe@ucdavis.edu , Yundi Xu Affiliation: University of California at Davis , Davis , USA , Christopher Castro Gaw Gonzalo Affiliation: University of California at Davis , Davis , USA , Zili Wang Affiliation: University of California at Davis , Davis , USA and Hao Chen OrcID: 0000-0002-4072-0710 Affiliation: The University of Hong Kong , China email: chenho@hku.hk 2018© , 2018; Abstract. Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive. Their robustness, however, is fragile. On invariant programs—semantically equivalent code written in different syntactic forms—their representations degrade substantially even though program behavior is unchanged. We present an empirical study of this robustness gap across four encoder baselines, two downstream tasks, and four datasets, together with a minimal code-only continued pretraining recipe that recovers a consistent part of it. Our method, invariant pretraining (InvPT), applies semantics-preserving transformations to the corpus and combines masked language modeling with multi-positive supervised contrastive learning that treats all augmentations of the same source function as positives, mixing self-contrast pairs (same code, different masks) with invariant-contrast pairs (transformed code) for positives of varying difficulty. Unlike prior contrastive code encoders, InvPT does not require paired natural-language data. Across our evaluation, InvPT improves robustness on transformed test sets in every model–dataset comparison, by a median of 8.18.1 percentage points on clone detection (up to 11.011.0) and 3.63.6 on code classification (up to 19.219.2), while matching or improving standard accuracy; ablations isolate multi-positive invariant contrast as the main source of the gains. Because the transformed test sets compose the same operator family used in pretraining, we claim invariance to that family rather than robustness in general. Our aim is not a new objective but a careful measurement of where encoder robustness breaks and how far a simple, code-only recipe can recover it. Keywords: Code Representation Learning, Model Robustness 1. Introduction Machine learning for code is now dominated by large generative models (6; 43; 17; 35). Encoder-based code representation models nonetheless remain the practical choice for discriminative tasks such as clone detection and code classification, which still sit inside modern pipelines: deduplicating training corpora, retrieving and ranking code, flagging near-duplicates in agent workflows. Models such as CodeBERT (11) and GraphCodeBERT (15) are far smaller than generation models while performing well on these tasks. With roughly 125M parameters, they can match or outperform generation models in the 7B–15B range while requiring less storage and lower inference cost (24; 30). Their low fine-tuning cost keeps them attractive wherever robust code representations are needed at scale, so their failure modes still matter. That robustness is not assured. Modern programming languages are syntactically flexible: developers routinely express the same computation in different ways because of coding style, conventions, or language idioms. We refer to such semantically equivalent but syntactically varied programs as invariant programs (or simply invariants), because their observable behavior is unchanged under these rewritings. For example, a counting for loop can often be rewritten as an equivalent while loop without changing program behavior. These variations are common in practice, yet code representation models often mishandle them because pretraining encourages sensitivity to surface tokens (50). 1 showed that perturbing natural-language elements in code can substantially degrade model performance. How large this degradation is across current encoders, and how much of it a simple training change can remove, has not been measured systematically. Prior work addresses parts of the problem but does not settle it. 52 introduced SPAT, a semantics-preserving transformation-based data-augmentation method that improved downstream performance without explicitly targeting robustness. 47 combined SPAT with curriculum learning, but focused on fine-tuning rather than pretraining. 32 proposed contrastive pretraining, but relied on transformations that can alter semantics or introduce syntax errors, and on bi-modal corpora with paired natural language and code (NL-PL pairs). From CodeBERT (11) to ContraBERT (32), these methods train on function–docstring pairs, and ContraBERT further designs natural-language transformations for its contrastive objective. For robustness against invariant programs, however, paired NL-PL data is not essential: the supervisory signal already lives in code through semantics-preserving transformations. Figure 1. Overview of invariant pretraining (InvPT). We contribute an empirical study, not a new objective. We quantify the robustness gap of encoder code models under invariant programs and ask how far a deliberately minimal, code-only recipe can close it. The recipe, invariant pretraining (InvPT), is continued pretraining that uses only programming-language data. InvPT applies semantics-preserving transformations to the pretraining corpus (Section 2.1), optimizes encoders with supervised contrastive learning (Section 2.2), and mixes self-contrast with invariant-contrast pairs to provide positives of varying difficulty (Section 2.2.3). Its only non-obvious design choice is a multi-positive contrastive mask that treats all augmentations of the same source function as positives, avoiding the false-negative problem of standard InfoNCE, which would push semantically equivalent variants apart. This code-only design removes the need for paired NL-PL corpora and enables continued pretraining on large PL-only code collections. The resulting models gain robustness on code-to-code tasks while preserving or improving standard downstream performance. In summary, our contributions are: (i) We measure the robustness of four encoder code models to semantics-preserving transformations across clone detection and code classification on POJ104 and three CodeNet subsets (Java250, Python800, C++1400), quantifying a consistent degradation under semantically equivalent rewrites. (i) We instantiate InvPT, a code-only continued pretraining recipe that combines invariant transformations with a multi-positive supervised contrastive objective. It gains up to 6.576.57 points of standard accuracy on the original clone-detection benchmark and improves robustness on every transformed model–dataset comparison, by a median of 8.078.07 points on clone detection and 3.563.56 on classification, recovering part but not most of the degradation. (i) We run ablations isolating the source of improvement, showing that multi-positive invariant contrastive learning is the dominant factor, that self-contrast adds consistent gains, and that paired natural-language descriptions are unnecessary for invariant pretraining. 2. Invariant Pretraining Figure 1overviews InvPT’s three components: invariant code transformations, code-only contrastive learning, and mixed-difficulty contrastive pairs. For each source snippet, we generate semantically equivalent but syntactically varied variants, and continue pretraining on both the original and its variants with a masked language modeling objective combined with an invariant contrastive objective. Self-contrast and invariant-contrast examples are mixed so the model sees positives of varying difficulty during training. 2.1. Invariant Code Transformations Invariant programs—semantically equivalent code that differs syntactically—arise naturally from coding styles, conventions, and language idioms (39), yet models often mishandle them because pretraining memorizes surface form (47; 32; 4). We address this by pretraining with invariant code transformations that produce semantically equivalent but syntactically diverse variants. Following previous work on naturally occurring invariants (3; 52), we design six transformation operators in two categories: syntax operators that modify surface-level tokens and a branching operator that alters control flow while preserving runtime behavior. Table 1 lists each operator with its transformation rule and supported languages, where p denotes a predicate (condition) and B a code block. We describe each operator below with its preconditions and a correctness argument. Table 1. Invariant transformation operators, transformation rules, and supported languages. Operator Category Transformation rule Lang. VarRe Syntax Replace all variable names with random valid identifiers C/C ++ W2F Syntax while(p)B⇒for(;p;)B while\ (p)\ \B\\; \; for\ (;\,p;\,)\ \B\ C/C ++ F2W Syntax for(,p,)B⇒;while(p)B; for\ (init;\,p;\,inc)\ \B\\; \;init;\ while\ (p)\ \B;\,inc\ C/C ++ P2A Syntax x++ ⇒ x += 1 C/C ++ A2EA Syntax x += 1 ⇒ x = x + 1 C/C ++ RevIf Branching if(p)B1elseB2⇒if(¬p)B2elseB1 if\ (p)\ \B_1\\ else\ \B_2\\; \; if\ ( p)\ \B_2\\ else\ \B_1\ C/C ++ We implement all operators at the AST level to ensure syntactic correctness: we reuse the Java operators of 52, use clang for C/C++, and the Python 3 ast module for Python (with 2to3 as a fallback for Python 2 code (55)). We discard code snippets with syntax errors that prevent AST parsing. Not all operators apply to every language (e.g., Python’s iterator-based for-loops preclude W2F and F2W), as shown in Table 1. Our operators target the language constructs that appear in our benchmarks (POJ104 and CodeNet); for Python we therefore omit the loop and increment operators. Variable Rename (VarRe) Programmers choose variable names freely without affecting runtime behavior; a name may be a whole word, an abbreviation, a single character, a term in another language, or a random string. VarRe replaces all variable names with randomly generated, non-repeating strings that comply with the naming conventions of the target language, keeping the transformed code syntactically valid. Variable names are purely syntactic identifiers with no effect on runtime behavior in any supported language (Java, C/C++, Python), and because VarRe performs a consistent, injective renaming of all occurrences within each scope, data and control flow are unchanged. Naming variation is pervasive in real code: style guides differ, internationalization yields non-English identifiers, and abbreviation conventions vary by domain (39), so VarRe models a naturally occurring transformation rather than an artificial adversarial perturbation. While-to-For (W2F) While- and for-loops are both common imperative constructs that programmers use interchangeably. Converting a while-loop to a for-loop reuses the condition while leaving the initialization and increment slots empty: while(p)B⇒for(;p;)B. while\ (p)\ \B\\; \; for\ (;\,p;\,)\ \B\. The for-loop for(; p;)B evaluates p before each iteration and executes B if true, identical to while(p)B; the empty slots add no computation, so semantics are preserved exactly. For-to-While (F2W) The reverse converts a for-loop into a while-loop, which is more involved because while-loops lack a built-in counter. We hoist the counter initialization before the loop and append the increment at the end of the body: for(,p,)B⇒;while(p)B;. for\ (init;\,p;\,inc)\ \B\\; \;init;\ while\ (p)\ \B;\,inc\. F2W applies only to for-loops whose body has no continue statement: with continue, the appended inc would be skipped, whereas the original for-loop runs inc before the next condition check regardless. Our clang-based implementation for C/C++ and the Java implementation of 52 both check this precondition. When init introduces a declaration, we wrap the rewrite in a block (i.e., ;while(p)B;\\,init;\ while\ (p)\ \B;\,inc\\,\) so loop-local declarations do not leak into the enclosing scope. Under this precondition, init runs once, p is checked before each iteration, B is the body, and inc runs after each complete iteration, matching the for-loop semantics exactly. PlusPlus-to-AddAssign (P2A) Self-increment operators are common and often interchangeable. P2A converts an increment x++ or ++x into x += 1 (in this direction only). For post-increment we check on the parent AST node that the result value is not consumed, since x++ evaluates to the old value whereas x += 1 evaluates to the new one; pre-increment needs no such precondition. We further restrict the operator to variables of primitive numeric type, so overloaded operator++ in C++ is never rewritten. AddAssign-to-EqualAdd (A2EA) A2EA converts “add-assign” into “equal-add”: x += 1 ⇒ x = x + 1. For primitive numeric types, which dominate our benchmarks (POJ104, CodeNet), x += expr and x = x + expr are equivalent in all supported languages. For C/C++ the clang-based implementation enforces this precondition directly, rewriting only variables whose declared type is primitive numeric. Python admits no such static check: the declared type is unavailable at the AST level, so our implementation rewrites every augmented assignment, and for objects that define __iadd__, such as lists and numpy arrays, x += e mutates in place while x = x + e rebinds a fresh object, which is observable under aliasing. The rewrite is sound for the primitive-scalar arithmetic that dominates our competitive-programming corpora, but it is a heuristic rather than a guaranteed invariant on arbitrary Python (Section 5). Reverse If (RevIf) RevIf reverses the condition of an if statement. For a single-branch if (no else), we negate the condition, leave an empty branch, and move the original body to a new else: if(p)B⇒if(¬p)elseB. if\ (p)\ \B\\; \; if\ ( p)\ \\\ else\ \B\. For a two-branch if, we negate the condition and swap the branches: if(p)B1elseB2⇒if(¬p)B2elseB1. if\ (p)\ \B_1\\ else\ \B_2\\; \; if\ ( p)\ \B_2\\ else\ \B_1\. For else if chains (syntactic sugar for nested ifs), we apply these rules recursively to each nested if node in the AST. Negating p and swapping branches preserves the mapping from condition values to blocks; for single-branch ifs, the empty block has no effect; and recursive application is correct because each nested if is reversed independently. 2.2. Invariant Contrastive Learning Because our semantics-preserving transformations preserve semantics under the preconditions of Section 2.1, the positive-pair gradient signal is unambiguous, and we use a single shared encoder rather than the momentum-decoupled encoder adopted by ContraBERT (32; 16). We train with the average MLM loss on the original code and its invariant, combined with an invariant contrastive loss. Since the transformations can produce multiple positives per anchor, we use supervised contrastive learning, which accommodates all positives without pushing equivalent augmentations apart as InfoNCE does. 2.2.1. Masked Language Modeling Masked language modeling (MLM), a common pretraining objective for language models, trains the model to predict masked tokens in the input sequence (9). Given an input sequence X=[CLS],C,[EOS]X=\[CLS],C,[EOS]\, where C is the tokenized code snippet, we randomly mask 15% of the tokens in X to obtain a masked sequence X′X , following previous work (15; 32; 9; 20). Let MX⊂XM_X⊂ X denote the masked tokens in X. The loss is the negative log-likelihood of the masked tokens given X′X : ℒMLM(X)=−1|MX|∑x∈MXlogP(x|X′).L_MLM(X)=- 1|M_X| _x∈ M_X P(x|X ). 2.2.2. Supervised Contrastive Mask Because our dataset applies multiple transformation operators to the same source function (e.g., both VarRe and RevIf), different augmentations of the same function can co-occur in a mini-batch. Under InfoNCE (45), these semantically equivalent augmentations would be pushed apart as negatives, contradicting the invariance objective. We therefore adopt supervised contrastive learning (SupCon) (26) with a multi-positive mask: each snippet receives a deterministic identifier computed from its original source code (a truncated SHA-256 hash), so all augmentations of the same function share an identifier and are recognized as positives. For a mini-batch of B code–invariant pairs, we concatenate the ℓ2 _2-normalized CLS embeddings into a pool of N=2BN=2B representations z with duplicated identifiers idsids, and form a multi-positive mask Mij=[idsi=idsj∧i≠j]M_ij=1[ids_i=ids_j i≠ j]. Let (i)=j:Mij=1P(i)=\j:M_ij=1\; every anchor has at least its paired augmentation as a positive, and when the same source function appears k times each anchor has up to 2k−12k-1 positives. We compute the identifier from the canonical, non-augmented source c as id(c)=int64(SHA-256(c)[:8])& 0,id(c)=int64 (SHA -256(c)[:8] ) \&\; 0x7F, the first 8 bytes of the digest read as a big-endian 64-bit integer with the sign bit masked. Because we hash the canonical source, all augmentations of a function, including self-contrast copies (Section 2.2.3), share an identifier. The assignment is deterministic and needs no coordination across training processes or data shards. Table 2 illustrates the resulting mask for B=4B=4 pairs drawn from three source functions A, B, C, where A appears twice through different operators (indices 00–33 are original codes, 44–77 their augmentations): anchors for A (indices 0,3,4,70,3,4,7) each have 33 positives, whereas anchors for B and C each have exactly 11. Table 2. Example positive mask M for B=4B=4 with source functions [A,B,C,A][A,B,C,A]. Rows/columns 0–3 are original codes; 4–7 are their augmentations. “∙ ”: the positives (Mij=1M_ij=1); “-”: the excluded diagonal. 0 1 2 3 4 5 6 7 0 (A) - ∙ ∙ ∙ 1 (B) - ∙ 2 (C) - ∙ 3 (A) ∙ - ∙ ∙ 4 (A) ∙ ∙ - ∙ 5 (B) ∙ - 6 (C) ∙ - 7 (A) ∙ ∙ ∙ - Let τ denote the temperature, and let sim(,)=/τsim(u,v)=u Tv/τ be the scaled cosine similarity of ℓ2 _2-normalized vectors. The supervised contrastive loss is ℒSupCon=−1|| _SupCon=- 1|A| ∑i∈1|(i)| _i 1|P(i)| ∑p∈(i)logexp(sim(i,p))∑j≠iexp(sim(i,j)), _p (i) (sim(z_i,z_p) ) _j≠ i (sim(z_i,z_j) ), where =i:|(i)|>0A=\i:|P(i)|>0\ is the set of anchors with at least one positive. When no same-function collisions occur, the mask reduces to the diagonal pairing of InfoNCE. For a batch of code X and invariants XinvX^inv, the overall objective is ℒ(X,Xinv)=ℒMLM(X)+ℒMLM(Xinv)2+αℒSupCon,L(X,X^inv)= L_MLM(X)+L_MLM(X^inv)2+ _SupCon, where α weights the contrastive loss. 2.2.3. Mixed-Difficulty Contrastive Pairs Self-contrast SimCSE (12) showed that passing the same input through an encoder with different dropout masks produces effective contrastive pairs for sentence representation learning. We introduce self-contrast to code representation learning, a signal that prior work (52; 47; 32) has overlooked, perhaps because small code edits can change semantics. In InvPT, we include identical code snippets as contrastive pairs during pretraining, relying on the 15%15\% MLM masking probability to produce varied positive pairs from the same code snippet. Self-contrast provides the easier signal in our mixed-difficulty strategy: the model learns to produce similar embeddings for the same code with different masked tokens. Table 3. Correspondence between baseline models and our InvPT models. Baseline CodeBERT GraphCodeBERT ContraBERT_C ContraBERT_G + InvPT InvCodeBERT InvGraphCodeBERT InvContraBERT_C InvContraBERT_G Invariant-contrast The single-operator transformations of Section 2.1 supply the hard end of the range. Whereas a self-contrast pair differs only in masked positions, an invariant pair can differ substantially in surface form while preserving semantics. For example, F2W rewrites a for-loop into a while-loop by hoisting the initialization before the loop and appending the increment to the body, and VarRe replaces every identifier with a fresh random name, so an anchor and its positive may share almost no tokens. Aligning such a pair forces the model past surface cues toward the underlying computation, a strictly harder objective than tolerating masked tokens on an otherwise identical sequence. Mixing difficulties We expose the model to both pair types simultaneously rather than staging them, presenting each anchor with its easy (self-contrast) and hard (invariant) positives in the same batch. A staged schedule that withheld the harder pairs early and introduced them later would shift the positive-pair distribution mid-training, risking instability and catastrophic forgetting (27) during continued pretraining of a large language model. We further start training with a low learning rate of 2×10−52×10^-5 so the model adapts gradually to the contrastive objective, then control the rate with standard warm-up and scheduling (25). The ablation in Section 3.4 supports this mix: keeping only the hard invariant pairs and removing self-contrast lowers performance on every original benchmark, so the easy signal complements the hard one rather than duplicating it. 3. Experiments 3.1. Tasks, Models, and Setup We evaluate InvPT on two code-to-code downstream tasks, where robustness to invariant transformations is particularly important. Clone detection retrieves semantically equivalent programs from a candidate pool using cosine similarity over encoder embeddings. We evaluate on POJ104 (36) (104 problems, 500 C/C++ solutions each) and three CodeNet (40) subsets (Java250, Python800, and C++1400) following 55. Code classification predicts the functional category (problem number) of a program; we evaluate on the same POJ104 and CodeNet benchmarks as clone detection. 11 1 We do not use BigCloneBench (44), due to label quality concerns (28; 29) and heavy data contamination. We also do not use Devign (56) defect detection due to serious data quality and label accuracy concerns (10). We compare CodeBERT (11), GraphCodeBERT (15), ContraBERT_C, and ContraBERT_G (32), together with their InvPT counterparts.22 2 We additionally include CodeSage (53) and ModernBERT (49) as reference points in the clone detection evaluation (Table 4). Since these models use different architectures, we do not apply InvPT continued pretraining to them. We follow the standard data splits and fine-tuning settings used in prior work (34; 40; 55); Table 3 lists the correspondence between each baseline and its InvPT variant. For clone detection, we report Mean Average Precision at R (MAP@R) (37), and for code classification we report accuracy. Relative improvements (%) over the corresponding baseline appear in gray. We pretrain on the Java and Python subset of CodeSearchNet (21): it provides high-coverage corpora for both languages, and all six transformation operators apply to them. We hold C/C++ out of pretraining on purpose. The cross-language evaluation in Section 3.2 and Section 3.3 then tests whether structural invariance learned on Java and Python transfers to a disjoint target language, rather than measuring in-distribution performance. We pretrain with maximum sequence length 512, batch size 256, temperature τ=0.1τ=0.1, and contrastive weight α=1.0α=1.0, using AdamW (33) with learning rate 2×10−52×10^-5, weight decay 0.010.01, and 10%10\% warmup, for 3 epochs with checkpoints selected by validation loss. For downstream fine-tuning, we follow CodeXGLUE (34) for the POJ104 split and 40 for the CodeNet splits, using learning rate 2×10−52×10^-5, batch size 8, and maximum sequence length 400. All pretraining runs use four NVIDIA H100 GPUs with 80 GB memory each, and all downstream runs use a single NVIDIA H100 GPU. Continued pretraining is inexpensive relative to pretraining from scratch: each InvPT model takes about 20 hours on the four H100s, or roughly 80 GPU-hours, for the three epochs over the augmented corpus. All reported numbers come from a single pretraining and fine-tuning run per configuration, with no repeats across random seeds (Section 5). We organize the evaluation around the claims made in the introduction. Throughout, we use standard accuracy to refer to accuracy on the original (untransformed) test set, in contrast to robustness, which we measure on the transformed test sets. We first test whether InvPT preserves standard downstream performance. We then evaluate robustness on transformed test sets to measure invariance under unseen compositions of transformations. Next, we use ablations to identify which design components drive the gains. Finally, we present representation visualizations as qualitative evidence that is consistent with the quantitative results. 3.2. Standard Downstream Performance Table 4. Clone detection results. “orig”: original test set; “+T”: transformed test set for robustness evaluation. Java250 Python800 C++1400 POJ104 Model orig + T orig + T orig + T orig + T CodeBERT 77.2877.28 51.1651.16 83.7183.71 62.1762.17 56.2156.21 25.5925.59 85.4785.47 55.9655.96 GraphCodeBERT 82.1582.15 53.5253.52 86.0686.06 65.8965.89 58.7658.76 30.2630.26 84.0984.09 58.6358.63 ContraBERT_C 82.1982.19 49.6449.64 85.6085.60 63.4163.41 57.6557.65 27.0927.09 89.5889.58 62.7662.76 ContraBERT_G 83.4783.47 53.7953.79 85.9785.97 65.9865.98 59.1559.15 29.1129.11 90.8090.80 64.2064.20 CodeSage 72.4372.43 25.9225.92 61.1161.11 25.8925.89 51.5351.53 21.3321.33 78.2078.20 52.8752.87 ModernBERT 80.5180.51 41.0641.06 83.1683.16 52.8552.85 59.0559.05 24.6624.66 88.2288.22 52.8652.86 InvCodeBERT 82.4482.44 59.2759.27 (+15.9%) 85.6385.63 70.5570.55 (+13.5%) 57.8057.80 31.4331.43 (+22.8%) 88.8188.81 62.0362.03 (+10.8%) InvGraphCodeBERT 84.4184.41 62.30 (+16.4%) 86.73 73.9173.91 (+12.2%) 59.23 33.68 (+11.3%) 90.6690.66 67.3267.32 (+14.8%) InvContraBERT_C 82.6382.63 60.6860.68 (+22.2%) 86.1686.16 74.3474.34 (+17.2%) 58.0958.09 32.0732.07 (+18.4%) 89.2889.28 63.5363.53 (+1.2%) InvContraBERT_G 84.53 62.2462.24 (+15.7%) 86.3486.34 74.60 (+13.1%) 59.2359.23 33.1033.10 (+13.7%) 91.25 68.63 (+6.9%) We first evaluate standard downstream performance to verify that InvPT does not buy robustness at the cost of standard accuracy. We then turn to transformed test sets in Section 3.3, where robustness to invariant programs is the main target. Table 4 shows the clone detection results. Applying InvPT improves all four baseline families on most benchmarks, and InvGraphCodeBERT and InvContraBERT_G achieve the strongest overall performance. The largest gain appears on POJ104, where InvGraphCodeBERT improves GraphCodeBERT by 7.8%7.8\%. Notably, these gains also transfer to the C/C++ benchmarks. Although the base encoders were pretrained on C/C++ (among other languages), our invariant continued pretraining uses only Java and Python programs from CodeSearchNet; no C/C++ code receives any invariant transformation during this stage. The C/C++ improvements therefore reflect cross-language transfer of the robustness signal acquired from Java and Python invariants alone. Code classification accuracy is nearly saturated on these benchmarks (orig columns of Table 5). Even in this regime, InvPT matches or slightly improves the corresponding baselines on all four datasets. Table 5. Code classification results. “orig”: original test set; “+T”: transformed test set for robustness evaluation. Java250 Python800 C++1400 POJ104 Model orig + T orig + T orig + T orig + T CodeBERT 97.8097.80 65.8165.81 98.9298.92 63.2663.26 89.6389.63 39.9239.92 98.5198.51 50.9750.97 GraphCodeBERT 97.9797.97 67.5167.51 99.0699.06 78.5278.52 89.7089.70 41.5241.52 98.6898.68 52.0352.03 ContraBERT_C 97.9097.90 67.3367.33 98.8998.89 72.6172.61 89.5489.54 41.4641.46 98.4898.48 51.1951.19 ContraBERT_G 98.1598.15 68.5768.57 99.1099.10 78.5978.59 89.5389.53 41.2341.23 98.6398.63 50.3850.38 InvCodeBERT 98.1698.16 70.1070.10 (+6.5%) 99.0099.00 82.4282.42 (+30.3%) 89.9489.94 43.6943.69 (+9.4%) 98.6398.63 53.6753.67 (+5.3%) InvGraphCodeBERT 98.0898.08 71.3071.30 (+5.6%) 99.0999.09 84.60 (+7.7%) 90.00 44.23 (+6.5%) 98.5498.54 52.6652.66 (+1.2%) InvContraBERT_C 98.0798.07 71.3571.35 (+6.0%) 98.9598.95 83.2883.28 (+14.7%) 89.8789.87 42.5142.51 (+2.5%) 98.7298.72 53.2453.24 (+4.0%) InvContraBERT_G 98.21 71.48 (+4.2%) 99.11 83.4283.42 (+6.1%) 89.7189.71 42.8842.88 (+4.0%) 98.73 53.74 (+6.7%) 3.3. Robustness Against Invariant Programs We evaluate robustness by cumulatively applying all six transformations from Section 2.1 to the test set. Because training uses only single-operator transformations, these compositions are unseen, so the setting measures invariance to the pretraining rewrite family under novel compositions rather than robustness to arbitrary semantics-preserving edits (Section 5). For clone detection, all InvPT models outperform their baselines on transformed data (Table 4), with relative gains up to 22.2%22.2\%. InvGraphCodeBERT also surpasses both ContraBERT_C and ContraBERT_G on all four benchmarks, and InvCodeBERT on Java250, Python800, and C++1400; the one exception is POJ104, where InvCodeBERT (62.0362.03) remains below ContraBERT_C (62.7662.76) and ContraBERT_G (64.2064.20). For code classification, InvPT again yields consistent gains on transformed data (+T columns of Table 5), improving over GraphCodeBERT by up to 7.7%7.7\% and over ContraBERT_G by up to 6.1%6.1\%. Notably, these gains extend to C++1400 (up to +3.77+3.77 on classification and +5.84+5.84 on clone detection), where no C/C++ code receives invariant transformations during pretraining, demonstrating cross-language transfer of structural robustness from Java and Python to a held-out target language. We do not claim cross-lingual representational alignment (i.e., mapping semantically equivalent programs across languages into nearby embeddings), which would require parallel multi-language data and is left to future work. Where InvPT helps least The smallest robustness gain on the transformed test sets is InvContraBERT_C vs. ContraBERT_C on POJ104 (+0.77+0.77 p, +1.2%+1.2\%), indicating that ContraBERT_C’s prior contrastive objective already captures much of the invariance signal on this comparatively saturated benchmark. The gain is largest where the baseline is weakest: on transformed C++1400, where C/C++ never appears in our pretraining corpus, InvCodeBERT improves over CodeBERT by +5.84+5.84 p (+22.8%+22.8\%). Why cross-language transfer works We attribute the transfer to two factors. First, our operators target language-agnostic structural concepts: loop equivalence (W2F, F2W), branch reversal (RevIf), scalar increment forms (P2A, A2EA), and identifier renaming (VarRe). Second, Java, Python, and C/C++ share a lexical substrate (common keywords, operators, and bracketing) that the encoder’s sub-word tokenizer represents consistently across languages. We do not provide a theoretical analysis of this transfer; these two factors are the strongest explanation our evidence supports. Absolute percentage-point improvements The tables above report performance and relative improvement (%) over each baseline. For readers who prefer absolute changes, Table 6 and Table 7 restate the robustness gains on the transformed (+T) test sets in percentage points (p), the simple arithmetic difference between each InvPT model and its corresponding baseline (e.g., InvCodeBERT on Java250 (+T): a +8.11+8.11 p gain over CodeBERT). InvPT improves in all 1616 model–dataset comparisons on each task, but the distributions differ sharply. On clone detection the gains are uniformly sizable (median +8.07+8.07 p, up to +11.04+11.04). On classification they are smaller and strongly skewed (median +3.56+3.56 p), with the largest value (+19.16+19.16, InvCodeBERT on Python800) well separated from the next (+10.67+10.67); we therefore treat clone detection as the more representative measure and the classification maximum as an outlier rather than a typical gain. Measured against the degradation itself, InvPT recovers only part of the drop from the original to the transformed test set: a median of 29.6%29.6\% on clone detection (range 2.92.9–49.3%49.3\%) and 8.7%8.7\% on classification (range 1.41.4–53.7%53.7\%), so the gap that remains is larger than the part we close. Table 6. Clone detection (+T): absolute p change vs. the corresponding baseline encoder (InvCodeBERT vs. CodeBERT, InvGraphCodeBERT vs. GraphCodeBERT, etc.). Model Java250 Python800 C++1400 POJ104 InvCodeBERT +8.11+8.11 +8.38+8.38 +5.84+5.84 +6.07+6.07 InvGraphCodeBERT +8.78+8.78 +8.02+8.02 +3.42+3.42 +8.69+8.69 InvContraBERT_C +11.04+11.04 +10.93+10.93 +4.98+4.98 +0.77+0.77 InvContraBERT_G +8.45+8.45 +8.62+8.62 +3.99+3.99 +4.43+4.43 Table 7. Code classification (+T): absolute p change vs. the corresponding baseline encoder. Model Java250 Python800 C++1400 POJ104 InvCodeBERT +4.29+4.29 +19.16+19.16 +3.77+3.77 +2.70+2.70 InvGraphCodeBERT +3.79+3.79 +6.08+6.08 +2.71+2.71 +0.63+0.63 InvContraBERT_C +4.02+4.02 +10.67+10.67 +1.05+1.05 +2.05+2.05 InvContraBERT_G +2.91+2.91 +4.83+4.83 +1.65+1.65 +3.36+3.36 3.4. Ablation Study Table 8. Ablation studies on clone detection using CodeNet and POJ104. “orig”: original test set; “+T”: transformed test set. Java250 Python800 C++1400 POJ104 Model orig + T orig + T orig + T orig + T CodeBERT 77.2877.28 51.1651.16 83.7183.71 62.1762.17 56.2156.21 25.5925.59 85.4785.47 55.9655.96 w/o contrastive loss 80.1680.16 56.1256.12 84.5584.55 68.2768.27 56.0556.05 28.4728.47 83.8783.87 57.5157.51 w/o self-contrast 81.9281.92 57.9257.92 85.0785.07 70.2770.27 57.1957.19 30.3330.33 87.6987.69 62.4262.42 InvCodeBERT ++ NL 82.7482.74 61.3561.35 85.5285.52 69.8769.87 57.6957.69 31.0131.01 88.0288.02 64.1164.11 InvCodeBERT 82.4482.44 59.2759.27 85.6385.63 70.5570.55 57.8057.80 31.4331.43 88.8188.81 62.0362.03 We ablate the contribution of each component of InvPT (Section 2) with three CodeBERT-based variants: (1) w/o contrastive loss: drops the supervised contrastive objective and trains with only MLM on original and transformed code; (2) w/o self-contrast: keeps only invariant-contrast pairs; (3) InvCodeBERT ++ NL: adds natural-language descriptions alongside code in pretraining. We evaluate the variants on clone detection (CodeNet, POJ104) with original and transformed (+T) test sets, and report the full result in Table 8. Contrastive learning is the primary driver of improvement Removing the contrastive loss and training only with MLM on original and transformed code still improves over CodeBERT on most benchmarks, but it remains clearly below full InvCodeBERT. On the original POJ104, it even drops below the CodeBERT baseline (83.8783.87 vs. 85.4785.47), indicating that exposure to transformed code without explicit representation alignment is insufficient. Self-contrast provides consistent additional gains Removing self-contrast lowers performance on all original benchmarks relative to InvCodeBERT, by roughly 0.50.5 to 1.11.1 points, and usually also reduces robustness on transformed test sets. The margins are modest, but the pattern is stable across datasets, suggesting that aligning different masked views of the same program complements invariant contrast rather than duplicating it. Natural language descriptions are not necessary Adding natural language descriptions yields mixed results. The effect is inconsistent across benchmarks and small relative to the other components: on the transformed sets, ++NL gains +2.08+2.08 p on both Java250 and POJ104 but loses 0.680.68 and 0.420.42 p on Python800 and C++1400, averaging 0.760.76 p above InvCodeBERT on transformed data (56.5856.58 vs. 55.8255.82) and 0.180.18 p below it on original data (78.4978.49 vs. 78.6778.67). We therefore do not claim that code-only pretraining outperforms ++NL. The ablation supports only a weaker claim: paired natural language moves performance by well under a point on average, and in opposite directions on the two settings. That gain does not justify a bi-modal NL-PL corpus, which restricts continued pretraining to the comparatively small set of functions carrying usable docstrings. Table 9 restates the ablation effects on the transformed (+T) test sets as absolute p changes relative to full InvCodeBERT, making each removal’s magnitude directly comparable: dropping the contrastive loss costs the most (−2.28-2.28 to −4.52-4.52 p), while removing self-contrast or adding NL descriptions matters less. Table 9. Ablations (+T) absolute p change. Variant Java250 Python800 C++1400 POJ104 w/o contrastive loss −3.15-3.15 −2.28-2.28 −2.96-2.96 −4.52-4.52 w/o self-contrast −1.35-1.35 −0.28-0.28 −1.10-1.10 +0.39+0.39 InvCodeBERT ++ NL +2.08+2.08 −0.68-0.68 −0.42-0.42 +2.08+2.08 Figure 2. Visualization of vector embeddings of 5 problems in POJ104. Top row: CodeBERT-based models; bottom row: GraphCodeBERT-based models. 3.5. Qualitative Representation Visualization We next present a qualitative view of the learned representation space. Following 32, we randomly select five problems from the test split of POJ104, apply invariant transformations to the code snippets within these problems, and use all 500 code snippets per problem for visualization. We then apply t-SNE (46) to project the high-dimensional code representations into two dimensions. Figure 2 shows the results for two model families. Across both families, the base models (CodeBERT and GraphCodeBERT) produce heavily overlapping embeddings with little cluster separation: invariant-transformed programs often lie closer to unrelated solutions than to their semantic equivalents. The ContraBERT variants (ContraBERT_C and ContraBERT_G) introduce more structure, with same-problem points forming local groupings, though considerable overlap remains. In contrast, our InvPT models (InvCodeBERT and InvGraphCodeBERT) yield tight, well-separated clusters for each problem, providing qualitative evidence consistent with the quantitative results. 4. Related Work Pretrained code models Code representation learning adapts encoder pretraining to programming languages, with models such as CodeBERT (11) and GraphCodeBERT (15) learning from large code corpora (17; 18). Subsequent work adds AST sequences (14), data-flow graphs 15, execution signals (20), and transformation-based augmentation (52). None of them explicitly trains for invariant embeddings under semantics-preserving transformations. Recent work confirms that encoders remain competitive and efficient for understanding tasks (24; 31; 49; 30), motivating InvPT’s encoder-based design. A complementary line of work scales code embedding models well beyond the 125M-parameter class (23; 41; 13; 38; 48); InvPT is architecture-agnostic, and we instantiate it on 125M-parameter encoders that dominate cost-sensitive deployments while including CodeSage (53) (1.3B) and ModernBERT (49) as larger references in Table 4. Contrastive learning for code Contrastive learning is a dominant paradigm for transferable representations; SimCLR (8) established that augmentation composition and a learnable projection head are critical ingredients. In the code domain, VarCLR (7) and CodeSage (53) show that contrasting transformed code improves representation quality, while NatGen (5) applies similar semantics-preserving transformations in a generative denoising setting. Two lines of work share our core premise that semantics-preserving transformations supply a contrastive signal for code: Corder (3) contrasts AST-level source-to-source variants, and ContraCode (22) contrasts compiler-generated JavaScript variants. We do not claim that premise as a contribution. InvPT differs in three respects. Both prior methods use pairwise InfoNCE, which treats two variants of the same function as negatives when they co-occur in a batch; since we apply several operators per function, such collisions are frequent, and we instead use a multi-positive supervised contrastive mask keyed on a hash of the canonical source (Section 2.2). We apply the recipe as continued pretraining on released encoders rather than pretraining from scratch, and we evaluate robustness explicitly, on transformed test sets and a held-out target language, rather than standard accuracy alone. We do not rerun their objectives on our backbone, so we report no controlled comparison; our ablation isolates the contrastive objective as a whole (Section 3.4) but not the multi-positive mask against pairwise InfoNCE. The closest prior work is ContraBERT (32), which combines MoCo with bi-modal NL-PL pretraining but evaluates robustness primarily on variable renaming. InvPT differs by using code-only continued pretraining with supervised contrastive learning over a shared encoder and a multi-positive mask, targeting naturally occurring semantics-preserving transformations (loop equivalence, branch reversal, scalar-increment forms, variable renaming) rather than dead-code insertion and random line deletion, and evaluating across multiple transformation types and downstream tasks. SPAT (52; 47) uses similar semantics-preserving transformations for fine-tuning augmentation; InvPT instead bakes them into pretraining as an objective. Adversarial robustness DAMP (51), MHM (54), and ALERT (50) use variable renaming and dead-code insertion as adversarial attacks on code models; 2 give a systematic treatment of code adversarial robustness, 19 formalize k-transformation robustness, and 42 frame it as generalizability under natural transformations. Unlike these fine-tuning defenses, InvPT builds invariance into pretraining, amortizing the benefit across downstream tasks. 5. Limitations Our robustness evaluation uses test-time compositions that are unseen but built from the same six operators used in pretraining. The results therefore establish invariance to this rewrite family, not robustness to semantics-preserving change in general; independently generated transformations and naturally occurring refactorings remain untested. We also do not compare against downstream-only augmentation, so we cannot separate building invariance into pretraining from augmenting the fine-tuning data with the same rewrites. All results come from a single pretraining and fine-tuning run per configuration, so differences of a point or less—including several ablation margins in Table 8—should be read with caution. Our operators cover Java, Python, and C/C++; extending them to languages with substantially different semantics (e.g., Haskell, Rust) requires non-trivial engineering to preserve equivalence. Because the rewrites are automated rather than formally verified, rare edge cases may alter runtime behavior, the clearest being A2EA on Python (Section 2.1); we expect the affected fraction of our corpora to be small but have not measured it. Finally, we evaluate discriminative tasks on competitive-programming benchmarks, which provide ground-truth labels but under-represent industrial codebases, and we instantiate InvPT on 125M-parameter encoder-only models pretrained on CodeSearchNet. Invariance is the appropriate criterion only for tasks whose output should not change under rewriting; generation requires the more general equivariance condition, which we leave to future work. 6. Conclusion We present InvPT, a code-only continued pretraining method that combines semantics-preserving transformations with supervised contrastive learning and mixed-difficulty positive pairs, eliminating the need for paired natural-language data. InvPT improves robustness over its baseline in every model–dataset comparison on clone detection and code classification, by a median of 8.078.07 and 3.563.56 percentage points respectively (up to 11.0411.04 and 19.1619.16), while matching or improving standard task performance. These gains recover part of the degradation, not most of it, and are measured on compositions of the operator family seen in pretraining; robustness to structurally different rewrites remains open. Ablations show that invariant contrastive learning is the primary driver of these gains, with self-contrast providing consistent additional improvement. Notably, pretraining on Java and Python invariants alone yields robustness gains on C/C++ downstream tasks, suggesting that the learned invariance transfers across languages. More broadly, InvPT demonstrates that robust code representations can be learned from code alone. Because the method modifies only the pretraining data and loss, extending it to richer transformation families and to encoder-decoder or decoder-only architectures is a natural next step. Generation, however, calls for a weaker criterion than invariance: under VarRe a model should rename consistently in its output rather than leave it unchanged. The general condition for a transformation T and a model g is equivariance, g(T(x))=MT(g(x))g(T(x))=M_T(g(x)) for a transformation-specific output map MTM_T, with invariance the special case MT=idM_T=id; learning equivariance under a known MTM_T is the direction we see as most promising. Acknowledgments We would like to thank the anonymous reviewers for their constructive comments. This work is supported by the UC Noyce Initiative. References Ahmed and Devanbu (2022) T. Ahmed and P. Devanbu Multilingual training for software engineering. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22. External Links: Link, Document Cited by: §1. Bielik and Vechev (2020) P. Bielik and M. Vechev Adversarial robustness for code. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 896–907. External Links: Link Cited by: §4. Bui et al. (2021) N. D. Q. Bui, Y. Yu, and L. Jiang Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, New York, NY, USA, p. 511–521. External Links: ISBN 9781450380379, Link, Document Cited by: §2.1, §4. Carlini et al. (2019) N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song The secret sharer: evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), SEC’19, USA, p. 267–284. External Links: ISBN 9781939133069, Link Cited by: §2.1. Chakraborty et al. (2022) S. Chakraborty, T. Ahmed, Y. Ding, P. T. Devanbu, and B. Ray NatGen: generative pre-training by "naturalizing" source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, New York, NY, USA, p. 18–30. External Links: ISBN 9781450394130, Link, Document Cited by: §4. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §1. Chen et al. (2022) Q. Chen, J. Lacomis, E. J. Schwartz, G. Neubig, B. Vasilescu, and C. Le Goues VarCLR: variable semantic representation pre-training via contrastive learning. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, New York, NY, USA, p. 2327–2339. External Links: ISBN 9781450392211, Link, Document Cited by: §4. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 1597–1607. External Links: Link Cited by: §4. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: §2.2.1. Ding et al. (2025) Y. Ding, Y. Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y. Chen Vulnerability detection with code language models: how far are we?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), ICSE. External Links: Document, Link Cited by: footnote 1. Feng et al. (2020) Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou CodeBERT: a pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, External Links: Document, Link Cited by: §1, §1, §3.1, §4. Gao et al. (2021) T. Gao, X. Yao, and D. Chen SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, p. 6894–6910. External Links: Link, Document Cited by: §2.2.3. Google (2025) Google EmbeddingGemma. Note: https://huggingface.co/googleOpen embedding model Cited by: §4. Guo et al. (2022) D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin UniXcoder: unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, p. 7212–7225. External Links: Link, Document Cited by: §4. Guo et al. (2021) D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. LIU, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou GraphCodeBERT: pre-training code representations with data flow. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2.2.1, §3.1, §4. He et al. (2020) K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9729–9738. External Links: Document, Link Cited by: §2.2. He et al. (2024) Y. He, J. Huang, Y. Rong, Y. Guo, E. Wang, and H. Chen UniTSyn: a large-scale dataset capable of enhancing the prowess of large language models for program testing. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, New York, NY, USA, p. 1061–1072. External Links: ISBN 9798400706127, Link, Document Cited by: §1, §4. He et al. (2025) Y. He, J. Wang, Y. Rong, and H. ChenC. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.) FuzzAug: data augmentation by coverage-guided fuzzing for neural test generation. Association for Computational Linguistics, Suzhou, China. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §4. Henkel et al. (2022) J. Henkel, G. Ramakrishnan, Z. Wang, A. Albarghouthi, S. Jha, and T. Reps Semantic robustness of models of source code. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Vol. , p. 526–537. External Links: Document Cited by: §4. Huang et al. (2024) J. Huang, J. Zhao, Y. Rong, Y. Guo, Y. He, and H. Chen Code representation pre-training with complements from program executions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Miami, Florida, US, p. 267–278. External Links: Link, Document Cited by: §2.2.1, §4. Husain et al. (2020) H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt CodeSearchNet challenge: evaluating the state of semantic code search. External Links: 1909.09436, Link Cited by: §3.1. Jain et al. (2021) P. Jain, A. Jain, T. Zhang, P. Abbeel, J. Gonzalez, and I. Stoica Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, p. 5954–5971. External Links: Link, Document Cited by: §4. Jina AI (2025) Jina AI Jina code embeddings. Note: https://huggingface.co/jinaai/jina-embeddings-v2-base-codeCode embedding model Cited by: §4. K et al. (2025) T. D. K, T. Fischer, and C. Biemann Large language models are overparameterized text encoders. p. 170–184. External Links: Link, Document, ISBN 979-8-89176-245-9 Cited by: §1, §4. Kalra and Barkeshli (2024) D. S. Kalra and M. Barkeshli Why warmup the learning rate? underlying mechanisms and improvements. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2.2.3. Khosla et al. (2020) P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan Supervised contrastive learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. External Links: ISBN 9781713829546 Cited by: §2.2.2. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. External Links: Document, Link, https://w.pnas.org/doi/pdf/10.1073/pnas.1611835114 Cited by: §2.2.3. Krinke and Ragkhitwetsagul (2022) J. Krinke and C. Ragkhitwetsagul BigCloneBench considered harmful for machine learning. In 2022 IEEE 16th International Workshop on Software Clones (IWSC), Vol. , p. 1–7. External Links: Document Cited by: footnote 1. Krinke and Ragkhitwetsagul (2025) J. Krinke and C. Ragkhitwetsagul How the misuse of a dataset harmed semantic clone detection. External Links: 2505.04311, Link Cited by: footnote 1. Lei et al. (2025) Y. Lei, S. He, A. Li, and A. Yates Making large language models efficient dense retrievers. arXiv preprint arXiv:2512.20612. External Links: Link Cited by: §1, §4. Lin et al. (2026) J. Lin, Y. Wang, Y. Yang, L. Zhang, and Y. Xie Towards better code understanding in decoder-only models with contrastive learning. Proceedings of the AAAI Conference on Artificial Intelligence 40 (38), p. 32006–32014. External Links: Link, Document Cited by: §4. Liu et al. (2023) S. Liu, B. Wu, X. Xie, G. Meng, and Y. Liu ContraBERT: enhancing code pre-trained models via contrastive learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 2476–2487. External Links: Document, Link Cited by: §1, §2.1, §2.2.1, §2.2.3, §2.2, §3.1, §3.5, §4. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §3.1. Lu et al. (2021) S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU CodeXGLUE: a machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), External Links: Link Cited by: §3.1, §3.1. Lyu et al. (2024) Y. Lyu, Y. Xie, P. Chen, and H. Chen Prompt fuzzing for fuzz driver generation. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, Salt Lake City, UT, USA. External Links: Document, Link Cited by: §1. Mou et al. (2016) L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin Convolutional neural networks over tree structures for programming language processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, p. 1287–1293. External Links: Document, Link Cited by: §3.1. Musgrave et al. (2020) K. Musgrave, S. Belongie, and S. Lim A metric learning reality check. In European Conference on Computer Vision, Cham, p. 681–699. External Links: ISBN 978-3-030-58595-2, Document, Link Cited by: §3.1. Nomic AI (2024) Nomic AI CodeRankEmbed. Note: https://huggingface.co/nomic-ai/CodeRankEmbedCode retrieval embedding model Cited by: §4. Ogura et al. (2018) N. Ogura, S. Matsumoto, H. Hata, and S. Kusumoto Bring your own coding style. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), Vol. , p. 527–531. External Links: Document Cited by: §2.1, §2.1. Puri et al. (2021) R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, V. Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss CodeNet: a large-scale AI for code dataset for learning a diversity of coding tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), External Links: Link Cited by: §3.1, §3.1, §3.1. Qwen Team (2025) Qwen Team Qwen3 embedding. Note: https://huggingface.co/QwenEmbedding model series Cited by: §4. Rabin et al. (2021) M. R. I. Rabin, N. D.Q. Bui, K. Wang, Y. Yu, L. Jiang, and M. A. Alipour On the generalizability of neural program models with respect to semantic-preserving program transformations. Information and Software Technology 135, p. 106552. External Links: ISSN 0950-5849, Document, Link Cited by: §4. Rozière et al. (2024) B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve Code llama: open foundation models for code. External Links: 2308.12950, Link Cited by: §1. Svajlenko et al. (2014) J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, p. 476–480. External Links: Document, Link Cited by: footnote 1. van den Oord et al. (2019) A. van den Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. External Links: 1807.03748, Link Cited by: §2.2.2. van der Maaten and Hinton (2008) L. van der Maaten and G. Hinton Visualizing data using t-sne. Journal of Machine Learning Research 9 (86), p. 2579–2605. External Links: Link Cited by: §3.5. Wang et al. (2022) D. Wang, Z. Jia, S. Li, Y. Yu, Y. Xiong, W. Dong, and X. Liao Bridging pre-trained models and downstream tasks for source code understanding. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, New York, NY, USA, p. 287–298. External Links: ISBN 9781450392211, Link, Document Cited by: §1, §2.1, §2.2.3, §4. Wang et al. (2023) Y. Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi CodeT5+: open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 1069–1088. Cited by: §4. Warner et al. (2025) B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, G. T. Adams, J. Howard, and I. Poli Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 2526–2547. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §4, footnote 2. Yang et al. (2022) Z. Yang, J. Shi, J. He, and D. Lo Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, New York, NY, USA, p. 1482–1493. External Links: ISBN 9781450392211, Link, Document Cited by: §1, §4. Yefet et al. (2020) N. Yefet, U. Alon, and E. Yahav Adversarial examples for models of code. Proc. ACM Program. Lang. 4 (OOPSLA). External Links: Link, Document Cited by: §4. Yu et al. (2022) S. Yu, T. Wang, and J. Wang Data augmentation by program transformation. Journal of Systems and Software 190, p. 111304. External Links: ISSN 0164-1212, Document, Link Cited by: §1, §2.1, §2.1, §2.1, §2.2.3, §4, §4. Zhang et al. (2024) D. Zhang, W. Ahmad, M. Tan, H. Ding, R. Nallapati, D. Roth, X. Ma, and B. Xiang Code representation learning at scale. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 47427–47446. External Links: Link Cited by: §4, §4, footnote 2. Zhang et al. (2020) H. Zhang, Z. Li, G. Li, L. Ma, Y. Liu, and Z. Jin Generating adversarial examples for holding robustness of source code processing models. Proceedings of the AAAI Conference on Artificial Intelligence 34 (01), p. 1169–1176. External Links: Link, Document Cited by: §4. Zhao et al. (2023) J. Zhao, Y. Rong, Y. Guo, Y. He, and H. Chen Understanding programs by exploiting (fuzzing) test cases. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, p. 10667–10679. External Links: Link, Document Cited by: §2.1, §3.1, §3.1. Zhou et al. (2019) Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: footnote 1.