Paper deep dive
Memorization Diagnostics for Code LLMs Should be Scale-Aware
Prateek Kumar Rajput, Abdoul Aziz Bonkoungou, Alberick Euraste Djiré, Xunzhu Tang, Yewei Song, Iyiola Emmanuel Olatunji, El Hacen Diallo, Jacques Klein, Tegawendé F. Bissyandé
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 8/14/2026, 5:10:16 AM
Summary
The paper argues that traditional memorization diagnostics for code Large Language Models (LLMs), such as synonym fuzzing and log-likelihood probes, fail to detect memorization in scaled models because these models absorb perturbations as 'representation load' rather than failing due to lack of understanding. The authors propose using invertible I/O isomorphisms to separate representation load from memorization, demonstrating that scaled models maintain algorithmic correctness even under heavy representational constraints, whereas smaller models fail entirely. This suggests that current evaluation methodologies entangle these phenomena and should be revised to focus on representational robustness.
Entities (10)
Relation Signals (6)
Scaled Models → absorbs → Representation Load
confidence 90% · scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions.
I/O Isomorphism → separates → Representation Load
confidence 90% · By applying invertible mathematical transforms to numeric problems, we isolate these two factors [representation load and memorization].
Synonym Fuzzing → failstodetect → Memorization
confidence 85% · Traditional encoder-style probes using perturbations such as synonym fuzzing... struggle to expose memorization in scaled models.
Small models → failsunder → Representation Load
confidence 85% · smaller models drop whole families of solutions altogether.
Scaled Models → maintains → Algorithmic Correctness
confidence 85% · scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions.
CoDeC → failstodetect → Memorization
confidence 80% · decoder-style probes that rely on log probabilities show similar performance degradation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.
Tags
Links
- Source: https://arxiv.org/abs/2608.12771v1
- Canonical: https://arxiv.org/abs/2608.12771v1
Trouble viewing inline? Open PDF directly →
Full Text
77,504 characters extracted from source content.
Expand or collapse full text
Memorization Diagnostics for Code LLMs Should be Scale-Aware Prateek Kumar Rajput 1* , Abdoul Aziz Bonkoungou 1 , Alberick Euraste Djir ́e 1 , Xunzhu Tang 1 , Yewei Song 1 , Iyiola Emmanuel Olatunji 1 , El Hacen Diallo 1 , Jacques Klein 1 , Tegawend ́e F. Bissyand ́e 1 1* University of Luxembourg, Esch-sur-Alzette, Luxembourg. *Corresponding author(s). E-mail(s): prateek.rajput@uni.lu; Contributing authors: abdoul.bonkoungou@uni.lu; euraste.djire@uni.lu; xunzhu.tang@uni.lu; yewei.song@uni.lu; emmanuel.olatunji@uni.lu; el-hacen.diallo@uni.lu; jacques.klein@uni.lu; tegawende.bissyande@uni.lu; Abstract The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature fre- quently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled mod- els, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial represen- tation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated bench- marks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations 1 arXiv:2608.12771v1 [cs.SE] 13 Aug 2026 must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them. Code and Data are available ∗ . Keywords: Code generation, LLM evaluation, memorization, contamination, representation load, robustness, metamorphic testing 1 Introduction LLMs are rapidly becoming the default engine for automating software engineering tasks, and code synthesis sits at the frontier of this transformation. They achieve impressive benchmark accuracy [1–4], yet a persistent question shadows their progress. Are they solving problems or recognizing them? Recent work sharpens the worry, with analyses of agentic coding benchmarks arguing that some leading results lean on recall more than reasoning [5]. The dominant strategy in the literature for probing this ques- tion is to apply semantics-preserving perturbations to the prompt, such as renaming variables [6, 7], paraphrasing docstrings [8, 9], and inserting dead code [6, 7], then reading the resulting pass@k drop as evidence bearing on memorization [10–12]. These encoder-side diagnostics operate in lexical and semantic space. They are intuitive and have no doubt enhanced our understanding, but they are also just convenient. Most studies say little about the extra demands these prompt changes place on the model, rarely examine scaled models where encoder capacity may absorb the perturbation entirely, and offer no formal guarantee that the rewritten prompt preserves the original task semantics. A complementary line of work targets the decoder. Methods such as CoDeC [13] measure how in-context examples shift token log-likelihoods to distinguish seen from unseen training data, but these probes are also often validated on small model families and do not address whether their discriminative signal survives as the model’s internal representation space scales and the probability mass spreads across a vastly larger set of plausible continuations. Other such decoder-style works include permutation tests on dataset ordering [14], multiple-choice contamination quizzes [15], and token-level likelihood analyses for verbatim code regurgitation [16]. We put the assumption that the probe keeps working as models grow under direct test, and on benchmarks with known contamination [2, 13, 17, 18], the encoder-side and decoder-side signals we studied faded as model capacity increased. This probe saturation pushed us toward a different question. Rather than asking whether a solu- tion was seen during training, we ask how faithfully a model carries its algorithmic knowledge through an unfamiliar way of stating the same task. We treat this capacity as distinct from memorization and term it representational load, and our experiments later confirm that the two are separable. The framing borrows from metamorphic test- ing, where a known relation between inputs constrains the relation between outputs. To add representational load while holding the core task fixed, we leverage invertible numerical transforms. We develop both ideas below. ∗ https://github.com/pkrajput/isomorphicprompts,https://doi.org/10.5281/zenodo. 19248561 2 Representational load. Representational load arises wherever the interface departs from training-like surface form. Domain-specific encodings, legacy data formats, unfa- miliar API contracts, and project-specific conventions all impose interface conditions that no benchmark captures. On the encoder side, it requires the model to parse aux- iliary constraints alongside the core task specification, mapping a richer, less familiar prompt to the correct underlying semantic representation. On the decoder side, it requires the model to honour those constraints token by token during autoregressive generation, interleaving novel formatting logic with the algorithmic solution it has internalized. A model that can only produce correct code under training-like surface conditions offers weaker guarantees than one that can transport its algorithmic knowl- edge through an unfamiliar representational channel. Building on this view, we first study one representative probe from each side of the literature, encoder and decoder, along a model-scale axis, and then propose a new perspective that leverages a property unique to software, the latent structure of code itself. I/O isomorphisms. To probe behaviour beyond surface text manipulation, and to keep the task algorithmically fixed under a mathematical guarantee, we use I/O isomorphisms. Methodologically this is a form of metamorphic and robustness test- ing [19, 20] adapted to code LLMs. We apply a bijective value transform to the test-case integers and append an explicit encode/decode contract to the prompt, wrapping the interface in a representational layer that demands explicit computation without alter- ing the underlying algorithmic task (f (x) = y). The metamorphic relation between the original test oracle and its isomorphic counterpart then serves as our oracle, so any deviation on a previously passing problem is, by construction, a representational- robustness failure rather than a change in the task itself. The model cannot memorize its way through this layer. It must decode every input, run the algorithm, and re- encode every output. For our main experiments we instantiate the transform as an affine integer isomorphism, T θ (t) = at + b, a̸= 0, T −1 θ (t ′ ) = t ′ − b a , and we additionally ablate over base-conversion and cubic bijection families. Here, t represents the original test-case integer input, while a and b are the randomly selected constants (with a̸= 0) that define the linear transformation parameterizing the affine isomorphism. Our study yields three findings. • Synonym substitution in prompts and token log-likelihood probes that rely on in- context examples both saturate at scale, losing their power to separate seen from unseen problems. • Representational load degrades performance even in scaled models, yet this drop does not by itself reveal memorization. • Opcode analysis shows that scaled models still reach the same solution families under added representational load, only less often, while smaller models drop whole families of solutions altogether. 3 A task under increasing representational load Every variant below states the same Woodall-number task. What differs is how far the prompt sits from the one that the model was trained on, and how much the model must transform to recover and serialize it. The five-segment bar beside each variant marks the representational load it imposes, ranging from the training-like surface form to five filled segments at the isomorphic contract. 1 Original Write a function to check if the given number is a Woodall number or not. A Woodall number has the form W k = k· 2 k − 1. Training-like surface form. No transformation required. 2 Synonym fuzz (∼40%) Compose aroutine toverify if thesupplied number is a Woodallnumeral or not. A Woodall number has the form W k = k· 2 k − 1. Light lexical drift. The model must read past degraded English back to the same task. 3 Dead code Write a function to check if the given number is a Woodall number or not. # unused scaffolding tmp = [i for i in range(10)] flag = len(tmp) > 0 The task text is untouched. The model must recognize and ignore an inert block. 4 LLM rewrite Given an integer n, decide whether some positive integer k makes n = k· 2 k − 1 hold, returning true when such a k exists and false otherwise. Same task, restructured far from the original wording. Cosine similarity to the original prompt is often used to bound the representation load in these cases. 5 I/O isomorphism Write a function to check if the given number is a Woodall number or not. All integers encoded via x ′ = 2x− 94. Decode inputs, solve, then re-encode every output. Heavy load. The model must invert the transform on every input, run the algorithm, and re-encode every output. Memorized retrieval cannot pass this layer. Fig. 1 The same task posed at increasing representational load, with the five-segment bar beside each variant indicating how much load it imposes. Synonym fuzzing, dead code, and a rewrite perturb the surface form only, while the isomorphic contract forces the model to invert a value transform, run the algorithm, and re-encode every output. Of the three, the third is the central phenomenon of this work, and Figure 1 builds the intuition behind it. A single Woodall task is restated at growing distance from its training-like form, moving from light synonym and dead-code edits through a free- form rewrite to a full isomorphic contract, with each variant forcing the model to do more work to recover the task and then serialize its answer. The pass@k drop that follows invites a naive reading in which the model has simply lost the algorithm, and we argue this reading is incomplete. What the added load strains is the route from a 4 MBPP Task 18 under a full isomorphic contract (T θ (t) = 2t− 94) Failure examples from 5 generations of the same I/O contract. Prompt and encoding contract Write a function to check if the given number is a Woodall number or not. A Woodall number has the form W k = k· 2 k − 1. All integers encoded via x ′ = 2x− 94. Decode inputs as x = (x ′ + 94)/2, solve, then re-encode every output. E.g. input−74 represents 10; if the answer is 0, output−94. Test cases: Original iswoodall(383)==True iswoodall(254)==False iswoodall(200)==False Iso iswoodall(672)==True iswoodall(414)==False iswoodall(306)==False Passing solution3/5 def is_woodall(n_enc): n = (n_enc + 94) / 2 if n <= 0 or not n.is_integer(): return False n = int(n) k = 1 while True: w = k * (2**k) - 1 if w == n: return True if w > n: return False k += 1 Failing solution2/5 def is_woodall(enc_n): n = (enc_n + 94) / 2 if not n.is_integer(): return (0*2)-94 == -94 n = int(n) if n <= 0: return (0*2)-94 == -94 k = 1 while True: w = k * (2**k) - 1 if w == n: return (1*2)-94 == -92 if w > n: return (0*2)-94 == -94 k += 1 Why it fails. The algorithmic loop is identical in both runs. The break is decoder-side: complying with the integer contract token-by-token, the model routes every boolean answer through an encoded arithmetic expression ((b*2)-94 == ...) which in Python evaluates back to True, so all four returns collapse to the same value. The load of following the contract during generation, not the algorithm, is what fails, and this is the load our RQs target. Fig. 2 Example: MBPP Task 18 under Iso contract. The failure mode is far from memorization, under heavy representational load. known solution to a correctly encoded one, not the model’s grasp of the solution itself. Section 2 makes this concrete on a single failing trace (Figure 2), where the algorithmic core is identical to that of a passing run and only the output encoding breaks. 2 Running Example Figure 2 traces one MBPP task under the isomorphic contract. Across both the passing and failing generations, the while loop that enumerates Woodall candidates 5 k· 2 k − 1 is identical. What separates a pass from a failure is the return statement. The failing generation pushes its boolean answer through the integer contract, writing comparisons such as (0 * 2) - 94 == -94. In Python, that expression is True on every branch, so the function reports every input as a Woodall number. The model has not lost the algorithm. It has confused the boundary between the encode-decode contract and Python’s own type system, and it cannot reliably serialize False once the native return type is boolean. Narrow channels. This is the behaviour we name a narrow channel. The path that carries a model’s algorithmic knowledge from specification to code stays open under representational load, but it tightens, so the same solution families still appear, only less often. A degraded pass@k then measures a constricted route rather than a for- gotten algorithm, and the Woodall trace is one instance of it, with the correct output surviving in three of five samples, while the loop never changes. Smaller models behave differently. Their channel closes rather than narrows, and entire solution families drop out, which is the collapse we separate from the graceful narrowing seen at scale. The same trace also tells the effect apart from memorization. A retrieved training solution should shatter under the isomorphic prompt, whose transformed I/O does not resemble any plausible training instance, yet the scaled models keep the task intact and break only at the contract. Prior probes would log this is nothing more than a lower pass@k [10–12]. To recover the algorithmic core that pass@k hides, we turn to opcode divergence between solutions under the original and isomorphic prompts, measured over passing solutions and overall generations. 3 Related Work 3.1 Memorization and Contamination in Code That memorization is real and measurable in large language models is by now well documented. Training data can be extracted near-verbatim from large models [21], and deduplicating the corpus reduces it while improving downstream behaviour [22]. For code in particular, contaminated benchmarks have been tied to inflated pass rates [12], models score higher on problems released before their training cutoff than on fresh ones [23], and the leakage of evaluation sets into training has proven both common and easy to overlook [24, 25]. The concern extends to the agentic setting, where recent analysis argues that some leading SWE benchmarks reflect recall rather than reasoning [5]. 3.2 Probes targeting the Encoder and Decoder On the encoder, the standard probe perturbs the prompt while preserving its meaning, then reads the pass@k drop as a memorization signal, through variable renaming [6, 7], docstring paraphrase [8, 9], or dead-code insertion [6, 7]. As we argue in Section 1, these manipulations stay in lexical and semantic space, rarely control for the extra parsing they impose, and carry no guarantee that the rewritten prompt preserves the original task. A complementary line attacks the question on the decoder. Membership inference asks whether a datum was seen, inferring it from the lowest-probability tokens of a 6 sequence [26] or from how a model scores a text against close neighbours [27], while counterfactual memorization [28] and influence functions [29] attribute behaviour to individual training examples. These methods sharpen what can be said about whether a specific input was seen, yet they answer a membership question rather than the capability question a practitioner cares about, which we try to target via separating the representational load. 3.3 Probe Reliability at Scale Memorization is known to grow with model size and data duplication [21] and to emerge in partly predictable, scale-dependent ways [30], so a probe calibrated on a small family is reading a moving quantity. Membership-advantage probes make this concrete, having been validated on a handful of small, open-code models where they separate likely-seen inputs from unseen ones [10, 11], though whether that separation survives at the scales now deployed in practice remains untested. Training dynamics complicate the picture further. Grokking shows that models can fit their data long before they generalize, with generalization arriving only after extended training uncov- ers latent structure [31], an effect later traced to the gradual formation of structured internal circuits [32] and explained as a shift toward the most efficient representa- tion [33], while memorization itself accrues across training in ways that do not track simple overfitting [34]. The duration and composition of training, not model size alone, may therefore be a decisive and largely uncontrolled variable, since two models of the same size can sit on opposite sides of a grokking-like transition. 3.4 Moving Targets and Our Perspective The community’s main response to contamination has been to build moving targets, whether contamination-free live benchmarks [35], harder test suites that expose brittle pass rates [36], or benchmarks that evolve known problems into fresh variants [37]. We take a different route. Because capability rises predictably with scale [38] and larger models adapt to unfamiliar prompt specifications through in-context learning [39, 40], we hold the algorithm fixed and change only how it is stated, asking what survives once representational load is separated from recall rather than chasing ever-cleaner data. 4 Research Questions What remains is to turn this intuition into a structured investigation, and we proceed by elimination. If the field’s standard probes still separated seen problems from unseen ones at frontier scale, the prevailing memorization reading would hold and little would need revising. We therefore put those probes to the test first, one on each side of the model, and treat their continued discriminative power as the hypothesis to be disproved. Only once that reading is shown to break do we introduce a load the model cannot have encountered in training, and ask what the resulting failure really is, whether the model has lost the algorithm or only the route that serializes it. The two leave the same trace in pass@k, so nothing short of this ordering can tell them apart. 7 Three research questions follow that path, two that close out the inherited account and one that opens the one we propose. RQ1: To what extent do encoder-side memorization probes work at scale? Synonym substitution and dead-code insertion stress lexical surface form. We test whether the resulting pass@k drop still tracks any diagnostic signal in frontier models, or whether scaled encoders absorb the perturbation. RQ2: To what extent do decoder-side memorization probes work at scale? Likelihood-based probes such as CoDeC compare token-level scores with and without in-context examples. We test whether the seen-vs-unseen gap they rely on survives as the model scale grows. RQ3: To what extent can representational load break performance? Using I/O isomorphisms, we keep the algorithmic task provably fixed and only change the interface. If pass@k still drops, the failure cannot be a memorization artifact and must come from the load itself. 5 Methodology This section describes the experimental setup common to all three diagnostic axes, covering the models and their selection rationale (Section 5.1), the datasets and evalua- tion conditions (Section 5.2), and the protocols specific to each axis (Sections 5.3–5.5). An overview of the full methodology can be seen in Figure 3. 5.1 Models (a) Table 1 Models used in our experiments. (a) Encoder-side lexical probing and I/O isomorphism experiments. (b) Decoder-side experiments (CoDeC). ModelParams Context Open-weight deepseek-coder-6.7b-instruct6.7B16K Llama-3.1-8B-Instruct8B128K CodeLlama-13B-Instruct13B16K StarCoder2-15B15B16K Codestral-22B-v0.122B32K Qwen2.5-Coder-32B-Instruct32B128K Llama-3.1-70B-Instruct70B128K Closed-source gpt-4o-mini-2024-07-18—128K gpt-4o-2024-08-06—128K gemini-2.0-flash—1M gemini-2.5-flash—1M (b) ModelParams Context Open-weight Pythia-410M0.4B2K Pythia-1.4B1.4B2K Pythia-12B12B2K Llama-3.1-70B70B128K Nemotron-4-340B-Instruct340B4K Llama-3.1-405B-Instruct405B128K Closed-source davinci-002—16K 8 Original Prompt Lexical perturbation Perturbed Prompt Perturbed Prompt Synonym substitution at 20/40 % For small models: pass@1 drops significantly (up to 37 points) For frontier models: drops are <5 points. The perturbation is absorbed Target sequence + In-context examples (same dataset) Token log- probabilities CoDeC score measures whether in- context examples depress or boost likelihood AUC: Seen vs Unseen seperation For small scale: AUC = 100% (perfect separation) At scale: AUC drops Original Task f(x) = y Apply Bijective Transform Transformed Task Bijection families: af fine, base conversion, cubic polynomial Encoding/Decoding contract appended to prompt Prompt-processing phase (global view) ... Read full prompt, read contract, transformed test cases, ... Token-by-token serialised generation: decoded transformed inputs, executes the core algorithm and re-encoded outputs Representational burden on input recognition Representational burden on output compliance: the narrow channel Generated Code pass@1 evaluation Opcode Analysis Variants: ISO (full) ISO (Enc only) ISO (Dec only) only inputs transformed only outputs transformed Model Saturates at scale Encoder-Side Probes: Synonym Fuzzing Decoder-Side Probes: CoDeC Contamination Discriminative power degrades at scale I/O Isomorphisms: Paper's Contribution Model Scale frontier models small models Model Model Same algorithmic code, brittle representational interface Fig. 3 Overview of diagnostic protocols. We evaluate three axes across model scale, namely encoder-side synonym fuzzing (top), decoder-side CoDeC contamination probing (middle), and our proposed I/O isomorphism protocol with opcode analysis (bottom). The isomorphism preservesthe algorithmic task exactly ( f ( x )= y ⇐⇒ T θ ( f ( T − 1 θ ( x ′ )))= y ′ ) while isolating representational load on both the prompt-processing and generation phases. 9 We select 11 models spanning a wide range of capabilities for the encoder-side and isomorphism experiments (Table 1a), prioritizing models commonly used for coding tasks and covering a broad parameter-count range. For the decoder-side contamina- tion analysis (Section 5.3), we construct a separate scale axis of 7 models (Table 1b) that permit teacher-forced scoring, computing per-token log-probabilities of a target sequence conditioned on its ground-truth prefix, without generating new tokens. For open-weight models (Pythia 410M, 1.4B, 12B; Nemotron-4-340B; Llama-3.1-405B), we use the vLLM library and access prompt-token log-probabilities directly through promptlogprobs. For davinci-002, we use OpenAI’s legacy completions endpoint, which exposes equivalent functionality via echo=True with logprobs, to include at least one commercial model in our study. A deliberate design choice guides our model selection. All models in our scale axis are dense architectures, not mixture-of-experts (MoE). Because MoE models activate only a fraction of their parameters per token, the relationship between nominal param- eter count and effective capacity is ambiguous. A 340B MoE model may behave more like a much smaller dense model on any given input. Since our analysis aims to study how memorization and generalization behaviours change as a function of scale, dense architectures provide a cleaner signal, as total parameters directly reflect the capac- ity available during inference. How decoder-side diagnostics should be adapted for MoE architectures, where routing decisions may interact with memorization patterns in ways that dense models do not exhibit, remains an open question. 5.2 Datasets, Conditions, and Generation Setup Table 2 Evaluation datasets across the study’s two diagnostic axes. The upper block covers the I/O isomorphism and encoder-side perturbations. The lower block covers the CoDeC probe, split into corpora that postdate every model’s training cutoff (Unseen) and corpora that are documented or probable members of the training mixtures (Seen). DatasetTasks / RoleDomainReleased I/O isomorphism and encoder-side perturbations MBPP [2]257Mostly-basic PythonAug 2021 EffiBench [41]338EfficiencyNov 2024 BigOBench [42]640Algorithmic complexityMar 2025 Decoder-side perturbations (CoDeC) gpqadiamond [43]UnseenGraduate-level Q&ANov 2023 livecodebenchv5 [35]UnseenCompetitive programmingJan 2025 wikipedia (Pile) [17]SeenEncyclopedic textDec 2020 hackernews (Pile) [17]SeenWeb forum textDec 2020 c 2024high [18]Seen † Common-Crawl web text2024 † Substituted for hackernews when probing Nemotron-4-340B- Instruct, following the corpus-overlap designation of Zawalski et al. [13]. 10 Datasets. We evaluate on three Python benchmarks spanning both a difficulty axis and a contamination axis (Table 2). MBPP sits at the easy end of the difficulty spectrum and is among the most widely used benchmarks in the code LLM litera- ture [10, 12, 35]. Every model in our study was released after its publication, making all potentially contaminated on this benchmark. EffiBench and BigOBench, by contrast, were published in 2024 and 2025, respectively, after the training cutoff of all models except Gemini-2.5-Flash. This creates a natural experiment. If memorization were the primary driver of performance, we would expect systematically larger Iso drops on the contaminated benchmark (MBPP), while drops on unseen benchmarks (EffiBench and BigOBench) should not be systematically large. We test this prediction directly in Section 6. All three benchmarks use Python exclusively, removing cross-language variation as a confound. Evaluation conditions. We utilize six conditions for inference. The first four probe representational invariance via I/O isomorphisms; the last two serve as encoder-side baselines operating in lexical space. Synonym fuzzing protocol. For the Syn-20% and Syn-40% conditions, we replace each non-connector, non-code-token word in the natural-language portion of the prompt with a WordNet synonym at the specified probability. This preserves task semantics while altering surface form, the same axis targeted directly in recent work [10] and by other similar probes like variable renaming [6, 7] and paraphras- ing [8, 9]. We work under the same hypothesis as theirs. If a model has merely memorized a solution, altering the surface form should degrade retrieval. Generation setup. For each (task, condition) pair, we generate n=5 completions at temperature T=0.0. Fixing the temperature at zero removes temperature-driven sampling as a source of variation, so that the behaviour we observe under each con- dition is not confounded by deliberate exploration of the output distribution. Greedy decoding is nonetheless not bitwise deterministic in practice, because floating-point reductions on accelerators are non-associative and sensitive to batch composition, and the hosted endpoints we query give no determinism guarantee, so repeated generations of the same prompt still diverge wherever two continuations lie close in probabil- ity. It is this residual nondeterminism, not temperature, that supplies the per-problem solution spread we analyse. We generate five completions for two reasons. First, our opcode-entropy analysis (Section 5.5) reads off a per-problem distribution of solu- tions, and pooling opcode frequency vectors across five generations enables meaningful per-problem Jensen-Shannon divergence estimates between conditions. Second, both EffiBench and BigOBench admit multiple valid solutions to each problem, so sev- eral generations help to characterise the solution space the model explores. The same decoding regime is held fixed across the Original and isomorphic conditions, so any divergence we report reflects the condition rather than a change in how solutions are drawn. 5.3 Decoder-Side Protocol (CoDeC) We reproduce CoDeC [13], a contamination diagnostic that operates on out- put log-probabilities. Each dataset receives a contamination score S CoDeC (D) = (1/N ) P i 1[∆(x i ) < 0], where ∆(x i ) is the change in mean token log-likelihood of 11 target x i when adding in-context samples from the same dataset. In-context exam- ples typically boost confidence for unseen data but may reduce it when the dataset was memorized during training by disrupting learned retrieval patterns. The dataset- level AUC quantifies separation between seen and unseen datasets. Formally, with seen-score set S + and unseen-score set S − , AUC = 1 |S + ||S − | X s + ∈S + X s − ∈S − 1(s + > s − ) + 0.5 1(s + = s − ) . To keep frontier-scale runs feasible, we use a constrained but consistent setup with two seen and two unseen datasets per run. We designate gpqa diamond [43] and livecodebenchv5 [35] as unseen, since both postdate the training cutoff of every model in our decoding experiments. For the seen datasets, we follow the designa- tions reported by Zawalski et al. [13] and use hackernews and wikipedia from the pile dataset [17] (with c2024high [18] replacing hackernews for Nemotron-4-340B- Instruct due to known corpus overlap). 5.4 I/O Isomorphism Protocol (Metamorphic Testing for Code LLMs) A coding-benchmark instance defines a mapping f : X → Y over textual I/O representations. We define a family of invertible value transforms parameterized by θ: T θ :V →V, T −1 θ T θ (v) = v. For the main experiments, we use affine integer isomorphisms: T θ (t) = at + b, a̸= 0, T −1 θ (t ′ ) = t ′ − b a . Every integer in the test I/O is mapped through T θ . Original tests (x,y) become (x ′ ,y ′ ) with x ′ = T θ (x), y ′ = T θ (y). The prompt is augmented with a contract specifying the values of a and b, along with instructions to decode inputs, solve, and return encoded outputs. We additionally ablate with two alternative bijection families, base conversion and cubic polynomials, to test whether the observed brittleness generalizes beyond affine transforms (described in Section 7.1). Correctness equivalence (the metamorphic relation). Because T θ is bijective, f (x) = y ⇐⇒ T θ f T −1 θ (x ′ ) = y ′ . Any accuracy difference between Original and Isomorphic variant (Iso) of a problem is therefore attributable to representational handling, not to a change in the underlying task. 12 5.5 Opcode Entropy as an Algorithmic Proxy Pass@k measures whether the model produces any correct solution but says nothing about what kind. To characterize the solution space beyond binary correctness, we compile each generated Python program to bytecode using dis and compute the Shannon entropy of its opcode-frequency distribution, H(π) =− X o∈O π(o) log 2 π(o), where π(o) is the normalized frequency of opcode o. Programs relying on similar algorithmic strategies produce similar opcode profiles, making entropy a lightweight proxy for the algorithmic family a solution belongs to. To quantify distributional divergence at the per-problem level, we pool the opcode frequency vectors from all 5 generations for a given (problem, condition) pair and compute the Jensen-Shannon divergence (JSD) against the corresponding pooled distribution under Original. JSD is bounded in [0, 1] (base 2) and symmetric, providing a principled measure of how far a model’s solution strategy shifts under perturbation. 6 Results Across all eleven models and three benchmarks, lexical probes (Syn) and likelihood probes (CoDeC) both saturate at scale. Our metamorphic Iso protocol drops frontier- model pass@1 by 14–30 points even though the algorithmic task is provably unchanged, yet these are not algorithmic failures. Pooled-opcode JSD between Original and Iso generations stays near zero for scaled models, which keep the same control flow and merely mis-serialize the interface, and spikes only for smaller models on harder benchmarks, which abandon the strategy outright. 6.1 Encoder-Side Probes Saturate at Scale Table 3 reports pass@1 under Syn-20 and Syn-40 across all eleven models and three benchmarks. The results show a clear saturation pattern. Synonym fuzzing remains diagnostic for small and some mid-tier models, but is nearly inert at the frontier. Frontier models absorb synonym perturbation. For frontier models, pass@1 drops remain small across benchmarks and conditions. Gemini-2.0-Flash loses at most 5.4 points, while Gemini-2.5-Flash loses even less, with a maximum drop of 2.1 points and occasional gains on BigOBench and EffiBench. GPT-4o shows the same overall pattern, with modest and consistent losses. Because these perturbations follow the synonym-fuzzing protocol of Djir ́e et al. [10], the result points to failure of lexical perturbations as a tool to study memorization, which was validated on smaller models but loses discriminative power at the frontier scale, where encoders readily collapse semantically equivalent surface forms. Mid-tier models show thresholded degradation. GPT-4o-mini, Llama-3.1-70B, and Qwen2.5-Coder-32B fall in an intermediate regime. Under Syn-20, drops are modest, but Syn-40 produces larger and benchmark-dependent losses. For example, Llama-3.1-70B drops 7.9 points on MBPP under Syn-20 but 27.3 under Syn-40, 13 Table 3 Pass@1 (%) across all conditions and models. Values in parentheses show the relative change from Original . ± denotes the half-width of a 95 % bootstrap CI (10000 resamples over problems). Tier labels: F = frontier, M = mid-tier, S = small. Dataset Model Orig Iso Iso (Enc) Iso (Dec) Syn-20 Syn-40 BigO gemini-2.5-flash F 43.2 ± 2.2 25.5 ± 4.1 (-41%) 35.4 (-18%) 30.1 (-30%) 45.1 (+4%) 46.2 (+7%) gemini-2.0-flash F 65.9 ± 2.0 50.3 ± 4.9 (-24%) 55.4 (-16%) 54.8 (-17%) 62.2 (-6%) 60.5 (-8%) gpt-4o-2024-08-06 F 50.9 ± 2.1 37.2 ± 3.1 (-27%) 46.5 (-9%) 40.1 (-21%) 49.4 (-3%) 48.0 (-6%) gpt-4o-mini-2024-07-18 M 41.5 ± 2.0 24.9 ± 4.4 (-40%) 33.8 (-19%) 27.5 (-34%) 37.4 (-10%) 33.2 (-20%) Llama-3.1-70B-Instruct M 42.6 ± 2.0 20.9 ± 4.7 (-51%) 32.4 (-24%) 27.3 (-36%) 37.5 (-12%) 31.1 (-27%) Qwen2.5-Coder-32B-Instruct M 30.1 ± 1.9 18.7 ± 2.8 (-38%) 24.4 (-19%) 22.3 (-26%) 27.7 (-8%) 26.8 (-11%) Codestral-22B-v0.1 S 22.0 ± 1.7 12.6 ± 3.9 (-43%) 17.8 (-19%) 13.9 (-37%) 17.2 (-22%) 16.1 (-27%) StarCoder2-15B S 0.6 ± 0.4 0.0 ± 0.6 (-100%) 0.3 (-50%) 0.0 (-100%) 0.3 (-50%) 0.0 (-100%) CodeLlama-13B-Instruct S 2.1 ± 0.6 1.7 ± 1.5 (-19%) 1.9 (-10%) 1.8 (-14%) 1.2 (-43%) 0.8 (-62%) Llama-3.1-8B-Instruct S 11.2 ± 1.3 3.8 ± 2.9 (-66%) 7.4 (-34%) 4.7 (-58%) 8.9 (-21%) 2.5 (-78%) deepseek-coder-6.7b-instruct S 14.5 ± 1.5 5.1 ± 3.3 (-65%) 10.2 (-30%) 6.5 (-55%) 8.3 (-43%) 4.8 (-67%) Effi gemini-2.5-flash F 26.2 ± 3.1 11.7 ± 5.1 (-55%) 21.0 (-20%) 15.4 (-41%) 27.0 (+3%) 28.1 (+7%) gemini-2.0-flash F 54.4 ± 3.1 29.3 ± 5.8 (-46%) 45.2 (-17%) 35.0 (-36%) 53.2 (-2%) 51.8 (-5%) gpt-4o-2024-08-06 F 62.2 ± 3.3 33.8 ± 6.2 (-46%) 52.1 (-16%) 40.2 (-35%) 60.5 (-3%) 58.4 (-6%) gpt-4o-mini-2024-07-18 M 48.0 ± 3.6 22.6 ± 5.3 (-53%) 36.5 (-24%) 26.4 (-45%) 43.7 (-9%) 39.4 (-18%) Llama-3.1-70B-Instruct M 36.2 ± 3.2 13.8 ± 5.0 (-62%) 23.5 (-35%) 17.7 (-51%) 32.2 (-11%) 24.6 (-32%) Qwen2.5-Coder-32B-Instruct M 39.5 ± 3.3 20.1 ± 4.0 (-49%) 30.8 (-22%) 27.3 (-31%) 35.9 (-9%) 28.4 (-28%) Codestral-22B-v0.1 S 21.3 ± 2.7 10.7 ± 4.4 (-50%) 16.2 (-24%) 12.1 (-43%) 18.9 (-11%) 7.6 (-64%) StarCoder2-15B S 3.9 ± 1.3 0.0 ± 3.1 (-100%) 1.2 (-69%) 0.3 (-92%) 0.3 (-92%) 0.6 (-85%) CodeLlama-13B-Instruct S 0.4 ± 0.4 0.0 ± 0.9 (-100%) 0.2 (-50%) 0.0 (-100%) 0.9 (+125%) 0.0 (-100%) Llama-3.1-8B-Instruct S 14.2 ± 2.2 3.7 ± 3.7 (-74%) 8.8 (-38%) 5.0 (-65%) 11.4 (-20%) 9.1 (-36%) deepseek-coder-6.7b-instruct S 9.6 ± 2.0 2.9 ± 3.1 (-70%) 6.4 (-33%) 3.8 (-60%) 7.8 (-19%) 6.2 (-35%) MBPP gemini-2.5-flash F 77.4 ± 2.0 47.2 ± 3.2 (-39%) 66.6 (-14%) 54.8 (-29%) 75.3 (-3%) 76.1 (-2%) gemini-2.0-flash F 80.8 ± 1.8 58.1 ± 4.1 (-28%) 71.1 (-12%) 59.0 (-27%) 79.2 (-2%) 76.4 (-5%) gpt-4o-2024-08-06 F 84.2 ± 1.8 58.5 ± 3.7 (-31%) 76.8 (-9%) 62.3 (-26%) 79.1 (-6%) 76.6 (-9%) gpt-4o-mini-2024-07-18 M 73.0 ± 2.1 43.8 ± 3.2 (-40%) 61.3 (-16%) 48.9 (-33%) 68.3 (-6%) 62.0 (-15%) Llama-3.1-70B-Instruct M 71.8 ± 2.1 41.4 ± 3.0 (-42%) 59.6 (-17%) 47.0 (-35%) 63.9 (-11%) 44.5 (-38%) Qwen2.5-Coder-32B-Instruct M 84.0 ± 1.8 45.4 ± 2.1 (-46%) 68.0 (-19%) 51.2 (-39%) 76.4 (-9%) 52.9 (-37%) Codestral-22B-v0.1 S 72.0 ± 2.1 44.1 ± 2.8 (-39%) 59.4 (-18%) 48.2 (-33%) 64.2 (-11%) 35.1 (-51%) StarCoder2-15B S 24.3 ± 2.0 0.8 ± 3.1 (-97%) 13.6 (-44%) 4.1 (-83%) 6.5 (-73%) 6.4 (-74%) CodeLlama-13B-Instruct S 12.8 ± 1.5 2.5 ± 2.1 (-80%) 7.8 (-39%) 3.8 (-70%) 13.2 (+3%) 4.3 (-66%) Llama-3.1-8B-Instruct S 48.2 ± 2.4 14.5 ± 3.1 (-70%) 32.4 (-33%) 19.8 (-59%) 41.2 (-15%) 23.6 (-51%) deepseek-coder-6.7b-instruct S 52.1 ± 2.4 16.7 ± 3.6 (-68%) 36.0 (-31%) 22.4 (-57%) 44.8 (-14%) 30.2 (-42%) 14 while Qwen2.5-Coder-32B shows a similar jump on MBPP. This non-linearity suggests that mid-tier encoders tolerate moderate lexical noise but degrade once perturbation density crosses a threshold. Small models degrade sharply, but the signal conflates fragility with mem- orization. Below 22B parameters, synonym fuzzing causes large drops across all benchmarks. Codestral-22B, for instance, loses 36.9 points on MBPP under Syn-40, 13.7 on EffiBench, and 5.9 on BigOBench, with similar trends for Llama-3.1-8B and deepseek-coder-6.7b. If synonym fuzzing primarily disrupted memorized retrieval, the largest effects should appear on MBPP, the only benchmark plausibly contaminated for all models, with smaller drops on EffiBench and BigOBench, which postdate their training cutoffs. The data does not support this. Relative losses on unseen bench- marks are often comparable or larger, but this is confounded by task difficulty that was untested in previous works. Small models already have low baseline accuracy on harder tasks, so even small absolute drops become large relative declines. The diagnos- tic, therefore, mixes encoder fragility with memorization and cannot cleanly separate the two. RQ1. To what extent do encoder-side memorization probes work at scale? Their diagnostic value diminishes as scale increases. Synonym fuzzing at 20% and 40% rates, following the protocol of Djir ́e et al. [10], produces <8- point pass@1 drops for frontier models across all three benchmarks, while mid- tier models degrade moderately and small models collapse. The probe retains some signal at smaller scales but appears to lose much of its discriminative power at the frontier, at least for the models and benchmarks we examine. 6.2 CoDeC Loses Discriminative Power at Scale Table 4 CoDeC contamination scores across model scale. Seen and Unseen report the mean contamination score for datasets designated as in or absent from the training corpus. AUC quantifies the separation between the two. ModelParams Seen (%) Unseen (%) AUC (%) Pythia-410M0.4B71.024.0100.0 Pythia-1.4B1.4B73.521.5100.0 Pythia-12B12B75.017.0100.0 Llama-3.1-70B70B8.09.025.0 davinci-002—43.531.075.0 Nemotron-4-340B340B26.022.575.0 Llama-3.1-405B405B7.52.075.0 Table 4 and Figure 4 show CoDeC contamination scores across model scale. Our unseen labels (gpqadiamond, livecodebenchv5) are guaranteed, since both datasets postdate the training cutoffs of all models in our decoder-side experiments. The seen 15 Pythia-410M Pythia-1.4B Pythia-12B Llama-3.1-70B davinci-002 Nemotron-340B Llama-3.1-405B 0 20 40 60 80 100 CoDeC Score (%) Seen vs Unseen Contamination Scores Seen Unseen (a) Pythia-410M Pythia-1.4B Pythia-12B Llama-3.1-70B davinci-002 Nemotron-340B Llama-3.1-405B 0 20 40 60 80 100 AUC (%) 100.0%100.0%100.0% 25.0% 75.0%75.0%75.0% Classification AUC (Seen vs Unseen) (b) Fig. 4 (a) Mean CoDeC scores across 4 datasets designated as Seen or Unseen (Section 5.3). (b) AUC quantifying the separation between seen and unseen scores across 4 datasets. 16 labels (hackernews, wikipedia, and c2024high for Nemotron) follow prior corpus- overlap analyses and documented training mixtures [13, 17, 44]. Although the seen status is therefore probable rather than certain, this asymmetry is shared across all models and does not affect the trend. Our reproduction matches the original CoDeC behaviour on Pythia: seen scores lie between 71–75%, unseen scores between 17–24%, and AUC remains 100% across all three checkpoints. The gap reduces with scale. As model size increases, both contamination scores and seen-unseen separation shrink (Figure 4a). davinci-002 narrows the gap to 12.5 points, Nemotron-4-340B to 3.5, and Llama-3.1-405B to 5.5. Llama-3.1-70B even inverts the expected ordering, yielding an AUC of 25%, though this is likely an anomaly given the small number of datasets. Even ignoring that inversion, seen-dataset scores drop from 71–75% at the Pythia scale to 7.5% at 405B, showing that in-context augmentation shifts token log-likelihoods much less in larger models. Why does the signal degrade? CoDeC measures how in-context examples shift token log-likelihoods. Memorized targets should be disrupted, while novel targets should benefit. This assumes memorized content occupies a concentrated region of probability space that context can perturb. As models scale, that signal weakens for two reasons. First, probability mass is spread across a richer representation space, diluting concentrated retrieval effects. Second, larger models have stronger in-context learning abilities [39, 40], allowing them to use added context constructively for both seen and unseen targets, which reduces the asymmetry CoDeC relies on. RQ2. To what extent do decoder-side memorization probes work at scale? Their discriminative power appears to weaken with scale. CoDeC’s seen-unseen gap shrinks from 58 points (Pythia-12B) to 5.5 points (Llama- 3.1-405B), and AUC drops from 100% to 75%. This trend is consistent with the dilution of localized retrieval signals in richer representation spaces and with the strengthening of in-context learning abilities that may override the memorization-specific asymmetry on which CoDeC depends. 6.3 Representational Load Degrades Performance Without Erasing the Algorithm To answer RQ3 we break it into two smaller sub-questions that can each be tested on their own. A pass@1 drop alone does not pin down the extent to which representational load breaks performance, because the same drop appears both when a model loses the algorithm under the unfamiliar interface and when it keeps the algorithm but mis- serializes the output. RQ3a isolates where in the model the degradation arises, and RQ3b establishes what kind of failure it is. 6.3.1 A Decoder-Side Bottleneck Drives the Degradation Table 3 reports pass@1 under all evaluation conditions. The full Iso contract drops frontier-model pass@1 by 14–30 absolute points on a provably fixed task, which on its own shows that representational load is sufficient to cause substantial degradation. 17 To localize the source, we decompose the contract into Iso (Enc only), which trans- forms only the inputs the model must decode, and Iso (Dec only), which transforms only the outputs the model must encode. For Gemini-2.0-Flash on MBPP, decod- ing inputs alone (Iso (Enc only)) costs a modest 9.7 points (80.8→71.1), whereas encoding outputs token by token (Iso (Dec only)) costs a far larger 21.8 points (80.8→59.0), close to the 22.7-point penalty of the full Iso condition (80.8→58.1). This split holds across datasets and across scaled models. The bottleneck is output- side contract compliance rather than input recognition, consistent with autoregressive generation struggling more with formatting constraints held over many decoding steps than the encoder does with parsing a transformed prompt once. RQ3a. Where does the degradation from representational load originate? Primarily on the decoder side. I/O isomorphisms drop pass@1 by 14–30 absolute points in frontier models, a vulnerability that purely lexical probes such as synonym fuzzing fail to surface. Decomposing the contract shows that most of this penalty comes from output-side encoding rather than input-side decoding, pointing to contract compliance during generation as the dominant cost. 6.3.2 Opcode Analysis Separates Compliance from Competence A pass@1 drop does not say whether the model has lost the algorithm or has kept it and only mis-serialized the interface. We read this distinction off the opcode-entropy analysis, treating a near-zero shift in the opcode distribution as evidence that the underlying strategy is preserved and a large shift as evidence that it is abandoned. Scaled models preserve core algorithms and control flow across all repre- sentational loads. For frontier models such as Gemini-2.0-Flash and GPT-4o, the per-problem JSD over all solutions, not just passing ones, under Iso and Iso (Dec only) stays near zero across all three benchmarks (Figure 6). The bytecode footprint under the metamorphic prompt is close to indistinguishable from that under the orig- inal prompt, and the model opts for the same loops, comparisons, and control flow on the exact problems where pass@1 has just dropped by 14–30 points. The combined distributions in Figure 5 track closely between Original and Iso, with the model selecting the same algorithmic families and producing near-identical opcode profiles while fumbling the final output representation more often. As a calibration check, JSD under Syn-20 and Syn-40 also stays near zero for all models, which indicates that the metric flags preserved core logic rather than any prompt change. Smaller models lose algorithmic stability as task complexity grows. The opcode distributions reveal a contrast between smaller and mid-tier models, with the latter heavily dependent on dataset difficulty. On easier datasets such as MBPP, mod- els like Codestral-22B behave much like scaled models and keep JSD relatively low under Iso. On harder datasets such as BigOBench and EffiBench, however, smaller models buckle under the representational load and their JSD spikes (Figure 6). This suggests that when the underlying task already sits at the boundary of their capabil- ity, the added interface load pushes them to abandon the correct strategy and diverge 18 to qualitatively different and often incorrect solution families, or to fail to produce valid logic at all. Scale expands algorithmic coverage, not just reliability. Comparing the Orig- inal opcode profiles across models reveals a further dimension of scaling. On MBPP, Gemini-2.0-Flash and Codestral-22B produce overlapping entropy distributions, and the algorithmic repertoire required for basic tasks is within reach of both. On EffiBench and BigOBench the distributions diverge before any perturbation is applied (Figures 5c and 5d). Gemini-2.0-Flash spans two peaks and performs better at pass@1, while the smaller Codestral-22B largely skips the second peak, consistent with a family of solutions present in the larger model’s repertoire but absent from the smaller one. RQ3b. Is the degradation a competence failure or a compliance failure? A compliance failure for scaled models and a competence failure for smaller ones. Despite large pass@1 drops, frontier models show near-zero opcode JSD under isomorphism, which indicates they attempt the same algo- rithmic core and mainly fumble the representational channel. Smaller models, by contrast, abandon their strategies under the same load on complex tasks, a pattern more consistent with shallow and fragile task comprehension than with mere mis-serialization. RQ3. To what extent can representational load break performance? Sharply, but in scaled models without erasing the algorithm. The Iso contract drops frontier-model pass@1 by 14–30 absolute points on a prov- ably fixed task, so the loss is the load rather than a memorization artifact. The penalty is concentrated on the decoder, where the model must serialize an encoded output token by token (RQ3a), and for scaled models the algorithmic core survives, surfacing as a narrowed rather than a closed route from specifi- cation to code (RQ3b). The competence failure that a bare pass@1 drop might suggest appears only in smaller models on tasks already near their capability limit. 19 2.02.53.03.54.04.5 Shannon Entropy (bits) 0 2 4 6 8 10 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 2.02.53.03.54.04.5 Shannon Entropy (bits) 0 1 2 3 4 5 6 7 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (a) Gemini-2.0-Flash:BigOBench 2.02.53.03.54.04.55.0 Shannon Entropy (bits) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 1.52.02.53.03.54.04.55.0 Shannon Entropy (bits) 0.0 0.2 0.4 0.6 0.8 1.0 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (b) Codestral-22B:BigOBench 2.42.62.83.03.23.43.63.8 Shannon Entropy (bits) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 2.42.62.83.03.23.43.63.8 Shannon Entropy (bits) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (c) Gemini-2.0-Flash:Effibench 2.42.62.83.03.23.43.63.8 Shannon Entropy (bits) 0 2 4 6 8 10 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 01234 Shannon Entropy (bits) 0 2 4 6 8 10 12 14 16 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (d) Codestral-22B:Effibench 2.02.53.03.54.0 Shannon Entropy (bits) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 1.52.02.53.03.54.0 Shannon Entropy (bits) 0 2 4 6 8 10 12 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (e) Gemini-2.0-Flash:MBPP 1.52.02.53.03.54.0 Shannon Entropy (bits) 0 5 10 15 20 25 Density Working (Pass) Original ISO ISO (dec only) Syn-20% Syn-40% 1.52.02.53.03.54.04.55.0 Shannon Entropy (bits) 0 2 4 6 8 10 Density Combined (All) Original ISO ISO (dec only) Syn-20% Syn-40% (f) Codestral-22B:MBPP Fig. 5 Static opcode-entropy distributions (KDE) for selected model-dataset pairs. The left column shows Gemini-2.0-flash producing nearly identical entropy profiles under Original and Iso, both for working solutions and for all generated code. The right column shows that Codestral-22b diverges substantially or produces no working solutions under Iso on harder datasets. 20 ISOISO (dec only)Syn-20%Syn-40% 0.0 0.2 0.4 0.6 0.8 1.0 JSD(Original Condition) Gemini-2.0-Flash ISOISO (dec only)Syn-20%Syn-40% Gemini-2.5-Flash ISOISO (dec only)Syn-20%Syn-40% Codestral-22B ISOISO (dec only)Syn-20%Syn-40% StarCoder2-15B (a) BigOBench ISOISO (dec only)Syn-20%Syn-40% 0.0 0.2 0.4 0.6 0.8 1.0 JSD(Original Condition) Gemini-2.0-Flash ISOISO (dec only)Syn-20%Syn-40% Gemini-2.5-Flash ISOISO (dec only)Syn-20%Syn-40% Codestral-22B ISOISO (dec only)Syn-20%Syn-40% StarCoder2-15B (b) EffiBench ISOISO (dec only)Syn-20%Syn-40% 0.0 0.2 0.4 0.6 0.8 1.0 JSD(Original Condition) Gemini-2.0-Flash ISOISO (dec only)Syn-20%Syn-40% Gemini-2.5-Flash ISOISO (dec only)Syn-20%Syn-40% Codestral-22B ISOISO (dec only)Syn-20%Syn-40% StarCoder2-15B (c) MBPP Fig. 6 Per-problem Jensen-Shannon divergence of pooled opcode distributions (Condition vs. Orig- inal). 21 7 Ablation Studies We conduct two ablation studies, one to test whether the observed brittleness generalizes beyond affine transforms, and one to control for prompt-length confounds. 7.1 Alternative Isomorphism Families Table 5 Pass@1 (%) under alternative isomorphism families. Data ModelOrigAffineBaseCubic BigO gemini-2.0-flash65.9 50.3 (-24%) 52.2 (-21%) 52.8 (-20%) Codestral-22B-v0.122.0 12.6 (-43%) 13.4 (-39%) 11.7 (-47%) Qwen2.5-Coder-32B-Instruct30.1 18.7 (-38%) 19.8 (-34%) 17.5 (-42%) Effi gemini-2.0-flash54.4 29.3 (-46%) 26.9 (-51%) 37.8 (-31%) Codestral-22B-v0.121.3 10.7 (-50%) 12.5 (-41%)9.8 (-54%) Qwen2.5-Coder-32B-Instruct39.5 20.1 (-49%) 22.5 (-43%) 18.0 (-54%) MBPP gemini-2.0-flash80.8 58.1 (-28%) 52.6 (-35%) 60.5 (-25%) Codestral-22B-v0.172.0 44.1 (-39%) 49.3 (-32%) 43.2 (-40%) Qwen2.5-Coder-32B-Instruct84.0 45.4 (-46%) 50.5 (-40%) 43.0 (-49%) To test whether the brittleness generalizes beyond affine transforms, we ablate with two alternative bijection families targeting distinct representational axes. 1. Base conversion: The decimal digit string of each integer is treated as if it were written in base b, yielding a different numeric value. For example, with b=7 the integer 25 has digit string "25", which read in base 7 gives 2× 7 + 5 = 19, so T (25) = 19. The inverse recovers the original value by inverting the conversion. The base-7 representation of 19 is "25", whose digits read in base 10 give 25. This replaces the arithmetic overhead of affine inversion with a format-parsing overhead, testing whether the model can reason about numeral systems rather than polynomial algebra. 2. Cubic polynomials: x ′ = x 3 + c, requiring explicit cube-root inversion, a qualitatively harder challenge than a linear inverse. We run three models across all datasets as shown in Table 5. All three families produce drops of comparable magnitude. On BigOBench, Gemini-2.0-Flash loses 21% (base) and 20% (cubic), closely tracking the 24% affine drop. On EffiBench, base conversion is steeper than affine (-51% vs. -46%), while the cubic transform is milder (-31%). On MBPP, both alternatives again bracket the affine result. The consistency across families confirms that the brittleness reflects general value-space sensitivity rather than an artifact of any particular transform. 22 Table 6 Pass@1 (%) under dead-code insertion controlling for the prompt-length confound. Dead-code blocks match the token-count increase of the Iso contract while leaving the task and I/O unchanged. Data ModelOrigIso Dead Code BigO gemini-2.0-flash65.9 50.3 (-24%)65.1 (-1%) Codestral-22B-v0.122.0 12.6 (-43%)20.5 (-7%) Llama-3.1-70B-Instruct42.6 20.9 (-51%)41.0 (-4%) MBPP gemini-2.0-flash80.8 58.1 (-28%)81.4 (+1%) Codestral-22B-v0.172.0 44.1 (-39%)68.5 (-5%) Llama-3.1-70B-Instruct71.8 41.4 (-42%)69.5 (-3%) 7.2 Prompt Length as a Confound The Iso condition appends an encoding contract to the prompt, increasing its length. To rule out prompt length as a confound, we insert semantically irrelevant Python code blocks that match the token-count increase of the Iso contract while leaving the task and I/O unchanged. If length alone drives the observed drops, dead-code insertion should produce comparable degradation. Table 6 shows it does not. Dead- code insertion causes negligible drops of 1 to 7% relative, while Iso degrades the same models by 24 to 51% on the same benchmarks. The gap is consistent across all model-dataset pairs. On BigOBench, gemini-2.0-flash loses 0.8 points with dead code versus 15.6 points under Iso, and Llama-3.1-70B loses 1.6 points versus 21.7. On MBPP, Gemini-2.0 under dead code actually gains 0.6 points (+1%), a fluctuation we attribute to noise. In every case, the Iso drop exceeds the dead-code drop by an order of magnitude, confirming that the performance degradation is driven by the representational load of contract compliance, not by prompt length. 8 Discussion 8.1 Why the Decoder-Side Bottleneck May Be Architectural Our decomposition experiments indicate that decoding accounts for most of the isomorphism penalty (Section 6.3.1). A plausible reading of this is the asymmetry between prompt understanding and generation. On the encoder side, the model can integrate the contract and task into a coherent representation before producing any output [45], so it can attend to the whole prompt at once. Decoding, by contrast, is sequential, and each token depends only on the prompt and prior outputs [46]. To satisfy the encoding contract, the model has to preserve an implicit arithmetic state across many steps while also maintaining correct Python syntax and contract com- pliance, and this kind of constrained multi-step generation plausibly stresses working memory in ways that standard next-token training does not explicitly support [47]. The same asymmetry may also help explain why synonym fuzzing saturates at scale (Section 6.1), since lexical variation is resolved in the context-rich regime where scaled representations tend to collapse equivalent forms [48], whereas isomorphism shifts the 23 load onto the decoder, which must serialize a novel computation token by token. If this account is correct, the bottleneck is closer to architectural than incidental and might persist in autoregressive models even as scale narrows the gap, though our experiments speak to the size of the effect rather than to its mechanism directly. 8.2 Representational Load as a Practical Test of Generalization The term “generalization” is highly ambiguous in the code LLM literature. At one extreme, it might require generating genuinely novel algorithms, that is, solutions that do not resemble any training instance in structure or strategy. Our work does not study this aspect of generalization, though some benchmarks, such as ARC-AGI [49, 50], do. At the other extreme, generalization can simply mean producing correct outputs on unseen inputs, which is trivially satisfied by any model that passes a held-out test set (qualitatively different from its training data). We study a point between these, generalization under a representational load the model could not have seen, and our results suggest that scaled models have internalized algorithmic strategies well enough to transport them across interfaces they have never encountered. This looks closer to what Chollet [51] terms developer-aware generalization, the ability to handle situations not anticipated by the system’s designers but lying within the span of its learned priors. We read this as a sign that the field’s methods for studying generalization may not have kept pace with the models, and it points toward evaluation frameworks that test whether a model can apply what it knows under novel conditions rather than only reproduce what it has seen. 8.3 Scope of the Metamorphic Claim The biconditional that gives our protocol its sharpness relies on bijective value trans- forms over numeric test inputs and outputs. Our findings, therefore, characterize code-LLM behavior specifically on numeric programming tasks, and what they indi- cate there is fairly narrow, that under representational stress, scaled models appear to retain the algorithmic core and mostly fail at re-serializing the encoded interface. Other classes of representational load, such as novel string-formatting contracts, alter- nate data layouts, or project-local API conventions, are equally important in practice, but the strict metamorphic-testing guarantee does not directly transfer, because con- structing a closed bijection on non-numeric input spaces is itself an open problem. Designing metamorphic-style oracles for those loads while preserving the same prov- able equivalence on the underlying task is a natural extension we leave for future work. 8.4 Solution Multiplicity Complicates the Memorization Narrative Our opcode analysis indicates that scaled models, particularly on BigOBench and EffiBench, span a broad, multi-modal entropy distribution under Original conditions (Section 6.3.2). We interpret these modes as distinct algorithmic families. A sorting problem might admit O(n logn) divide and conquer, O(n 2 ) comparison-based, and O(n) counting-based solutions, each with a characteristic opcode signature. Smaller 24 models, by contrast, tend toward narrow, often unimodal distributions, frequently col- lapsing to a single solution strategy per problem. If this reading holds, the multiplicity has a bearing on how memorization is interpreted, since methodologies targeting the decoder may be confounded when a model can stochastically switch between several “correct” solutions. 9 Threats to Validity Construct validity. Static opcode entropy is a proxy for algorithmic strategy and not a direct measurement of it. Two programs with the same opcode histogram can differ in runtime behavior, and two behaviourally identical programs can compile to different bytecode, for instance, a list comprehension against an explicit loop. The Jensen-Shannon divergence we report is invariant to opcode ordering, so it tracks the compositional makeup of a solution rather than its control-flow structure, and a model could, in principle, reorganize its control flow while leaving the histogram unchanged. We mitigate these effects by reporting divergence over both passing and all generations and by reading the metric only as relative movement between matched conditions rather than as an absolute fingerprint. A second construct concern is that representational load is observable only through the transforms we instantiate, so the affine, base-conversion, and cubic families stand in for a property we cannot measure directly. Internal validity. Iso lengthens the prompt, and our dead-code ablation (Section 7.2) bounds the share of degradation attributable to length alone at roughly 7%. We decode greedily at T =0.0 to strip sampling variance, which collapses pass@5 toward pass@1 and may understate the headroom a frontier model would show under sampling. The exact wording of the encode-decode contract is a further free variable, since a dif- ferently phrased instruction could prime the decoder more or less effectively, and we hold the phrasing fixed rather than searching over it. For the closed-source models, we query hosted endpoints whose system prompts and checkpoint versions are not dis- closed, so a silent provider-side change would surface as a behavioral shift we could not attribute. Finally, the Seen labels in the CoDeC probe are probable rather than certain, as noted in Section 5.3, though the asymmetry is shared across every model and does not move the trend. External validity. Our isomorphisms cover three integer-valued bijection families, all of our prompts are Python, and the metamorphic guarantee holds only over numeric input and output. Whether the same narrow-channel behavior appears under string- formatting contracts, structured data layouts, or project-local API conventions is untested, and constructing a closed bijection on those spaces is itself an open problem, as we discuss above. Our model panel is dense throughout and excludes mixture-of- experts architectures, where routing may interact with memorization in ways a dense model does not exhibit. The three benchmarks also favor self-contained algorithmic tasks, so our results speak to function-level synthesis rather than repository-scale or agentic coding. Broader transform families, additional languages, MoE models, and larger task units remain future work. 25 10 Conclusion We do not claim that memorization is absent from scaled code LLMs, only that the diagnostics commonly used to detect it appear to stop discriminating at scale. In our experiments, synonym fuzzing and dead-code insertion on the encoder side, and CoDeC on the decoder side, lose much of their signal on frontier models, so a pass@1 drop or a likelihood shift is, on its own, weak evidence of memorization. Holding the algorithmic task fixed under an I/O isomorphism lets us separate representational load from memorization, and scaled models tend to keep the same solution families while mis-serializing the encoded interface. The takeaway we would offer is practical. Treat a bare pass@k drop as a start- ing point rather than a memorization finding, separate representational load before attributing any drop to recall, and report solution-space stability alongside correct- ness. For most software engineering settings the more useful question is not whether a solution was seen in training but whether a model can carry a known algorithm through an unfamiliar interface. And if memorization is to be studied in the stronger sense the term usually implies, we would suggest approaching it from a different van- tage point than the surface and likelihood probes inherited from prior work, which give little to build on at current scales. Declarations Funding. This research was funded by the Luxembourg National Research Fund (FNR) in collaboration with the University of Luxembourg. Competing interests. The authors have no competing interests to declare that are relevant to the content of this article. Data availability. The dataset generated and analysed in this study is openly available in Zenodo at https://doi.org/10.5281/zenodo.19248561. Code availability. The code is openly available at https://github.com/pkrajput/ isomorphic prompts. Author contributions. Prateek Kumar Rajput framed the methodology and wrote the manuscript. Alberick Euraste Djir ́e and Abdoul Aziz Bonkoungou helped run the experiments. Yewei Song and Iyiola Emmanuel Olatunji contributed to reviewing and framing the manuscript. Jacques Klein and Tegawend ́e F. Bissyand ́e served as project advisors and principal investigators. All authors read and approved the final manuscript. References [1] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021) [2] Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021) 26 [3] Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.: Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023) [4] Zhang, L., He, S., Zhang, C., Kang, Y., Li, B., Xie, C., Wang, J., Wang, M., Huang, Y., Fu, S., et al.: Swe-bench goes live! arXiv preprint arXiv:2505.23419 (2025) [5] Liang, S., Garg, S., Moghaddam, R.Z.: The swe-bench illusion: When state-of- the-art llms remember instead of reason. arXiv preprint arXiv:2506.12286 (2025) [6] Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P.T., Ray, B.: Natgen: generative pre-training by “naturalizing” source code. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 18–30 (2022) [7] Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al.: Recode: Robustness evaluation of code genera- tion models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13818–13843 (2023) [8] Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., Bavota, G.: On the robustness of code generation techniques: An empirical study on github copilot. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 2149–2160 (2023). IEEE [9] Chen, J., Zhenhao, L., Xing, H., Xin, X.: Nlperturbator: Studying the robust- ness of code llms to natural language variations. ACM Transactions on Software Engineering and Methodology (2024) [10] Djir ́e, A.E., Kabor ́e, A.K., Barr, E.T., Klein, J., Bissyand ́e, T.F.: Memorization or interpolation? detecting llm memorization through input perturbation analysis. arXiv preprint arXiv:2505.03019 (2025) [11] Euraste, D.A., Kader, K.A., Samhi, J., Barr, E.T., Klein, J., Bissyand ́e, T.F.: Learned or memorized? quantifying memorization advantage in code llms. arXiv preprint arXiv:2604.13997 (2026) [12] Riddell, M., Ni, A., Cohan, A.: Quantifying contamination in evaluating code generation capabilities of language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14116–14137 (2024) [13] Zawalski, M., Boubdir, M., Ba lazy, K., Nushi, B., Ribalta, P.: Detecting data contamination in llms via in-context learning. arXiv preprint arXiv:2510.27055 (2025) 27 [14] Oren, Y., Meister, N., Chatterji, N.S., Ladhak, F., Hashimoto, T.: Proving test set contamination in black-box language models. In: The Twelfth International Conference on Learning Representations (2023) [15] Golchin, S., Surdeanu, M.: Data contamination quiz: A tool to detect and esti- mate contamination in large language models. Transactions of the Association for Computational Linguistics 13, 809–830 (2025) [16] Nie, Y., Wang, C., Wang, K., Xu, G., Xu, G., Wang, H.: Decoding secret mem- orization in code llms through token-level characterization. In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), p. 2880–2892 (2025). IEEE [17] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al.: The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020) [18] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21(140), 1–67 (2020) [19] Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology (1998) [20] Segura, S., Fraser, G., Sanchez, A.B., Ruiz-Cort ́es, A.: A survey on metamorphic testing. IEEE Transactions on Software Engineering 42(9), 805–824 (2016) [21] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al.: Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21), p. 2633–2650 (2021) [22] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., Carlini, N.: Deduplicating training data makes language models better. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8424–8445 (2022) [23] Coignion, T., Quinton, C., Rouvoy, R.: A performance study of llm-generated code on leetcode. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, p. 79–89 (2024) [24] Matton, A., Sherborne, T., Aumiller, D., Tommasone, E., Alizadeh, M., He, J., Ma, R., Voisin, M., Gilsenan-McMahon, E., Gall ́e, M.: On leakage of code gen- eration evaluation datasets. In: Findings of the Association for Computational Linguistics: EMNLP 2024, p. 13215–13223 (2024) 28 [25] Sainz, O., Campos, J., Garc ́ıa-Ferrero, I., Etxaniz, J., Lacalle, O.L., Agirre, E.: Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In: Findings of the Association for Computational Linguistics: EMNLP 2023, p. 10776–10787 (2023) [26] Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., Zettle- moyer, L.: Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789 (2023) [27] Mattern, J., Mireshghallah, F., Jin, Z., Sch ̈olkopf, B., Sachan, M., Berg- Kirkpatrick, T.: Membership inference attacks against language models via neighbourhood comparison. In: Findings of the Association for Computational Linguistics: ACL 2023, p. 11330–11343 (2023) [28] Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tram`er, F., Carlini, N.: Counter- factual memorization in neural language models. Advances in Neural Information Processing Systems 36, 39321–39362 (2023) [29] Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al.: Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296 (2023) [30] Biderman, S., Prashanth, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., Raff, E.: Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems 36, 28072–28090 (2023) [31] Power, A., Burda, Y., Edwards, H., Babuschkin, I., Misra, V.: Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 (2022) [32] Nanda, N., Chan, L., Lieberum, T., Smith, J., Steinhardt, J.: Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 (2023) [33] Liu, Z., Kitouni, O., Nolte, N.S., Michaud, E., Tegmark, M., Williams, M.: Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems 35, 34651–34663 (2022) [34] Tirumala, K., Markosyan, A., Zettlemoyer, L., Aghajanyan, A.: Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems 35, 38274–38290 (2022) [35] Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., Stoica, I.: Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 (2024) [36] Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt 29 really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, 21558–21572 (2023) [37] Xia, C.S., Deng, Y., Zhang, L.: Top leaderboard ranking= top coding profi- ciency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint arXiv:2403.19114 (2024) [38] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020) [39] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022) [40] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) [41] Huang, D., Qing, Y., Shang, W., Cui, H., Zhang, J.M.: Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems 37, 11506–11544 (2024) [42] Chambon, P., Roziere, B., Sagot, B., Synnaeve, G.: Bigo (bench)–can llms generate code with controlled time and space complexity? arXiv preprint arXiv:2503.15242 (2025) [43] Rein, D., Hou, B.L., Stickland, A.C., Petty, J., Pang, R.Y., Dirani, J., Michael, J., Bowman, S.R.: Gpqa: A graduate-level google-proof q&a benchmark. In: First Conference on Language Modeling (2024) [44] Biderman, S., Schoelkopf, H., Anthony, Q.G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M.A., Purohit, S., Prashanth, U.S., Raff, E., et al.: Pythia: A suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, p. 2397–2430 (2023). PMLR [45] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) [46] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019) [47] Dziri, N., Lu, X., Sclar, M., Li, X.L., Jiang, L., Lin, B.Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al.: Faith and fate: Limits of transformers on 30 compositionality. Advances in neural information processing systems 36, 70293– 70332 (2023) [48] Ethayarajh, K.: How contextual are contextualized word representations? com- paring the geometry of bert, elmo, and gpt-2 embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 55–65 (2019) [49] Chollet, F., Knoop, M., Kamradt, G., Landers, B.: Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604 (2024) [50] Chollet, F., Knoop, M., Kamradt, G., Landers, B., Pinkard, H.: Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831 (2025) [51] Chollet, F.: On the measure of intelligence. arXiv preprint arXiv:1911.01547 (2019) 31