Paper deep dive

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, Ravi Mangal, Gagandeep Singh, Corina Pasareanu

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 92

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/20/2026, 4:28:33 AM

Summary

This paper evaluates the robustness of AI code agents against semantics-preserving transformations (SPTs). The authors introduce a random variant sampler to generate perturbed codebases and test two agentic scaffolds (mini-SWE agent, OpenCode) backed by four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B) on SWE-bench Verified and Pro. Results show small but statistically significant degradation in resolve rates (up to 6.7 percentage points) and increased effort. Crucially, model robustness rankings are not consistent across scaffolds, revealing a 'jagged frontier' where robustness is a joint property of the model, scaffold, and workload.

Entities (10)

Claude Opus 4.5 · llm-model · 99%Kimi K2.5 · llm-model · 99%MiniMax-M2.5 · llm-model · 99%Qwen 3.6-27B · llm-model · 99%mini-SWE agent · agentic-scaffold · 98%OpenCode · agentic-scaffold · 98%SWE-bench Pro · benchmark · 97%SWE-bench Verified · benchmark · 97%Semantics-Preserving Transformations · methodology · 95%Random Variant Sampler · tool · 92%

Relation Signals (6)

Semantics-Preserving Transformations → causesdegradationin → resolve_rate

confidence 95% · up to 6.7 percentage points mean resolve-rate drop in the most affected configurations

Random Variant Sampler → generates → perturbed_variants

confidence 95% · We introduce a random variant sampler that applies common semantics-preserving transformations... to produce perturbed variants.

Semantics-Preserving Transformations → increases → agent_effort

confidence 90% · perturbations raise the effort an agent expends (increasing step counts and token cost by up to 9.9% and 22.9% respectively)

mini-SWE agent → ismorerobustthan → OpenCode

confidence 90% · The simpler scaffold (mini-SWE agent) is more robust to perturbation.

Qwen 3.6-27B → ismostbrittleunder → OpenCode

confidence 85% · yet the most brittle under OpenCode

Qwen 3.6-27B → ismostrobustunder → mini-SWE agent

confidence 85% · Qwen is among the most robust under mini-SWE agent on SWE-bench Verified

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

88,280 characters extracted from source content.

Expand or collapse full text

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations Hasan Najib Mahmud Shreya Gupta Isha Chaudhary Nathaniel Enis Ravi Mangal Gagandeep Singh Corina Pasareanu Abstract AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs)—spanning control-flow rewrites, dead-code injection, and identifier renaming—to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds—Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode—revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases. Code — https://github.com/CSU-TrustLab/jagged-frontier Introduction In the span of a few years, AI-powered coding tools have moved from research prototypes into mainstream software development, with developer surveys reporting that a majority of professionals now use them regularly (31; 11; 14; 12). Their progress is charted on challenging coding benchmarks such as SWE-bench Verified (24) and SWE-bench Pro (7), against which nearly every new model and agent release is measured. While performance numbers on these benchmarks can be instructive, they do not account for the phenomenon that neural models, in general, are known to learn shortcuts (10) and therefore be susceptible to small input perturbations. Two concerns follow. First, if agent behavior shifts with superficial changes to the surrounding code, then benchmark numbers may overstate deployment reliability. Second, such shifts would suggest that the underlying models rely on shallow syntactic patterns rather than an understanding of program semantics. Prior work has evaluated the robustness of code LLMs to semantics-preserving perturbations in single-turn, non-agentic settings (38; 28; 35). However, the robustness of repository-level code agents, which interact with the LLM over many turns, has received little systematic study. We present, to our knowledge, the first such evaluation, and find that robustness is not a fixed property of a model but a jagged frontier: a model that holds up under one scaffold or codebase can be the most brittle under another, so benchmark rankings do not transfer to deployment. For robustness evaluation, we compare agent performance on unperturbed code repositories with semantically equivalent perturbed variants. The perturbations are injected via local, semantics-preserving transformations (SPTs) of the code. We develop a library of SPTs that either mimic refactorings a developer may routinely perform or are intended to stress-test the agent. We use the library to build a randomized algorithm, i.e., a sampler, that draws perturbed variants of a base repository. Our sampler randomly chooses which SPTs to apply and where to apply them. Each variant is constructed independently, and unlike algorithms that aim to find adversarial examples (30), the sampling process is not guided by feedback computed from model outcomes. This non-adversarial design is intentional. Our aim, in this work, is to compute lower bounds for the impact of SPTs on code agents. A feedback-guided adversary can only do more damage. These bounds can inform developer choices about whether and which code agents to use in deployment. They can also help guide engineers training models and building agentic scaffolds towards robustness-enhancing designs. The stochastic nature of LLM-based agents makes the robustness evaluation challenging. If an agent has a failing run on a perturbed variant but a passing run on the unperturbed repository, this observation is not sufficient to conclude that the perturbation was the cause; the observation might simply be an artifact of agent stochasticity. Experiment design is further constrained by the cost of each agentic run. Accordingly, we design the experimental methodology to isolate the perturbation effect from intrinsic run-to-run variability while balancing the statistical validity of our claims against the cost of the experiments. We do this by running the agent repeatedly on both the unperturbed seed and its variants and pairing the outcomes. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across a total of 54 instances drawn from SWE-bench Verified and SWE-bench Pro. Throughout, we study repository-level issue resolution: each task instance pairs a repository at a base commit with an issue description, the tests an accepted patch must satisfy, and a gold patch, the reference edit that resolves the issue. We measure the degradation in issue resolve rates across unperturbed and perturbed versions of a repository, as well as the change in step count and the cost of an agent run. Our experiments reveal the following insights: (1) localized perturbations applied in a simple, non-feedback-guided manner can cause small degradations in most configurations—up to 6.7 percentage points in the most affected configurations and statistically significant degradations in 6 of 16 configurations; (2) perturbations raise the effort an agent expends (increasing step counts and token cost by up to 9.9% and 22.9% respectively even in configurations where resolve rate is left largely unchanged), so an outcome-only view understates their impact; (3) comparing models within a scaffold, those such as Claude Opus 4.5 with higher capability on unperturbed versions can be impacted more drastically by perturbations than lower-capability models, whereas comparing across scaffolds, the simpler scaffold tends to be the more robust one; (4) the agentic scaffold and the benchmark shift the robustness ranking of models, so that Qwen, for instance, is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode; and (5) the impact of the perturbations varies dramatically across different repositories, concentrating in a small set of instances while leaving others untouched. Together these observations indicate that robustness is a joint property of the model, the scaffold, and the workload, not of the model alone, revealing a jagged robustness frontier. Semantics-Preserving Transformations Definition A transformation T is semantics-preserving if the transformed program T⁡(P)T(P) produces the same observable behavior as the original program P on every possible program input. Concretely, under any input, P and T⁡(P)T(P) must (i) return the same value or raise the same exception, and (i) produce the same externally observable effects. If either halts, so must the other. In the context of evaluating coding agents, we operationalize this definition through functional test-suite equivalence. Specifically, a transformation is considered semantics-preserving if T⁡(P)T(P) yields the same per-test outcome as P across a project’s test-suite. Transformation Catalog Table 1 lists the 14 semantics-preserving transformations (SPTs) we implement. We represent this catalog using the notation T. The catalog includes a mixture of SPTs that either mimic refactorings a developer might routinely perform (e.g., reordering commutative operands, swap if and else branches) or are intended to stress-test the agent by introducing unnatural or behaviorally inert code fragments (e.g., splitting string literals, dead code/method injection). Each SPT is specified by a structural pattern that needs to match for the SPT to be applicable and by a rewrite rule. We call the locations in a file where an SPT’s pattern matches its candidate sites. Two SPTs, namely Dead String Assignment and Dead Method Injection, must be bound to a target keyword, either a string literal or a method name, before rewriting. Given such a keyword, the former injects an unread assignment of that string and the latter appends a dead method of that name. Both plant a decoy: dead code that contains the keyword, so it surfaces whenever the agent searches the repository for that term. An agent that localizes by keyword must tell the decoy apart from genuine sites of interest. We write kw⊂T_kw for this keyword-bound subset of the catalog T. Full implementation details of SPTs are provided in Appendix A. Validation We validate that our SPTs are semantics-preserving empirically through differential testing (19) against the test suites of three projects drawn from our experimental benchmark, SWE-bench (16), spanning distinct domains: SymPy (symbolic mathematics; 12,994 tests), sqlfluff (SQL linting; 10,060 tests), and xarray (labeled N-dimensional arrays; 19,917 tests). Each transformation is validated in isolation: for a given transformation, we apply the transformation at every applicable site, run the full test-suite, and compare the per-test outcome against the unmodified baseline. Validating one transformation at a time renders any divergence in behavior attributable to a single transformation type. Across all three projects, every test retained its outcome under all 14 transformations. This evidence is bounded by the coverage of the underlying test-suites and is therefore not a proof of equivalence. As each transformation is independently validated to be preserving, we rely on this property rather than re-verifying the test outcome of every perturbed task instance. Table 1: Catalog of semantics-preserving transformations (SPTs). Transformation Summary If Else Switcher Swaps if/else branches and negates the condition For Loop Rewriting Rewrites a for loop using an explicit iterator And Condition Splitter Decomposes if A and B into nested ifs Comparison Swapper Swaps operands and inverts the operator While Loop Unrolling Unrolls one iteration of a while loop Double Negation Injector Wraps a condition in not not (⋅·) Commutative Operand Permuter Reorders commutative operands Local Variable Renamer Renames safe local variables to synonyms If True Wrapper Wraps a block in a permanently true guard Try Except Injector Wraps a block in a redundant try/except Dead Code Injector Inserts an unreachable block Dead String Assignment Inserts an unread variable assignment Dead Method Injection Appends an unreachable method to a class String Literal Splitter Splits a string literal into a concatenation Composition and Scope Individual transformations act on candidate sites within a single file, whereas evaluating an agent on a code repair task requires perturbing a whole repository. We therefore apply a finite sequence ⟨t1,…,tm⟩ t_1,…,t_m of SPTs to the source files, excluding test suite files, of a repository. Each tjt_j is individually semantics-preserving, and because observational equivalence is transitive, their composition is semantics-preserving as well. The perturbed repository is therefore observationally equivalent to the original. The exclusion of test suite files is what keeps such a perturbed repository usable as an evaluation target. The oracle that defines task success is identical before and after perturbation, and no change in an agent’s score can be attributed to a moved target. Sampling Semantics-Preserving Variants Given a seed repository associated with a task instance, the variant sampler (Algorithm 1; its subroutine AssignTargetNames is given as Algorithm 2 in Appendix B) applies a sequence of SPTs at randomly chosen sites in randomly chosen files and returns a population of variants: repositories that, by the composition argument of the previous section, are semantically equivalent to the seed. Each variant becomes an independent task instance used to evaluate the agent. For each variant, the sampler makes four random decisions: Algorithm 1 Random Variant Sampler 1: Seed repository CseedC_seed, Sample count N, List of transformations T, Files modified by the gold patch ℱgoldF_gold, Issue description I, Number of transformations to apply NtN_t, Maximum number of keywords selected NkN_k, Fraction of candidates transformed ϕφ, Maximum files per keyword-bound transformation NfN_f 2: Variant population V 3: 4: procedure GenerateVariants(CseedC_seed, N, T, ℱgoldF_gold, I, NtN_t, NkN_k, ϕφ, NfN_f) 5: ←∅V← 6: ←NoneK← None 7: for j←1j← 1 to N do 8: Cmut←Clone(Cseed)C_mut← Clone(C_seed) 9: selected←RandomSelect(,Nt)T_selected← RandomSelect(T,N_t) 10: pfile(t)∼U⁡(0,1)p_file(t) U(0,1) for each t∈selectedt _selected 11: if selected∩kw≠∅T_selected _kw≠ and =NoneK= None then 12: ←ExtractKeywords(I)K← ExtractKeywords(I) ⊳ LLM call; cached 13: end if 14: ℬ←AssignTargetNames(selected,,Nk)B← AssignTargetNames(T_selected,K,N_k) 15: for each (t,τ)∈ℬ(t,τ) do 16: c←0c← 0 17: for each f∈SourceFiles(Cmut)f∈ SourceFiles(C_mut) do 18: if f∉ℱgoldf _gold and (Random>pfile(t)CLOSE ( Random>p_file(t) or 19: (t∈kwCLOSE(t _kw and OPENOPENc≥Nf))c≥ N_f) ) then continue 20: c←c+1c← c+1 21: S←GetCandidates(f,t,τ)S← GetCandidates(f,t,τ) 22: if S=∅S= then continue 23: S′←RandomSelect(S,⌈ϕ|S|⌉)S ← RandomSelect(S, φ|S| ) 24: Cmut[f]←ApplyTransform(f,S′,t,τ)C_mut[f]← ApplyTransform(f,S ,t,τ) 25: end for 26: end for 27: .append(Cmut)V.append(C_mut) 28: end for 29: return V 30: end procedure 1. Which transformations? A subset of NtN_t transformations is drawn uniformly from the list of transformations T. 2. Which files? For each selected transformation, an inclusion probability pfile∼U⁡(0,1)p_file U(0,1) is drawn and each file, excluding test suite files, is included with probability pfilep_file. The files, ℱgoldF_gold, modified by the gold patch (i.e., the reference edit that resolves the issue) are exempt from this filter and from the NfN_f cap introduced below. So they are always included. These are the files an agent must locate and edit to resolve the issue. So every variant targets the solution-relevant region rather than targeting it by chance. 3. Which sites? Within each included file, a fraction ϕφ of the candidate sites is chosen uniformly for rewriting. 4. Which keywords? For each selected keyword-bound transformation t, up to NkN_k targets are drawn uniformly from its extracted candidates. We extract the targets for kwT_kw from the issue description with a single LLM call: method names it mentions for Dead Method Injection, literal strings for Dead String Assignment. Because pfilep_file is resampled for every transformation, different transformations reach different portions of the repository, and the population spans from a single localized edit to pervasive, repository-wide perturbation. Notation. Random∼U⁡(0,1) Random U(0,1) is a uniform random number generator. RandomSelect(X,m) RandomSelect(X,m) returns m elements drawn uniformly without replacement from X. SourceFiles(C) SourceFiles(C) returns the source files of C, excluding test suite files. Clone(C) Clone(C) copies repository C. ExtractKeywords(I) ExtractKeywords(I) invokes an LLM to obtain the keyword map, K, where K holds keyword candidates for t∈kwt _kw. AssignTargetNames(⋅) AssignTargetNames(·) binds each selected t∈kwt _kw to up to NkN_k of its candidates, emitting one pair (t,τ)(t,τ) per keyword; every other t, and any t with no candidates, is emitted as (t,None)(t, None) (Algorithm 2). GetCandidates(f,t,τ) GetCandidates(f,t,τ) returns sites in f where t can be applied. ApplyTransform(f,S′,t,τ) ApplyTransform(f,S ,t,τ) rewrites f at every site in S′S . Hyperparameters. Five hyperparameters govern the sampler. N is the number of variants drawn from each seed instance. NtN_t is the number of transformations sampled for each variant. NkN_k caps how many keywords each keyword-bound transformation is bound to. ϕ∈(0,1]φ∈(0,1] is the fraction of candidate sites rewritten within each included file. Finally, NfN_f caps the number of files that a single keyword-bound transformation may select. This last bound is a practical necessity. We observed that agents frequently detected the perturbation and reverted the repository outright (for instance via git reset) when a variant contained too many decoys. Capping the reach of the keyword-bound transformations keeps variants within the range an agent treats as ordinary code. The one quantity that is not fixed, the file-inclusion probability pfilep_file, is drawn from U⁡(0,1)U(0,1) for every transformation. The population V returned by Algorithm 1 is consumed by the agent evaluation loop (Algorithm 3 in Appendix C), which provisions an isolated environment for each variant, runs the agent, and evaluates its patch against the instance’s test oracle. Experiments Research Questions Our evaluation is organized around the following questions. RQ1 To what extent do SPTs degrade agent performance, and how does this effect vary across models and scaffolds? RQ2 How much additional effort do SPTs induce on agents, controlling for task outcome? RQ3 What failure patterns emerge when agents fail on semantics-preserving variants? Setup Benchmarks and instance selection. We draw task instances from two popular repository-level program-repair benchmarks: SWE-bench Verified (24), a human-validated 500-instance subset of SWE-bench (16), and SWE-bench Pro (7), which contains 731 instances. Constrained by a compute budget, we selected 28 task instances from SWE-bench Verified and 26 from SWE-bench Pro. Our selection procedure is detailed in Appendix C. Each task instance pairs a repository at a base commit with FAIL_TO_PASS and PASS_TO_PASS test sets that an accepted patch must satisfy. The FAIL_TO_PASS tests are those that fail at the base commit but must pass once the issue is fixed. PASS_TO_PASS tests, by contrast, are those that already pass at the base commit but must remain passing to demonstrate that the patch fixes the issue without causing any regression. Agents and models. We evaluate two agentic scaffolds, mini-SWE agent (36) and OpenCode (25), each driven by one of four backing models: Claude Opus 4.5 (1), Kimi K2.5 (32), MiniMax M2.5 (21), and Qwen 3.6-27B (27). Crossing two scaffolds, four models, and two benchmarks yields 16 configurations, each of which we evaluate independently. Within a configuration the agent is run under a fixed setting, identical on the seed and on its variants, so that any change in outcome is attributable to the perturbation rather than to the agent. Variant generation. We configure the Random Variant Sampler (Algorithm 1) with N=20N=20, Nt=3N_t=3, Nk=5N_k=5, Nf=10N_f=10, and ϕ=0.7φ=0.7. Under this configuration the sampler produces 20 variants per instance. For keyword-bound transformations, the target keywords are extracted from the issue description alone, using the same model that backs the agent under evaluation. Perturbation magnitude. Averaged over a task instance’s variants, the transformations target a median of 6.9% of lines of source code on SWE-bench Verified and 7.7% on SWE-bench Pro. These percentages are lower bounds, since each transformation also inserts new code at the sites it touches. Metrics All metrics are computed per configuration, a (scaffold, model, benchmark) triple. Metrics are reported both per instance and in aggregate. Resolve rate. A run resolves an instance if its patch passes both the FAIL_TO_PASS and PASS_TO_PASS test sets. For instance i, the baseline resolve rate r0(i)r_0(i) is the fraction of the N=20N=20 unperturbed runs that resolve it, and the perturbed resolve rate rp(i)r_p(i) is the fraction of the N=20N=20 variant runs that resolve it. Both are proportions in [0,1][0,1]. We report them in percentage points. Degradation. Our primary metric is the per-instance degradation Δ⁡(i)=r0(i)−rp(i), (i)\;=\;r_0(i)-r_p(i), where a positive value means the agent resolves the issue less often after perturbation. We summarize each configuration by the mean degradation Δ¯=1|ℐ|∑iΔ⁡(i) = 1|I| _i (i) (ℐI is the set of instances) and compare Δ¯ across configurations. Effort and cost. To capture overhead that resolve rate alone misses, we report the mean difference between perturbed and baseline runs in two quantities: agent steps and cost. We use s⁡(⋅)s(·) for the number of agent steps in a run (model turns or tool invocations) and c⁡(⋅)c(·) for its token cost. Let s¯0(i) s_0(i) and s¯p(i) s_p(i) be the mean step count over the baseline and perturbed runs of instance i. We report the relative change δstep(i)=s¯p(i)−s¯0(i)s¯0(i)×100%, _step(i)\;=\; s_p(i)- s_0(i) s_0(i)× 100\%, and define δcost(i) _cost(i) the same way from c⁡(⋅)c(·). We summarize a configuration by the mean over instances, δ¯=1|ℐ|∑iδ⁡(i) δ= 1|I| _iδ(i). When conditioning these metrics on whether the issue was resolved or not, we exclude instances where the baseline has no resolved runs. Experimental Protocol For each instance we perform N=20N=20 runs on the unperturbed seed and one run on each of N=20N=20 sampled variants. Every run, unperturbed or perturbed, executes in a freshly provisioned, isolated environment, so no state carries between runs. Concretely, for each instance we extract the seed repository, gold-patch file list, and issue description; execute the N unperturbed runs; generate N variants with Algorithm 1; and then, for each variant, provision a fresh environment, inject the variant, run the agent, and evaluate the resulting patch against the instance’s test oracle. Algorithm 3 in Appendix C gives the full procedure. Two sources of randomness. A perturbed run varies for two independent reasons. Variant randomness: the sampler draws transformations, files, and sites at random, so variants of the same instance differ in difficulty. Writing pVp_V for the agent’s resolve probability on variant V, this is the spread σ2=Var⁡(pV)σ^2=Var(p_V). Agent randomness: with the variant held fixed, a single run is a Bernoulli⁡(pV)Bernoulli(p_V) draw, contributing average within-variant noise ν=⁡[pV(1−pV)]ν=E\! [p_V(1-p_V) ]. The unperturbed condition carries only the second. The quantity rp(i)r_p(i) targets is μ=⁡[pV]μ=E[p_V], the resolve probability averaged over the variant population, and degradation contrasts it with the unperturbed resolve probability. One run per variant is the efficient split. With a budget of R perturbed runs per instance, the design choice is how to split them into N variants of K runs each (R=NKR=NK). Averaging the per-variant rates is unbiased for μ at every K, since a single run already has conditional mean pVp_V; repetition tightens each per-variant estimate around a center that is already correct rather than moving it. The variance does depend on the split: Var⁡(μ^)=(Kσ2+ν)/RVar( μ)=(Kσ^2+ν)/R, strictly increasing in K whenever σ2>0σ^2>0. Intuitively, a run spent on a fresh variant samples both sources of randomness, whereas re-running a variant resamples only the agent. We therefore set N=R=20N=R=20, K=1K=1. Appendix E gives the derivation. Statistical protocol. Our inference is fixed-population: the instance set of each benchmark is held fixed and all intervals quantify run-to-run variability on those instances, not sampling of instances from the benchmark. We refer to the unperturbed and perturbed runs of an instance as its two conditions. For the configuration-level means (degradation, step count, and cost) we report 95%95\% bootstrap percentile intervals from B=20,000B=20,000 resamples, resampling runs within each instance while leaving the instance set intact. For degradation, we redraw each instance’s per-run outcomes in each condition from a binomial model, Binomial⁡(n,p^)/nBinomial(n, p)/n, with p p the observed resolve rate over that condition’s n runs. For step count and cost, we draw the per-run values within each condition with replacement and recompute the instance’s relative change. For per-instance degradation, a difference of two proportions, each estimated from 2020 runs, we instead report 95%95\% Newcombe intervals (23; 9). RQ1: Degradation of Agent Performance Figure 1: Mean degradation Δ¯ (percentage points) for each model under each scaffold on both benchmarks. Error bars are fixed-population 95% bootstrap confidence intervals. Robustness rankings do not transfer across scaffolds or benchmarks. Figure 1 gives the mean degradation Δ¯ for each scaffold-model pair on each benchmark with its fixed-population 95% bootstrap confidence interval. Perturbation reduces resolve rates in most configurations: Δ¯ is positive in 13 of the 16, and the interval excludes zero in 6 of the 16. No scaffold-model pair is significant on both benchmarks, so the effect is configuration-dependent rather than uniform. No single ordering of models by robustness holds across the two scaffolds, and none holds across the two benchmarks either. On SWE-bench Verified, Qwen is the most robust model under mini-SWE agent, degrading only 0.2 points, yet the most brittle under OpenCode at 5.5 points. MiniMax moves the other way, from 2.7 points under mini-SWE agent down to 0.5 under OpenCode. The benchmark axis is just as jagged. Opus under mini-SWE agent degrades 1.8 points on SWE-bench Verified but 6.7 points on Pro, the largest drop in the study, while under OpenCode on Pro it is the least degraded of the four models. Kimi stays at or below one point in three of its four cells but is the most degraded OpenCode model on Pro at 4.2 points. Every model is among the most robust in at least one cell and among the most brittle in another. A mean degradation measured on one scaffold and benchmark therefore does not predict behavior on another. A practitioner who picks a model for how it holds up under mini-SWE agent on SWE-bench Verified may get the opposite outcome after switching scaffold or codebase. We call this a jagged robustness frontier: robustness is a joint property of the model, the scaffold, and the workload, not of the model alone. The simpler scaffold is consistently more robust. Averaged over the four models, mini-SWE agent degrades less than OpenCode on both benchmarks: 1.34 points against 1.88 on SWE-bench Verified, and 1.88 against 3.65 on SWE-bench Pro. Its absolute strength, by contrast, is benchmark-dependent. It resolves 89.05% of unperturbed runs on Verified against OpenCode’s 81.20%, but 76.35% on Pro against OpenCode’s 79.33%. The robustness gap survives this capability flip, which is what makes it informative. The simpler scaffold loses less under perturbation. Figure 2: Mean relative change in agent steps (δ¯step δ_step) and cost (δ¯cost δ_cost) after perturbation, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right), restricted to runs the agent resolved in both cases. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals. Degradation is concentrated, not diffuse. Appendix D reports the per-instance degradation Δ⁡(i) (i) with its Newcombe 95% interval for every instance-configuration. At the level of an individual instance the comparison is underpowered. With 20 runs per condition, only swings of roughly 25 points or more can exclude zero. Only 14 out of 432 instance-configuration items exclude zero in their interval. The mass of point estimates is shifted toward positive values. The aggregate effect is carried by a small heavy tail of instance-configuration items. Three instances alone account for roughly two-thirds of the 6.7-point mean degradation of mini-SWE agent with Opus on SWE-bench Pro. The concentration mirrors the jaggedness we saw at the aggregate level. No instance degrades under all eight configurations on either benchmark, and a few cells swing sharply the other way. Brittleness attaches to particular instance-configuration pairs rather than to instances, models, or repositories alone. RQ2: Effort Induced by Perturbation Perturbation inflates effort more broadly than it degrades outcomes. Figure 2 reports δ¯step δ_step and δ¯cost δ_cost for each configuration on both benchmarks, restricted to the runs the agent resolved, so that the comparison cannot be driven by long failing runs. On SWE-bench Verified, cost rises in all 8 configurations, from 4.0% (mini-SWE agent with Opus) to 22.9% (OpenCode with MiniMax), and the interval excludes zero in every one. On SWE-bench Pro the increments are smaller and concentrated in the richer scaffold. All four OpenCode configurations pay 8.8–12.6% more with intervals excluding zero, while the four mini-SWE agent configurations sit near zero. The same holds over all runs (Appendix D): cost rises in all eight Verified configurations and seven of eight on Pro. Because the trend is already present when only successful runs are counted, it is not an artifact of failure. Measured by resolve rates, most configurations are impacted modestly by the perturbations, but, measured by what the agent spends, the impact is more drastic. The agent spends more per step, not only more steps. Steps and cost do not move together. Cost per step rises in twelve of the sixteen configurations. On SWE-bench Verified, step overhead never exceeds 9.9% while cost overhead reaches 22.9%. The agent is not merely taking more turns after a rewrite. It is consuming more context on each turn, consistent with longer files and heavier reading under perturbation. Among runs the agent resolved, input tokens per step rise in 13 of the 16 configurations. The scaffold contrast from the degradation analysis reappears here on Pro. Restricted to resolved instances, the four OpenCode configurations pay 4.6–7.2% more in cost per step, while the mini-SWE agent configurations are flat or slightly cheaper. Opus compresses its trajectory rather than extending it. Opus under mini-SWE agent on SWE-bench Verified responds to perturbation by taking fewer steps. This behavior is uniform. For every instance, it takes fewer steps on perturbed runs (mean −19.8%-19.8\%, range −6%-6\% to −39%-39\%). But output tokens per step more than double, causing cost to rise on every instance (median +101%+101\%), while input tokens per step fall by 17%. Rather than taking more turns, Opus emits fewer, denser actions, for a net +30.4%+30.4\% cost per step and +4.0%+4.0\% total cost. This behavior is also benchmark-specific. On SWE-bench Pro the step reduction shrinks to (−1.29%-1.29\%). RQ3: Observed Failure Patterns We manually analyze 20 agent trajectories across both scaffolds, both benchmarks (14 distinct instances), and all 4 models to check for behavioral patterns and how SPTs influence agent handling of tasks. These tasks are selected for displaying high variance (3), high degradation (3), large increase in step count (1), file coverage (2) or cost (1). Additionally, cases with one mutant failure out of 20 (2) and those with high likelihood of SPT interference (8) are also picked, with a focus on covering a diverse set of model-scaffold-benchmark combinations. Details on selection can be found in Appendix F. Here, we present 4 interesting behavior patterns that we observed. For additional observations, refer to Appendix F. Detecting and reverting SPTs. In the cases where there is significant overlap between the SPT edited files and the files that the agent touches, there are instances where the agent recognizes that the code is "obfuscated". For example, in one case, with MiniMax on Qutebrowser and mini-SWE agent, the agent checks the commit history and spots that the transformations were added in the latest commit. It then reverts the repository state before it makes the final changes so that all SPTs are cleared. Degradation of code localization capabilities. SPTs may make it harder for agents to localize relevant sections of the code, either by diluting grep results or making core files lose credibility. For example, Opus with mini-SWE agent on Openlibrary tries to grep for a keyword in the issue description, but ends up getting thrown off by multiple noisy dead string assignments which use the same keyword and turn up in the search. Editing/"fixing" of perturbed code. The agent occasionally simplifies the perturbed code during patching, for example, Opus with mini-SWE agent on Qutebrowser mentions that it has recognized and fixed the perturbed code in multiple places. This additional editing may also increase the risk of the patch failing due to the increased volume of changes. Corrupted patch validation. With Kimi on Pytest-dev, one of the unit tests in the test suite fails. The part of the code the unit test refers to contains a string literal splitter in one of the logs. The agent gets confused and incorrectly attributes the failure to the split string, reasons that this was not due to an edit it made, and then submits the incorrect patch without attempting to resolve the error. Related Work Robustness of code language models. A substantial body of work shows that neural models of code are brittle to surface-level perturbations that preserve semantics. Identifier renaming attacks have been demonstrated against neural code models (38), formalized under robustness constraints (2), and scaled to black-box settings against pretrained models like CodeBERT (37). Structural perturbations such as dead code insertion and operator substitution have been used to attack program classifiers (30). 28 combined renaming, dead code, and operand swapping to expose fragility across method-name prediction and code summarization. On the defense side, contrastive learning on equivalent programs (13) and naturalizing transformations as a pretraining objective (3) have been shown to improve robustness. At the benchmark level, perturbations to docstrings, signatures, and syntax have been used to evaluate code generation models (35), attack frameworks have been proposed across defect detection and clone detection (15), and minor refactorings in prompt context have been shown to significantly alter Copilot’s completions (18). All of the above evaluate single-turn model inferences on classification or short-generation tasks where the perturbation is applied to a fixed-length input. Our work extends this to multi-turn agents that localize faults, navigate entire repositories, and synthesize patches across dozens of reasoning steps. The perturbations occur randomly over the codebase. Adversarial robustness of LLMs. LLMs are fragile to semantics-preserving transformations of general natural-language inputs. Prompt formatting changes alone (spacing, delimiters, option ordering) cause up to 76-point accuracy swings (29), and systematic perturbation benchmarks confirm this across character-, word-, and sentence-level attacks (42; 33). The phenomenon extends to structured problem solving. GSM-Symbolic (22) and GSM-Plus (17) create variants of math benchmarks by changing surface features (names, numbers, phrasing) while preserving problem structure, observing up to 10% performance drops. Similarly, reordering multiple-choice options (40; 6) or generating structurally equivalent reasoning problems (41) significantly shifts model rankings. Formal certification of LLM robustness has also been proposed (4; 34; 5). Geirhos et al. (10) frame these failures as shortcut learning, where networks exploit spurious surface statistics rather than learning the intended decision rule. Dziri et al. (8) provide evidence that transformers linearize compositional reasoning, offering a mechanistic explanation for why surface changes can derail multi-step tasks. Our findings echo this pattern in the code domain. We show coding agents are sensitive to how the surrounding code is structured. A further dimension absent from prior work is the scaffold interaction. We find that robustness rankings can change when the same model is embedded in a different agent architecture. Robustness certification for agentic systems has also been studied for simple, non-code tool-selection (39). Conclusion In this work, we contribute a library of local, semantics-preserving transformations (SPTs) for code repositories and a randomized sampler that draws semantically equivalent variants of a repository without feedback from the agent; an experimental methodology that isolates the perturbation effect from the intrinsic stochasticity of LLM-based agents through paired seed-and-variant runs while balancing statistical validity against the cost of agentic runs; and, to our knowledge, the first systematic robustness evaluation of repository-level code agents. Our experiments reveal a jagged robustness frontier across models, scaffolds, and repositories. Generative AI Usage Statement Generative AI tools were used in the preparation of this manuscript to improve the clarity of the writing, to assist in writing Python scripts for the experiments, and to develop the arguments in Section E. Everything was carefully reviewed by the authors and they remain fully accountable for this work. References Anthropic (2025) Anthropic Introducing Claude Opus 4.5. Note: https://w.anthropic.com/news/claude-opus-4-5System card: https://assets.anthropic.com/m/64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf Cited by: Agents and models.. Bielik and Vechev (2020) P. Bielik and M. Vechev Adversarial robustness for code. In Proceedings of the 37th International Conference on Machine Learning (ICML), Cited by: Robustness of code language models.. Chakraborty et al. (2022) S. Chakraborty, T. Ahmed, Y. Ding, P. Devanbu, and B. Ray NatGen: generative pre-training by “naturalizing” source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), Cited by: Robustness of code language models.. Chaudhary et al. (2025a) I. Chaudhary, Q. Hu, M. Kumar, M. Ziyadi, R. Gupta, and G. Singh Certifying counterfactual bias in LLMs. In International Conference on Learning Representations (ICLR), Cited by: Adversarial robustness of LLMs.. Chaudhary et al. (2025b) I. Chaudhary, V. Jain, P. Parhar, K. Sachdeva, A. Singh, S. Ranu, and G. Singh Lumos: let there be language model system certification. arXiv preprint arXiv:2512.02966. Cited by: Adversarial robustness of LLMs.. Chaudhary et al. (2024) I. Chaudhary, V. V. Jain, and G. Singh Certifying knowledge comprehension in LLMs. arXiv preprint arXiv:2402.15929. Cited by: Adversarial robustness of LLMs.. Deng et al. (2025) X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, et al. Swe-bench pro: can ai agents solve long-horizon software engineering tasks?. arXiv preprint arXiv:2509.16941. Cited by: Introduction, Benchmarks and instance selection.. Dziri et al. (2023) N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, P. West, C. Bhagavatula, R. Le Bras, J. D. Hwang, et al. Faith and fate: limits of transformers on compositionality. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Adversarial robustness of LLMs.. Fagerland et al. (2015) M. W. Fagerland, S. Lydersen, and P. Laake Recommended confidence intervals for two independent binomial proportions. Statistical methods in medical research 24 (2), p. 224–254. Cited by: Statistical protocol.. Geirhos et al. (2020) R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), p. 665–673. Cited by: Introduction, Adversarial robustness of LLMs.. GitHub (2025) GitHub Octoverse 2025: the state of open source and rise of AI. Note: https://github.blog/news-insights/octoverse/Accessed: 2026-07-22 Cited by: Introduction. Google DORA (2025) Google DORA 2025 dora report: state of AI-assisted software development. Note: https://dora.dev/research/2025/Accessed: 2026-07-22 Cited by: Introduction. Jain et al. (2021) P. Jain, A. Jain, T. Zhang, P. Abbeel, J. Gonzalez, and I. Stoica Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Robustness of code language models.. JetBrains (2025) JetBrains The state of developer ecosystem 2025. Note: https://devecosystem-2025.jetbrains.com/Accessed: 2026-07-22 Cited by: Introduction. Jha and Reddy (2023) A. Jha and C. K. Reddy CodeAttack: code-based adversarial attacks for pre-trained programming language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: Robustness of code language models.. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024, p. 54107–54157. Cited by: Validation, Benchmarks and instance selection.. Li et al. (2024) Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi GSM-Plus: a comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: Adversarial robustness of LLMs.. Mastropaolo et al. (2023) A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota On the robustness of code generation techniques: an empirical study on GitHub Copilot. In Proceedings of the 45th International Conference on Software Engineering (ICSE), p. 2149–2160. Cited by: Robustness of code language models.. McKeeman (1998) W. M. McKeeman Differential testing for software. Digital Technical Journal 10 (1), p. 100–107. Cited by: Validation. Miller (1995) G. A. Miller WordNet: a lexical database for english. Communications of the ACM 38 (11), p. 39–41. Cited by: §A. MiniMax AI (2026) MiniMax AI MiniMax-M2.5. Note: https://huggingface.co/MiniMaxAI/MiniMax-M2.5 Cited by: Agents and models.. Mirzadeh et al. (2024) I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar GSM-Symbolic: understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229. Cited by: Adversarial robustness of LLMs.. Newcombe (1998) R. G. Newcombe Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in medicine 17 (8), p. 873–890. Cited by: Statistical protocol.. OpenAI (2024) OpenAI Introducing SWE-bench Verified. Note: https://openai.com/index/introducing-swe-bench-verified/Accessed: 2026-07-10 Cited by: Introduction, Benchmarks and instance selection.. OpenCode Contributors (2024) OpenCode Contributors OpenCode: an open-source ai coding agent for the terminal.. Note: https://github.com/anomalyco/opencodeAccessed: 2026-07-10 Cited by: Agents and models.. Pylint contributors (2026) Pylint contributors astroid: an abstract syntax tree library for Python. Note: https://github.com/pylint-dev/astroid Cited by: §A. Qwen Team (2026) Qwen Team Qwen3.6-27B: flagship-level coding in a 27b dense model. External Links: Link Cited by: Agents and models.. Ramakrishnan et al. (2022) G. Ramakrishnan, J. Henkel, Z. Wang, A. Albarghouthi, S. Jha, and T. Reps Semantic robustness of models of source code. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), p. 526–537. Cited by: Introduction, Robustness of code language models.. Sclar et al. (2024) M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr Quantifying language models’ sensitivity to spurious features in prompt design. In International Conference on Learning Representations (ICLR), Cited by: Adversarial robustness of LLMs.. Srikant et al. (2021) S. Srikant, S. Liu, T. Mitrovska, S. Chang, Q. Fan, G. Zhang, and U. O’Reilly Generating adversarial computer programs using optimized obfuscations. In International Conference on Learning Representations (ICLR), Cited by: Introduction, Robustness of code language models.. Stack Overflow (2025) Stack Overflow 2025 stack overflow developer survey. Note: https://survey.stackoverflow.co/2025/Accessed: 2026-07-22 Cited by: Introduction. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: Agents and models.. Vega et al. (2023) J. Vega, I. Chaudhary, C. Xu, and G. Singh Bypassing the safety training of open-source LLMs with priming attacks. arXiv preprint arXiv:2312.12321. Cited by: Adversarial robustness of LLMs.. Wang et al. (2026) C. Wang, I. Chaudhary, Q. Hu, W. Ruan, R. Gupta, and G. Singh How catastrophic is your LLM? certifying risks in conversation. In International Conference on Learning Representations (ICLR), Cited by: Adversarial robustness of LLMs.. Wang et al. (2023) S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Kumar, S. Tan, B. Ray, P. Bhatia, et al. ReCode: robustness evaluation of code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: Introduction, Robustness of code language models.. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Agents and models.. Yang et al. (2022) Z. Yang, J. Shi, J. He, and D. Lo Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering (ICSE), p. 1482–1493. Cited by: Robustness of code language models.. Yefet et al. (2020) N. Yefet, U. Alon, and E. Yahav Adversarial examples for models of code. Proceedings of the ACM on Programming Languages (OOPSLA) 4. Cited by: Introduction, Robustness of code language models.. Yeon et al. (2025) J. Yeon, I. Chaudhary, and G. Singh Quantitative certification of agentic tool selection. arXiv preprint arXiv:2510.03992. Cited by: Adversarial robustness of LLMs.. Zheng et al. (2024) C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), Cited by: Adversarial robustness of LLMs.. Zhu et al. (2024) K. Zhu, J. Chen, J. Wang, N. Z. Gong, D. Yang, and X. Xie DyVal: dynamic evaluation of large language models for reasoning tasks. In International Conference on Learning Representations (ICLR), Cited by: Adversarial robustness of LLMs.. Zhu et al. (2023) K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Z. Gong, and X. Xie PromptRobust: towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528. Cited by: Adversarial robustness of LLMs.. A Semantics-preserving Transformation Implementation Details The appendix details the implementation of each SPT. It describes the applicability condition of each SPT that restricts the application of the SPT to the candidate sites. Writing these conditions is where the difficulty lies. They must hold across diverse repositories, where the same construct appears in many different implementations, so a condition has to rule out every unsafe variant rather than the common ones. Writing an SPT for a function-level code snippet is comparatively easy. The snippet is short and self-contained, its bindings and callers are visible in the code being rewritten, and the only observable behavior is the value it returns. None of this holds in a repository. Names resolve through nested scopes and imports, so a builtin such as iter, next, or Exception may be shadowed at the rewrite site. Types are not written down either. Addition commutes on numbers, but not on strings, lists, or a class with a custom __add__. A program can also observe its own structure. It can do so through locals(), frame introspection, metaclasses, attribute-dispatch hooks, and framework base classes that fix what a class namespace may contain. Code that never runs can therefore still change behavior. Scale makes this more difficult. A single variant transforms thousands of sites in a repository. So a rule that is not semantics-preserving in a rare case will meet that case. We therefore keep every SPT conservative. When an applicability condition cannot be established from the code, we skip the site. We group the transformations of Table 1 by the operation they perform: rewrites of existing structure and insertions of inert code. Rewrites If Else Switcher negates the condition and exchanges the two branches. It is applied only to if statements that have a direct else clause. An if whose else branch is another if (i.e. an elif) is not switched. Although an elif clause carrying its own else is itself a valid target. This SPT is also applied to inline ternary expressions, where an else is syntactically required. The condition, and in the ternary case both result expressions, are wrapped in parentheses before negation to maintain correct operator precedence. And Condition Splitter decomposes a compound if statement whose condition is a Boolean conjunction (A and B) into semantically equivalent nested if statements. When the statement has an else clause, that clause is duplicated onto both levels so that failure of either conjunct reaches it. Comparison Swapper exchanges the operands of an asymmetric relational comparison (<, >, <=, >=) and inverts the operator to preserve semantic equivalence. It is restricted to simple two-operand comparisons: chained comparisons and symmetric operators are excluded. Double Negation Injector wraps an if or while condition in not not (Condition). It skips literal True/False conditions and conditions that are themselves a top-level negation, so that a condition such as not x is left alone while a compound condition that merely contains a not (e.g. not a and b) remains eligible. While Loop Unrolling unrolls one iteration of a while loop: the loop body is duplicated at the head of the new body, followed by an inner while carrying the original test and body, followed by an unconditional break. The outer test is therefore evaluated the same number of times as in the original loop, which matters when the test has side effects. Only loops with an indented block body are transformed, and loops carrying an else clause are excluded, since the injected break would suppress it. For Loop Rewriting rewrites a for loop into a semantically equivalent while True loop using the explicit iterator protocol. It materializes an iterator with iter(...) and advances it with next(...) inside a try/except StopIteration: break block. The introduced iterator and flag names carry a random suffix so they cannot collide with existing bindings. It also emulates the for ... else contract via a sentinel flag. Specifically, each break statement belonging to the rewritten loop—breaks inside nested loops are left untouched—is rewritten to flag = False; break, so that the flag correctly records whether the loop exited early. If the iterable is an implicit tuple, it is explicitly parenthesized before being passed to iter() to avoid the two-argument iter(callable, sentinel) misinterpretation. The transformation is skipped when any of iter, next, or StopIteration is shadowed by a binding in an enclosing scope. async for loops are excluded, as their asynchronous protocol cannot be emulated with synchronous iter()/next(). Commutative Operand Permuter swaps the operands of commutative binary operations: addition, multiplication, bitwise AND, OR, and XOR. Because commutativity holds for these operators only over specific types in Python (e.g., string and list + are order-dependent, and a user-defined __add__/__radd__ need not commute), the transformation performs type inference using Astroid (26) before swapping. The swap is applied only when both operands are independently inferred to lie within the allowed type set for that operator. If inference is inconclusive or the inferred type lies outside the allowed set, the transformation is skipped. Logical and/or are never permuted. Local Variable Renamer renames local variables to semantically related synonyms. Candidates are the targets of plain assignment statements inside function bodies. Module-, class-, lambda- and comprehension-level bindings are never renamed. Candidate names are looked up in WordNet (20) for a synonym. If none is suitable, or the synonym would collide with an identifier already present in the module, a random five-letter lowercase name is generated. All occurrences within the scope of the binding are updated consistently, including nested scopes that do not rebind the name. We apply a conservative exclusion list to avoid renaming anything whose identity may be externally meaningful or whose rename could not be confined to one scope: built-ins, keywords, very short names, function names, parameters, imported and decorator names, names declared global or nonlocal, loop targets, and variables used in a call position, as an argument, or as part of an attribute access or the base of a subscript. We additionally exclude variables that alias another name (a = b), variables assigned the result of a call, and any variable bound more than once in its scope, since control flow then determines which binding a reference observes. String Literal Splitter fragments a string literal into a +-joined concatenation at a randomly chosen split point, exploiting that Python evaluates the concatenation of literals to the original value. Because the literal no longer appears as one contiguous token, this frustrates localization strategies that search for the whole string. Splitting is skipped for literals inside type annotations, prefixed byte/raw/unicode strings (escape semantics), strings shorter than a safe split length of four characters, literals used as the base of an attribute access, and literals already part of an implicit concatenation, a non-+ binary operation, or a match pattern. Formatted (f-)strings are never targeted. Inert Insertions These transformations add code that would never alter program behavior. Two of them—Dead String Assignment and Dead Method Injection—insert code that is inert but lexically salient, forcing an agent that relies on surface cues such as identifier or method names to distinguish genuine edit sites from decoys. If True Wrapper encloses an indented block in a guard built from a grammar of randomly generated tautologies, whose complexity is set by a depth parameter. Because the guard always evaluates to True, every wrapped statement still executes. Candidate blocks are those spanning between 5 and 100 lines, and blocks whose first statement is a docstring are skipped. Try Except Injector wraps an indented block in try ... except Exception: raise. The except clause unconditionally re-raises any exception it catches, so all exceptions are propagated to the same handler as in the unmodified code. We use the except Exception: raise pattern instead of except: raise to avoid catching BaseException subclasses such as KeyboardInterrupt and SystemExit. As with the previous transformation, candidates are blocks of 5 to 100 lines and blocks whose first statement is a docstring are skipped; injection is additionally skipped when Exception is shadowed by a binding at the injection site, which would otherwise make the handler itself raise TypeError. Dead-Code Injector inserts an if False block containing a dummy assignment at the head of an indented block. The guard renders the block statically unreachable. When the block begins with a docstring, the injection is placed after it so that __doc__ is preserved. Dead String Assignment injects an unread assignment of the form <name> = "<keyword>" before selected statements inside a function body. The keyword-like string serves as a distractor resembling content an agent might search for, while remaining a no-op. The <name> is drawn from a fixed pool, chosen so as not to shadow any identifier used in the enclosing function, and is never read. Injection never displaces a leading docstring and is not placed after an unconditional terminator (return/raise/break/continue) in the same block. Any function that reflects on its local namespace is excluded, as introducing a new binding would be observable: this covers locals(), argument-less vars(), eval/exec calls that do not pass explicit globals and locals, and frame introspection via sys._getframe, inspect.currentframe, inspect.stack, and related APIs. Dead Method Injection appends a structurally complete method, def <name>(self, *args, **kwargs) whose entire body is guarded by if False, to a class body, serving as a highly plausible edit decoy. The method is callable but its body is unreachable, so it always returns None without side effects. Crucially, appending a method is not unconditionally behavior-preserving, because a class may observe its own attribute namespace. The transformation is therefore skipped whenever a class controls that namespace: classes declaring a custom metaclass, classes that override attribute dispatch (__getattr__, __getattribute__, __setattr__, __delattr__), property-only namespaces, sentinel classes whose body is a bare pass or ..., and classes whose base list mentions an attribute-constraining framework type (e.g., Pydantic BaseModel, Enum, NamedTuple, TypedDict, Protocol, ABC), matched by the simple name of the base expression. A class is also skipped when it already declares the target name as a method or class attribute, and dunder target names are excluded outright. B Sampler Procedure Details Algorithm 2 describes the sub-procedure invoked by the sampler described in Algorithm 1. Algorithm 2 Target Assignment 1: Selected transformations selectedT_selected, Keyword map K, Maximum number of keywords to select NkN_k 2: Bound (transformation, target) pairs ℬB 3: procedure AssignTargetNames(selected,,NkT_selected,K,N_k) 4: ℬ←∅B← 5: for each transformation t∈selectedt _selected do 6: if t∉kwt _kw or =NoneK= None then 7: ℬ←ℬ∪(t,None)B ∪\(t, None)\ ⊳ no target required 8: else 9: ←⁡[t]names [t] ⊳ candidates extracted for t 10: if =∅names= then 11: ℬ←ℬ∪(t,None)B ∪\(t, None)\ 12: else 13: ←RandomSelect(,min⁡(Nk,||))targetedKeywords← RandomSelect(names, (N_k,|names|)) 14: for each τ∈τ do 15: ℬ←ℬ∪(t,τ)B ∪\(t,τ)\ 16: end for 17: end if 18: end if 19: end for 20: return ℬB 21: end procedure C Experimental Procedure Details Sampling Procedure To construct our task sample, we first identified instances resolved by Claude Opus 4.5 using official SWE-bench Verified leaderboard results, ensuring that our sampled tasks were solvable under standard (unperturbed) conditions. We then performed stratified sampling over repository–difficulty combinations, drawing one instance per stratum to ensure balanced coverage of the instance space. Since official instance wise results were unavailable for SWE-bench pro, we started by sampling an equal number of instances from each of the repos, and running Claude Opus 4.5 on the set. Then, 26 passing instances were sampled across all repos that contained at least one valid python file to which the SPTs could be applied. Full Experimental Procedure Algorithm 3 gives the full experimental procedure summarized in the Protocol paragraph of the experiments section. For each instance it proceeds in four phases: seed retrieval, baseline evaluation (N runs on the unperturbed seed), variant generation via the sampler of Algorithm 1 of the variant sampler section, and perturbed evaluation (one run per variant, each in a freshly provisioned environment). Algorithm 3 Agent Robustness Experiment 1: Set of Instance IDs ℐI, Number of samples N, Configuration C, Agent A, Evaluator ℰE, Transformations T, Sampler hyperparameters Nt,Nk,ϕ,NfN_t,N_k,φ,N_f 2: Baseline Reports ℛ0R_0, Perturbed Reports ℛpR_p 3: procedure RunExperiment(ℐI, N, C, A, ℰE, T, NtN_t, NkN_k, ϕφ, NfN_f) 4: ℛ0←∅R_0← , ℛp←∅R_p← ⊳ Initialize report collections 5: for each instance i∈ℐi do 6: ⊳ Phase 1: Seed Retrieval 7: (Cseed,ℱgold,desc)←ExtractInstance(i)(C_seed,F_gold,desc)← ExtractInstance(i) 8: ⊳ Phase 2: Baseline Evaluation 9: for k←1k← 1 to N do 10: Eeval←ProvisionEnvironment(i)E_eval← ProvisionEnvironment(i) ⊳ unperturbed seed, no injection 11: ℛ←RunAndEvaluate(,ℰ,Eeval,i,C)R← RunAndEvaluate(A,E,E_eval,i,C) 12: ℛ0←ℛ0∪ℛR_0 _0 13: end for 14: ⊳ Phase 3: Variant Generation (Algorithm 1) 15: ←S← GenerateVariants(CseedC_seed, N, T, ℱgoldF_gold, descdesc, NtN_t, NkN_k, ϕφ, NfN_f) 16: ⊳ Phase 4: Perturbed Evaluation 17: for each sample s∈s do 18: Eeval←ProvisionEnvironment(i)E_eval← ProvisionEnvironment(i) 19: InjectCode(Eeval,s) InjectCode(E_eval,s) 20: ℛ←RunAndEvaluate(,ℰ,Eeval,i,C)R← RunAndEvaluate(A,E,E_eval,i,C) 21: ℛp←ℛp∪ℛR_p _p 22: end for 23: end for 24: return (ℛ0,ℛp)(R_0,R_p) 25: end procedure 26: 27: procedure RunAndEvaluate(,ℰ,E,i,CA,E,E,i,C) 28: Patch←ExecuteAgent(,E,i,C)Patch← ExecuteAgent(A,E,i,C) 29: Report←EvaluatePatch(ℰ,i,Patch)Report← EvaluatePatch(E,i,Patch) 30: return ReportReport 31: end procedure Computational Infrastructure All experiments were run on one lab server. It has two AMD EPYC 9554 64-core processors (128 cores, 256 threads), 768 GB RAM, 6 Nvidia RTX PRO 6000 Blackwell GPUs (96 GB VRAM per GPU), and a 7 TB NVMe drive. The operating system is Ubuntu 24.04.4 LTS. Each agent run executes in its own Docker container (Docker version 29.5.2). We serve Qwen 3.6-27B on this machine with vLLM (v0.20.1) using 2 GPUs. We access Claude Opus 4.5( AWS bedrock ID: anthropic.claude-opus-4-5-20251101-v1:0), Kimi K2.5 (AWS bedrock ID: moonshotai.kimi-k2.5) and Minimax M2.5 (AWS bedrock ID: minimax.minimax-m2.5) using Amazon Bedrock API. All 4 models used temperature 1.0. Our code is Python 3.10, using LibCST v1.8.5 for SPTs, Astroid v3.3.0 for type inference, and NLTK v3.6.0 for WordNet lookups. The scaffolds are mini-SWE agent v1.17.5 and OpenCode v1.18.4. Cost accounting of Qwen 3.6. Reported costs are token costs. For the three models accessed through Bedrock we use the provider’s published per-token prices. Qwen 3.6-27B is served locally and to keep costs comparable across models we price its input and output tokens at the OpenRouter rates for the same model. The same price schedule is applied to the unperturbed and perturbed runs of an instance, so δcost _cost is an internally consistent within-instance comparison. Randomness and Reproducibility We do not fix a random seed. Our pipeline is random in two places. The sampler (Algorithm 1) draws the transformation subset, the file-inclusion probability pfilep_file, the sites within each file, and the target keywords. And all four models are sampled at temperature 1.0, so a run is stochastic even on a fixed repository. A seed fixes which 20 variants we draw, not what the agent does on them: rerunning the agent on the same variant can give a different outcome. Reproducing our results therefore means drawing 20 fresh variants per instance at the same hyperparameters and recovering our reported effects. The bootstrap of the statistical protocol is the exception: it is seeded. Code and Instance Lists Our code and data are publicly available at https://github.com/CSU-TrustLab/jagged-frontier. The repository contains the full implementation of the experiment and analysis. This includes the SPT implementations, the variant sampler of Algorithm 1, the evaluation harness of Algorithm 3, the analysis scripts that produce the reported statistics and figures, and an interactive dashboard for exploring the results. The prompts used are included there as well, as is the list of the 28 SWE-bench Verified and 26 SWE-bench Pro instance IDs used in the study. Agent Scaffold Comparison This table 2 compares between mini-SWE agent and OpenCode scaffold. Table 2: Comparison of mini-SWE and OpenCode Agent Scaffolds Category mini-SWE OpenCode Architecture Single, centralized agent Primary agents invoke tools and subagents for specific tasks (e.g., Build, Plan, General, Explore) Available Tools bash bash, edit, write, read, grep, glob, lsp, apply_patch, skill, todo_write, web_fetch, web_search, question Access Restrictions None Available tools determined by an agent’s role and user configuration Context Management Continuous, append-only Multiple, dynamically changing contexts for different subagents Selection of hyperparameter values We did not conduct a search over the sampler’s hyperparameter space. A single configuration at a single hyperparameter setting costs 54×40=2,16054× 40=2,160 agent runs. So even a coarse grid over (Nt,Nk,ϕ,Nf)(N_t,N_k,φ,N_f) would multiply the cost of the study by an order of magnitude. We instead fixed values for each parameter. Nt=3N_t=3 of the 14 transformations makes a variant a composition of several SPTs. ϕ=0.7φ=0.7 leaves a minority of candidate sites in a file untouched. So a variant contains both transformed and untransformed instances of the same pattern. Nk=5N_k=5 and Nf=10N_f=10 are the only values set from observation. Without them, agents frequently detected the perturbation and reverted the repository outright (e.g. via git reset), which destroys the measurement rather than making it harder. D Additional Experiment Results Figure 3: Mean resolve rate (%) on unperturbed and perturbed runs for each model under each scaffold, on SWE-bench Verified (top) and SWE-bench Pro (bottom). Resolve rates fall, but capability rankings survive. Figure 3 shows the mean resolve rate on unperturbed and perturbed runs for each model under each scaffold, separately on SWE-bench Verified and SWE-bench Pro. In 13 of the 16 scaffold-model-benchmark combinations the agent resolves fewer issues after perturbation. The three remaining combinations move slightly in the other direction, each by less than one point, within run-to-run noise. Perturbation, however, barely disturbs the models’ capability ordering. In three of the four scaffold-benchmark panels, the ranking of models by resolve rate is identical before and after perturbation. In the fourth (OpenCode on SWE-Pro), the only change is a swap between MiniMax and Qwen, which are separated by just 0.2 points at baseline. This shows that a leaderboard-style comparison would not register the impact of SPTs. The ranking that perturbation does scramble is the ranking by robustness, as we show next. Effort Overhead over All Runs Figure 4 repeats the RQ2 effort analysis of the main text over all runs, rather than only the runs the agent resolved. The trends match the resolved-only figure. On SWE-bench Verified cost rises in all eight configurations, from 4.5% (mini-SWE agent with Opus) to 25.4% (OpenCode with MiniMax), with the interval excluding zero in seven of the eight. On SWE-bench Pro cost rises in seven of the eight, by up to 14.7% (OpenCode with Kimi), with six intervals excluding zero. The one exception, mini-SWE agent with Opus, is flat at −0.1%-0.1\%. Step counts move less than cost in both benchmarks, and Opus under mini-SWE agent on Verified again takes 19.7% fewer steps while still costing more. Because the two views agree, the overhead cannot be attributed to long failing runs, and we report the resolved-only view in the main text as the more conservative one. Figure 4: Mean relative change in agent steps (δ¯step δ_step) and cost (δ¯cost δ_cost) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right). Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals. Per-Instance Degradation Plots Figures 5 and 6 give the per-instance degradation Δ⁡(i) (i) with its Newcombe 95% interval for every instance-configuration cell. This section reports the underlying numbers for the concentration claim of the RQ1 results. With 20 runs per condition, a Newcombe interval is wide by construction: even an instance resolved in every run of both conditions carries a ±16± 16-point interval. The interval excludes zero in only 7 of the 224 cells on SWE-bench Verified and 7 of the 208 on SWE-bench Pro. Most intervals are therefore too wide to establish degradation for a specific instance, although their mass is shifted toward positive values. Point estimates are positive in 65 cells against 35 negative on SWE-bench Verified, and 79 against 44 on SWE-bench Pro. The individually significant cells concentrate in the tail. On SWE-bench Verified, scikit-learn__scikit-learn-14983 loses 40 points under OpenCode with Qwen, matplotlib__matplotlib-20859 loses 35 under mini-SWE agent with Qwen, and sympy__sympy-12489 loses 30 under mini-SWE agent with MiniMax. On SWE-bench Pro, element-web_dae13 loses 50 points, openlibrary_f3b26 40, and openlibrary_a7b7d 35, all under mini-SWE agent with Opus. These three Pro cells alone account for roughly two thirds of that configuration’s 6.7-point mean degradation. Across configurations, qutebrowser_ef5ba and ansible_9142b degrade in seven of eight, and sympy__sympy-11618 in five while never improving. Single cells also swing the other way: mwaskom__seaborn-3069 improves by 55 points under OpenCode with MiniMax and by 35 under mini-SWE agent with Qwen, both with intervals excluding zero. With a single mwaskom__seaborn instance in the sample, such swings should not be read as a repository-level effect. Figure 5: Per-instance degradation Δ⁡(i) (i) for the 28 SWE-bench Verified instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20n=20 runs per condition). Figure 6: Per-instance degradation Δ⁡(i) (i) for the 26 SWE-bench Pro instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20n=20 runs per condition). E Unbiasedness of the Degradation Estimator We estimate perturbed performance by running the agent once on each of N=20N=20 sampled variants. Because a single run of a stochastic agent is a noisy measurement of that variant’s difficulty, it is reasonable to ask whether one run per variant is enough. An alternative would be to draw fewer variants and run each several times, averaging away the agent’s randomness. This appendix shows that no such averaging is needed. Per-instance degradation is unbiased for any number of runs per variant including one, and under a fixed run budget one run per variant is in fact the variance-minimizing allocation. Setup and notation Consider, a task instance i and a configuration (scaffold, model, benchmark). Variant sampling. A variant V is drawn by the variant sampler from the population of semantics-preserving variants of instance i. Each variant carries a true resolve probability pV=Pr⁡(agent resolves V∣V),p_V\;=\; (agent resolves V V), (1) the probability that the agent resolves that specific variant, averaged over the agent’s own randomness. Because V is itself random, pVp_V is a random variable. It is the difficulty of a randomly chosen semantically equivalent rewrite of instance i. Agent sampling. Given a fixed variant v, a single agent run yields a resolve/no-resolve indicator Y∣(V=v)∼Bernoulli(pv),[Y∣V=v]=pv.Y (V=v)\; \;Bernoulli(p_v), [Y V=v]\;=\;p_v. (2) The agent’s stochasticity is conditionally independent of how the variant was chosen, given the variant. This is enforced by the experimental protocol. Every run executes in a freshly provisioned, isolated environment, so no state carries between runs. Estimands. The population quantity that rp(i)r_p(i) targets is the mean per-variant resolve probability, μ=⁡[pV],μ\;=\;E[p_V], (3) and its companion, the spread of difficulty across variants, is σ2=Var⁡(pV).σ^2\;=\;Var(p_V). (4) Here μ summarizes average performance on variants, while σ2σ^2 measures consistency — how much the agent’s success probability swings across semantically identical inputs. Degradation is Δ⁡(i)=p0−μ (i)=p_0-μ, where p0p_0 is the resolve probability on the unperturbed seed. Δ⁡(i)=p0−μ (i)=p_0-μ is estimated by Δ^(i)=r0(i)−rp(i) (i)=r_0(i)-r_p(i), with rp(i)=μ^r_p(i)= μ. Data. We draw N variants V1,…,VNV_1,…,V_N independently and run the agent K times on each, recording the pass count and empirical pass rate Xj∣(Vj=vj)∼Binomial(K,pvj),p^j=Xj/K.X_j (V_j=v_j)\; \;Binomial(K,p_v_j), p_j\;=\;X_j/K. (5) Our protocol is N=20N=20, K=1K=1. A useful identity. One identity relates the within-variant noise to the two estimands and is used repeatedly below. Write ν=⁡[pV(1−pV)]ν\;=\;E [p_V(1-p_V) ] (6) for the average within-variant Bernoulli variance. Expanding and applying ⁡[pV2]=Var⁡(pV)+(⁡[pV])2=σ2+μ2E[p_V^2]=Var(p_V)+(E[p_V])^2=σ^2+μ^2, ν=⁡[pV]−⁡[pV2]=μ−(σ2+μ2)=μ⁡(1−μ)−σ2,ν\;=\;E[p_V]-E[p_V^2]\;=\;μ-(σ^2+μ^2)\;=\;μ(1-μ)-σ^2, (7) so that σ2+ν=μ⁡(1−μ).σ^2+ν\;=\;μ(1-μ). (8) The total variability in a single perturbed run therefore splits into a between-variant part σ2σ^2 and a within-variant part ν. The estimator The estimator of μ is the average of the per-variant rates, μ^=1N∑j=1Np^j. μ\;=\; 1N _j=1^N p_j. (9) At K=1K=1 we have p^j=Yj∈0,1 p_j=Y_j∈\0,1\ and this collapses to μ^=#variants resolvedN, μ\;=\; \#\variants resolved\N, (10) an ordinary binomial proportion. We use this formula to calculate resolve rate for perturbed runs. Why it is unbiased for any K, including K=1K=1 The argument conditions on the variant first and averages over variants second, using the tower property [⋅]=[[⋅∣V]]E[\,·\,]=E [E[\,· V] ]. Step 1: collapse the agent randomness. Condition on the j-th variant VjV_j and write Yj1,…,YjKY_j1,…,Y_jK for the K runs on it. By Eq. (2) each has conditional mean pVjp_V_j, so ⁡[p^j∣Vj]=⁡[1K∑k=1KYjk|Vj]=1K∑k=1K⁡[Yjk∣Vj]=1K⋅KpVj=pVj. splitE[ p_j V_j]&\;=\;E [ 1K _k=1^KY_jk\, |\,V_j ]\;=\; 1K _k=1^KE[Y_jk V_j]\\ &\;=\; 1K· K\,p_V_j\;=\;p_V_j. split (11) The factors of K and 1/K1/K cancel, so this holds regardless of K. Even a single draw, K=1K=1, is already unbiased for its own pVjp_V_j. Step 2: collapse the variant randomness. Because VjV_j is itself random, the conditional mean in Eq. (11) is a random variable, and averaging it over the variant population gives, by the tower property and Eq. (3), ⁡[p^j]=⁡[⁡[p^j∣Vj]]=⁡[pVj]=μ.E[ p_j]\;=\;E [\,E[ p_j V_j]\, ]\;=\;E [\,p_V_j\, ]\;=\;μ. (12) Step 3: average over variants. By linearity of expectation, applied to Eq. (9), ⁡[μ^]=1N∑j=1N⁡[p^j]=1N⋅Nμ=μ.E[ μ]\;=\; 1N _j=1^NE[ p_j]\;=\; 1N· Nμ\;=\;μ. (13) Consequence. The parameter K never appears in Eqs. (11)–(13). Repetition is a variance-reduction operation. It tightens each p^j p_j around the center pVjp_V_j, but that center was already correct, so tightening cannot remove a bias that was never there. One run per variant is exactly as unbiased as a thousand. Variance, and why K=1K=1 is the efficient allocation Unbiasedness alone does not justify K=1K=1. An unbiased estimator can still be noisy. We now show that K=1K=1 is not only admissible but optimal. Step 1: variance of one per-variant rate. Decompose Var⁡(p^j)Var( p_j) with the law of total variance, Var⁡(p^j)=Var⁡(⁡[p^j∣Vj])+⁡[Var⁡(p^j∣Vj)].Var( p_j)\;=\;Var (E[ p_j V_j] )\;+\;E [Var( p_j V_j) ]. (14) The first term is Var⁡(pVj)=σ2Var(p_V_j)=σ^2 by Eqs. (11) and (4). For the second, p^j=Xj/K p_j=X_j/K with Xj|Vj∼Binomial⁡(K,pVj)X_j V_j (K,p_V_j), so Var⁡(p^j∣Vj)=pVj(1−pVj)/KVar( p_j V_j)=p_V_j(1-p_V_j)/K, whose expectation over variants is ν/Kν/K by Eq. (6). Hence Var⁡(p^j)=σ2+νK.Var( p_j)\;=\;σ^2+ νK. (15) Step 2: variance of the estimator. The variants are drawn independently and each is run in an isolated environment, so the p^j p_j are i.i.d. and Var⁡(μ^)=1N2∑j=1NVar⁡(p^j)=σ2N+νNK=σ2N+νR,Var( μ)\;=\; 1N^2 _j=1^NVar( p_j)\;=\; σ^2N+ νNK\;=\; σ^2N+ νR, (16) writing R=NKR=NK for the total number of agent runs. The within-variant noise term ν/Rν/R depends only on the total budget R; how those runs are distributed across variants is irrelevant to it. The between-variant term σ2/Nσ^2/N depends only on the number of distinct variants N, and K does not appear in it at all. Step 3: optimize under a fixed budget. Assume a fixed run budget R and substitute N=R/KN=R/K into Eq. (16): Var⁡(μ^)=Kσ2R+νR=Kσ2+νR.Var( μ)\;=\; Kσ^2R+ νR\;=\; Kσ^2+νR. (17) This is strictly increasing in K whenever σ2>0σ^2>0, and is therefore minimized at K=1K=1. Under a fixed number of agent runs, the variance-optimal design for estimating the mean is to spend every run on a new variant and run the agent once. Step 4: the K=1K=1 specialization. At K=1K=1 the estimator’s variance simplifies. Substituting the identity (8) into Eq. (17) with K=1K=1 and R=NR=N, Var⁡(μ^)=σ2+νN=μ⁡(1−μ)N.Var( μ)\;=\; σ^2+νN\;=\; μ(1-μ)N. (18) This has a direct interpretation. Each YjY_j is marginally Bernoulli⁡(μ)Bernoulli(μ) and the N trials are independent, since each uses a distinct variant and no two share a draw. So μ μ is a textbook binomial proportion with standard error μ^(1−μ^)/N μ(1- μ)/N, and rp(i)r_p(i) may be compared to r0(i)r_0(i) with a standard two-proportion procedure. No clustering correction is needed at the variant level precisely because every trial is a different variant. F Qualitative Analysis Details Instance Selection Procedure 12/20 of these instances are manually picked by evaluating characteristics like high variance (3), high degradation (3), large increase in step count (1) or file coverage (2) or cost (1), or cases with 1/20 mutant failures (2). The remaining 8/20 are picked by passing mutants that the agent had failed to resolve through an LLM prompt designed to take the SPT log as well as a condensed version of the agent trajectory as input, and predict possible SPT impact based on the SPTs that were applied, intersection of files touched by both the agent and the SPTs, and specific snippets from agent thoughts or tool outputs that have SPT markers. This is done for 115 failing trajectories across all 16 experiments. The resulting output is then manually evaluated to shortlist 8 trajectories for final human review. AI-Assisted Trajectory Analysis The same prompt also extracts a sequence of events from the trajectory, citing snippets from the raw trajectory for each, while also classifying each event as one of the following: Localization (exploring the repo to find relevant code), Debugging (reasoning, specifically based on observed outputs), Planning (forming a plan for patching), Patching (Implementation of changes), Validation (Testing of changes), Recovery (Backtracking on implemented changes or proposed plan) or General (additional events that don’t come under any of the 6 aforementioned types, for example, patch submission). A final HTML is generated, containing the raw trajectory, a graph with the extracted events, including snippets from the actual trajectory, a record of files touched by both the agent and the SPTs, and script based results like git statements, Human evaluators then use this html to get a comprehensive view of the trajectory while analyzing. This is strictly an assistive method, and all qualitative analysis examples mentioned in the paper have involved actual human review of all relevant portions of the raw trajectories. Additional Qualitative Insights Variability in SPT detection. There is high variance in whether SPTs are detected at all even in cases where the agent repeatedly accesses mutated code. While some SPTs like double negation are easier to miss, SPTs which have been detected in other cases, like if true wrapper, are ignored in cases like Opus on Psf-Requests with mini-SWE agent. In this particular case, even though the agent accesses mutated code multiple times, it never acknowledges any SPT presence. Moreover, consistent with the qualitative results, SPT impact appears to be variable across instances, with more impact seen in repos like qutebrowser, openlibrary, ansible, pytest and sympy as compared to repos like nodeBB or astropy, based on qualitative reviews. Additional examples of agent response to SPTs. There are multiple other examples of how agents choose to respond to SPTs. In Qwen on Matplotlib with OpenCode, after detecting SPTs, the agent decides to reset the repository. However, it claims to have reset it multiple times without actually doing so, and only later ends up proceeding with it. In another case, Opus on Qutebrowser with mini-SWE agent, the agent notices complicated booleans introduced by the if true wrapper in the relevant code, and spends additional time in resolving the booleans into final true/false values. It later also "fixes" them in the final patch, thereby investing significant effort in interacting with the SPTs. In contrast, MiniMax with mini-SWE agent on Qutebrowser chooses to avoid core files that have been transformed by SPTs, and instead tries to get information from other sources in the repository, subsequently increasing the time taken to localize the issue as well as file coverage. Input prompt impacts agent jaggedness. We observe that issue description quality and nature of issue may impact agent jaggedness. We analyse Qwen, Kimi and Opus on the same NodeBB issue which exhibits high variance across various model and scaffold pairs. The issue description contains two conflicting statements — editing the code to satisfy issue requirements causes one of the unit tests to break, and the description explicitly states not to alter any unit tests. This leads to all three agents falling into a loop of patching, validating, and debugging, while going back to the issue description multiple times, which substantially adds to both the step count and cost. In a passing trajectory for the same issue, the agent comes across the same conflict and fails the same test, but doesn’t backtrack on the code and therefore ends up passing. So the conflicting nature of the issue instructions greatly contributes to jaggedness in agent performance, as it is left to its own judgement to resolve the conflict. Robustness of test suite formation varies. For Astropy, Kimi uses a three-level test suite, complete with custom tests for basic functionality, project test suite, and a dedicated edge cases script. However, in Pytest and Qutebrowser, the validation framework is brittle, with insufficient debugging and incorrect failure attribution to SPTs in the former and shifting test cases during debugging in the latter. Similarly, for MiniMax, validation setup is comprehensive in Sympy, but has incorrect assertions in custom test scripts in Qutebrowser.

Instant research discovery