Paper deep dive
Evaluating LLM-Based Test Generation Under Software Evolution
Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 1:47:47 AM
Summary
This paper presents a large-scale empirical study evaluating the robustness of LLM-based unit test generation under software evolution. By analyzing 22,374 program variants across eight LLMs, the authors demonstrate that while LLMs perform well on original code, they struggle with regression awareness when code undergoes semantic-altering changes (SACs) or semantic-preserving changes (SPCs). The study reveals that LLMs often rely on surface-level patterns rather than deep semantic reasoning, leading to test instability and residual alignment with original program behaviors.
Entities (5)
Relation Signals (3)
LLM-based test generation ā exhibitsinstabilityunder ā Semantic-preserving changes (SPC)
confidence 100% Ā· SPC edits typically introduce larger syntactic changes than SAC edits while preserving semantics, yet they trigger greater instability in generated test suites.
LLM-based test generation ā exhibitsperformancedegradationunder ā Semantic-altering changes (SAC)
confidence 100% Ā· Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%.
LLMs ā relieson ā surface-level patterns
confidence 90% Ā· Overall, our findings show that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are increasingly used for automated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit weaknesses such as reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests and how those tests respond to code evolution is therefore essential. We present a large-scale empirical study of LLM-based test generation under program changes. Using an automated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC) across eight LLMs and 22,374 program variants. LLMs achieve strong baseline results, reaching 79% line coverage and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original behavior rather than adaptation to updated semantics. Performance also declines under SPCs despite unchanged functionality: pass rates fall to 79% and branch coverage to 69%. Although SPC edits preserve semantics, they often introduce larger syntactic changes, leading to instability in generated test suites. Models generate more new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our results indicate that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve.
Tags
Links
- Source: https://arxiv.org/abs/2603.23443v1
- Canonical: https://arxiv.org/abs/2603.23443v1
Trouble viewing inline? Open PDF directly ā
Full Text
71,880 characters extracted from source content.
Expand or collapse full text
Evaluating LLM-Based Test Generation Under Software Evolution Sabaat Haroon Virginia Tech USA Mohammad Taha Khan Carnegie Mellon University USA Muhammad Ali Gulzar Virginia Tech USA Abstract Large Language Models (LLMs) are increasingly used for auto- mated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit important weaknesses, including reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests for a program and how those tests respond to code evolution is therefore essential. In this work, we present a large-scale empirical study of LLM-based test generation under program evolution. Using an auto- mated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC). Our evaluation spans eight LLMs and 22,374 pro- gram variants derived from widely used benchmarks. LLMs achieve strong baseline results, reaching 79% line cover- age and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original program behavior rather than adaptation to the updated semantics. Performance also drops under SPCs, despite unchanged function- ality: test pass rates fall to 79% and branch coverage to 69%. SPC edits typically introduce larger syntactic changes than SAC edits while preserving semantics, yet they trigger greater instability in generated test suites. On average, models produce 1.2Ćmore new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our find- ings show that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve. 1 Introduction Large Language Models (LLMs) are increasingly integrated into software engineering workflows, with automated test generation emerging as a prominent use case [12]. Early studies and bench- marks suggest that LLMs can generate syntactically valid and often semantically correct test cases for well-known public programming tasks [8,18,25,45]. However, producing complete and reliable automated test suites requires deep reasoning about control flow, execution paths, and the specific functional state of the provided implementation. Real-world software is inherently dynamic, code is frequently reused, refactored, and slightly tweaked to serve dif- ferent functional purposes. When developers supply this modified Conferenceā17, Washington, DC, USA 2026. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n code to an LLM to generate a high-coverage test suite, they expect the resulting tests to accurately reflect the current code. Problem. Current studies on LLM-based test generation largely overlook how code evolution impacts model robustness and be- havior [18,40,41]. This gap leaves a critical question: do LLMs genuinely comprehend the semantics of the provided code, or are they performing shallow pattern replication of programs seen in pre-training? Consider a scenario in which a developer takes an existing open- source function and makes a minor semantic-altering modification to adapt it to a new use case. Even when explicitly prompted to gen- erate high-coverage tests for this new code, an ideal test generator (e.g., random test generators [34], Fuzzers [27,45] or Symbolic exe- cution [5]) should adapt to the provided logic. However, if the LLM relies heavily on memorized structures, it may implicitly assume the modified code is a "buggy" version of the original program. It may also completely overlook the semantic implication of the code change. Consequently, it might generate test cases that fail on the provided code but paradoxically pass on the original, unmodified version seen during training. Conversely, if a developer applies a semantic-preserving mutation, such as renaming variables or refactoring logic without changing the output, the generated test coverage and passing rates should remain stable. If an LLM cannot maintain resilience under these harmless lexical changes, its utility in real-world test generation scenarios is compromised. Current State of the Art. Traditional test generation systems [14, 20,26,38] and fuzzers [27,45], are designed to maximize code coverage, while modern LLM code generation benchmarks (e.g., HumanEval, LiveCodeBench) focus almost exclusively on func- tional correctness in code generation tasks [11]. Neither approach characterizes how an LLMās test generation behavior shifts as soft- ware evolves. Although mutation testing is a standard technique for assessing test suite adequacy, it has not been systematically adapted to evaluate LLM-based test generation under program changes. It is currently unknown whether LLMs can successfully adapt tests to functional changes while remaining resilient against changes that do not affect behavior. Contributions. A challenge in evaluating LLM-based test genera- tion is distinguishing genuine semantic reasoning from alignment with patterns observed during training. High coverage on static benchmarks alone cannot resolve this question. A model may ap- pear effective simply because the program under test resembles implementations encountered during training, allowing it to repro- duce familiar test structures without reasoning about the specific code instance. To address this limitation, we evaluate LLM behavior under controlled program changes. We introduce a mutation-driven evaluation framework that ap- plies two classes of code changes. Semantic-altering changes (SACs) produce program variants with modified behavior, requiring tests to adapt to new semantics. Semantic-preserving changes (SPCs) modify arXiv:2603.23443v1 [cs.SE] 24 Mar 2026 Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar the code surface while preserving functionality. Because the pro- gram behavior remains unchanged, degradation under SPCs reveals sensitivity to structural or lexical variation rather than semantic reasoning. By analyzing LLM behavior under both change types, we isolate whether performance differences arise from semantic misunderstanding or reliance on superficial code patterns. This dual perspective allows us to characterize three properties of LLM- based test generation: sensitivity to semantic change, resilience to non-functional structural change, and stability of generated test suites across evolving programs. We design an automated end-to-end evaluation framework that generates baseline tests, measures coverage, injects SACs and SPCs, and evaluates test quality across program variants. Beyond cover- age metrics, we analyze failing tests to determine whether their assertions align with the mutated program behavior or with the original program semantics. This analysis enables us to directly assess whether LLM-generated tests reflect true reasoning about program behavior or shallow pattern replicationāinsights that static public benchmarks cannot provide. Experimental Results. We evaluated 8 state-of-the-art LLMs us- ing 22,374 program variants sourced from the Project CodeNet dataset [35], specifically focusing on Java and Python implementa- tions. Our baseline evaluation reveals that LLMs perform well on the original, unmodified programs, achieving an average of 79.2% line coverage and 76.1% branch coverage with fully passing test suites, containing 13.1 tests per program, on average. However, this performance degrades sharply under code changes. When subjected to SACs, the pass rate of newly generated tests plummets to 66.5%, and branch coverage falls to 60.6%. Crucially, further analysis of failing test cases finds that, among 119,163 tests generated under SACs, over 99% of the 23,977 tests that failed on the mutated code pass on the original program when executing the modified region. These results support our hypothesis that LLMs lack the precision to reason about semantic code changes and instead rely heavily on pat- terns observed during training. As a result, they often fail to capture behavioral changes introduced by evolving programs and rarely generate tests that reflect the updated semantics. Furthermore, when subjected to SPCs, the test pass rate decreases from 100% to 79%, and branch coverage drops to 69%. Despite the same functionality, LLMs interpret SPC-modified code as substan- tially different programs, triggering 1.2Ćmore new test generations than under SACs. We attribute this behavior to the slightly larger syntactic changes introduced by SPCs, which typically span 1-2 lines of code, whereas SACs are restricted to single-line changes. As a result, changes with higher syntactic impact but no seman- tic effect lead to larger differences in generated test suites, while changes with real semantic impact but minimal syntactic change produce smaller shifts. However, a model capable of meaningful code reasoning should remain robust to such SPCs. Instead, our findings show that larger edit distancesāeven when semantically irrelevantādisrupt LLMsā pattern matching and trigger exploratory regeneration of tests, leading to unstable test suites. Overall, these findings indicate that LLM-based test generation relies heavily on surface-level patterns and memorized structures, revealing a limited ability to comprehend and adapt to evolving code. Data Availability. The artifacts and datasets used in this study are publicly available at https://doi.org/10.5281/zenodo.18898624. 1 n = int(input ()) 2 goods1 , goods2 = [], [] # ... up to goods10 3 count = 0 4 5 # SAC: Shifted to exclusive range logic 6 for _ in range(n - 1): # originally range(n) 7 check = input () 8 finder = len(check) 9 if finder == 1: 10 if check not in goods1: 11 goods1.append(check) 12 count += 1 13 # SPC: Redundant Else injected here 14 else: 15 pass 16 17 if finder == 2: 18 # <ADDITIONAL CODE ON REMAINING LENGTH CHECKS> 19 print(count) Figure 1: Fragment of CodeNet program showing injected SAC and SPC 2 Motivating Example We present a motivating example based on an instance from the CodeNet Python dataset (ID:p02701_s025427336). The program reads an integerķfollowed byķstrings, groups them by length (from 1 to 10 characters), and prints the total number of distinct strings across all lengths. We first prompt GPT-5.2 to generate a test suite for the original, unmodified program. The model successfully generates a suite of 7 test cases, achieving 100% line coverage and 81.0% branch coverage, with a 100% test pass rate. This establishes a strong baseline, demonstrating the problemās tractability and that LLMs can meet high code-coverage objectives by generating executable, passing test cases for the string-bucketing logic. Next, we simulate a functional extension by injecting a code change that alters semantics. Specifically, we modify the input- reading logic to process only the firstķ ā1 entries, effectively treating the final input as a terminator or metadata rather than a data point. This change establishes a new valid functional re- quirement: the program intentionally ignores the last string in the sequence. Note that we do not aim for functional alignment, as LLMs are not provided with the codeās functional requirements. Instead, LLMs are asked to achieve code coverage as the primary objective, as in the automated test generation literature [40, 41]. We then provide this modified version to a separate, independent instance of the LLM, instructing it with generating a high-coverage test suite from scratch. By using a fresh session, we ensure that the model has no memory of the previous baseline code, forcing it to rely entirely on the provided code to reason upon. When evaluating the tests generated for this mutant, we observe that line coverage drops moderately to 68%, and branch cover- age falls to 50%. However, the test pass rate drops to 82%, mean- ing several tests fail. Upon closer inspection, the specific failing test case shown in Figure 3 stands out as particularly revealing: test_all_lengths_unique. This test generates inputs that feed exactly 10 distinct strings through the bucketing logic. However, despite receiving no external specification in the prompt, the model explicitly asserts that the program will output"10". This reveals that rather than deriving the expected behavior from the provided Evaluating LLM-Based Test Generation Under Software EvolutionConferenceā17, July 2017, Washington, DC, USA mutated code, the LLM forcefully matches the assertion to the original algorithmic specification it observed during pre-training. On the original program, this test passes. When subjected to SAC the loop terminates after 9 iterations, the 10th string ("j") is never read, and the output is"9", causing the assertion to fail. The LLMās failure to adjust the expected output to"9"demonstrates a lack of semantic grounding. Rather than deriving assertions from the provided code, the model displays residual alignment: it halluci- nates the requirements of the standard algorithm from its training data, ignoring the explicit functional logic of the current instance. To further evaluate robustness, we return to the original baseline code and apply a semantic-preserving code change (SPC). We insert a purely redundantelse: passblock immediately following the first length check (see Figure 1). Because this change preserves the exact same control flow and program behavior, a robust LLM should generate a test suite comparable to the baseline. Instead, the SPC causes line coverage to drop to 79% and branch coverage to 64%. More concerningly, the test pass rate drops to 80%. This degradation reveals an over-reliance on syntactic visibility at the expense of semantic reasoning. Because SPCs often introduce highly visible structural modifications such as inserting an entirely new code block, the LLM easily detects the surface-level disrup- tion but struggles to recognize its semantic impact, prompting it to unnecessarily discard valid tests and generate new, inferior suites. Conversely, SACs typically possess a minimal syntactic footprint (e.g., modifying a single operator) but carry larger semantic impli- cations. Because these behavioral shifts are visually subtle, LLMs frequently overlook them. 3 Research Questions To guide this study of LLM-based testing under the program changes, we investigate the following research questions: ā¢RQ1: To what extent can LLMs generate structurally valid and high-coverage test suites for software benchmarks in their initial state? ā¢RQ2: How sensitive are LLMs to semantic-altering code changes, where meaningful changes should cause tests to adapt? ā¢RQ3: How resilient are LLMs to semantic-preserving code changes, where program behavior is functionally identical? ā¢RQ4: What explains the failures of LLM-generated tests under Semantic Altering Code Changes? ā¢RQ5: To what extent do LLM-generated test suites demon- strate regression awareness? 4 Methodology We design an automated pipeline that benchmarks the adaptability of LLM-generated test suites. As shown in Figure 2, our framework subjects the model to both SACs and SPCs to quantify its ability to remain aligned with the current code state. In the first phase, we establish a baseline to assess the modelās initial generation capabilities. We prompt the LLM to generate test suites for a set of seed programs and strictly retain only those programs for which all generated tests compile and pass perfectly. It ensures that any degradation observed in later stages is directly attributable to the injected code changes, rather than a general inability to comprehend the original codebase. The second phase evaluates the modelās adaptability to code changes. We apply SACs, detailed in Table 1, to represent functional changes, such as changing a loop boundary or an arithmetic operator. Second, we apply SPCs, also outlined in Table 1, to represent non-functional code refactoring, such as inserting a redundant condition or renaming a variable. While real- world commits and pull-requests reflect natural developer behavior, utilizing them introduces a critical issue of data contamination i.e., LLMs have already been exposed to these open-source commits, along with their corresponding regression tests, during pre-training. Furthermore, manually crafting novel, realistic pull requests at scale is infeasible. Therefore, systematically applied mutations serve as a scalable and robust proxy for code changes. To facilitate a uniform evaluation, we employ a standardized two- shot prompting strategy, where the second shot performs a repair cycle for the LLM in cases where the generated tests fail to compile due to errors, and this strategy is applied consistently across all phases of the pipeline. The prompt instructs the model to act as an expert developer and generate a high-coverage, successfully passing test suite based entirely on the provided code snippet. Instead of a custom prompt design, we employ prompting strategies that are commonly used in prior LLM-based unit test generation studies, where the model is instructed to generate tests directly from the provided implementation[40,49]. If the tests remain erroneous after 2-shot attempt, the test generation process for that specific program version is marked as a failure, and the program is excluded. 4.1 Seed Programs Procurement We procure our seed programs from the Project CodeNet dataset [35], a large scale, widely recognized benchmark. We focus our empir- ical study on Java and Python, two widely used programming languages heavily represented in both real-world software systems and LLM pretraining corpora. By initializing our pipeline with 5,723 candidate programs (3,231 Java and 2,492 Python) sourced from CodeNet, we evaluate models at a scale that significantly ex- ceeds standard test generation benchmarks, such as HumanEval (164 problems)[7] or MBPP (974 problems)[4]. This large scale se- lection ensures our findings are robust and builds upon a verified dataset successfully utilized in numerous previous empirical soft- ware engineering studies [19,30]. We follow the following criteria. ā¢Dataset Criteria: We enforce a strict quality filter requiring all selected programs to be natively compilable, executable, and strictly self-contained. This ensures the model receives the complete logical state without relying on external depen- dencies. ā¢Scale: We initialize our pipeline with 5,723 candidate pro- grams (3,231 Java and 2,492 Python). ā¢ Stratified Sampling: We partition the dataset into four systematic strata based on Lines of Code (LOC) percentiles: 0-25%, 25-50%, 50-75%, and 75-100%. By defining the bound- aries using these quartiles, each band contains an identical number of candidate projects rather than relying on arbi- trary line count limits. Uniformly sampling passing projects from these equal-frequency bands guarantees a balanced evaluation across all code complexities and prevents our results from skewing toward small or large files. Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar Baseline Formation Seed Programs ā¢Java ā¢Python LLM Based Test Generation (Original Code) Repair Prompt with error details TRUE FALSE ā ā X ā ā Filtering only 100% test passing Projects Mutations Injection Injecting Semantic Altering Code Changes Injecting Semantic Preserving Code Changes LLM Based Test Generation (Mutated Code) Repair Prompt with error details Compliable? FALSE TRUE 70%100% 100%100% 0%0% 30%45% 100%100% Record Coverage% & Test Passing Rates 2 Regression Awareness 3 Residual Alignment 4 Compare Original Tests from with Mutated Tests from Finds # of Old, New and Same Tests Finds Tests that are Failing on Mutated but Passing on Original Code 1 2 Executing Failing Mutated Tests from on Original Code from 2 1 1 Compliable? Figure 2: Methodology Overview ModificationTypeDescriptionExample Semantic-Altering Code Changes(SACs) Boundary ShiftSemantic-AlteringModifies the inclusive or exclusive range of a loop or array index. range(n)ā range(n-1) Changed Boolean LogicSemantic-AlteringSwitches boolean operators to alter decision logic. a && bā a || b Changed ArithmeticSemantic-AlteringSwitches arithmetic operators to alter calculation values. a + bā a - b Argument SwapSemantic-Altering Reorders arguments of the same type in a function call, shifting the functional role of the inputs within the program logic. f(sum, price)ā f(price, sum) Variable Role RebindingSemantic-AlteringReassigns a variableās logical role. for x in data: total += xā for total in data: total += total Semantic-Preserving Code Changes(SPCs) Void Loop InjectionStructuralInserts dummy loops that execute but perform no operation. for i in range(1): pass Void ConditionStructuralInserts dummy conditions that always execute but do nothing.if (true) ... Redundant ElseStructuralAdds empty else blocks to statements lacking them. if x: y else: pass Equivalent ComparisonStructuralRewrites comparison operators to logically equivalent forms. x < 10ā !(x >= 10) Unused ParameterStructuralAdds an unused parameter to method signature and updates calls. def f(x):ā def f(x, null): Misleading VariablesIdentifierRenames variables to generic or misleading names. countā sum Misleading CommentsAnnotativeAdds comments that misdescribe code behavior. // Returns sum (above diff code) Misleading MandarinAnnotativeAdds misleading comments translated into Mandarin Chinese.//čæåę»å Remove CommentsAnnotativeRemoves all single-line and multi-line comments. /* comment */ā (empty) Table 1: Types of Semantic Altering and Semantic Preserving Code Changes Applied to Seed Programs 1 def test_all_lengths_unique(self , capsys): 2 """Test that strings of all lengths 1-10 are correctly counted.""" 3 inputs = ["10", "a", "b", "c", ..., "j"] 4 mod = self._import_module_with_input(inputs) 5 out = capsys.readouterr ().out.strip () 6 # Fails under SAC: out is "9" instead of "10" 7 assert out == "10", "All 10 unique strings should be counted" 8 assert mod.goods10 == ["j"] Figure 3: The intriguingtest_all_lengths_uniquetest case generated by the LLM 4.2 Baseline Test Generation and Program Filtering For each seed program, we instruct a set of 8 LLMs (listed in Table 2) to generate high-coverage test suites targeting the original codeās logic. These models were selected for their prominence in current literature and include both widely used open-source and proprietary alternatives. Open-source models were executed on local servers equipped with NVIDIA L40S GPUs, while closed-source models were accessed via their respective APIs. Once generated, the test ModelCreatorSizeType GPT-OSS [33]OpenAI20 BOpen-source Nemotron-3-Nano [31]NVIDIA30 BOpen-source GPT-5 [43]OpenAIUndisclosedClosed-source GPT-5.2 [32]OpenAIUndisclosedClosed-source Claude 4.5 Haiku [2]AnthropicUndisclosedClosed-source Claude 4.6 Sonnet [3]AnthropicUndisclosedClosed-source Gemini 2.5-Flash [15]GoogleUndisclosedClosed-source Gemini 3.1-Pro [16]GoogleUndisclosedClosed-source Table 2: LLMs Evaluated suites are executed against the original programs to establish a baseline for correctness and code coverage. We systematically limit our final baseline set to exactly 100 fully passing projects per LLM. Using the four line-count bands defined during procurement, we continuously generate and evaluate base- line test suites until we successfully identify 25 passing projects from each of the four size bands. This stratified sampling guaran- tees our evaluation is thoroughly balanced across varying code complexities and is not skewed by a disproportionate number of trivial snippets or excessively large files. Crucially, we only select 100 programs because each baseline program is independently sub- jected to 14 distinct code changes (comprising all categories of SACs Evaluating LLM-Based Test Generation Under Software EvolutionConferenceā17, July 2017, Washington, DC, USA and SPCs), an initial set of 100 baseline programs expands to nearly 1,400 independent test suites per evaluated LLM. This massive mul- tiplier effect ensures a statistically rigorous, large-scale evaluation of code evolution while remaining computationally feasible across 8 different models. We enforce a strict filtering condition: a program is only selected for the final 100-project baseline if the LLM successfully generates a test suite with a 100% pass rate on the original code. Any program for which the model produces syntax errors, compilation failures, or at least one failing test is discarded. If a model fundamentally fails to comprehend the original program to generate valid tests, evaluating its response to a subsequent code changes yields confounded results. This rigorous filtering step creates a flawless control group. It guarantees that the LLM demonstrates absolute baseline compe- tence for every retained program. Consequently, we can definitively attribute any subsequent degradation in test pass rates or coverage observed in the later stages directly to the injected mutations, com- pletely isolating the impact of code evolution from inherent task ambiguity or baseline generation deficits. 4.3 Semantic-Altering Code Changes We apply Semantic-Altering Code Changes to the filtered seed pro- grams. These changes represent functional divergences from the original code, simulating common scenarios where a programās logic is updated to meet new operational requirements or bound- ary conditions. In this phase, we provide the modified code to an independent instance of the LLM and instruct it to generate a high- coverage test suite. This generation process ensures the model treats the code as a primary reference, allowing us to evaluate its semantic alignment without the influence of prior versions. This allows us to measure how semantic changes affect the test generation, including changes in test suite size, structural diversity, and code coverage. Table 1 lists the SACs used. Consider two specific examples from our methodology. First, a Boundary Shift code change changes a loop condition fromrange(n)torange(n-1). The expected behav- ior is that the LLM will generate tests reflecting one fewer iteration in the expected output. Second, a Changed Arithmetic mutation swaps an addition operator (a + b) for a subtraction operator (a - b), obligating the LLM to generate assertions that check for a completely different mathematical result. 4.4 Semantic-Preserving Code Changes To evaluate the robustness of LLM-based test generation, we apply Semantic-Preserving Code Changes (SPCs) to the filtered baseline pro- grams. Real-world software maintenance frequently involves code refactoring, variable renaming, and documentation updates that do not alter the underlying execution logic. A reliable, production- ready testing agent must maintain consistent performance across these functionally identical variations. We provide the refactored code to an independent instance of the LLM and instruct it to generate high-coverage tests. Because SPCs strictly preserve program semantics, the expected behavior is that LLM will recognize the unchanged execution logic and generate a test suite that achieves coverage and pass rates comparable to the baseline scenario. Any degradation in coverage or passing% serves as empirical evidence that the model is over-reliant on superficial syntactic or lexical cues rather than active code comprehension. Table 1 lists SPCs. Across our mutation pipeline, SACs modify an average of 1.0 line per program, whereas SPCs modify 2.4 lines on average (with some affecting up to 7 lines); we later analyze how this larger syntactic edit size influences test-suite stability (Section 5.5). For SPCs, consider two specific examples. First, an Equivalent Comparison mutation rewrites a conditional statement fromx < 10to its logical equivalent!(x >= 10). The expected behavior is that the LLM will recognize the identical boundary condition and generate the exact same test assertions. Second, a Misleading Variables mutation might rename a simple loop counter fromcounttosum. A robust model should derive its understanding from the actual operations applied to the variable, rather than being tricked by the deceptive identifier into generating irrelevant assertions about addition. We simulate SPCs into three distinct groups, with full details provided in Table 1: ā¢Structural Refactoring: We introduce control-flow trans- formations that strictly preserve behavior (e.g., injecting a Void Loop or Redundant Else) to test whether the LLM be- comes disoriented by the presence of harmless, non-operational execution paths. ā¢Identifier Refactoring: We systematically rename vari- ables to generic or contextually deceptive identifiers to eval- uate whether the model relies heavily on semantic naming conventions rather than programmatic logic. ā¢Annotative Refactoring: We modify, add, or remove com- ments to manipulate the modelās textual context, including injecting cross-lingual Misleading Mandarin comments, to assess if logical reasoning can be derailed by textual artifacts that contradict actual code behavior. 4.5 Failure Attribution Analysis Consider RQ4 from Section 3, we extend our evaluation pipeline with a failure attribution analysis applied to tests generated by LLMs on programs with SACs. For each SAC, we collect all LLM- generated tests that fail when executed on the changed program. For each such failing test, we perform two additional checks: Original-Program Execution. The failing test is executed on the original, unmodified version of the program. If the test also fails on the original program, the failure is attributed to poor or malformed test generation rather than misalignment with programās logic. Code Change Execution Coverage. We verify whether the test executes the changed statement or control-flow region, ensuring that the observed failure is causally related to the code change rather than an unrelated execution path. A failing test is classified as residually aligned with the original program semantics if it satisfies both of the following conditions: (1) it passes when executed on the original program, and (2) it executes the mutated code region while failing on the mutated program. This analysis validates our hy- pothesis that LLMs tend to match code to specifications resembling programs seen during training. Consequently, they often overlook code changes with substantial semantic impact and generate tests that remain aligned with the original program behavior, even when the modified program exhibits diverging semantics. Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 Percentage (%) 47.5 56.4 66.1 63.0 69.0 65.6 Changed Boolean Logic Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 48.3 56.7 68.8 62.7 68.9 65.0 Changed Arithmetic Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 55.3 62.2 75.1 74.4 79.6 71.5 Boundary Shift Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 60.5 66.0 72.8 54.4 64.2 58.7 Argument Swap Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 48.7 55.9 61.3 64.9 72.1 72.7 Variable Role Rebinding JavaPythonBranch Cov.Line Cov.Tests Pass. Figure 4: Performance degradation of all LLM-generated test suites across Semantic-Altering Code Changes 500100015002000 Number of Programs Tried to Achieve 100 Passing Suites GPT-5 Mini Gemini 3.1 Pro GPT-5.2 Claude 4.6 Sonnet GPT-OSS Claude 4.5 Haiku Nemotron 3 Nano Gemini 2.5 Flash LLM Language Java Python Figure 5: Programs evaluated per model to achieve exactly 100 fully passing baseline test suites. The horizontal gap high- lights the increased difficulty of test generation in Python compared to Java. 5 Experimental Results This section presents the empirical results of our large-scale evalu- ation of LLM-based test generation under code changes. Across the entire evaluation pipeline, we ran 22,374 test generation tasks on the evaluated models, consuming approximately 346 million tokens (including both input prompts and generated outputs). 5.1 RQ1: Baseline Performance on Unmodified Programs We begin by establishing a control. For each seed program, we prompt the LLMs to generate test suites for the original, unmodified code. To ensure we are evaluating the impact of mutations rather than the inherent modelās ability to generate tests for that program, we strictly retain only those programs for which the generated tests achieve a 100% pass rate. While this filtering step guarantees a control group for later phases, tracking the number of attempts required to secure these 100 passing programs per model reveals significant variations in baseline test generation capabilities. As illustrated in Figure 5, the efficiency of LLMs varies drastically across both models and pro- gramming languages. Highly capable models like GPT-5, Gemini 3.1 Pro, and Claude 4.6 Sonnet required relatively few attempts to hit the target threshold (e.g., Claude 4.6 Sonnet tried 180 programs to secure 100 passing Java suites). Conversely, we observe a severe performance bottleneck in Python test generation across several models. For example, Gemini 2.5 Flash required 2,155 attempts in Python compared to only 323 in Java. This indicates that producing fully self-contained, syntactically correct, and logically sound test suites in Python poses a distinctly higher baseline challenge for current LLMs. We attribute this discrepancy primarily to the fundamental dif- ferences in the languagesā type systems. Javaās static and strong typing enforces explicit type declarations for all variables, method arguments, and return types. This rich syntactic structure provides the LLM with definitive deterministic context, dramatically reduc- ing the search space for valid test inputs and expected behaviors. In contrast, Pythonās dynamic typing requires the model to implicitly infer expected data types and object structures from the surround- ing control flow. This reliance on implicit type inference forces the LLM to make frequent assumptions, significantly increasing the probability of generating type-mismatched inputs, invoking incompatible methods, or hallucinating incorrect object structures during test generation. Beyond type systems, this performance gap may also stem from a scarcity of formal Python tests in the modelsā training corpora. Python is heavily utilized in exploratory environ- ments such as computational notebooks, where developers rarely write structured unit tests [29]. Analyzing Baseline Test Failures. Analyzing the discarded programs sheds light on where LLMs struggle before any code changes are even introduced. The failures that prevented test suites from achiev- ing a 100% pass rate generally fall into two categories: ā¢Syntactic and Environmental Errors: A significant por- tion of failures i.e., 39% of failures, particularly in Python, resulted from structural hallucinations. Models frequently generated tests that attempted to import nonexistent test- ing utilities or hallucinated external file dependencies that violated our self-contained dataset criteria. In Java, failures often involved incorrect class instantiation or mismatched accessibility modifiers (e.g., attempting to test private helper methods without reflection). ā¢Logical Assertion Failures: In 61% of discarded cases, the tests compiled successfully but failed during execution be- cause the LLM fundamentally misunderstood the programās edge cases. These logical mismatches prove the necessity of our filtering phase because if a model cannot correctly assert the behavior of the original algorithm, its performance on a changed version cannot yield reliable insights. Evaluating LLM-Based Test Generation Under Software EvolutionConferenceā17, July 2017, Washington, DC, USA Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 Percentage (%) 62.7 67.8 88.0 76.1 80.7 69.8 Equivalent Comparison Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 63.5 68.7 89.3 83.4 86.0 77.9 Misleading Comments (EN) Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 63.0 68.5 89.1 81.9 84.6 76.1 Misleading Comments (ZH) Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 63.0 68.5 89.7 80.4 83.5 75.6 Misleading Variables Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 63.9 68.8 90.6 80.1 83.9 76.1 Redundant Else Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 Percentage (%) 62.4 68.1 86.7 66.5 69.3 60.9 Remove Comments Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 60.0 66.1 88.9 55.4 63.9 47.0 Unused Parameter Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 60.7 68.4 88.5 50.4 55.1 49.6 Void Condition Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 62.7 68.3 90.0 82.3 85.5 77.3 Void Loop JavaPythonBranch Cov.Line Cov.Tests Pass. Figure 6: Performance degradation of all LLM-generated test suites across Semantic-Preserving Code Changes Key Takeaway: LLMs struggle to generate syntactically valid tests because the task requires deep code comprehension, not just code generation. Consequently, models perform drastically better in Java, where strong typing reduces ambiguity compared to Python. 5.2 RQ2: Impact of Semantic-Altering Code Changes For each program with SAC, we prompt the LLMs in an isolated in- stance to generate test suites for the updated code and evaluate their effectiveness directly on the changed program. Across all evaluated programs, test generation effectiveness deteriorates noticeably once program behavior changes. The average test pass rate drops from 100% in the baseline to 66.5%, representing a 33.4 percentage-point decrease. Coverage also declines substantially. Line coverage de- creases from 79.3% to 67.4% (an 11.9 percentage-point reduction, or 15.0% decrease), while branch coverage falls from 76.1% to 60.6% (a 15.5 percentage-point reduction, or 20.4% decrease). These declines indicate that once program behavior diverges from the original version, LLM-generated tests frequently fail to capture the updated logic and execute fewer relevant program paths. Beyond the overall drop in performance, several trends emerge. First, branch coverage degrades more than line coverage. This sug- gests that LLMs struggle particularly with reasoning about altered control-flow decisions introduced by functional changes, especially when the semantic-altering changes occur directly within branch predicates. While the generated tests may still execute parts of the program, they often fail to construct inputs that explore newly introduced branches or modified conditional logic. Second, we observe a phenomenon we term Scattershot Testing, where the total number of generated tests actually increases slightly (from an average of 13.2 to 13.9) despite a severe drop in cover- age. This suggests that when LLMs encounter altered logic that diverges from recognized program patterns, they lose their ability to conceptualize an optimal testing strategy. Instead of targeting new execution paths, the models attempt to brute-force coverage by generating a higher volume of shallow tests. This scattershot approach artificially inflates the test count but fails to engage with the altered control flow, perfectly explaining the simultaneous rise in test volume and sharp decline in branch coverage. Figure 4 further illustrates these patterns across different cate- gories of semantic-altering code changes. Boundary shifts retain notably higher line coverage (73.4%) compared to other mutations, indicating that the generated tests still manage to execute portions of the affected loops or bounds. However, a persistent gap exists: branch coverage remains substantially lower than line coverage across every single SAC category. This reinforces the observation that while LLMs successfully generate inputs to trigger the altered code blocks along primary execution paths, they struggle to provide the diverse test cases necessary to explore all alternative branches and conditional outcomes of the new logic. Key Takeaway: Even small functional changes can hurt test performance. Models struggle with new control-flow paths and tend to generate tests based on the old program structure. 5.3 RQ3: Impact of Semantic-Preserving Code Changes As with SACs, we regenerate test suites after applying SPC and evaluate them on the refactored code. Although the underlying semantics remain identical to the baseline, we still observe a mea- surable decline in test generation performance. Compared to the baseline, the average test pass rate drops from 100% down to 78.9%, representing a 21.0 percentage-point decrease. Coverage metrics ex- hibit a corresponding degradation. Average line coverage decreases from 79.3% to 73.7% (a 5.6 percentage-point reduction), while branch coverage falls from 76.1% to 69.2% (a 6.9 percentage-point reduc- tion). Although these drops are smaller than those observed under semantic-altering changes, they are alarming given that the execu- tion behavior has not changed at all. Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 Percentage (%) 75.3 61.5 77.6 79.3 67.4 80.2 88.4 63.6 100.0 Claude Haiku Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 73.8 62.3 75.3 78.5 68.5 78.2 90.7 71.9 100.0 Claude Sonnet Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 66.2 62.8 80.6 69.6 69.3 81.0 80.4 64.2 100.0 Gemini 2.5 Flash Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 71.2 62.1 80.6 74.0 68.4 82.2 71.6 70.1 100.0 Gemini 3.1 Pro Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 Percentage (%) 73.6 69.2 80.0 78.6 76.0 83.5 90.9 83.0 100.0 GPT-5 Mini Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 68.1 55.8 80.9 74.5 64.2 85.6 82.6 56.2 100.0 GPT-5.2 Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 67.3 61.0 77.3 71.9 67.3 79.9 74.0 61.6 100.0 GPT-OSS Branch Cov. Line Cov. Tests Pass. 0 20 40 60 80 100 43.6 36.3 51.5 53.7 46.7 59.2 55.9 40.7 100.0 Nemotron 3 SPCSACOriginal Figure 7: Test pass rates and branch coverage across the original baseline, SPCs, and SACs, broken down by evaluated LLM. Several patterns emerge from these results. First, branch cover- age again declines more than line coverage, suggesting that even purely structural refactorings disrupt the modelsā ability to rea- son about control-flow exploration. While the generated tests still execute many statements in the program, they frequently fail to construct inputs that exercise alternative decision paths. Second, in our previous analysis of semantic-altering changes, models over- compensated by generating more tests. Conversely, under semantic- preserving changes, the average number of generated tests drops from 13.2 down to 12.1. This truncation is most extreme under the Unused Parameter change, where the models generate an av- erage of only 8.1 tests, causing line coverage to plummet to 65.7%. This suggests a "distraction effect", when presented with noisy or extraneous structural elements, the model expends its reasoning capacity trying to interpret the dead logic, resulting in an abruptly truncated and incomplete test suite. These observations show that LLMsā test generation is largely influenced by the syntactic āsurface areaā of a modification rather than its semantic effect. Larger syntactic changes with no seman- tic impact (e.g., SPCs) cause substantial deviations in generated tests, while low-syntax, high-semantic changes (e.g., SACs) often go unnoticed. As illustrated in Figure 6, SPCs consistently reduce branch coverage more than line coverage, highlighting the modelās sensitivity to structural noise rather than true program semantics. Key Takeaway: LLM-based test generation is sensitive to the syntactic and lexical footprint of code changes rather than their true semantic impact. 5.3.1 Discussion: Code Change Sensitivity Across LLMs. Figure 7 provides a consolidated, model-specific view of how different types of code changes impact LLM-generated tests. First, the aggregate data confirms a universal hierarchy of degradation where behav- ioral changes or SACs predictably cause the most catastrophic failures across all models, and purely structural refactorings or SPCs consistently induce a degradation, given that the code is func- tionally the same. No model evaluated was completely immune to the structural noise introduced by SPCs. Even state-of-the-art models like Claude 4.6 Sonnet and GPT-5 Mini experienced roughly a 10 percentage-point drop in pass rates due solely to structural refactoring. This reinforces the takeaway that current LLM test generation relies heavily on surface-level pattern matching rather than robust semantic understanding. Second, the breakdown reveals profound differences in resilience. We observe three distinct profiles of code-change sensitivity: Highly Resilient (e.g., GPT-5 Mini, Claude 4.6 Sonnet): These models demonstrate the strongest adaptability. GPT-5 Mini, in par- ticular, maintained an 82.9% pass rate under SACs and a 90.9% pass rate under SPCs, the highest retention among all evaluated models. They exhibit a stronger capacity to update their internal context when prompted with altered code. High Baseline but Higher Sensitivity (e.g., GPT-5.2): Surpris- ingly, while GPT-5.2 achieved the highest baseline line coverage (85.6%), it proved fragile to semantic evolution. Its pass rate dropped by nearly 44 percentage points (down to 56.2%) when faced with SACs. This suggests a severe degree of "memorization over-fitting." Size Bottleneck (e.g., Nemotron-3-Nano): Smaller models strug- gle with evolutionary context. Nemotron-3-Nanoās pass rate dropped to 40.7% under SACs and 55.9% under SPCs, showing that limited- parameter models lack the reasoning depth to handle even minor code changes reliably. Comparing these conditions side-by-side reveals a critical limi- tation in AI-assisted software engineering. While LLMs are highly proficient at two-shot test generation for public benchmarks, their sensitivity to both behavioral shifts and syntactic noise makes them highly unreliable companions for continuous integration environ- ments where code is constantly evolving and refactoring. 5.4 RQ4: Failure Attribution under SACs Figure 8 disambiguates the cause of test failures observed under semantic-altering code changes. For each change category, the figure reports the total number of generated tests, the number of tests failing on the modified program, and the subset of failing tests that pass when re-executed on the original program. Evaluating LLM-Based Test Generation Under Software EvolutionConferenceā17, July 2017, Washington, DC, USA Total TestsFailing on Altered Functionality Passing on Original Test Categories 0 20k 40k 60k 80k 100k 120k Number of Tests 119,163 23,977 23,737 Figure 8: Failure analysis of generated tests under SACs New tests /project Lost tests /project Matched tests /project Test Match Rate % Metrics 0 2 4 6 8 10 12 Average Number of Tests per Project 9.74 11.78 8.76 10.89 5.71 2.43 SAC SPC 0 5 10 15 20 25 30 Test Match Rate (%) 29.33 18.14 Figure 9: Regression awareness of LLM-generated test suites. Across all categories, we analyze 119,163 generated tests, of which 23,977 fail when executed on the modified programs. When these failing tests are re-executed on the corresponding original programs, 23,737 tests pass while still executing the modified code region, yielding a residual-alignment rate above 99%. This pattern is consistent across all semantic-altering change types, including arithmetic changes, logical condition changes, boundary shifts, argument swaps, and variable role rebinding, with attribution rates ranging from 98.6% to 100%. The near-perfect re- covery of failing tests on the original code indicates that the tests themselves are not malformed; instead, they remain aligned with the behavior of the original implementation. Key Takeaway: LLM-generated tests remain aligned to the original code rather than adapting to updated logic. 5.5 RQ5: Regression Awareness Ideally, when code evolves, a test generator should preserve previ- ously valid, high-quality testing behavior and only add or modify tests necessary to exercise the updated behavior. To analyze this property, we track test-suite continuity across program versions by identifying matched, newly generated, and lost tests. We use the coverage profiles of two test cases to determine whether they are matched, i.e., they exercise the same lines of code. 5.5.1 Test Suite Evolution under Code Changes. Figure 9 summa- rizes how test suites evolve under both SACs and SPCs. We also measured the number of lines modified by each change category. Semantic-altering changes modify an average of 1.0 line per pro- gram, while semantic-preserving changes modify 2.4 lines on av- erage, with some changes affecting up to 7 lines. SPCs typically introduce larger structural edits that may cause the LLM to treat the altered program as a different artifact, increasing the likelihood that fewer tests are reused for semantic-preserving changes. Across 8,585 SAC-enabled programs, we observe high levels of test suite instability. When functionality changes, LLMs generate an average of 9.7 entirely new tests while simultaneously discarding 8.8 previously valid tests, resulting in a test-suite churn of 18.5 tests per evaluation. Consequently, the average test match rate between the baseline and the mutated code is a mere 29.3%. This indicates that when program logic shifts, models fail to adapt existing tests, opting instead to regenerate the majority of the suite from scratch. As a result, many high-coverage tests are lost, and the newly generated tests achieve lower overall coverage, indicating that LLMs produce less effective tests than prior high-value tests. Alarmingly, this instability worsens under 10,563 SPC evalua- tions. Although semantic-preserving changes leave the underlying program semantics unchanged, LLMs exhibit even higher churn (22.7 tests per evaluation), generating 11.8 new tests and discard- ing 10.9 existing ones. This drives the average match rate down to a 18.1%. And since the overall coverage of the regenerated test suite after SPCs always decreases, the discarded tests were more coverage-effective than the newly generated ones. Key Takeaway: LLMs rely on syntactic cues during code changes; they often regenerate test suites from scratch, dis- carding high-coverage tests and replacing them with lower- coverage ones. 6 Discussion The results of our study reveal two key insights about the behavior of LLM-based test generation under evolving code. First, the models rely heavily on surface-level syntactic cues rather than the true semantic impact of a change. As a result, even when program be- havior remains unchanged, structural refactorings can significantly disrupt the generated test suites. Second, larger surface-level edits appear to push the models away from simple pattern matching and encourage them to attempt deeper reasoning about the code logic. This suggests that the magnitude of code changes strongly influences how LLMs interpret and respond to modified programs. These observations point toward potential directions for improv- ing the LLM-driven test generation. One promising approach is to preprocess code changes before presenting them to the model. For example, computing a code diff and explicitly highlighting the modified regions could help guide the modelās attention toward the parts of the program. Prior work on generating natural language summaries of code diffs and automated commit messages [24,52] may also help communicate the intent behind modifications [13], enabling the model to better understand how program behavior has evolved. Additionally, static program analysis techniques could extract structural information such as control-flow and data-flow differences between program versions, providing a clearer signal about the functional impact of the change. Overall, these directions suggest that future systems should incorporate mechanisms that help LLMs reason about the magnitude and nature of code changes rather than relying solely on raw source code inputs. Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar 7 Threats to Validity Internal Validity: To mitigate the risk of adding unintended func- tional shifts during SPCs, we test updated code against the original baseline test suites and verify a 100% pass rate on the original code, ensuring that SPCs are behavior-neutral and do not cause any func- tional shifts. Another threat is the inherent non-determinism of Large Language Models. Model outputs can vary across attempts, potentially affecting test pass rates and coverage metrics. We ad- dressed this by using fresh, independent model instances for each generation task to prevent context leakage, strictly evaluating im- mediate outputs to capture the modelsā baseline reasoning. External Validity: Our evaluation is grounded in 22,374 pro- gram variants sourced from the Project CodeNet dataset, focusing specifically on Java and Python implementations. While these rep- resent widely used languages and common algorithmic tasks, the findings may not fully generalize to highly complex, multi-file enterprise codebases where deep contextual dependencies exist. However, because the models already struggle to maintain seman- tic grounding on these relatively simple, self-contained programs, we consider our observed performance degradation to be a conser- vative baseline, introducing greater architectural complexity would likely only further reduce test reliability. Additionally, our study evaluates a specific subset of 8 state-of-the-art LLMs. As model ar- chitectures rapidly evolve, future iterations could display different sensitivities to code evolution. Construct Validity: We relied on line coverage, branch cover- age, and test pass rates to evaluate test suite quality. While these are standard test quality metrics, they do not fully capture a test suiteās fault-finding capability or the exact developer intent. How- ever, in the context of evaluating semantic grounding and residual alignment, these metrics, combined with our failure attribution analysis, provide a robust and objective framework for quantifying the modelsā sensitivity to functional and structural changes. Related Work Traditional automated test generation. Automated test genera- tion is extensively studied with traditional techniques like feedback- directed random testing, such as Randoop [34] and JCrasher [9], search-based software testing (SBST), such as EvoSuite [14] and Pynguin [26], and symbolic execution frameworks like KLEE [5] and DART [17]. These approaches generate test inputs to maximize code coverage, such as branches or statements. Most existing ap- proaches are coupled with specific programming languages and require substantial engineering effort to design and maintain test generation frameworks. Despite this complexity, they lack many features necessary for automated and efficient test generation. LLM-based automated test generation. Recent work has ex- plored using LLMs for test generation with some proposing prompt- ing strategies and fine-tuning to generate developer-like unit tests that achieve both high structural coverage and semantic readabil- ity [21,23]. For example, Sch"afer et al. [40] and Lops et al. [25] provide comprehensive empirical evaluations and generation sys- tems for LLM-driven unit testing. Further advancing these genera- tion techniques, SymPrompt [39] uses a multi-stage, code-aware prompting process aligned with execution paths, while IntUT [28] uses explicit test intentions (e.g., inputs, mocks, and expected out- comes) to guide generation. Similarly, Yuan et al. [49] proposed ChatUniTest, an automated framework leveraging adaptive focal context to minimize hallucinations when generating test suites for complex Java projects. Metaās unit test improvement framework [1] improves and extends existing tests in industrial workflows, and E-Test [36] continuously augments test suites using execution sce- narios harvested from production logs. Extensive benchmarking efforts have also sought to quantify these generation capabilities. Siddiq et al. [42], Yuan et al. [50], and Jiang et al. [18] evaluated various LLMs on their baseline ability to generate unit tests. Despite these promising results, current evaluations predominantly rely on well-known, publicly available benchmarks. Existing studies fail to isolate this potential memorization from true semantic reasoning. Code modifications and LLM robustness. The fragility of LLMsā generative capability is also studied when presented with variations of standard programming tasks. For instance, Wang et al. [44] intro- duced ReCode, a framework for evaluating the robustness of code generation models against semantic-preserving mutations. Simi- larly, Rabin et al. [37], and Yefet et al. [48] investigated the generaliz- ability of neural program analyzers under semantic-preserving mu- tations, demonstrating that simple changes like variable renaming can severely degrade model performance. Other empirical studies, such as those by Dong et al. [10] and Yang et al. [47], assess how adversarial refactoring and natural mutations impact downstream tasks like vulnerability detection and code summarization. These works primarily evaluate whether a model can still produce the correct code snippet or identify a bug when the code or prompt is altered, mostly testing LLM generative capabilities. They do not evaluate the LLM code comprehension ability needed for test gen- eration. Memorization and semantic grounding. A fundamental chal- lenge in evaluating Large Language Models is distinguishing gen- uine reasoning from the memorization of pre-training data. Carlini et al. [6] and Lee et al. [22] demonstrate that LLMs can exactly reproduce substantial portions of their training corpora. Recent sur- veys and contamination-focused studies [46,51] also highlight how benchmark contamination can inflate the perceived performance of code generation models. 8 Conclusion We examined how code changes affect the reliability of LLM-based test generation. Using an automated mutation-driven evaluation framework on over 22,374 program variants, we evaluate model behavior under both semantic-altering and semantic-preserving changes. While LLMs achieve strong baseline performance on un- modified programs (79.3% line and 76.1% branch coverage with fully passing tests), test quality degrades substantially once code changes are introduced. Under semantic-altering changes, many generated tests remain aligned with the original program behavior, with over 99% of failing tests passing on the original code. Under semantic-preserving changes, coverage and pass rates decline de- spite unchanged functionality. These results suggest that current LLM-based test generation fails to reason about the semantic im- pact of code changes and instead responds mainly to the magnitude of syntactic differences in the code. Evaluating LLM-Based Test Generation Under Software EvolutionConferenceā17, July 2017, Washington, DC, USA References [1]Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated Unit Test Improvement using Large Language Models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024). Association for Computing Machinery, New York, NY, USA, 185ā196. doi:10.1145/3663529.3663839 [2]Anthropic. 2025. Claude Haiku 4.5 System Card. https://w.anthropic.com/ claude-haiku-4-5-system-card. [3]Anthropic. 2026. Claude 4.6 Sonnet System Card. https://w.anthropic.com/ claude-4-6-sonnet-system-card. [4]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732 [5] Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al.2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs.. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), Vol. 8. 209ā224. [6]Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633ā2650. [7] Mark Chen and et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). arXiv:2107.03374 [cs.LG] [8] Alex Chudic and GĆ¼l Ćalıklı. 2026. Automated Test Suite Enhancement Us- ing Large Language Models with Few-shot Prompting. arXiv e-prints, Article arXiv:2602.12256 (Feb. 2026), arXiv:2602.12256 pages. doi:10.48550/arXiv.2602. 12256 arXiv:2602.12256 [cs.SE] [9]Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robust- ness tester for Java. In Software: Practice and Experience, Vol. 34. Wiley Online Library, 1025ā1050. [10] Yihong Dong, Junda Jiang, et al.2023. Understanding the Robustness of Large Language Models for Code. arXiv preprint arXiv:2305.14886 (2023). [11] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ā24). Association for Computing Machinery, New York, NY, USA, Article 81, 13 pages. doi:10.1145/3597503.3639219 [12]Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engi- neering: Survey and Open Problems. 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) (2023), 31ā53. https://api.semanticscholar.org/CorpusID:263671720 [13]Minying Fang, Xing Yuan, Yuying Li, Haojie Li, Chunrong Fang, and Junwei Du. 2025. Enhanced Prompting Framework for Code Summarization with Large Language Models. Proc. ACM Softw. Eng. 2, ISSTA, Article ISSTA072 (June 2025), 24 pages. doi:10.1145/3728949 [14]Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE ā11). Association for Computing Machinery, New York, NY, USA, 416ā419. doi:10.1145/2025113.2025179 [15]Gemini Team. 2025. Gemini 2.5: Advancing Multimodal Capabilities for Fast Inference. Google DeepMind Technical Report (2025). [16]Gemini Team. 2026. Gemini 3.1: Next-Generation Reasoning and Agentic Capa- bilities. Google DeepMind Technical Report (2026). [17]Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed auto- mated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation. 213ā223. [18] Zongze Jiang, Ming Wen, Jialun Cao, Xuanhua Shi, and Hai Jin. 2024. Towards Understanding the Effectiveness of Large Language Models on Directed Test Input Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ā24). Association for Computing Machinery, New York, NY, USA, 1408ā1420. doi:10.1145/3691620. 3695513 [19]Mohammad Khajezade et al.2024. Investigating the Efficacy of Large Language Models for Code Clone Detection. In Proceedings of the 32nd IEEE/ACM Interna- tional Conference on Program Comprehension (ICPC). IEEE. [20]Vladimir Kovalenko et al.2023. UnitTestBot: Automated Unit Test Generation for C Code in Integrated Development Environments. In IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE. [21]Shuvendu K Lahiri et al.2022. Interactive code generation via test-driven user- intent formalization. arXiv preprint arXiv:2208.05950 (2022). [22]Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Sirivianos. 2022. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 8424ā8445. [23]Caroline Lemieux et al.2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In Proceedings of the 45th International Conference on Software Engineering. [24]Yucen Lin et al.2024. Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond. IEEE Transactions on Software Engineering (2024). [25]Andrea Lops, Fedelucio Narducci, Azzurra Ragone, Michelantonio Trizio, and Claudio Bartolini. 2025. A System for Automated Unit Test Generation using Large Language Models and Assessment of Generated Test Suites. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 29ā36. doi:10.1109/ICSTW64639.2025.10962454 [26]Stephan Lukasczyk, Christian KroiĆ, and Gordon Fraser. 2020. Pynguin: Au- tomated unit test generation for Python. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 168ā179. [27]Valentin J. M. Manes, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering of Fuzzing: A Survey. arXiv:1812.00140 [cs.CR] https://arxiv.org/ abs/1812.00140 [28]Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. IEEE Press, 1026ā1038. https://doi.org/10.1109/ ICSE55347.2025.00243 [29] Tien Nguyen, Waris Gill, and Muhammad Ali Gulzar. 2025. Are the Majority of Public Computational Notebooks Pathologically Non-Executable?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 396ā407. doi:10.1109/msr66628.2025.00070 [30] Luca Nicoletti et al.2024. Cross-lingual Code Clone Detection: When LLMs Fail Short Against Embedding-based Classifier. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE. [31]NVIDIA. 2025. Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hy- brid Mamba-Transformer Model for Agentic Reasoning. arXiv preprint arXiv:2512.20848 (2025). [32]OpenAI. 2025. Advancing science and math with GPT-5.2. https://openai.com/ index/gpt-5-2-for-science-and-math/. [33]OpenAI. 2025.gpt-oss-120b & gpt-oss-20b Model Card. arXiv preprint arXiv:2508.10925 (2025). [34]Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th international conference on Software Engineering (ICSE). IEEE, 75ā84. [35]Ruchir Puri, David S Kung, Gaetan Janssen, Wei Zhang, Giacomo Bury, Jason Griguer, Luca Fanebust, Bryce Dykeman, Tanveer Shah, Rishit Kamani, et al.2021. Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. arXiv preprint arXiv:2105.12655 (2021). [36]Ketai Qiu, Luca Di Grazia, Leonardo Mariani, and Mauro PezzĆØ. 2025. E-Test: Eāer- Improving Test Suites. arXiv:2510.19860 [cs.SE] https://arxiv.org/abs/2510.19860 [37]Md Rafiqul Islam Rabin, Amin Al-Haj, and Amin Alipour. 2021. On the general- izability of neural program models with respect to semantic-preserving transfor- mations. Information and Software Technology 135 (2021), 106552. [38]Sanghoon Rho, Philipp Martens, Seungcheol Shin, and Yeoneo Kim. 2024. Taming the Beast: Fully Automated Unit Testing with Coyote C++. arXiv:2401.01073 [cs.PL] https://arxiv.org/abs/2401.01073 [39]Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proc. ACM Softw. Eng. 1, FSE, Article 43 (July 2024), 21 pages. doi:10.1145/3643769 [40]Max SchƤfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. arXiv:2302.06527 [cs.SE] https://arxiv.org/abs/2302.06527 [41]Amrit Siddiq, Michael D. Ernst, and Mauro PezzĆØ. 2025. Do LLMs generate useful test oracles? An empirical study with an unbiased dataset. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025). To appear. [42]Mohammed Latif Siddiq and Joanna CS Santos. 2023. Exploring the effective- ness of large language models in automated unit test generation. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 332ā342. [43] Aaditya Singh et al.2026.OpenAI GPT-5 System Card. arXiv preprint arXiv:2601.03267 (2026). [44]Shuo Wang, Pei Zheng, et al.2023. ReCode: Robustness Evaluation of Code Generation Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2634ā2649. [45]Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ā24). Association for Computing Machinery, New York, NY, USA, Article 126, 13 pages. doi:10.1145/3597503.3639121 Conferenceā17, July 2017, Washington, DC, USASabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar [46]Shuo Yang et al.2024. Data Contamination in Large Language Models: A Survey. arXiv preprint arXiv:2402.10825 (2024). [47]Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering. 1482ā1493. [48]Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. In Proceedings of the ACM on Programming Languages, Vol. 4. 1ā30. [49]Zhiqiang Yuan, Yiling Bai, Xuan Huo, Cuiyun Chen, et al.2023. Chatunitest: a chatgpt-based automated unit test generation framework. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1ā13. [50]Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023. Evaluating Instruction-Tuned Large Language Models on Code Comprehen- sion and Generation. arXiv:2308.01240 [cs.CL] https://arxiv.org/abs/2308.01240 [51]Lizhe Zhang, Wentao Chen, Li Zhong, Letian Peng, Zilong Wang, and Jingbo Shang. 2025. Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting. arXiv:2503.02296 [cs.AI] https://arxiv.org/abs/2503.02296 [52]Linghao Zhang, Jingshu Zhao, Chong Wang, and Peng Liang. 2024. Using Large Language Models for Commit Message Generation: A Preliminary Study. arXiv:2401.05926 [cs.SE] https://arxiv.org/abs/2401.05926