Paper deep dive
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
Bhanu Prakash Vangala, Tanu Malik
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/2/2026, 9:04:16 AM
Summary
This paper introduces a three-layer framework to measure environment reproducibility in AI-generated software, identifying a significant gap between functional code correctness and accurate dependency specification. The authors evaluate three coding agents across four languages and fifty tasks, finding that agents systematically misspecify dependencies with low agreement rates (7%) and no improvement in newer models. The study highlights that divergence between declared and runtime dependencies is driven by training data priors, establishing environment specification as a critical, distinct axis for evaluating LLM coding agents.
Entities (13)
Relation Signals (11)
Bhanu Prakash Vangala → affiliatedwith → University of Missouri–Columbia
confidence 99% · Bhanu Prakash Vangala | Tanu Malik 1 Department of Electrical Engineering and Computer Science, University of Missouri–Columbia
Tanu Malik → affiliatedwith → University of Missouri–Columbia
confidence 99% · Bhanu Prakash Vangala | Tanu Malik 1 Department of Electrical Engineering and Computer Science, University of Missouri–Columbia
Three-Layer Dependency Framework → includeslayer → D(1) Declared Dependencies
confidence 95% · introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies
Three-Layer Dependency Framework → includeslayer → D(2) Installed Dependencies
confidence 95% · introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies
Three-Layer Dependency Framework → includeslayer → D(3) Traced Dependencies
confidence 95% · introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies
C → usesmanifestformat → CMakeLists.txt
confidence 95% · For C++, it collects every library name from FetchContent_Declare and find_package calls in CMakeLists.txt
Java → usesmanifestformat → pom.xml
confidence 95% · For Java, it extracts every <artifactId> from the <dependencies> section of pom.xml
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, defined as the accurate identification of the dependencies required to execute generated code, is equally critical. We develop an agent protocol for environment specification and introduce a three-layer framework comprising declared, runtime-installed, and necessary-and-sufficient dependencies to systematically assess coding agents for environment specification. Using this protocol, we evaluate the extent to which coding agents systematically misspecify software environment dependencies and how this misspecification varies across three agents, four languages, and fifty programming tasks. Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared and runtime dependency layers, implicating environment priors learned from the models' training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.
Tags
Links
- Source: https://arxiv.org/abs/2610.00425v1
- Canonical: https://arxiv.org/abs/2610.00425v1
Trouble viewing inline? Open PDF directly →
Full Text
92,426 characters extracted from source content.
Expand or collapse full text
AI Magazine Article Code That Works, Environments That Don’t: Measuring Environment Reproducibility in AI-Generated Software Bhanu Prakash Vangala | Tanu Malik 1 Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA | Correspondence: Bhanu Prakash Vangala (bv3hz@missouri.edu); Tanu Malik (tanu@missouri.edu) Received: Revised: Accepted: Keywords: LLM coding agents | dependency specification | environment gap | code generation | software supply chain ABSTRACT Code generation has emerged as a central capability of large language models, with coding agents now able to produce function- ally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, which is defined as the accurate identification of the dependencies re- quired to execute generated code, is equally critical. We develop an agent protocol for environment specification, and introduce a three-layer framework comprising declared , runtime-installed, and necessary and sufficient dependencies to systematically assess coding agents for environment specification. Using this protocol, we evaluate the extent to which coding agents systematically mis- specify software environment dependencies, and how does this misspecification vary across three agents, four languages, and fifty programming tasks. Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared (퐷 (1) ) and runtime (퐷 (3) ) dependency layers, implicating environment priors learned from the models’ training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current bench- marks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability. 1Introduction Code generation has emerged as one of the most consequential ca- pabilities of large language models. Advanced coding agents [1, 2] can now synthesize complete, multi-file software repositories from natural language descriptions, representing a qualitative leap beyond the line-level autocomplete of earlier tools [3, 4]. However, the benchmarks used to evaluate these agents have not kept pace with their evolving capabilities [5, 6]. Current evaluation frameworks [3, 7, 8] focus almost exclusively on functional code correctness. They fail to measure the cor- rectness of the underlying software environment, such as depen- dencies, package versions, and system configurations, which is strictly required for execution. One might argue that code and its environment are two sides of the same coin, implying that traditional benchmarks implic- itly evaluate both. We counter that software environments are fundamentally distinct from source code. While code dictates logic, environments manage external, stateful ecosystems that are prone to silent runtime failures and complex version con- flicts [9, 10]. Consequently, evaluating environment generation requires a qualitatively different benchmarking paradigm. Before AI scientists build such benchmarks, we believe it is important to first systematically measure how effectively current agents specify these environments. To address this gap, this paper introduces a framework to eval- uate and quantify how reliably coding agents generate software environments. We propose an iterative evaluation protocol. First, AI Magazine, 2026;v0:1–16 https://doi.org/00001 of 16 arXiv:2610.00425v1 [cs.SE] 30 Sep 2026 the agent is prompted to generate a complete software artifact containing both source code and its environment specification, establishing a baseline performance metric. Second, we evaluate the agent’s repair capabilities by forcing it to iteratively diagnose and resolve its own environment failures. The protocol ultimately determines the agent’s baseline accuracy, its self-repair efficiency, and the execution bottlenecks common to modern LLMs. Toward this goal, this paper introduces a novel evaluation frame- work to measure, assess, and quantify the quality of environment specifications generated by coding agents. We propose an itera- tive evaluation protocol divided into two primary phases. First, an agent is prompted to generate a complete software artifact comprising both source code and its environment specification, establishing a baseline metric for its initial generation capabil- ities. Second, if the initial environment is under-specified or broken, the agent enters a self-repair phase where it must itera- tively diagnose and resolve its own environment failures. Once an agent achieves a successfully executing artifact, the overall quality of the environment specified across all iterations is inde- pendently validated using application virtualization. By tracing the necessary and sufficient dependencies loaded into memory during runtime, our framework establishes an absolute ground truth, allowing us to explicitly measure the degree of dependency over-specification. Our framework thus provides a holistic, three-dimensional as- sessment: it benchmarks initial environment generation; quan- tifies the repair friction required to resolve under-specification; and exposes over-specification by comparing the agent’s declared dependencies against the lean, optimal runtime ground truth. We instantiate this framework in a controlled empirical study spanning three production coding agents, four language ecosys- tems, and 1,000 curated project instances. Each instance is exe- cuted in a clean, isolated environment with full dependency trac- ing from manifest through runtime. Using this rich dataset, we systematically characterize the root causes of environment speci- fication failures across different agents, programming languages, and task domains. Our results produce the first systematic evidence that depen- dency declarations are profoundly inconsistent across agents solv- ing identical tasks. Critically, we find that LLM environment generation is non-deterministic; a single agent rarely reproduces its own dependency choices across independent runs. In fact, the unanimous consensus rate for identical tasks across all languages is exactly zero. Furthermore, the gap between declared and actual runtime dependencies is wide, directionally biased, and heavily ecosystem-dependent. These variations implicate artifacts within the models’ training distributions rather than the inherent diffi- culty of the tasks themselves. Our earlier study [11] offered first evidence that agent-generated code is not reproducible at the dependency level; here we formalize the underlying evaluation protocol and measurement primitives, and expand the analysis across agents, ecosystems, and stochastic trials. These findings have direct, urgent implications for the repro- ducibility of agent-generated software and the validity of using agents for autonomous software engineering. Specifically, our study indicates that software environment configuration must be treated as a core target training objective for LLMs. Further- more, precision, recall, and F1-scores on dependency declara- tions—measured against traced runtime ground truth—must be treated as first-class evaluation criteria alongside functional code correctness. We make the following primary contributions: ∙ Three-Layer Dependency Framework: We introduce a formal paradigm for evaluating coding agents by decoupling dependencies into three layers: what the agent declares, what the package manager installs, and what the program actually loads into memory at runtime. ∙ Comprehensive Evaluation Dataset: We construct and open-source a dataset spanning three production coding agents, four language ecosystems, and 1,000 verified project instances, complete with execution traces in isolated environ- ments. ∙ Empirical Analysis of Environment Failures: We deliver a systematic characterization of LLM environment genera- tion, proving that under- and over-specification represent systemic algorithmic gaps in training rather than isolated edge cases. The rest of this paper is organized as follows. Section 2 formalizes the agent evaluation protocol and defines the core agent capa- bilities required for environment specification. Section 3 details the primitive functions that underpin the three-layer dependency framework. Section 4 outlines our experimental design, followed by the evaluation dimensions in Section 5 and a comprehensive evaluation and metrics analysis in Section 6. Finally, Section 7 reviews related work, and Section 8 concludes the paper. 2Agent Evaluation Protocol Given a hardware and operating environment, if a software en- vironment is incorrectly specified, a developer will aim to fix it till it results in executable code. An agent is similar. Our agent evaluation protocol divides this agent behavior into two skills, a generation skill, in which it produces code and a dependency manifest for a given task, and a repair skill, in which it iteratively revises both specified code and dependency manifest in response to runtime failures. Formally, given a coding agent 퐴 and a programming task 푇 with corresponding prompt 휋 푇 , the agent’s generation skill produces the initial code 퐶 and dependency manifest 푀, specified as: (퐶,푀) = Gen(퐴,휋 푇 )(1) Let the (퐶,푀) produced by the Gen skill be indexed as (퐶 (0) ,푀 (0) ). Executing (퐶 (0) ,푀 (0) ) in an isolated environment 퐼 yields an initial signal: 푒 (0) = Execute(퐶 (0) ,푀 (0) ,퐼) If 푒 (0) = ∅ the agent has reliably executed the code given the dependency manifest, and it denotes successful execution. Oth- erwise it denotes failed execution with 푒 (0) specifying the error. If 푒 (0) ≠ ∅, the Repair skill iteratively generates (퐶 (푖) ,푀 (푖) ) by feeding the previous code, manifest, and error signal to 퐴: (퐶 (푖) ,푀 (푖) ) = Repair(퐴,퐼,[퐶 (푖−1) ,푀 (푖−1) ,푒 (푖−1) ]), 푖 = 1,...,푘 푒 (푖) = Execute(퐶 (푖) ,푀 (푖) ,퐼) 2 of 16AI Magazine, 2026 퐷 (1) Declared 퐷 (2) Installed 퐷 (3) Traced repair-skill gap ∆: 퐷 (1) ↔퐷 (2) bloat: 퐷 (2) ⧵퐷 (3) hidden: 퐷 (3) ⧵퐷 (2) phantom: 퐷 (1) ⧵퐷 (3) hidden: 퐷 (3) ⧵퐷 (1) F I G U R E 1 | Relationships among the three dependency layers. 퐷 (1) is extracted from the initial manifest, 퐷 (2) from the final successfully installed environment, and 퐷 (3) from independent runtime tracing via strace. No fixed set-inclusion is assumed between퐷 (1) and퐷 (2) . Pairwise gaps define phantom dependencies (declared but not needed), hidden de- pendencies (needed but not declared or installed), and bloat (installed but unused). The loop halts at the first 푖 ≤ 푘 for which 푒 (푖) = ∅, or exhausts the budget 푘 otherwise. The final artifact is denoted (퐶 (푘) ,푀 (푘) ), where (퐶 (푘) ,푀 (푘) ) = (퐶 (푖) ,푀 (푖) ) for the smallest 푖 at which 푒 (푖) = ∅, or the last iterate if no successful execution is reached within budget. From these artifacts, obtained iteratively, we derive three depen- dency sets: ∙ The declared set 퐷 (1) ∶= ParseManifest 푙 (푀 (0) ) captures the agent’s initial claims about the environment the code needs given the programming task. ∙ The installed set 퐷 (2) ∶= Resolve 푙 (푀 (푘) ,퐼) captures the environment installed in 퐼 at the end of the iterative cy- cle. In both definitions, 푙 denotes the language ecosystem, which is necessary to understand the language code-specific dependencies. ∙ The traced set 퐷 (3) ∶= Trace(퐶 (푘) ,퐼) is defined only when 푒 (푘) = ∅, i.e., when the program has been success- fully installed and executes without error. Trace is agent- independent and captures the necessary and sufficient en- vironment for the observed execution of 퐶 (푘) in 퐼. We often use 퐶 (푘) to refer to both the source code and its installed exe- cutable in퐼 interchangeably, as the distinction is not material to the dependency analysis. These three sets 퐷 (1) , 퐷 (2) , and 퐷 (3) define our three-layer frame- work. To use these as a basis for measuring the generative and repair skills of coding agents, we do a pairwise comparison of 퐷 (1) , 퐷 (2) and 퐷 (3) . In this comparison, 퐷 (3) serves as a proxy for ground truth since obtaining 퐷 (3) is independent of the agent and this truth is obtained via the reference run of the executable cor- responding to the code, which identifies dependencies for the specific execution run. Comparing 퐷 (1) and 퐷 (3) in both directions reveals two com- plementary gaps: dependencies present in 퐷 (1) but absent from 퐷 (3) are phantom dependencies i.e., declared by the agent’s Gen skill but not actually needed by the code, while dependencies present in 퐷 (3) but absent from 퐷 (1) are hidden dependencies i.e., necessary for execution but never declared. Comparing 퐷 (2) and 퐷 (3) exposes the agent’s blindness to deeper dependencies: packages in 퐷 (2) but not 퐷 (3) constitute depen- dency bloat. These are dependencies installed by the package manager but unused at runtime. Alternatively, packages in 퐷 (3) but not 퐷 (2) remain hidden even after full resolution. Finally, comparing 퐷 (1) and 퐷 (2) quantifies what the Repair skill contributes: packages added to or removed from the manifest across 푘 iterations, isolating the effect of iterative error correction from the agent’s initial declaration. Figure 1 summarizes each pairwise comparison and the gap types each exposes. 3 Protocol Primitives: Dependency Generation and Extraction We now describe the different primitives used in our protocol, namely how Gen and Repair are configured with prompts and tasks, and how ParseManifest, Resolve and Trace lead to 퐷 (1) , 퐷 (2) , and 퐷 (3) . Figure 2 shows the use of these primitives as part of an agentic pipeline that includes Gen and Repair. 3.1Dependency Generation Primitives The quality of the results from Gen and Repair depends on two inputs: the complexity of the programming task 푇 and the de- sign of the prompt휋 푇 . We address each through principled design criteria. Tasks are drawn from common real-world programming prob- lems, subject to two constraints: First, each task must be platform- independent i.e., implementable in every language under study so that cross-language comparisons are controlled at the task level rather than confounded by structural differences between tasks. Second, each task must also be self-contained, requiring no external services, APIs, or credentials, so that dependency failures are attributable to the agent and not to environmental inaccessibility. The prompt template instantiated for each task is shown in Figure 3. Every instance shares the same structure of task ti- tle, task description, and functional requirements. Further, there are two explicit framing instructions, differing only in the tar- get language runtime and manifest filename, so that cross-agent comparisons are controlled at the prompt level. The two framing instructions specify that the agent must generate a complete project for a clean environment; the clean environ- ment is language-specific and stated explicitly in every instance. Completeness is treated as an experimental variable instantiated in two prompt variations. In the implicit variant, no complete- ness specification is given, measuring the agent’s default behavior when a developer is succinct. In the explicit variant, the agent is instructed to produce source code, a dependency manifest cov- ering both direct and transitive dependencies pinned to exact versions, and a README — measuring behavior when a devel- oper is meticulous. These two extremes reflect the observation that real-world developer prompts cluster toward one or the other rather than occupying a graded middle ground. A prompt 휋 푇 is considered valid if the agent produces code that passes an accompanying test suite specified within the functional requirements. This grounds prompt adequacy in an objective, reproducible criterion: passing tests confirm the prompt was 3 of 16 F I G U R E 2 | Environment Evaluation Pipeline. Left: the agent 퐴 exercises its Gen skill to produce an initial project (퐶 (0) ,푀 (0) ), which is executed in an isolated environment 퐼. On failure, the Repair skill is invoked with the runtime error 푒 (푖) for up to 푘 rounds, yielding the final artifact (퐶 (푘) ,푀 (푘) ). Right: three extraction primitives operate on the pipeline outputs to produce the three dependency layers: 퐷 (1) via manifest parsing of the initial declara- tion 푀 (0) ; 퐷 (2) via package-manager resolution of the final manifest 푀 (푘) ; and 퐷 (3) via strace-based runtime tracing of 퐶 (푘) in 퐼, serving as ground truth. TASK: task_title task_description FUNCTIONAL REQUIREMENTS: functional_requirements Create a complete language project for a clean Ubuntu 22.04 machine with only language_runtime installed. Include: – Source code – dependency_file with all dependencies (direct and transitive) pinned to exact versions – README.md with setup instructions, dependency explanations, build steps, run commands, and expected output F I G U R E 3 | A prompt template. The template is instantiated with language, manifest filename, and runtime; all functional content is iden- tical across languages. sufficiently specified, ensuring that any dependency misspecifi- cation in the output is attributable to the agent rather than to an underspecified prompt. 3.2 Dependency Extraction Primitives Amongst the three extraction primitives ParseManifest 푙 , Resolve 푙 , and Trace, the first two are language-specific al- though we describe a generalized specification. The Trace primitive is language-independent, but relies on the clean environment. 3.2.1Parse Primitive A dependency manifest is a language-ecosystem-specific configu- ration file that formally declares the external packages a software project requires to execute, and serves as the input artifact that package managers read to install and resolve dependencies. In our framework, the initial manifest 푀 (0) is the agent’s explicit machine-readable claim of what the code needs in response to 휋 푇 , while the final manifest 푀 (푘) is the manifest produced after iterative repair. ParseManifest 퓁 comprises parsing rules and takes as input 푀 (0) to produce 퐷 (1) , the declared dependency set. The parsing rules are language-specific and currently support four language ecosys- tems. For Python, it reads each line of requirements.txt and strips version constraints to normalize package names for cross- agent comparison. For Java, it extracts every <artifactId> from the <dependencies> section of pom.xml. For JavaScript, it reads the keys of the "dependencies" object in package.json, delib- erately excluding "devDependencies" since build-time tools are outside the runtime supply chain; this exclusion is applied consis- tently across all agents and tasks. For C++, it collects every library name from FetchContent_Declare and find_package calls in CMakeLists.txt. Note that C++ differs architecturally from the other three languages in that CMake does not have a canoni- cal package manager, which has implications for Resolve 퓁 dis- cussed below. Each rule is a deterministic extraction from a single field of a for- mally specified file format and involves no heuristic inference; the complete parsing implementation is released in our artifact. Because the same manifests are subsequently consumed by each ecosystem’s package manager in the Resolve 퓁 step, any system- atic parsing error would surface as a mismatch between 퐷 (1) and the resolver’s reported dependency set during execution. 4 of 16AI Magazine, 2026 3.2.2Resolve primitive The primitive Resolve 퓁 determines 퐷 (2) , the set of installed dependencies, within the isolated environment 퐼. The Docker instance that instantiates 퐼 is built from a base OS image with the language runtime, i.e., compiler, interpreter, or virtual ma- chine, and the language-specific package manager. No additional packages are pre-installed beyond this toolchain baseline. Within퐼, the primitive Resolve 퓁 executes the final manifest푀 (푘) , causing the language-specific package manager to install both the dependencies explicitly declared in 푀 (푘) and any additional transitive dependencies they require. Resolve 퓁 then queries the package manager for the complete list of packages installed in 퐼, and reports this as 퐷 (2) : 퐷 (2) ∶= pkg_list 푝표푠푡 (퐼,푀 (푘) ) The correctness of 퐷 (2) rests on the package manager itself serv- ing as the measurement instrument: the reported set is the ecosystem’s own authoritative enumeration of installed packages, queried inside 퐼 immediately after installation, rather than an in- ference made by our tooling. The clean toolchain baseline of 퐼 is fixed and identical across all runs, so the enumeration reflects only packages installed from 푀 (푘) . A failed or partial resolution is not silently absorbed: it surfaces as an installation error in the protocol and is returned to Repair as 푒 (푖) . Given the declared dependency set 퐷 (1) obtained from ParseManifest 퓁 , Resolve 퓁 also measures the transitive overhead introduced by the package manager: ∆ = 퐷 (2) ⧵ 퐷 (1) ;휌 = |퐷 (2) | |퐷 (1) | where ∆ is the set of packages installed that the agent did not explicitly declare, and 휌 is the inflation ratio measuring the mul- tiplicative factor by which the package manager expands the agent’s declared dependency set. 휌 = 1 indicates no transitive resolution; 휌 ≫ 1 indicates aggressive transitive expansion. For example, if 푀 (푘) declares only pandas and the build succeeds on 푀 (0) such that 푀 (푘) = 푀 (0) , then pip additionally installs numpy, python-dateutil, pytz, tzdata, and six as transitive dependencies, giving |퐷 (1) | = 1, |퐷 (2) | = 6, |∆| = 5, and 휌 = 6.0. Java and JavaScript behave analogously and are described in Section 4. C++ is a special case. Since CMake has no canoni- cal package manager, no transitive dependency resolution occurs at the package manager layer. Consequently, Resolve 퓁 for C++ reduces to 퐷 (2) ∶= 퐷 (1) , meaning ∆ = ∅ and 휌 = 1 by construc- tion rather than by agent behavior. The 퐷 (1) -vs-퐷 (2) comparison is therefore structurally uninformative for C++ and is excluded from cross-language analyses of transitive dependency gaps. 3.2.3 Trace primitive The Trace primitive operates on the final successfully installed program(퐶 (푘) ,푀 (푘) ) within the isolated environment퐼, determin- ing which packages are actually loaded during a specific exe- cution run. Unlike ParseManifest 퓁 and Resolve 퓁 , the Trace method is independent of both the language runtime and the coding agent: It is independent of the language-runtime due to the choice of the tracing method; It is independent of the cod- ing agent as it does not query any agent-produced artifact but instead observes program behavior directly for used dependen- cies. It does however require that Resolve 퓁 has succeeded, since a successfully installed program is a precondition for tracing. Trace produces 퐷 (3) , the set of dependencies necessary and suf- ficient for the observed execution of 퐶 (푘) in 퐼. Comparing 퐷 (3) against퐷 (1) and퐷 (2) exposes overclaimed dependencies i.e., pack- ages declared or installed but never loaded at runtime. Formally: 퐷 (1) ⧵ 퐷 (3) (declared but unused); 퐷 (2) ⧵ 퐷 (3) (installed but unused) (2) While modern interpreted language IDEs often warn developers of unused declared dependencies, the problem is equally present in compiled languages such as Java and C++, where a package may be linked but never exercised on a given execution path [12]. Trace uses ptrace to intercept all file- and process-related sys- tem calls made by the running program and any child processes it spawns. Each dependency file accessed during execution is recorded in a log. Dependency-specific files are then filtered us- ing a dictionary of language-specific installation paths, such as dist-packages for Python and dependency for Java to map ac- cessed file paths to package names, enabling direct comparison with 퐷 (1) and 퐷 (2) . Language-specific path-to-package mapping rules are described in Section 4. Listing 1 Example code showing reference execution int main(int argc , char* argv []) if (std:: string(argv [1]) == "--matrix") // uses Eigen Eigen :: MatrixXd m = Eigen :: MatrixXd :: Random(3, 3); else if (std:: string(argv [1]) == "--json") // uses nlohmann/json nlohmann ::json j = nlohmann ::json::parse(""); Two limitations apply to 퐷 (3) . First, since tracing captures only dependencies loaded during a specific execution run, condition- ally imported packages not exercised on the observed execution path are absent from 퐷 (3) . See example in Listing 1. We assume such conditional dependencies are rare in practice and do not materially affect aggregate agent behavior across tasks. Second, Trace operates at the file access level and therefore cannot ob- serve header-only libraries, which are included at compile time rather than loaded at runtime. Such libraries may appear in 퐷 (1) or퐷 (2) but will be systematically absent from퐷 (3) , which is partic- ularly relevant for C++ where header-only libraries are common. We acknowledge this as a scope limitation of dynamic over static analysis. 4 Experiment Setup In this section, we first instantiate the protocol of Section 2 with tasks, agents, and language ecosystems. We then describe how the agent protocol is conducted via configured primitives of Section 3. The experimental setup of tasks and extracted depen- dencies will be made available as an open reproducible artifact. 5 of 16 T A B L E 1 | The 50 benchmark programming tasks span eight appli- cation domains. Domain푛Representative tasks Data Processing 10 CSV statistical analyzer, JSON schema validator, XML-to-JSON converter, YAML config merger Cryptography7 AES-256 file encryption, RSA digital signa- tures, TOTP generator, bcrypt hasher Image Processing 6 Format converter, watermarking, his- togram analyzer, EXIF metadata extractor Networking6 HTTP downloader with retry, port scanner, DNS resolver, static file server Text Processing6 Markdown-to-HTML, spell checker, TF- IDF search engine, template engine Compression5 ZIP archive manager, GZIP tool, file dedu- plicator, compression benchmark Math/Scientific5 Matrix operations, linear regression, Dijk- stra shortest path, FFT analyzer System Utilities 5 QR code generator/reader, PDF report builder, Merkle tree verifier, cron parser 4.1Protocol Instantiation and Execution Programming Tasks 푇. We instantiate the task set 푇 with 50 benchmark programming tasks spanning eight application do- mains, as shown in Table 1. Tasks are selected to satisfy the two criteria established in Section 3.1: platform independence, en- suring each task can be implemented identically across all four language ecosystems inℒ, and self-containment, requiring no ex- ternal services, APIs, or credentials so that dependency misspeci- fication is attributable to the agent’s Gen and Repair skills rather than environmental inaccessibility. The eight domains are cho- sen to provide broad coverage of real-world software categories that developers routinely delegate to coding agents, ranging from data processing and cryptography to system utilities and scientific computing. The distribution across domains, which is weighted toward data processing and cryptography reflects the relative fre- quency of such tasks in open-source coding benchmarks [3, 7]. The 50-task scale is chosen to ensure statistical power for cross- agent and cross-language comparisons while remaining feasible for the fully manual execution protocol described below. Language Ecosystems ℒ. The language ecosystem set consists of ℒ = Python, Java, JavaScript, C++. These language ecosys- tems are used for configurations of ParseManifest 퓁 , Resolve 퓁 , and Trace defined in Section 3. These four ecosystems were selected to represent diversity in runtimes (interpreted vs com- piled), package manager design (centralized vs decentralized), and transitive resolution behavior (휌 ≫ 1 for npm vs 휌 ≈ 1 for CMake), enabling cross-ecosystem analysis of the environment specification gap. Coding agents 풜. The agent set is 풜 = 푎 1 ,푎 2 ,푎 3 , instantiated as: ∙ 푎 1 : Claude Code (Anthropic, Claude Opus with extended thinking). ∙ 푎 2 : Codex (OpenAI, the latest Codex model via GitHub Copilot). ∙ 푎 3 : Gemini Code Assist (Google, Gemini with the highest reasoning configuration). These three agents are selected representing the current frontier of coding agent capability, covering the three dominant providers, and are accessed exclusively through their command-line inter- faces at default settings. This mirrors the experience of a devel- oper using these agents in practice. All agents receive the same prompt 휋 푇 for each task, with the iteration budget 푘 = 10 repair cycles specified in each agent’s configuration file as described in Section 2, yielding a maximum of 11 agent interactions per in- stance — one Gen call followed by up to 10 Repair calls. Further, agents preserve full history within the context window during the Repair cycle, enabling the agent to reason about the sequence of previously attempted repairs. Trial Structure and Study Scale. Every agent 푎 ∈ 풜 receives every prompt 휋 푇 for every task-language pair (푇,퓁) ∈ 풯 × ℒ, yielding |풯|× |ℒ|× |풜| = 50×4×3 = 600 primary instances. To assess the stochastic variability of the Gen and Repair skills, 푎 1 is additionally executed for two further independent trials per task- language pair, each initialized from a fresh conversation with no memory of prior attempts, contributing 50 × 4 × 2 = 400 stochas- tic trials. 푎 1 is selected for variance analysis as it is the only agent achieving 100% final execution success (Section 6), guarantee- ing complete data across all task-language pairs for a statistically meaningful variance analysis. The total study scale is thus 1000 evaluation runs (Table 2). Section 3.1 defines two prompt variations: implicit, in which no completeness specification is provided, and explicit, in which the agent is instructed to produce source code, a pinned dependency manifest, and a README. All 600 primary instances use the ex- plicit variant, so the reported results characterize agent behavior under maximal prompt specificity; this constitutes the agent’s best case, and the misspecification rates reported in Section 6 should therefore be read as lower bounds on what less specific prompting would produce. A systematic comparison against the implicit variant is left to future work. T A B L E 2 | Study scale. DimensionCount Benchmark programming tasks50 Target language ecosystems4 LLM coding agents3 Primary configurations (50×4×3)600 Stochastic trials (Claude Code, 2 extra × 200)400 Maximum agent interactions per instance11 Total evaluation runs1000 Isolated Environment 퐼. The isolated environment 퐼 is realized as a per-language Docker container built from the ubuntu:22.04 base image and containing only the language toolchain necessary to execute Resolve 퓁 and Trace 퓁 : ∙ 퐼 푃푦푡ℎ표푛 (tmlr-python-eval): Python 3.10, pip ∙ 퐼 퐽푎푣푎 (tmlr-java-eval): JDK 17, Maven 3 ∙ 퐼 퐽푆 (tmlr-js-eval): Node.js 20 LTS, npm ∙ 퐼 퐶++ (tmlr-cpp-eval-v2): G++ 12, CMake 3.22, make, git, zlib1g-dev, and standard headers No image contains common system development libraries be- yond those listed: packages such as OpenSSL, libcurl, or SQLite3 development headers are absent from every container, so any agent assumption of their presence surfaces as an observ- able failure rather than silently succeeding. The C++ container 6 of 16AI Magazine, 2026 necessarily includes zlib1g-dev and standard C/C++ headers as these are required by the CMake toolchain; these are recorded in pkg_list 푝푟푒 (퐼) and subtracted from 퐷 (2) via the baseline pro- cedure defined in Section 3, ensuring they do not contaminate the installed dependency measurement. Each image additionally contains Sciunit [13], whose ptrace-based engine implements the Trace 퓁 primitive inside the same container; Sciunit is part of the fixed baseline, and its own file accesses are excluded from 퐷 (3) so that the runtime trace reflects only the agent-generated artifact. Manual execution. All 1000 instances were conducted manu- ally, with each instance comprising a multi-turn interaction: the initial Gen prompt followed by up to ten Repair turns, for a max- imum of eleven agent interactions per instance (Table 2). This means the following actions were undertaken by hand: (i) the initial prompt and every subsequent repair turn were submitted to each agent’s CLI, (i) generated artifacts were transferred into the Docker containers hosted on an AWS EC2 instance, (i) run- time errors were transcribed verbatim into the ongoing agent conversation, and (iv) outcomes were recorded into structured CSV files. Automation was deliberately avoided since in our expe- rience the diversity of build failures across four ecosystems was too broad for reliable scripting, and an automated harness would either mask real failure modes through generic recovery heuris- tics or accumulate so many special-case rules that the harness itself would become a confounding factor, especially in the evalu- ation of the Repair skill. Once the manual execution of a project is completed and the necessary dependencies collected, each con- tainer is destroyed via docker run --rm and a fresh container is instantiated for the next programming task. 4.2 Primitive Configuration ParseManifest 푙 and Resolve 푙 . Table 3 shows concrete instanti- ations of ParseManifest 푙 and Resolve 푙 per ecosystem. Trace. We realize Trace(퐶 (푘) ,퐼) using Sciunit [13], compiled from source and installed in every Docker image. Sciunit attaches to the target process via the Linux ptrace API and intercepts all open(), openat(), read(), and execve() system calls, writing the absolute path of each accessed file to a provenance log. READ- tagged entries are filtered using ecosystem-specific path rules to map accessed file paths to package names, enabling direct comparison with 퐷 (1) and 퐷 (2) . A critical implementation challenge arises for Java. LLM- generated Java projects frequently use the Maven Shade Plugin to produce a fat JAR bundling all dependency classes into a sin- gle archive. Under fat JAR invocation, the JVM issues exactly one openat() call for the archive itself, making the provenance log opaque to dependency analysis. Sciunit cannot determine which packages were loaded since all are bundled inside the same file. We refer to this as the fat JAR opacity problem. To resolve it, we replace fat JAR execution with classpath-based execution before tracing: dependency JARs are unpacked as physically sep- arate files via mvn dependency:copy-dependencies, and the program is traced against the unpacked classpath. With each JAR a distinct file, the JVM issues a separate openat() per loaded JAR, which appears individually in the provenance log. We verified that classpath-based execution produces functionally identical output to fat JAR execution in all traced cases. For C++, the compiled binary is traced directly. At startup, the dynamic linker issues openat() calls for every shared library the binary was linked against. We retain .so entries and exclude the seven base system libraries loaded by every C++ binary re- gardless of application code: libc.so, libstdc++.so, libm.so, libgcc_s.so, libpthread.so, libdl.so, and librt.so, leav- ing only application-level libraries in 퐷 (3) . The header-only library limitation noted in Section 3 is most acute here: nlohmann_json, the most frequently declared C++ dependency across all agents (65 successful manifests), is compiled directly into the binary and produces no .so file, and therefore no prove- nance log entry. For projects whose declared dependencies are entirely header-only, 퐷 (3) = ∅ even when 퐷 (1) is non-empty. We return to this limitation in Section 6.6. 5 Evaluation Dimensions We evaluate the agent evaluation protocol and the extraction primitives across five dimensions: ∙ Protocol Convergence. The Gen→ Repair cycle produces a successful artifact (퐶 (푘) ,푀 (푘) ) through iterative resolution. We evaluate whether and how quickly an agent converges, i.e., reaches 푒 (푘) = ∅ within budget, and how convergence rate varies across agents, languages, and task domains. This includes measuring both the binary outcome indicating if the cycle converges, and the number of iterations consumed via repair depth 푖 ∗ = min푖 ≤ 푘 ∶ 푒 (푖) = ∅. ∙ Environment Specification Quality. The three depen- dency sets 퐷 (1) , 퐷 (2) , and 퐷 (3) produced by the extraction primitives form the basis for measuring specification qual- ity. We evaluate two complementary measures. First, pair- wise comparison of 퐷 (1) ,퐷 (2) ,퐷 (3) against 퐷 (3) as ground truth, quantifying the extent of phantom (퐷 (1) ⧵ 퐷 (3) ), hid- den (퐷 (3) ⧵ 퐷 (1) ), bloated (퐷 (2) ⧵ 퐷 (3) ), and transitive (∆ = 퐷 (2) ⧵ 퐷 (1) ) dependencies, reported as precision, recall, and F1 scores. Second, the direction and magnitude of misspec- ification i.e., over- versus under-specification, measured via ∆ and 휌. We additionally assess stochastic variability within a single agent using the repeated trials, determining whether misspecification is a deterministic property of the agent’s knowledge or a stochastic artifact of generation. measured via ∆ and 휌. We additionally assess stochastic variability within a single agent using the repeated trials , determining whether misspecification is a deterministic property of the agent’s knowledge or a stochastic artifact of generation. ∙ Cross-Agent Comparison. Given per-agent specification quality, we evaluate how agents differ from one another for identical task-language pairs. This covers pairwise depen- dency set agreement across agents and whether differences are systematic or task-dependent, isolating agent-specific be- havior from task and language effects. ∙ Cross-Language Variation. We evaluate whether specifi- cation quality varies systematically across the four language ecosystems ℒ = Python, Java, JavaScript, C++. If agents perform poorly and that poor performance is language- specific, this evaluation informs whether agents require 7 of 16 T A B L E 3 | Per-ecosystem realization of ParseManifest 퓁 , Resolve 퓁 , and Trace 퓁 for the three dependency layers. EcosystemLayer 1: 퐷 (1) = ParseManifest 퓁 (푀 (0) )Layer 2: 퐷 (2) = Resolve 퓁 (푀 (푘) )Layer 3: 퐷 (3) = Trace 퓁 (퐶 (푘) ,퐼) PythonParse requirements.txt line by line; strip version constraints Diff of pip list before vs. after pip installSyscall READs under dist-packages/ or site-packages/ Java <artifactId> values in pom.xml <dependencies> mvn dependency:list artifact IDs (full transitive closure) Syscall READs of target/dependency/*.jar via classpath execution JavaScript Keys of "dependencies" in package.json npm list --all --json; full node_modules/ tree Syscall READs under node_modules/ C++ FetchContent_Declare and find_package names in CMakeLists.txt CMake configure log (퐷 (2) ≈ 퐷 (1) ; transitive expansion typically absent) Syscall READs of .so files; seven base system libraries excluded language-specific training or whether general environment specification ability is the primary deficiency. ∙ Task Complexity Sensitivity. We evaluate whether speci- fication quality degrades with task complexity or varies by programming domain. This determines whether agents fail consistently on particular task types, providing diagnostic signal about which categories of software projects are most vulnerable to environment specification failure. 6 Evaluation Results We present our results across the five evaluation dimensions of Section 5. The building blocks are already in place: the depen- dency sets 퐷 (1) , 퐷 (2) , 퐷 (3) come from Section 2, and the inflation ratio 휌 from Section 3. Each subsection below only adds the small amount of extra notation it needs. We fix three conventions used throughout. First, subscripts iden- tify the setting:퐷 (푗) 푎,푝,퓁 is the dependency set of agent푎 on task푝 in language퓁. Second, a project counts as successful when its final artifact (퐶 (푘) ,푀 (푘) ) runs without error in the clean environment 풟 퓁 within the repair budget; we write Success(퐶,푀,풟 퓁 ) for this condition and 풮 푎,퓁 for the set of tasks where agent 푎 succeeds in language퓁. Third, unless stated otherwise, every reported number is a mean over successful projects. 6.1 Protocol Convergence Figures 4–6 examine the agents for protocol convergence. We first measure the performance of the Gen skill, i.e., obtaining a successful software artifact for a programming task in the first attempt. We measure this across the 600 instances, ordered first by agent and then by language, so that each cell in Figure 4 rep- resents one (agent, language, task) triple. Dense green regions show that Claude and Codex have a strong Gen skill in Python and JavaScript but a much weaker one in Java and C++, while Gem- ini inverts the pattern: it is weakest in Python (32% first-attempt success) yet the strongest of the three agents in C++ (80%). No agent is strong in all four languages, and the failure patterns dif- fer across agents, motivating a closer examination of the Repair skill. Figure 5 shows how much of the Repair skill is exercised per agent. For each (agent, language) pair, it reports the fraction of tasks resolved on the first attempt and the fraction requiring 푘 repair cycles, with 푘 bucketed into the ranges [1,3], [4,7], and [8,10]; the final category marks tasks that failed within the repair budget. The vast majority of repairs complete within 1–3 cycles in every ecosystem. Deep repair chains of four or more cycles are rare and appear almost exclusively in C++. Claude repairs every one of its failed first attempts in all four languages, whereas Codex recovers 89% of its failed C++ attempts and Gemini recovers 92% in Java and 90% in C++; outright failures are therefore confined to Codex and Gemini in C++ and to Gemini in Java. Figure 6 aggregates these results by language. To aggregate, we define a success rate at iteration 푖 for agent 푎 and language퓁 as SR (푖) 푎,퓁 = 1 |풫| ∑ 푝 ∈풫 ퟏ [ Success ( 퐶 (푖) 푎,푝,퓁 , 푀 (푖) 푎,푝,퓁 , 풟 퓁 )] , (3) so that SR (0) 푎,퓁 is the first-attempt success rate, measuring the Gen skill alone, and SR (≤푘) 푎,퓁 = max 푖≤푘 SR (푖) 푎,퓁 is the final success rate after up to 푘 repair cycles. The self-correction gain SR (≤푘) 푎,퓁 − SR (0) 푎,퓁 measures the contribution of the Repair skill to eventual convergence. Figure 6 shows solid bars indicating SR (0) and hatched bars in- dicating the self-correction gain for each agent and language. First-attempt success varies widely: Claude ranges from 96% in JavaScript and 86% in Python down to 36% in C++ and only 18% in Java, Codex spans 44% (C++) to 88% (Python and JavaScript), and Gemini runs from 32% (Python) to 84% (JavaScript). After re- pair, nearly every configuration converges: Claude reaches 100% in all four ecosystems, Codex falls short only in C++ (94%), and Gemini falls slightly short in Java (96%) and C++ (98%). The Repair skill thus rescues most first-attempt failures, but the size of the hatched region shows how much iterative feedback each configuration consumes to get there. The corresponding analysis by task domain is deferred to Section 6.5. These results carry a direct implication for how coding agents are evaluated. A benchmark that scores only final success would rate all three agents as near-perfect, yet the cost of reaching that success differs by up to a factor of five in repair cycles, and every repair cycle is additional latency, compute, and opportu- nity for the agent to mutate the dependency set. First-attempt environment specification is therefore a hidden cost driver that functional-correctness benchmarks do not surface, and the first manifest, not the final one, determines how much of that cost a deployment pays. 6.2 Environment Specification Quality All measures in 4 are aggregated as means per stratum (푎,퓁) and per ecosystem퓁 pooled over agents. Because F1 is computed per project and then averaged, the tabulated F1 need not equal the harmonic mean of the tabulated precision and recall; the discrepancy is largest in C++, where many projects have zero declared–runtime overlap. 8 of 16AI Magazine, 2026 Claude Code Python SR 0 =86% Java SR 0 =18% JavaScript SR 0 =96% C++ SR 0 =36% Codex Python SR 0 =88% Java SR 0 =68% JavaScript SR 0 =88% C++ SR 0 =44% 050100150200 Instance index (50 tasks × 4 languages = 200 per agent) Gemini Python SR 0 =32% Java SR 0 =50% JavaScript SR 0 =84% C++ SR 0 =80% Gen Skill — First-Attempt Success Across 600 Instances Gen success (k = 0) Gen failure (k ≥ 1) F I G U R E 4 | Gen skill: first-attempt success across all 600 instances (3 agents × 4 languages × 50 tasks). Each cell is one (agent, language, task) triple; green indicates first-attempt success and red indicates failure. Claude Code Python 100% Java 100% JavaScript 100% C++ 100% Codex 100%100%100%89% 050100150200 Instance index (50 tasks × 4 languages = 200 per agent) · % = share of failed-Gen tasks repaired Gemini 100%92%100%90% Repair Skill — Iterations k to Convergence No repair needed (k = 0)k = 1–3k = 4–7k = 8–10No conv. (k > 10) F I G U R E 5 | Repair skill: distribution of repair cycles needed to converge for each (agent, language) pair, bucketed into first-attempt success, 1–3, 4–7, and 8–10 cycles, and failure within the budget. PythonJava JavaScript C++ Programming Language 0% 20% 40% 60% 80% 100% Success Rate Protocol Convergence by Language Claude Code Codex Gemini SR 0 (first attempt) Repair gain F I G U R E 6 | Protocol convergence by language. Solid bars show the first-attempt success rate SR (0) and hatched extensions show the self- correction gain contributed by Repair. We measure environment specification quality in two comple- mentary ways, building directly on the three-layer framework PythonJavaJavaScriptC++ Programming Language 0% 20% 40% 60% 80% 100% Mean Rate Manifest Misspecification — Phantom, Hidden, and Bloat Rates Phantom (φ) Hidden (η) Bloat (β) Rates pooled over agents on successful projects; zero-denominator cases counted as zero F I G U R E 7 | Phantom, hidden, and installed-but-unused bloat rates by ecosystem, averaged over successful projects with zero-denominator cases included as zero. of Section 2. First, we operationalize the phantom, hidden, and bloat categories of Section 2 as rates, together with the inflation 9 of 16 PythonJavaJavaScriptC++ Programming Language 1× 2× 5× 10× 20× 50× Inflation Ratio ρ med = 1.5 med = 1.7 med = 8.5 med = 1 Environment Growth — Transitive Inflation by Ecosystem ρ= |D (2) | / |D (1) | per successful project · log scale F I G U R E 8 | Initial-to-final environment growth by ecosystem. Each box summarizes the distribution of the inflation ratio 휌 of Section 3 over all successful projects in that ecosystem (log scale). T A B L E 4 | Initial-manifest accuracy: treating runtime dependencies 퐷 (3) as ground truth. AgentLanguagePrecisionRecallF1 Claude Python0.9900.642 0.733 Claude Java0.9350.739 0.758 Claude JavaScript0.9290.406 0.493 Claude C++0.0370.781 0.031 CodexPython0.9380.827 0.837 CodexJava0.9390.684 0.735 CodexJavaScript0.2140.143 0.143 CodexC++0.0620.658 0.004 Gemini Python0.8990.721 0.756 Gemini Java0.9470.523 0.624 Gemini JavaScript0.0400.182 0.005 Gemini C++0.0590.708 0.004 AllPython0.9440.7130.765 AllJava0.9400.6530.708 AllJavaScript0.4130.2580.219 AllC++0.0490.7290.018 ratio 휌 of Section 3. These metrics characterize the nature and magnitude of misspecification independent of the agent, reveal- ing which language ecosystems are structurally harder to specify. Second, retrieval measures of precision, recall, and F1 consider 퐷 (3) as the ground truth and characterize per-agent accuracy, enabling the direct cross-agent comparison of Section 6.3. For a successful project (푎,푝,퓁), let 푛 푗 = |퐷 (푗) 푎,푝,퓁 | for 푗 ∈ 1,2,3. The three mismatch rates are 휙 = |퐷 (1) ⧵퐷 (3) | 푛 1 , 휂 = |퐷 (3) ⧵퐷 (1) | 푛 3 , 훽 = |퐷 (2) ⧵퐷 (3) | 푛 2 ,(4) with each rate set to zero when its denominator is zero. The phantom rate 휙 measures over-specification: packages declared but never loaded at runtime. The hidden rate 휂 measures under- specification: packages loaded at runtime but absent from the initial manifest. The bloat rate 훽 measures transitive excess: pack- ages installed by the package manager but never accessed during execution. Treating 퐷 (1) as a retrieval system predicting 퐷 (3) , per-project precision, recall, and F1 are P = |퐷 (1) ∩ 퐷 (3) | 푛 1 , R = |퐷 (1) ∩ 퐷 (3) | 푛 3 , F1 = 2PR P + R , (5) with the conventions P = 1 when 푛 1 = 0 and 푛 3 > 0, R = 1 when 푛 3 = 0 and 푛 1 > 0, and projects with 푛 1 = 푛 3 = 0 excluded entirely. Precision and recall jointly characterize the direction of agent error: high precision but low recall indicates systematic under-specification, while the reverse indicates systematic over- specification. All measures are aggregated as means per stratum (푎,퓁) and per ecosystem퓁 pooled over agents. Figure 7 shows the character of misspecification per ecosys- tem. Python and Java have relatively low phantom rates (about 5%), but both still show meaningful hidden-dependency rates because transitive packages become active at runtime without be- ing named in the initial manifest. JavaScript is much worse on all three dimensions: about 49% of declared packages are phan- tom, about 62% of runtime packages are hidden, and about 22% of installed packages are never touched. C++ is dominated by a different pathology: many dependencies are compile-time or header-only and therefore disappear from the runtime trace, in- flating phantom and bloat rates under a runtime-ground-truth definition. Figure 8 quantifies how large the eventual installed environment becomes relative to what the agent initially declared, using the inflation ratio 휌 defined in Section 3. JavaScript shows by far the largest environment growth, with ̄ 휌 JavaScript = 11.40 and a median of 8.5, meaning that small initial declarations routinely lead to much larger installed environments. Java also shows heavy inflation ( ̄ 휌 Java = 3.97), while Python is noticeably milder ( ̄ 휌 Python = 2.06). C++ remains near 휌 = 1.0, which matches its non-centralized dependency workflow (Section 3), with only mi- nor deviations in rare fetched-project cases. The figure shows that the size of the initial declared dependency set is a poor proxy for actual footprint in resolver-heavy ecosystems. Table 4 reports the retrieval measures. Python achieves the strongest overall initial-manifest accuracy with macro F1 of 0.765, followed by Java at 0.708. JavaScript drops sharply to 0.219 be- cause both over-declaration and under-declaration are common. C++ falls to 0.018, but that value should be interpreted together with the methodological limitation that many C++ libraries leave little or no runtime footprint after compilation (Section 6.6). The impact of these mismatch rates extends beyond code qual- ity. A dependency manifest is the primary input to software bills of materials, vulnerability scans, and license audits; when nearly half of a JavaScript manifest is phantom and nearly two thirds of what actually runs is hidden, every downstream audit inherits that inaccuracy. Phantom declarations inflate the apparent attack surface and generate false positives in security review, while hid- den dependencies are worse: they are runtime-active code that no scan ever examines. At the accuracy levels of Table 4, an agent-produced manifest cannot be treated as a trustworthy au- dit artifact in JavaScript or C++ without independent runtime verification. 6.3 Cross- and Intra-Agent Comparison We now measure cross-agent and intra-agent agreement. For both assessments, we define the pairwise declared-dependency agreement between agents 푎 푖 and 푎 푗 on task 푝 in language퓁 as 퐽(푎 푖 ,푎 푗 ,푝,퓁) = |퐷 (1) 푎 푖 ∩ 퐷 (1) 푎 푗 | |퐷 (1) 푎 푖 ∪ 퐷 (1) 푎 푗 | ,(6) 10 of 16AI Magazine, 2026 PythonJavaJavaScriptC++ Programming Language 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mean Jaccard on D (1) 0.07 half of packages shared Cross-Agent Agreement — Declared Dependencies Claude–Codex Claude–Gemini Codex–Gemini Pairwise Jaccard averaged over tasks where both agents succeed F I G U R E 9 | Cross-agent agreement on declared dependencies. Each bar reports the mean Jaccard similarity between the declared dependency sets extracted from two agents’ initial manifests for the same task in the same ecosystem; the dotted line marks the level at which half of the de- clared packages would be shared. T A B L E 5 | Stochastic variability: Claude’s three independent trials produce near-zero agreement. LanguageMean ̄ 퐽 intra MedianUCR ̄ 푉 ∪ ̄ 푉 ∩ Python0.1130.0830/50 (0.0%) 5.5 0.2 Java0.1380.1110/48 (0.0%) 5.6 0.1 JavaScript0.0680.0000/50 (0.0%) 5.9 0.0 C++0.0930.0670/35 (0.0%) 5.3 0.1 with 퐽 = 1 when both sets are empty and 퐽 = 0 when exactly one is. The mean cross-agent Jaccard ̄ 퐽 퓁 and mean intra-agent Jaccard ̄ 퐽 intra are averages of퐽 over all valid agent pairs and, for the intra-agent case, over the three independent Claude Code trials of Section 4. The complete disjointness rate CDR 퓁 is the fraction of pairs for which 퐽 = 0, and the unanimous consensus rate UCR 퓁 is the fraction of tasks for which all three Claude trials produce an identical 퐷 (1) . Figure 9 shows that agents consistently solve the same program- ming task with substantially different declared dependency sets. Python is the most convergent ecosystem, yet even there mean pairwise Jaccard similarity remains below 0.5 across all agent pairs, indicating that fewer than half of declared packages are shared on average. JavaScript is the most divergent, with Codex and Gemini agreeing on as little as ̄ 퐽 = 0.073, meaning the two agents share fewer than one in thirteen declared packages for the same task. The complete disjointness rate is highest in JavaScript (63.3%), followed by C++ (57.7%) and Java (56.2%), with Python lowest at 36.0%, yielding a macro-average CDR of 53.3% across the four ecosystems. Complete disagreement between agents is therefore not an edge case but the majority outcome. Table 5 asks whether this divergence is vendor-specific or whether it reflects a deeper instability, by repeating Claude Code three independent times on the same tasks. The unanimous consensus rate is 0% in every language, meaning Claude never converged to the same dependency declaration across all three tri- als for any task in any ecosystem. Intra-agent Jaccard scores are low throughout, with JavaScript again the worst case. Crucially, the mean vocabulary union ̄ 푉 ∪ substantially exceeds the mean unanimous core ̄ 푉 ∩ in every language, confirming that the agent draws from a much larger pool of plausible packages than it con- sistently commits to. Taken together, the figure and table show T A B L E 6 | Environment Gap: C++ system library assumption fail- ures and recovery. “Fail” counts first-attempt failures; “SysLib” counts those whose terminal error is a missing system library; “Rec.” counts SysLib failures eventually repaired. AgentFailSysLibSLARRec.EGAR Claude 322887.5% 28 100.0% Codex 28517.9% 5 100.0% Gemini 1000.0%0– that dependency selection is neither reproducible across vendors nor stable within a single model, a result that implicates the gener- ative process itself rather than any vendor-specific design choice, and suggests that dependency specification is not retrieved from a stable internal knowledge structure but sampled stochastically on each generation. The practical consequence is that agent-generated software is not reproducible at the supply-chain level: two developers prompting the same agent with the same task, or one developer prompting it twice, receive different dependency trees, so every fresh gener- ation effectively produces a different supply chain. Dependency review, pinning decisions, and security approvals performed on one generation do not transfer to the next. Because the instability persists within a single vendor, it also cannot be mitigated by stan- dardizing on one agent; it must be addressed in the generation process itself. 6.4Cross-Language Variation The preceding results consistently separate the four ecosystems, and we now draw the language-level picture together. Python is the easiest ecosystem to specify: it combines the highest first- attempt success, the lowest phantom rates, mild environment inflation ( ̄ 휌 Python = 2.06), and the best initial-manifest accuracy (macro F1 of 0.765). Java sits in the middle: first-attempt success is weaker, but its declaration quality remains high (macro F1 of 0.708) despite heavy transitive inflation. JavaScript is the patho- logical case on every specification metric: the largest inflation ( ̄ 휌 = 11.40), the highest phantom and hidden rates, the lowest manifest accuracy (0.219), and the weakest cross- and intra-agent agreement. Poor specification quality is therefore not merely an agent property; it interacts strongly with how aggressively an ecosystem’s package manager expands and obscures the declared set. C++ varies along a different axis: agents fail not because the resolver inflates their declarations, but because they assume system development libraries that the clean container does not provide. For every C++ project that fails at round 푖 = 0, we classify the terminal error 푒 (0) into one of three types: SystemLib, a find_package failure for a system library ab- sent from 풟 퓁 (e.g., OpenSSL, libcurl, SQLite3); FetchFail, a FetchContent_Declare network or archive error; and Other, covering compilation, missing-header, linker, or runtime failures unrelated to system library absence. From this taxonomy we report two rates per agent: the system library assumption rate (SLAR), the fraction of an agent’s first-attempt C++ failures whose terminal error is of type SystemLib, and the Environment Gap recovery rate (EGAR), the fraction of those SystemLib failures that the agent eventually repairs within the budget. 11 of 16 Data Proc. Cryptography Image Proc. Networking Text Proc. Compression Math/Sci Sys Utility Task Domain 0% 20% 40% 60% 80% 100% Success Rate (pooled across languages) Protocol Convergence by Task Domain (pooled languages) Claude Code Codex Gemini SR 0 (first attempt) Repair gain F I G U R E 10 | Protocol convergence by task domain, with solid and hatched bars as in Figure 6. Table 6 reports the clearest C++ manifestation of the environ- ment gap. Claude’s initial C++ failures are overwhelmingly due to incorrect system-library assumptions (SLAR of 87.5%), and every one of those failures is eventually repaired after feed- back. Codex shows a lower but still meaningful assumption rate (17.9%), again with perfect recovery once the missing package is revealed. Gemini’s C++ failures arise from other causes rather than missing system libraries. The table shows that the clean Docker image is not just making the benchmark harder; it is ex- posing a concrete mismatch between the environments agents implicitly assume and the environments they are actually asked to run in. Together, these ecosystem differences imply that mitigation can- not be uniform. JavaScript projects need the strongest specifica- tion guardrails, because there the manifest is least trustworthy precisely where the resolver amplifies it most. C++ points to a different remedy: since nearly all system-library assumption failures are repaired once the environment reports the missing package, feedback from the actual deployment target, rather than a richer assumed environment, is the effective corrective. Code that runs on a developer’s fully provisioned machine will still fail in the minimal containers that production systems favor, and the environment gap measures exactly that deployment risk. 6.5 Task Complexity Sensitivity Finally, we ask whether specification quality depends on what kind of software is being built. Figure 10 repeats the conver- gence analysis of Section 6.1 per task domain. Final success is at or near 100% in every domain (never below 95% for any agent– domain pair), and no domain is intrinsically hard for all agents: first-attempt success spans 40%–88% across domain–agent pairs, but the weakest domain differs per agent — System Utilities for Claude (40%), Image Processing for Codex (50%), and Data Pro- cessing for Gemini (48%), which is simultaneously the strongest domain for the other two agents. Convergence behavior is there- fore driven primarily by language ecosystem rather than by task domain: C++ is hard for every agent regardless of what the task computes. A sharper test of task sensitivity isolates the tasks that need no external libraries at all. Following Section 4, a subset of benchmark tasks in each language is identified a priori by the authors as entirely solvable with the standard library (e.g., Python’s csv, hashlib, and json modules; Java’s java.util and java.security packages; Node’s fs, crypto, and path built- ins). The unnecessary dependency rate (UDR) is the fraction of these standard-library-sufficient tasks for which the agent never- theless declared at least one external package. On these standard-library-sufficient tasks, Claude introduced ex- ternal packages in 91.8% of these tasks on macro average, reach- ing 100% in C++ and remaining at or above 84% in every other ecosystem. Gemini is similarly aggressive in JavaScript. Codex is much more conservative in Python and JavaScript, though it still adds unnecessary libraries in more than half of eligible C++ tasks. This result isolates over-dependence from task difficulty: the extra libraries are not required by the benchmark itself. Each unnecessary package is avoidable attack surface, mainte- nance burden, and audit scope added for zero functional gain, and on standard-library-sufficient tasks that cost is pure waste. Because the behavior persists on tasks where no library is needed, it reflects a generation prior rather than a response to task de- mands, which makes it a natural target for training objectives that reward minimal, sufficient environment specification alongside functional correctness. 6.6 Limitations The strongest measurement limitation is in C++. Runtime trac- ing undercounts header-only and purely compile-time dependen- cies, which pushes C++ manifest precision and F1 downward relative to ecosystems where runtime loading leaves a clearer file-level footprint. Similarly, although C++ usually has |퐷 (2) | ≈ |퐷 (1) |, a small number of fetched projects recursively pull an ad- ditional library during their own build process, so exact equality should be interpreted as an ecosystem-level trend rather than as a strict invariant. More broadly, the study uses a controlled bench- mark of 50 tasks rather than fully open-ended real-world projects, so the reported rates should be interpreted as evidence about structured code-generation settings rather than as exhaustive es- timates for all software development contexts. All results are additionally conditioned on the prompt template of Section 3.1; because the template is held identical across agents and lan- guages, comparative findings are controlled at the prompt level, but absolute rates may vary under different phrasings, and a sys- tematic prompt-robustness analysis remains future work. The manual evaluation protocol and fixed annotations for standard- library tasks and C++ failure labels improve control, but they also introduce a degree of judgment that future larger-scale repli- cations should further test. Finally, because commercial coding agents evolve rapidly, the results should be understood as a snap- shot of the evaluated agent configurations at evaluation time rather than as immutable model properties. 7 Related Work LLM Code Generation and Its Evaluation Gaps. The era of LLM code generation began with Codex [3], which solved 28.8% of HumanEval problems. Subsequent models such as Al- phaCode [8], StarCoder [14], Code Llama [15], and commercial agents like Claude Code [1], GitHub Copilot, and Gemini Code Assist [2] have expanded from single-function completion to full project synthesis. Surveys by Fan et al. [6] and Jiang et al. [5] cover this landscape comprehensively. 12 of 16AI Magazine, 2026 The critical gap in this literature is that nearly all benchmarks evaluate functional correctness of isolated snippets [3, 7]. A smaller body of work examines code quality [16], bug patterns [17, 18], and security vulnerabilities [4]. Ouyang et al. [19] studied non-determinism in ChatGPT’s code output but restricted their analysis to functional behavior; they did not examine whether de- pendency choices are stable across runs. Our stochastic analysis fills this gap directly: we show that a single agent’s dependency declarations are unstable across independent runs of the same task, with unanimous consensus rate zero across all tasks and languages — a form of non-determinism invisible to functional correctness metrics. Recent work has identified package halluci- nations, where models recommend packages that do not exist in registries, linked to practical supply-chain attacks [20]. Our work is complementary but distinct: we study the misspecification of existing packages, over-declaration, under-declaration, and tran- sitive blindness, which is a different and broader failure mode that persists even when every named package exists. None of this work examines dependency management as a first-class evalua- tion concern across the full lifecycle from declaration through res- olution to runtime execution. Our own earlier study [11] provided initial evidence of dependency gaps in agent-generated code; the present article formalizes the evaluation protocol underlying that study and extends the analysis. Dependency Ecosystems and Their Distinct Challenges. Software dependency management is not a uniform problem: each ecosystem has its own resolution semantics, transitive be- havior, and failure modes, and these differences matter pro- foundly for LLM-generated code. A key contribution of this paper is measuring these ecosystem-specific failure modes under a uni- fied framework, rather than studying any single ecosystem in isolation. We breifly review relevant works within each ecosys- tem: JavaScript/npm.. Zimmermann et al. [21] found that installing a median npm package transitively imports 79 packages main- tained by 39 individuals. The npm ecosystem’s deeply nested dependency graphs mean that a single declared package can pull in dozens or even hundreds of transitive dependencies, creating a vast and largely invisible attack surface. Decan et al. [9] con- firmed that npm exhibits the most aggressive growth and fragility among seven ecosystems studied. Our results quantify this ag- gression in the context of agent-generated code specifically: we observe a mean inflation ratio ̄ 휌 = 11.40 for JavaScript, with indi- vidual projects expanding a three-package declaration into over 140 installed packages, a transitive footprint no agent explicitly reasoned about. Java/Maven.. Soto-Valero et al. [22] found that 75.1% of Maven Central artifacts contain at least one bloated dependency — a li- brary declared in the build file but never used at runtime. Maven’s transitive resolution is opaque: a developer declares a handful of dependencies in pom.xml, and Maven silently resolves an entire tree of transitive artifacts. Wang et al. [10] further documented the risks of third-party library management in Java projects. Whereas these studies measure bloat in human-authored arti- facts, we measure it in agent-generated code where the selection decisions are made by a model rather than a developer, finding that bloat is pervasive and that agents show no awareness of the transitive footprint their declarations induce. C++/CMake.. C++ dependency management is fundamen- tally different from the other three ecosystems. There is no single dominant package manager; dependencies are handled through CMake’s FetchContent, find_package, or manual ven- doring. The key issue for LLM-generated code is the system library assumption gap: agents trained on code from rich development environments assume that system libraries such as libssl-dev or libopencv-dev are already installed. We provide the first systematic measurement of this assumption gap across multiple agents, showing that it accounts for the majority of C++ build failures and that its prevalence differs substantially across agents — implicating training distribution rather than task difficulty as the primary driver. Python/pip.. Python’s ecosystem is comparatively well- behaved for dependency management. The pip freeze convention produces exact version pins, and transitive resolu- tion is relatively flat. However, system library issues still arise for packages with native extensions — pyzbar requires libzbar0 and opencv-python requires libGL.so — and our results show that even in this most tractable ecosystem, agents achieve macro F1 of only 0.765 against runtime ground truth, leaving meaningful specification gaps. Cross-ecosystem analysis. Kikas et al. [23] studied the struc- tural evolution of dependency networks in JavaScript, Ruby, and Rust. Cox [24] articulated the fundamental tension between code reuse and dependency risk. Supply-chain attacks exploiting de- pendencies are a major concern [25, 26], prompting government mandates for Software Bills of Materials [27, 28]. Our paper pro- vides the first cross-ecosystem comparison of dependency specifi- cation quality in agent-generated code, using a unified three-layer measurement framework that enables direct comparison across Python, Java, JavaScript, and C++ rather than treating each ecosystem as an isolated problem. Software Bloat and Unused Dependencies. Bruce et al. [29] found 14.8% of compiled Java methods unreachable. Quach et al. [12] proposed piece-wise compilation for C/C++ attack surface reduction. Soto-Valero et al. [30] showed that most transitive Maven dependencies have very low runtime utilization. These studies measure bloat in human-authored artifacts using static or coverage-based analysis. Our study is the first to measure dependency bloat in agent-generated code, where the selection decisions are made by a model rather than a developer, and to use dynamic runtime tracing as the ground truth rather than static reachability, capturing the actual runtime footprint rather than an approximation of it. Static vs Dynamic Dependency Analysis. Tools for automati- cally inferring dependencies from source code, such as pipreqs for Python, jdeps for Java, and import-graph analyzers more broadly, represent the non-agent baseline for dependency speci- fication. These tools use static analysis, which has known limita- tions: they cannot capture dynamically imported packages, condi- tionally loaded modules, or runtime-only dependencies exercised on specific execution paths. Dynamic tracing via system call inter- position addresses these limitations at the cost of execution-path specificity — capturing only dependencies exercised during a particular run. We use dynamic tracing via ptrace-based prove- nance capture [31, 32] as our runtime ground truth precisely because it captures what the program actually loads. 13 of 16 Software Bills of Materials and Manifest Accuracy. Regu- latory mandates for Software Bills of Materials [27, 28] have prompted a growing literature on SBOM generation accuracy and completeness [33, 34]. Existing SBOM generation tools operate on human-authored codebases; the accuracy of agent-generated manifests as SBOMs has not been studied. Our three-layer frame- work is directly applicable to this question: the declared depen- dency set 퐷 (1) is precisely what an agent-generated SBOM would contain, and our precision, recall, and F1 measurements against runtime ground truth 퐷 (3) provide the first empirical charac- terization of agent-generated SBOM accuracy. The finding that macro F1 ranges from 0.765 for Python to 0.018 for C++ has direct implications for the trustworthiness of agent-generated SBOMs in security-critical contexts. Computational Reproducibility and Provenance. Repro- ducibility remains a persistent challenge in computational sci- ence [35, 36]. Docker containers [37] capture static environment snapshots but cannot reveal what a program actually loads at run- time — a limitation that makes them insufficient as a ground truth for dependency analysis. Provenance-based systems go fur- ther. ReproZip [31] traces system calls to build reproducible archives. Sciunit [13] uses its PTU engine building on CDE’s system call interposition approach [38] to intercept open(), openat(), read(), and execve() via ptrace, recording every file accessed during execution. We use Sciunit as our runtime de- pendency oracle, extending its application from reproducibility packaging to dependency ground-truth extraction in the context of agent-generated code evaluation. LLM Evaluation Methodology. Beyond code generation specif- ically, our work contributes to the broader literature on LLM evaluation methodology. Holistic evaluation frameworks such as HELM [39] argue that model capabilities should be measured across multiple dimensions simultaneously rather than through single-metric proxies. Our three-layer dependency framework in- stantiates this principle for the code generation setting: rather than collapsing environment specification quality into a single pass/fail execution outcome, we decompose it into declaration ac- curacy, resolution behavior, and runtime footprint, each of which implicates a different failure mode and a different remediation. We argue that environment specification should be incorporated as a standard evaluation dimension in code generation bench- marks alongside functional correctness, and that the metrics introduced here - phantom rate, hidden rate, bloat rate, and manifest F1 - provide a reusable evaluation vocabulary for this dimension. 8 Conclusion This study shows that dependency generation by current LLM coding agents is highly inconsistent and often misaligned with the environments their code actually requires. Although these agents can frequently repair projects to functional success, they do not reliably produce correct or minimal dependency specifi- cations. Our results directly answer the central research ques- tion: current LLMs systematically misspecify software environ- ment dependencies, and this misspecification varies substantially across ecosystems and across agents. The problem appears both as under-specification, where required dependencies are missing from the manifest, and as over-specification, where unnecessary libraries inflate the software environment. These findings have immediate implications for both research and practice. Success- ful code generation should not be treated as evidence of envi- ronment correctness without additional validation. More reliable and portable code generation will require evaluation frameworks, validation pipelines, and training objectives that explicitly target accurate and minimal dependency specification alongside task completion. One concrete direction is to treat dependency speci- fication as a first-class learning target: future systems could incor- porate environment-grounded fine-tuning, dependency-aware re- ward signals, or manifest-verification steps in iterative generation and repair loops so that dependency correctness is optimized alongside functional task completion. References 1. Anthropic 2024. “Claude Code: Agentic Coding Tool” https:// w.anthropic.com/claude-code, Accessed: 2024. 2. Gemini Team, Google. “Gemini: A Family of Highly Capable Multimodal Models”. arXiv preprint arXiv:2312.11805 (2023). 3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ry- der, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Moham- mad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Eliz- abeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Woj- ciech Zaremba. “Evaluating Large Language Models Trained on Code”. arXiv preprint arXiv:2107.03374 (2021). 4. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions” In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), 754–768. IEEE. 5. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. “A Survey on Large Language Models for Code Genera- tion”. arXiv preprint arXiv:2406.00515 (2024). Accepted to ACM TOSEM 2025; doi:10.1145/3747588. 6. Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. “Large Language Models for Software Engineering: Survey and Open Problems”. arXiv preprint arXiv:2310.03533 (2023). 7. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. “Program Synthesis with Large Language Models”. arXiv preprint arXiv:2108.07732 (2021). 8. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cy- prien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. “Competition-Level Code Generation with Alpha- Code”. Science 378, no. 6624 (2022): 1092–1097. 9. Alexandre Decan, Tom Mens, and Philippe Grosjean. “An Em- pirical Comparison of Dependency Network Evolution in Seven 14 of 16AI Magazine, 2026 Software Packaging Ecosystems”. Empirical Software Engineer- ing 24, no. 1 (2019): 381–416. 10. Ying Wang, Bihuan Chen, Kaifeng Huang, Bowen Shi, Congy- ing Xu, Xin Peng, Yang Liu, and Yijian Wu. 2020. “An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects” In Proceedings of the 2020 IEEE International Con- ference on Software Maintenance and Evolution (ICSME), 35–45. IEEE. 11. Bhanu Prakash Vangala, Ali Adibifar, Ashish Gehani, and Tanu Malik. 2026. “AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents” , . 12. Anh Quach, Aravind Prakash, and Lok Yan. 2018. “Debloat- ing Software through Piece-Wise Compilation and Loading” In Proceedings of the 27th USENIX Security Symposium (USENIX Security ’18), 869–886. USENIX Association. 13. Quan Pham, Tanu Malik, and Ian Foster. 2013. “Using Prove- nance for Repeatability” In Proceedings of the 5th USENIX Workshop on the Theory and Practice of Provenance (TaPP ’13), Lombard, IL: USENIX Association. 14. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muen- nighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christo- pher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhat- tacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hai- ley Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jen- nifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bah- danau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. “StarCoder: may the source be with you!”. arXiv preprint arXiv:2305.06161 (2023). 15. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Fer- rer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nico- las Usunier, Thomas Scialom, and Gabriel Synnaeve. “Code Llama: Open Foundation Models for Code”. arXiv preprint arXiv:2308.12950 (2024). 16. Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tan- tithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. “Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues”. ACM Transactions on Software Engineering and Methodology 33, no. 5 (2024): 116:1–116:26. 17. Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, and Emily Morgan. 2023. “Large Language Models and Simple, Stupid Bugs” In Proceedings of the 2023 IEEE/ACM 20th In- ternational Conference on Mining Software Repositories (MSR), 563–575. IEEE. 18. Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. “Bugs in Large Language Models Generated Code: An Empir- ical Study”. Empirical Software Engineering 30, no. 3 (2025). Preprint: arXiv:2403.08937, 2024. 19. Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. “LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation”. arXiv preprint arXiv:2308.02828 (2023). 20. Bar Lanyado, Ortal Keizman, and Yair Divinsky. 2023. “Can You Trust ChatGPT’s Package Recommendations?” Vulcan Cy- ber Voyager18 Research Team blog post, https://vulcan.io/blog/ ai-hallucinations-package-risk, Accessed: 2024. 21. Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019. “Small World with High Risks: A Study of Security Threats in the npm Ecosystem” In Proceedings of the 28th USENIX Security Symposium (USENIX Security ’19), 995–1010. USENIX Association. 22. César Soto-Valero, Nicolas Harrand, Martin Monperrus, and Benoit Baudry. “A Comprehensive Study of Bloated Dependen- cies in the Maven Ecosystem”. Empirical Software Engineering 26, no. 3 (2021): Article 45, 1–44. 23. Riivo Kikas, Georgios Gousios, Marlon Dumas, and Dietmar Pfahl. 2017. “Structure and Evolution of Package Dependency Networks” In Proceedings of the 14th International Conference on Mining Software Repositories (MSR), 102–112. IEEE. 24. Russ Cox. “Surviving Software Dependencies”. Communications of the ACM 62, no. 9 (2019): 36–43. 25. Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023. “SoK: Taxonomy of Attacks on Open-Source Soft- ware Supply Chains” In Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP), 1509–1526. IEEE. 26. Chengwei Liu, Sen Chen, Lingling Fan, Bihuan Chen, Yang Liu, and Xin Peng. 2022. “Demystifying the Vulnerability Prop- agation and Its Evolution via Dependency Trees in the NPM Ecosystem” In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22), 672–684. ACM. 27. National Telecommunications and Information Adminis- tration 2021. “The Minimum Elements for a Software Bill of Materials (SBOM)” U.S. Department of Commerce, https://w.ntia.doc.gov/files/ntia/publications/sbom_ minimum_elements_report.pdf, Pursuant to Executive Order 14028 §4(f). 28. The White House 2021.“Executive Order on Im- proving the Nation’s Cybersecurity” Executive Order 14028 of May 12, 2021, https://w.whitehouse. gov/briefing-room/presidential-actions/2021/05/12/ executive-order-on-improving-the-nations-cybersecurity/, . 29. Bobby R. Bruce, Tianyi Zhang, Jaspreet Arora, Guoqing Harry Xu, and Miryung Kim. 2020. “JShrink: In-Depth Investiga- tion into Debloating Modern Java Applications” In Proceedings of the 28th ACM Joint European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020), 135–146. ACM. 30. César Soto-Valero, Thomas Durieux, Nicolas Harrand, and Benoit Baudry. “Coverage-Based Debloating for Java Bytecode”. ACM Transactions on Software Engineering and Methodology 32, no. 2 (2023): Article 38, 1–34. 31. Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. 2016. “ReproZip: Computational Reproducibility With Ease” In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data (SIGMOD ’16), 2085–2088. ACM. 32. Dai Hai Ton That, Gabriel Fils, Zhihao Yuan, and Tanu Ma- lik. 2017. “Sciunits: Reusable Research Objects” In Proceedings of the 2017 IEEE 13th International Conference on e-Science (e- Science ’17), 158–167. IEEE. 33. Nusrat Zahan, Elizabeth Lin, Mahzabin Tamanna, William Enck, and Laurie Williams. “Software Bills of Materials Are Re- quired. Are We There Yet?”. IEEE Security & Privacy 21, no. 2 (2023): 82–88. 34. Boming Xia, Tingting Bi, Zhenchang Xing, Qinghua Lu, and Liming Zhu. 2023. “An Empirical Study on Software Bill of Materials: Where We Stand and the Road Ahead” In Proceed- ings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE), 2630–2642. 35. Monya Baker. “1,500 Scientists Lift the Lid on Reproducibility”. Nature 533, no. 7604 (2016): 452–454. 15 of 16 36. Christian Collberg and Todd A. Proebsting. “Repeatability in Computer Systems Research”. Communications of the ACM 59, no. 3 (2016): 62–69. 37. Carl Boettiger. “An Introduction to Docker for Reproducible Re- search”. ACM SIGOPS Operating Systems Review 49, no. 1 (2015): 71–79. 38. Philip J. Guo and Dawson Engler. 2011. “CDE: Using System Call Interposition to Automatically Create Portable Software Packages” In Proceedings of the 2011 USENIX Annual Technical Conference (USENIX ATC ’11), Portland, OR: USENIX Associa- tion. 39. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. “Holistic Evalu- ation of Language Models”. Transactions on Machine Learning Research (2023). arXiv:2211.09110. APPENDIX ASupplementary Structural Metrics This appendix reports supplementary results that support inter- pretation of the primary findings. Manifest size vs. first-attempt success.. We test whether projects with larger declared dependency sets fail more frequently on their first execution attempt. For each ecosystem퓁, we com- pute the Spearman rank correlation between |퐷 (1) 푎,푝,퓁 | and the round-zero success indicator ퟏ[Success(퐶 (0) ,푀 (0) ,풟 퓁 )] over all (푎,푝) pairs. A negative correlation would indicate that manifest bloat is itself a source of build fragility. T A B L E A1 | Spearman correlation between manifest size and first- attempt success. Ecosystem휌 푆 푝-value Python−0.276 0.0006 Java−0.260 0.0013 JavaScript −0.093 0.2564 C++−0.410 0.0000 Table A1 shows that larger declared dependency sets are as- sociated with lower first-attempt success in Python, Java, and C++. The strongest negative association appears in C++, while JavaScript shows the same direction but not a statistically mean- ingful effect in the current benchmark. Summary.. Table A2 cross-references the eight primary metrics against the results subsections where they are reported. B Complete Problem List Most frequent phantom and hidden packages.. We compile ranked frequency tables of the packages that appear most of- ten in 퐷 (1) ⧵ 퐷 (3) (phantom) and 퐷 (3) ⧵ 퐷 (1) (hidden) across all projects in each ecosystem. These tables serve as qualitative diag- nostics: recurring phantom packages such as pycparser, yargs, and nlohmann_json reveal systematic over-declaration patterns, while recurring hidden packages such as six, python-dateutil, T A B L E A2 | Metric summary: definitions, key quantities, and result section. IDMetricKey quantityResults M1 Success rate under self-correction SR (0) , SR (≤푘) §6.1 M2 Environment growth휌§6.2 M3 Phantom / hidden / bloat rates 휙, 휂, 훽§6.2 M4 Manifest accuracyP, R, F1§6.2 M5 Cross-agent consistency ̄ 퐽 퓁 , CDR§6.3 M6 Stochastic variability ̄ 퐽 intra , UCR§6.3 M7 Unnecessary external libraries UDR§6.5 M8 Environment gap failuresSLAR, EGAR §6.4 jackson-core, and jackson-annotations reveal systematic omissions in dependency reasoning. 16 of 16AI Magazine, 2026 T A B L E B3 | All 50 programming problems. IDProblemDomainCx.IDProblemDomainCx. p_01 CSV Statistical AnalyzerData Processing Mp_26 DNS ResolverNetworkingM p_02 JSON Schema ValidatorData Processing Mp_27 HTTP REST Client w/ RetryNetworkingH p_03 XML to JSON ConverterData Processing Mp_28 Static HTTP File ServerNetworkingH p_04 YAML Config MergerData Processing Mp_29 Network Latency MonitorNetworkingM p_05 Log File Pattern AnalyzerData Processing Mp_30 Markdown to HTML Converter Text Processing M p_06 SQLite Database ManagerData Processing Hp_31 Spell CheckerText Processing M p_07 Data Profiling ToolData Processing Hp_32 TF-IDF Search EngineText Processing H p_08 Multi-Format Data Converter Data Processing Hp_33 Text Diff ToolText Processing M p_09 Time Series Trend Detector Data Processing Hp_34 HTML to Plain TextText Processing M p_10 Duplicate Record FinderData Processing Mp_35 Template EngineText Processing H p_11 AES-256 File EncryptionCryptographyHp_36 ZIP Archive ManagerCompressionM p_12 RSA Digital SignaturesCryptographyHp_37 GZIP Compression ToolCompressionM p_13 HMAC Integrity CheckerCryptographyMp_38 Tar Archive HandlerCompressionM p_14 TOTP GeneratorCryptographyMp_39 File DeduplicatorCompressionM p_15 Password Strength Analyzer CryptographyMp_40 Compression BenchmarkCompressionH p_16 X.509 Certificate ParserCryptographyHp_41 Matrix Operations Calculator Math/Scientific H p_17 Bcrypt Password HasherCryptographyMp_42 Linear Regression FitterMath/Scientific H p_18 Image Format ConverterImage Processing Mp_43 Dijkstra Shortest PathMath/Scientific M p_19 Image Resizer / Thumbnails Image Processing Mp_44 FFT Spectrum AnalyzerMath/Scientific H p_20 Image Watermarking ToolImage Processing Mp_45 Numerical IntegrationMath/Scientific H p_21 Image Histogram Analyzer Image Processing Mp_46 QR Code Generator / ReaderSystem Utilities M p_22 EXIF Metadata ExtractorImage Processing Mp_47 PDF Report GeneratorSystem Utilities H p_23 Color Palette ExtractorImage Processing Hp_48 Merkle Tree VerifierSystem Utilities M p_24 HTTP File DownloaderNetworkingMp_49 Cron Expression ParserSystem Utilities M p_25 TCP Port ScannerNetworkingMp_50 Structured Log ProcessorSystem Utilities M Cx. = complexity: M = Medium, H = High. T A B L E B4 | Most frequent phantom packages 퐷 (1) ⧵ 퐷 (3) by ecosystem. EcosystemMost frequent phantom packages Python pycparser (6), pandas (5), tabulate (2), tzdata (1), python-dateutil (1) Java gson (2), zxing-javase (2), zxing-core (2), slf4j-api (1), javalin (1) JavaScript yargs (46), sharp (8), fast-csv (4), lodash (4), js-yaml (3) C++ nlohmann_json (65), openssl (16), opencv (11), yaml-cpp (5), mbedtls (5) T A B L E B5 | Most frequent hidden packages 퐷 (3) ⧵ 퐷 (1) by ecosystem. EcosystemMost frequent hidden packages Python six (31), python-dateutil (23), pytz (20), packaging (14), kiwisolver (11) Java jackson-annotations (21), jackson-core (20), slf4j-api (14), commons-logging (12), error_prone_annotations (12) JavaScript ansi-styles (15), supports-color (13), has-flag (13), emoji-regex (9), is-fullwidth- code-point (9) C++ crypto (16), z (14), zstd (6), nettle (6), krb5 (5) 17 of 16