Paper deep dive
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, David Lo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:42:42 AM
Summary
The paper introduces CLI-Tool-Bench, a structure-agnostic benchmark designed to evaluate the 0-to-1 generation of CLI tools by LLMs. It addresses limitations in existing benchmarks by using a black-box differential testing framework that validates software through execution in isolated sandboxes, comparing terminal outputs and system-level side effects against human-written oracles.
Entities (4)
Relation Signals (2)
CLI-Tool-Bench → evaluates → LLM
confidence 95% · Evaluating seven state-of-the-art LLMs, we reveal that top models achieve under 43% success
CLI-Tool-Bench → uses → Docker
confidence 95% · It utilizes black-box differential testing in isolated Docker environments
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are driving a shift towards intent-driven development, where agents build complete software from scratch. However, existing benchmarks fail to assess this 0-to-1 generation capability due to two limitations: reliance on predefined scaffolds that ignore repository structure planning, and rigid white-box unit testing that lacks end-to-end behavioral validation. To bridge this gap, we introduce CLI-Tool-Bench, a structure-agnostic benchmark for evaluating the ground-up generation of Command-Line Interface (CLI) tools. It features 100 diverse real-world repositories evaluated via a black-box differential testing framework. Agent-generated software is executed in sandboxes, comparing system side effects and terminal outputs against human-written oracles using multi-tiered equivalence metrics. Evaluating seven state-of-the-art LLMs, we reveal that top models achieve under 43% success, highlighting the ongoing challenge of 0-to-1 generation. Furthermore, higher token consumption does not guarantee better performance, and agents tend to generate monolithic code.
Tags
Links
- Source: https://arxiv.org/abs/2604.06742v1
- Canonical: https://arxiv.org/abs/2604.06742v1
Trouble viewing inline? Open PDF directly →
Full Text
69,699 characters extracted from source content.
Expand or collapse full text
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios Ruida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 1 Harbin Institute of Technology, Shenzhen, China 2 Independent Researcher, China 3 Singapore Management University, Singapore 200111107@stu.hit.edu.cn,200111115@stu.hit.edu.cn,chao.peng@acm.org,gaocuiyun@hit.edu.cn,davidlo@smu.edu.sg Abstract The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to architect and deliver complete, runnable software systems entirely from scratch. However, existing benchmarks fail to adequately assess this promising 0-to-1 soft- ware generation capability due to two fundamental limitations. First, they predominantly rely on predefined structural scaffolds, regarding the task as mere file-filling and failing to evaluate the LLM’s crucial ability in repository structure planning. Second, they heavily depend on rigid white-box unit testing, which forces agents to conform to specific internal implementations and lacks scalable, end-to-end behavioral validation from a user-centric perspective. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure- agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by a black-box differential testing framework, CLI-Tool-Bench comprises 100 high- quality, real-world repositories spanning diverse difficulty levels, programming languages, and application domains. For each repos- itory, our automated pipeline synthesizes a robust suite of end- to-end test cases. We evaluate the agent-generated software by executing it in isolated sandboxes and comparing its system-level side effects and terminal outputs against human-written oracles using a rigorous, multi-tiered equivalence metric. Leveraging CLI-Tool-Bench, we conduct an extensive evaluation of seven state-of-the-art LLMs deployed within prominent agent frameworks. Our experiments reveal that top-tier models achieve less than 43% overall success, highlighting that 0-to-1 generation remains a highly challenging frontier. Furthermore, we discover that higher token consumption does not necessarily yield better performance, and agents exhibit a strong tendency to generate monolithic code structures. CCS Concepts • Software and its engineering→Software verification and validation. † Corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X Keywords LLM Agent, Software Generation, Evaluation ACM Reference Format: Ruida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 . 2018. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 11 pages. https://doi.org/X.X 1 Introduction The advent of Large Language Models (LLMs) has catalyzed a para- digm shift in automated software engineering [1,5,11,16]. Moving beyond simple code completion, recent advancements have spurred the development of autonomous LLM-based agents [3,4,14,24] capable of tackling complex programming tasks. This evolution has given rise to the era of “Vibe Coding” or intent-driven develop- ment [15,18,21], where users, ranging from professional developers to non-technical individuals, can generate functional software sim- ply by expressing their high-level requirements in natural language. In this new paradigm, autonomous agents are expected to take a high-level natural language requirement and generate a complete, runnable software repository entirely from scratch. To measure the capabilities of these agents, robust evaluation benchmarks are indispensable. However, existing benchmarks fall short of evaluating the true potential of agents in real-world soft- ware creation. Traditional benchmarks, such as HumanEval [6] and MBPP [2], are confined to function-level or snippet-level genera- tion. While recent efforts like SWE-bench [12] have elevated the evaluation to the repository level, they predominantly focus on software maintenance tasks—such as resolving issues or adding features to existing codebases—rather than the creation of software from scratch. More recently, attempts like NL2Repo-Bench [7] have explored repository-level generation from scratch; however, their evaluation methodologies reveal critical limitations that hinder the accurate assessment of modern LLM agents. Specifically, we identify three major challenges in the current evaluation landscape: Challenge 1: Reliance on Predefined Repository Structures. Despite rapid advancements, evaluating the true 0-to-1 generation capabilities of LLM agents remains an open challenge. Existing repository-level benchmarks predominantly focus on issue resolu- tion or feature addition within already established repositories [12]. Even in generation-focused benchmarks, the evaluation heavily relies on a fixed, predefined repository structure [7]. These bench- marks typically provide agents with pre-built file skeletons and arXiv:2604.06742v1 [cs.SE] 8 Apr 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 directory scaffolds, reducing the complex task of software genera- tion to mere code-filling. This structure-dependent paradigm fun- damentally bypasses a critical step in software creation: repository structure planning. In real-world 0-to-1 development, developers must autonomously decide how to organize directories, modular- ize files, and manage dependency configurations. By constraining agents to predefined structures, current benchmarks fail to assess whether LLMs can independently plan and construct a coherent repository from scratch. Challenge 2: Absence of End-to-End Black-Box Testing. Furthermore, existing evaluations heavily rely on white-box unit testing. These tests are tightly coupled with the internal implemen- tation details of the software, forcing the generated code to conform to specific function signatures or class definitions. However, from a user-centric perspective, software utilities, such as Command-Line Interface (CLI) tools, are always consumed as black boxes. Users care about whether the tool correctly parses command-line argu- ments, produces the expected terminal outputs, and executes the correct system-level side effects (e.g., modifying the file system), regardless of how the internal code is structured. The reliance on rigid white-box testing not only stifles the structural autonomy of LLMs but also fails to provide a realistic, end-to-end validation of the software’s functional correctness. To address these limitations, we introduce CLI-Tool-Bench, a novel benchmark specifically designed to evaluate the 0-to-1 gen- eration of CLI tools. CLI-Tool-Bench shifts the evaluation para- digm from structure-dependent white-box testing to structure- agnostic black-box differential testing. We curate a high-quality dataset of 100 real-world CLI repositories across three program- ming languages (Python, JavaScript, Go) and complexity levels. To overcome the reliance on predefined structures (Challenge 1), the agent is provided only with a natural language requirement and an empty workspace. This unconstrained setting forces the LLM to autonomously handle repository structure planning, dependency management, and logic implementation from scratch. To provide realistic, end-to-end validation (Challenge 2), we evaluate the gen- erated tool purely from a user-centric perspective. We execute the tool in an isolated sandbox and compare its terminal outputs and system-level side effects (e.g., file system state changes) against a human-written oracle. To ensure a fair assessment, we propose a multi-tiered equivalence metric—encompassing Execution, Ex- act, Fuzzy, and Semantic Match—to accurately gauge behavioral correctness without penalizing the agent’s architectural diversity. In summary, the main contributions of this paper are as follows: •We introduce CLI-Tool-Bench, the first benchmark for end-to- end software generation. It challenges agents to build functional CLI tools from scratch, granting them complete autonomy without predefined scaffolds. • We propose an automated pipeline for repository-level gen- eration. It utilizes black-box differential testing in isolated Docker environments to rigorously assess execution reliability, behavioral equivalence, and system-level side effects. •We evaluate advanced LLMs and agent frameworks on CLI- Tool-Bench, revealing their great struggles with autonomous system- level generation. We also uncover critical behavioral patterns, such as monolithic design preferences and infinite generation loops. 2 Construction and Evaluation Pipeline To evaluate the 0-to-1 software generation capabilities of LLM agents, we propose an automated benchmark construction and evaluation pipeline. As illustrated in Figure 1, it consists of three core modules: (1) Repository Curation, (2) Schema-Guided Task Synthesis, and (3) Black-Box Differential Evaluation. 2.1 Repository Curation To construct a high-quality and representative dataset, we target three mainstream languages widely adopted for CLI development, including Python, JavaScript, and Golang [13]. To ensure the re- liability of the oracle repositories serving as ground truth in our benchmark, we design a rigorous curation pipeline, with the details as follows: 2.1.1Static Metadata Filtering. We initially retrieve candidate repos- itories from GitHub based on specific criteria to ensure benchmark quality and relevance: (1) Stars >10; (2) Primary language purity >60%; (3) Presence of CLI-related keywords in repository descrip- tions; (4) Presence of an open-source license (e.g., MIT); and (5) Exis- tence of language-specific build configuration files (e.g., “setup.py” or “pyproject.toml” for Python, “package.json” for JavaScript, and “go.mod” for Golang). 2.1.2Dynamic Entry Identification. To ensure the practical utility of the selected repositories, we introduce a dynamic verification step. We attempt to install each candidate repository globally in an isolated environment using standard package managers (e.g., “pip install .”, “npm install -g .”). After a successful installation, it is crucial to determine the exact command name used to invoke the CLI tool. While this command name is typically identical to the repository name, variations can occasionally occur. To robustly and autonomously identify the correct entry point for subsequent test- ing, we design a heuristic matching algorithm. Specifically, let푁 푟푒푝표 be the repository name and퐷 푏푖푛 be the target binary directory. Dur- ing installation, we monitor퐷 푏푖푛 to capture the set of newly added executables, denoted as퐸 푛푒푤 . We then select the target executable 푒 ∗ ∈ 퐸 푛푒푤 by matching against푁 푟푒푝표 using a prioritized sequence: (1) Exact match; (2) Case-insensitive match; (3) Acronym match; and (4) Maximum Levenshtein similarity. Once an executable푒 ∗ is identified, we validate it by executing the command “푒 ∗ --help”. A successful execution, indicated by a zero exit code and the output of standard usage documentation, confirms that the identified binary is indeed a functional CLI entry point. 2.1.3 Stratified Manual Validation. Finally, to further ensure the quality of the selected repositories, two software engineering ex- perts, each with over five years of professional programming expe- rience, conduct a rigorous and independent sanity check on every repository in the filtered dataset. This manual review specifically verifies two critical aspects: first, that the heuristically identified command serves as the correct entry point for the repository’s primary CLI tool; and second, that the tool can execute its com- mands correctly without encountering underlying environmental or dependency errors. To eliminate subjective bias, a repository is retained in the final benchmark if and only if both experts reach a unanimous consensus on its absolute correctness. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosConference acronym ’X, June 03–05, 2018, Woodstock, NY DynamicEntry Identification StaticMetadata Filtering Stratified Manual Validation Raw GitHub Repos 100Oracle Repositories 1. Repository Curation 2. Task Synthesis Iterative SchemaExtraction Command Schema Black-box TestCases LLM-Directed Fuzzing for Test Generation TaskPrompts LLM PromptTemplate Standardized Task Prompt Construction 3. DifferentialEvaluation Execution EnvironmentInitialization (푀 !"#$ ) Behavior Side-Effect (푀 "%& ) (푀 #'(( ) Equivalence Evaluation [Exit Code, Stdout, File System] [Exit Code, Stdout, File System] OracleEnv AgentEnv Figure 1: Overview of the CLI-Tool-Bench framework. 2.2 Schema-Guided Task Synthesis A critical challenge in benchmark construction is generating com- prehensive test cases at scale without heavy manual intervention. We address this challenge by designing an automated task synthe- sis pipeline that leverages LLMs to extract a structured command schema from the oracle repository and subsequently generate both the evaluation prompts and a massive suite of test cases based on this extracted schema. 2.2.1Iterative Schema Extraction. Given an oracle repository푅 표푟푎푐푙푒 , we employ an LLM to iteratively parse itsREADMEdocumenta- tion and “--help” outputs. Since modern CLI tools often feature complex, nested command structures, the LLM explores the CLI level by level (e.g., running “tool --help”, then “tool subcommand --help”). It extracts a hierarchical metadata schema푆for the CLI, defined as a tuple: 푆=⟨퐶,푆 푠푢푏 ,푃,퐾⟩(1) where퐶is the command name,푆 푠푢푏 is the set of nested subcom- mands,푃represents the accepted parameters and flags along with their data types (e.g., string, path, boolean), and퐾signifies execu- tion constraints (e.g., required arguments). 2.2.2 LLM-Directed Fuzzing for Test Generation. Based on the ex- tracted schema푆, we implement an LLM-directed fuzzing mech- anism to generate a diverse set of test intents and corresponding command strings. To systematically cover the CLI’s capabilities, we first unroll the hierarchical schema푆into a set of distinct Com- mand Classes. A command class represents a unique functional path, defined by a specific command/subcommand and a valid com- bination of parameters that satisfies the constraints퐾(e.g., required flags or mutually exclusive arguments). For each repository, the LLM utilizes a predefined Python-based fuzzing framework, which provides standardized utilities for command execution, output cap- ture, and assertion checking, to automatically generate customized fuzzing scripts. These scripts generate test cases across various dimensions for the identified classes: (1) Common usage; (2) Bound- ary conditions; and (3) Error handling (e.g., intentionally omitting required parameters to test exception outputs). To ensure the validity of the generated test cases, we introduce an execution-feedback loop aimed at instantiating every identified command class. For a given command class퐶 푖 , let푡 푖 be a generated concrete test command. We execute푡 푖 against푅 표푟푎푐푙푒 . If the exe- cution fails unexpectedly (e.g., returning a non-zero exit code for a normal usage intent), the error trace (stderr) is fed back to the LLM to refine the command string. This process iterates until we successfully discover at least one valid, working execution instance for퐶 푖 . Once this verified template is established, the LLM further mutates its parameters to generate a broader suite of test cases. Crucially, this automated fuzzing also serves as a strict quality filter for the oracle repositories. If there exists any command class explicitly claimed in the “--help” documentation for which the LLM consistently fails to find a successful execution instance despite repeated refinements (e.g., reaching a maximum retry limit), we deem the oracle implementation incomplete, buggy, or misaligned with its own documentation. In such cases, the entire repository is discarded. Furthermore, while our LLM-directed pipeline is capa- ble of generating an arbitrarily large volume of test cases, execut- ing end-to-end differential tests within isolated Docker sandboxes incurs non-trivial computational overhead. To strike an optimal balance between evaluation comprehensiveness and execution ef- ficiency, we configure the pipeline to synthesize a robust suite of exactly 50 end-to-end test cases for each identified command class. To ensure thorough behavioral verification, this suite deliberately encompasses both positive test cases (where the oracle executes successfully with a zero exit code) and negative test cases (designed to trigger expected errors or exceptions). Ultimately, this automated, schema-guided approach enables the scalable generation of highly diverse, end-to-end test cases, far exceeding the coverage typically achieved through manual crafting. 2.2.3 Standardized Task Prompt Construction. We construct the task prompt using a unified template (detailed in our repository). To strictly mitigate the risk of data contamination, where the eval- uated LLM might recognize the target and rely on memorized code from its pre-training corpus rather than generating it from scratch, we apply a rigorous de-identification process. Specifically, we auto- matically and manually scrub all identifying metadata, including author names, email addresses, GitHub repository links, and spe- cific project branding, from the source texts. The final anonymized prompt comprises three core components to simulate a realistic, Conference acronym ’X, June 03–05, 2018, Woodstock, NYRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 greenfield development requirement: (1) The sanitized functional context derived from theREADME; (2) The complete, iteratively ex- tracted “--help” documentation, which serves as the strict external interface specification; and (3) One concrete, verified successful exe- cution example for each command class (obtained from the fuzzing stage) to unambiguously demonstrate the expected behavior. 2.3 Black-Box Differential Evaluation To evaluate the LLM-generated software푅 푡푒푠푡 without relying on predefined repository structures or internal code implementations, we design a black-box Differential Evaluation Engine based on isolated Docker sandboxes. The proposed engine consists of three key phases: 2.3.1 Environment Initialization. For a given task, we instanti- ate two identical Docker containers based on the latest language- specific base images (e.g.,python:latest). We mount the oracle repository푅 표푟푎푐푙푒 and the generated repository푅 푡푒푠푡 into their re- spective containers at a unified workspace path. After executing the standard installation commands, we capture the exact initial state of the workspace as a clean snapshot. This ensures strict stateless isolation and prevents any persistent environmental changes, such as newly created files or modified configurations (i.e., system-level side effects), from polluting subsequent tests. We denote these per- sistent, restorable base environments as퐸 표푟푎푐푙푒 and퐸 푡푒푠푡 for the human-written oracle and the LLM-generated tool, respectively. 2.3.2Differential Execution. For each test case푡 ∈ 푇, we execute푡 independently in both퐸 표푟푎푐푙푒 and퐸 푡푒푠푡 . To comprehensively capture the behavior of the CLI tool, our engine monitors the execution process and records the resulting state. Specifically, the execution yields a state transition tuple: 퐸푥푒푐(푅,푡,퐸) → ⟨퐶 푟푒푡 ,푂 푠푡푑 ,Δ푆⟩(2) where퐶 푟푒푡 is the return code,푂 푠푡푑 is the standard output, and Δ푆represents the raw system-level side effects (i.e., file system mutations). This tuple serves as the foundational data structure for our subsequent equivalence analysis. 2.3.3 Equivalence Evaluation. Based on the captured execution states, we evaluate the functional equivalence using three rigorous metrics, including Execution Reliability, Behavioral Equivalence, and System-Level Side-Effect Consistency: Execution Reliability (푀 푐표푑푒 ): We strictly focus on test cases that represent valid functional paths, defined as those where the oracle executes successfully with a zero exit code (퐶 표푟푎푐푙푒 푟푒푡 ==0). For these expected-to-work commands, we verify if the LLM-generated software also completes successfully, i.e.,퐶 푡푒푠푡 푟푒푡 ==0. This ensures that the LLM correctly implements the core functionalities explicitly claimed in the documentation, while avoiding the ambiguity of matching diverse non-zero error codes across different internal implementations. Behavioral Equivalence (푀 표푢푡 ): We capture the standard out- put푂 푠푡푑 generated by each test case, deliberately ignoring standard error streams to focus purely on the functional payload. Recogniz- ing that LLMs may generate functionally identical CLI tools with slight formatting variations in their terminal outputs, our differ- ential evaluation engine employs a multi-tiered output compari- son mechanism. This mechanism evaluates behavioral equivalence across three progressive relaxation levels: (1) Exact Match that demands strict string equivalence after basic whitespace normal- ization; (2) Fuzzy Match that utilizes algorithmic similarity metrics (e.g., normalized edit distance) to tolerate minor formatting diver- gences while preserving core data integrity; and (3) Semantic Match that leverages an LLM-as-a-judge [8] to verify the equivalence of the core informational payload, completely disregarding superficial stylistic differences. The detailed definitions of these metrics are provided in Section 3. System-Level Side-Effect Consistency (푀 푑푖푓 푓 ): Beyond termi- nal outputs, real-world CLI tools often interact with the file system (e.g., creating, modifying, or deleting files). To evaluate these exter- nal impacts, we track the state of the workspace before and after a command is executed. By comparing these two states, we ex- tract the exact file system changes, which form the raw side-effect Δ푆. Furthermore, during execution, tools often generate trivial in- termediate artifacts (such as hidden cache folders or temporary logs) that are irrelevant to the core functionality. To prevent these non-essential files from interfering with the evaluation, our engine automatically ignores hidden paths. LetΦ 푖푔푛표푟푒 be the set of these ignored paths. The effective side-effect is calculated by filtering them out:Δ푆 ′ =Δ푆\Φ 푖푔푛표푟푒 . A specific test case passes this metric only if the effective side-effects of the LLM perfectly match those of the oracle (Δ푆 ′ 표푟푎푐푙푒 ==Δ푆 ′ 푡푒푠푡 ). To provide a clear overview of the constructed benchmark, Table 1 summarizes the statistical distribution of the curated repositories across three key dimensions: task difficulty, programming language, and application domain. Following the taxonomy established by NL2Repo-Bench, we stratify the task difficulty into three levels based on the oracle’s Lines of Code (Easy≤1500, Medium 1500− 4000, Hard≥4000), and classify the repositories into nine distinct application domains. For programming languages, we specifically focus on Python, JavaScript (Node.js), and Go. These languages are selected to provide a representative mix of both interpreted and compiled paradigms that are extensively utilized in modern CLI tool development and system automation. This diverse composition ensures that our evaluation comprehensively reflects an LLM’s general-purpose software engineering capabilities. To ensure the high quality and representativeness of our bench- mark, we manually curated a final set of 100 repositories from the initial candidate pool. During this selection process, we carefully balanced the dataset across three key dimensions: programming language, difficulty level (measured by Lines of Code, LOC), and application domain. For the domain classification, we adopted the taxonomy introduced by NL2Repo-Bench [7]. The detailed statisti- cal distribution of the curated dataset is presented in Table 1. As illustrated, the final benchmark maintains a relatively even and highly diverse composition, comprehensively covering three main- stream CLI programming languages, various project complexities, and nine distinct real-world application scenarios. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 1: Statistical distribution of our benchmark. Dimension Sub-categoryCount Avg. LOC Difficulty Easy (≤ 1500 LOC)42623.76 Medium (1500− 4000 LOC)242,597.33 Hard (≥ 4000 LOC)3418,445.91 Language Python384,495.34 JavaScript165,448.56 Go469,949.89 Domain Web Development85,500.75 Testing91,921.78 Utility Libraries208,607.75 Machine Learning123,332.33 Data Analysis & Processing123,894.58 Database Interaction63,509.17 Networking Tools76,462.57 Batch File Processing1420,651.29 System Tools123,342.00 3 Experimental Setup 3.1 Selected LLMs and Agent Frameworks To comprehensively evaluate the state-of-the-art in autonomous software generation, we select 7 cutting-edge LLMs, encompass- ing both leading closed-source models and highly capable open- source models: GPT-5.4, Claude-Sonnet-4.6, DeepSeek-V3.2, Qwen- 3.5-plus, GLM-5, MiniMax-M2.5, and Kimi-k2.5. To effectively evaluate these models’ capabilities as software engineers, we employ two representative agent frameworks specif- ically designed for repository-level tasks: •OpenHands (with CodeAct) [23]: A prominent open-source agent framework that utilizes the CodeAct paradigm, allowing the LLM to iteratively execute code, interact with a bash terminal, and observe environmental feedback. • Mini-SWE-Agent [22]: A streamlined adaptation of the popular SWE-agent framework, specifically optimized for iterative reposi- tory construction and terminal-based debugging. By evaluating seven models across two frameworks, we con- duct a total of 14 distinct agent configurations for each of the 100 repositories in our benchmark. 3.2 Evaluation Metrics and Scoring Mechanism As established in methodology, evaluating the true functional cor- rectness of generated CLI tools requires a rigorous, multi-layered approach. To capture the agent’s capabilities, we design an evalua- tion funnel consisting of progressive metrics. 3.2.1The Evaluation Funnel and Equivalence Metrics. Before com- paring any terminal outputs, the generated repository푅 푡푒푠푡 must pass two fundamental system-level prerequisites: •Build (Global Installation Success Rate): We first verify if푅 푡푒푠푡 can be successfully installed in the isolated environment using standard package managers. A failure here indicates a funda- mentally broken repository. •Exec (Execution Reliability,푀 푐표푑푒 ): We strictly focus on the subset of test cases where the oracle executes successfully (퐶 표푟푎푐푙푒 푟푒푡 == 0). This metric measures the proportion of these valid test cases where the agent-generated tool also completes without runtime errors (퐶 푡푒푠푡 푟푒푡 ==0). A generated test case is considered valid for further output comparison if and only if it passes the Exec check and perfectly matches the oracle’s system-level side effects (Δ푆 ′ 표푟푎푐푙푒 ==Δ푆 ′ 푡푒푠푡 ). Building upon these strict prerequisites, we define three progressive metrics to evaluate the Behavioral Equiva- lence (푀 표푢푡 ) of the standard outputs: • Exact Match (EM): This metric represents the most rigid functional alignment. The standard outputs of the oracle and the agent are first normalized by stripping all whitespace characters (e.g., , , and spaces). A match is recorded only if the resulting strings are strictly identical. •Fuzzy Match (FM): To accommodate minor formatting diver- gences (e.g., different table alignments or spacing) while preserving core data integrity, we calculate the normalized Levenshtein edit distance between the two outputs. A match is recorded if the string similarity score meets or exceeds an empirical threshold of휏=0.8. •Semantic Match (SM): For cases where the output structure differs significantly but the underlying information is correct, we employ GPT-5.4 as an automated judge. The LLM is prompted to evaluate whether the core semantic content and informational pay- load of the two outputs are equivalent, completely disregarding superficial stylistic differences. To rigorously validate the reliabil- ity of this LLM-as-a-judge approach, we conducted a large-scale human annotation study. Two software engineering experts—each with over five years of professional experience—independently reviewed a random sample of 1,000 pairs of oracle and agent execu- tion outputs, manually labeling them for semantic equivalence. We then compare the human consensus labels against the automated judgments produced by GPT-5.4. This extensive analysis yielded a Cohen’s Kappa coefficient of휅>0.9, demonstrating that our Semantic Match metric strongly aligns with human judgment, en- abling a more flexible yet accurate evaluation of output consistency. 3.2.2 Multi-Level Scoring Mechanism. To systematically quantify the performance of an agent-generated repository푅 푡푒푠푡 and pop- ulate our final evaluation tables, we compute final scores for the aforementioned metrics. The Build score is simply the binary instal- lation success rate averaged across all repositories. For the remain- ing metrics (Exec, EM, FM, and SM), we employ a macro-averaging aggregation strategy. To prevent command classes with simpler logic from disproportionately dominating the evaluation, we cal- culate the scores as follows. Let퐶denote the set of all identified command classes for a given repository푅 푡푒푠푡 . For each command class푐 ∈ 퐶, let푇 푐 be its corresponding suite of 50 test cases. The pass rate of a specific command class under a given evaluation metric (denoted as푚푒푡푟푖푐 ∈ Exec, EM, FM, SM) is calculated as: 푃 푚푒푡푟푖푐 (푐)= 1 |푇 푐 | ∑︁ 푡∈푇 푐 I 푚푒푡푟푖푐 (푡)(3) whereI 푚푒푡푟푖푐 (푡) ∈ 0,1is the binary indicator of whether test case t successfully passes the specified metric. The final aggregated score for the repository푅 푡푒푠푡 under the chosen metric is then formally defined as: 푆푐표푟푒 푚푒푡푟푖푐 (푅 푡푒푠푡 )= ( 1 |퐶| Í 푐∈퐶 푃 푚푒푡푟푖푐 (푐)if installation succeeds 0if installation fails (4) Conference acronym ’X, June 03–05, 2018, Woodstock, NYRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 This formulation ensures that a repository fundamentally failing the initial package manager installation is penalized with a score of 0. For successfully installed repositories, the macro-averaging mechanism guarantees that the agent is evaluated on its compre- hensive ability to implement the entire spectrum of the CLI tool’s functionalities evenly. The final values reported in our subsequent experimental results represent the average푆푐표푟푒 푚푒푡푟푖푐 across all evaluated repositories. 3.3 Implementation Details Following the evaluation protocol established by NL2Repo-Bench, we aim to fully assess the repository-level generation capabili- ties of the evaluated LLMs. Therefore, rather than restricting the agents with default hyperparameters, we remove all artificial con- straints—such as maximum iteration limits or token budgets—across all models and frameworks. The generation process is entirely open- ended, allowing the agent to autonomously determine when the software construction is complete and voluntarily terminate the exe- cution. For all auxiliary tasks within our automated pipeline—including schema extraction, fuzzing test generation, and the Semantic Match judge—we strictly utilize GPT-5.4. To eliminate generation random- ness and ensure deterministic evaluation, the temperature for these auxiliary GPT-5.4 API calls is set to zero. 3.4 Research Questions We investigate the following four research questions (RQs): • RQ1: How do state-of-the-art LLMs perform in ground-up CLI generation, and how do different agent frameworks influence their success rates? •RQ2: How does the software generation capability of these agents scale or degrade across different task difficulty levels? •RQ3: What are the differences in computational overhead and generation efficiency among the models under an unconstrained, open-ended generation setting? • RQ4: Given complete structural freedom, how do the internal structures of agent-generated repositories compare to human- written oracles? 4 Experimental Result In this section, we present the evaluation results of the selected LLMs and agent frameworks to answer our four research questions. 4.1 RQ1: Overall Performance and Framework Impact To answer RQ1, we evaluate seven state-of-the-art LLMs across two distinct agent frameworks, revealing a clear performance hierarchy (Table 2). Kimi-k2.5 emerges as the absolute frontrunner, achieving the highest average Semantic Match (SM) score of 42.74%, followed closely by MiniMax-M2.5, which excels particularly within the Mini-SWE-Agent framework. A competitive middle tier consists of GPT-5.4, Qwen-3.5-plus, and GLM-5, all hovering around 29% to 30% SM. Conversely, Claude-Sonnet-4.6 exhibits abnormally poor performance (10.48% SM), primarily bottlenecked at the initial Build stage. Furthermore, the choice of framework significantly impacts success rates; Mini-SWE-Agent outperforms OpenHands across (a)Results on OpenHands.(b)Results on Mini-SWE-Agent. Figure 2: Performance comparison of seven models across three programming languages. The radar charts illustrate the varying capabilities of each model, with the radial axes representing the evaluation scores of Semantic Match Score. almost all models (31.94% vs. 25.75% average SM), suggesting its workspace management and interaction design provide a more conducive environment for the complex reasoning required in CLI tool development. Beyond individual model capabilities, the progressive evaluation funnel exposes fundamental bottlenecks in current agentic work- flows. Across all models, there is a steep degradation from the Build stage (average 76.93%) to Execution Reliability (57.90%), followed by a drastic plunge when evaluated by Exact Match (EM, 23.07%). This massive drop highlights that while agents can successfully write syntactically correct and executable code, they struggle to perfectly replicate the exact string outputs of the Oracle. However, the notable recovery in scores observed in the FM score of 37.69% and the SM score of 28.85% validates our methodological design: agents frequently generate functionally correct and semantically equivalent CLI schemas that are unfairly penalized by stringent EM metrics. A deeper analysis of the programming language distribu- tion (Figure 2) further reveals a pronounced language bias inherent in current LLMs. As depicted in the radar charts, almost all evalu- ated models exhibit larger coverage areas for interpreted languages like Python and JavaScript, while Golang emerges as a severe bot- tleneck (represented by the constricted inner green polygons). It suggests that models frequently fail to navigate Golang’s strict type matching and rigid compilation constraints, often generating code that fails to build. Overall, while top-tier models demonstrate promising capabilities, the highest average Semantic Match score remains below 43%, indicating that end-to-end CLI tool develop- ment remains a highly challenging task with substantial room for future improvement. Finding 1: Despite the promising capabilities of top-tier models, their steep performance degradation across the evaluation fun- nel and severe struggles with compiled languages cap the highest overall success rate at under 43%, highlighting that end-to-end software generation remains a highly challenging frontier with substantial room for future improvement. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 2: Overall performance of the evaluated LLMs across two agent frameworks. The rightmost columns and bottom row present macro-averages. Best results are highlighted in bold. Model OpenHandsMini-SWE-AgentAverage (Both Frameworks) Build Exec EM FM SM Build Exec EM FM SM Build Exec EM FM SM GPT-5.480.00 58.63 21.18 37.31 28.1486.00 60.96 24.25 40.30 31.7983.00 59.79 22.72 38.81 29.97 Claude-Sonnet-4.6 38.00 17.395.598.806.7853.00 31.43 11.98 20.77 14.1845.50 24.418.7914.79 10.48 DeepSeek-V3.281.00 59.51 21.18 36.45 26.7973.00 57.79 20.90 38.83 26.7077.00 58.65 21.04 37.64 26.74 Qwen-3.5-plus74.00 58.00 23.09 37.21 28.1377.00 60.41 23.53 38.68 29.7975.50 59.20 23.31 37.94 28.96 GLM-576.00 56.07 20.76 35.27 26.8380.00 62.08 26.02 42.26 32.2178.00 59.08 23.39 38.76 29.52 MiniMax-M2.578.00 63.55 21.60 39.74 28.30 96.00 78.73 31.34 50.84 38.7287.00 71.14 26.47 45.29 33.51 Kimi-k2.589.00 68.05 28.98 45.11 35.31 96.00 78.01 42.51 56.11 50.18 92.50 73.03 35.74 50.61 42.74 Average73.7154.4620.3434.2725.7580.1461.3425.7941.1131.9476.9357.9023.0737.6928.85 (a)Results on OpenHands.(b)Results on Mini-SWE-Agent. Figure 3: Correlation between repository complexity and agent performance. Each scatter point represents an individ- ual test case. The color of the points denotes the difficulty level: Green for Easy, Orange for Medium, and Red for Hard. The dashed black line represents the LOESS trend line. 4.2 RQ2: Impact of Task Complexity To answer this, we analyze the correlation between the Oracle LOC and the Semantic Match Score, as visualized in Figure 3. Intuitively, one might expect a strict negative correlation: as the repository grows larger, the agent’s performance should monotonically de- grade due to context window limitations and complex dependency resolutions. However, the trend lines reveal a counterintuitive non- monotonic (U-shaped) trajectory. In the transition from Easy (< 1500 LOC) to Medium (1500 - 4000 LOC) repositories, we observe an obvious decline in performance. As projects evolve from simple scripts to multi-module structures, the cognitive load on the LLM increases significantly. Agents strug- gle with cross-file context retrieval and often suffer from the “lost in the middle” phenomenon, leading to a rapid drop in semantic accuracy. Surprisingly, as the LOC scales into the Hard category (> 4000 LOC), the trend line stabilizes and eventually trends up- ward. We attribute this phenomenon to the structural standardiza- tion of enterprise-scale repositories. Extremely large CLI projects rarely rely on ad-hoc argument parsing; instead, they heavily utilize standardized, well-documented CLI frameworks (e.g., “Cobra” in Go, “Click” in Python). LLMs are highly proficient at recognizing these boilerplate structures from their pre-training corpora. Conse- quently, even if the agent fails to execute the tool perfectly, it can accurately reconstruct the semantic schema (commands, flags, and arguments) based on framework conventions, thereby achieving a higher Semantic Match Score. (a)Results on OpenHands.(a)Results on Mini-SWE-Agent. Figure 4: Cost-effectiveness analysis of LLM agents across dif- ferent frameworks. The scatter plots illustrate the trade-off between task performance and execution cost. Small translu- cent dots represent individual task runs, while the large cross markers denote the centroid for each model. The dashed gray lines indicate the overall mean score and mean token con- sumption, dividing the space into four quadrants. Finding 2: Agent performance exhibits a non-monotonic, U- shaped trend with repository complexity; while performance drops sharply in mid-sized repositories, the observed rebound in massive projects suggests that agents might benefit from the highly standardized structures typical of large-scale frameworks. 4.3 RQ3: Agent Cost and Generation Efficiency To answer RQ3, we analyze the cost-effectiveness of different LLMs by plotting their Semantic Match Scores against their total token consumption, as shown in Figure 4. The dashed lines represent the average score and token usage, dividing the performance space into four quadrants. Ideally, a highly capable agent should fall into the top-left quadrant, achieving above-average scores with below- average token costs. Across both frameworks, GPT-5.4 and GLM- 5 consistently demonstrate this optimal behavior. They achieve highly competitive semantic scores while maintaining a minimal token footprint. This suggests that these models possess strong zero- shot reasoning capabilities and can generate accurate CLI schemas without relying on extensive, token-heavy trial-and-error loops. Similarly, Kimi-k2.5 emerges as a standout performer, particularly in the Mini-SWE-Agent framework (Figure 4 (b)), where it dominates the top-left quadrant by achieving the highest overall score with remarkably low token consumption. Conference acronym ’X, June 03–05, 2018, Woodstock, NYRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 Table 3: Generation overhead and estimated API cost per repository, categorized by LLM and agent framework. ModelFramework StepsPrompt Tokens (K)Comp. Tokens (K)Cost ($) Avg Min MaxAvg MinMaxAvg Min MaxAvg Min Max GPT-5.4 OpenHands12.38626161.8643.75708.429.321.6925.890.540.151.94 Mini-SWE-Agent2.332616.902.8156.453.130.308.260.090.010.21 Claude-Sonnet-4.6 OpenHands7.76319194.1552.37690.857.651.0181.780.700.183.30 Mini-SWE-Agent6.9121985.888.96483.826.860.5120.920.360.031.77 DeepSeek-V3.2 OpenHands60.88251322,057.02352.049,851.3618.874.30130.310.580.102.77 Mini-SWE-Agent41.7914112914.8743.565,626.7416.971.8251.700.260.011.60 Qwen-3.5-plus OpenHands35.613941,161.520.515,104.6314.022.0865.330.500.012.10 Mini-SWE-Agent41.2781311,076.3918.976,907.6419.731.3271.860.480.012.88 GLM-5 OpenHands25.70382239.542.871,073.989.420.6637.560.270.011.19 SWE-Agent43.582116264.389.451,156.6614.330.9859.560.310.011.22 MiniMax-M2.5 OpenHands84.09201693,783.14285.0811,736.3128.063.8284.451.170.093.58 Mini-SWE-Agent68.23101532,399.2421.379,503.5032.941.04105.400.760.012.93 Kimi-k2.5 OpenHands45.7041261,259.5724.926,844.4020.581.5475.980.820.034.33 Mini-SWE-Agent48.683193343.856.044,883.8535.360.46342.210.310.013.96 Conversely, the right half of the scatter plots reveals a phenom- enon of “diminishing returns” in agent trajectories. Models like Minimax-M2.5 and DeepSeek-V3.2 frequently fall into the right-side quadrants, consuming massive amounts of tokens (often exceeding 2 million) but failing to achieve top-tier performance. This high token consumption is typically indicative of “thrashing”, situations where the agent gets trapped in repetitive debugging cycles or generates overly verbose, unhelpful commands without making actual progress toward the task resolution. On the other extreme, Claude-Sonnet-4.6 consistently occupies the bottom-left quadrant. Its extremely low token usage, coupled with the lowest semantic scores, implies a tendency to “fail fast”; the agent likely encounters an early error it cannot resolve and prematurely terminates the trajectory before consuming significant context. Finding 3: Higher token consumption does not equate to better task resolution; while some models get trapped in costly debug- ging loops with diminishing returns, models like GPT-5.4 and Kimi-k2.5 demonstrate superior cost-effectiveness by achieving top-tier performance with minimal token footprints. 4.4 RQ4: structural Autonomy and Diversity To answer RQ4, we investigate how different LLMs organize repos- itory structures and manage workspace complexity when granted full structural autonomy. Figure 5 compares the file count distribu- tions of agent-generated repositories against the human Oracle. Human Modularity vs. Agent Monolithic Preference. The evident trend is the divergence in structural design between human developers and autonomous agents. The human Oracle exhibits a broad distribution, reflecting standard software engineering prac- tices where code is modularized into distinct components. In con- trast, all evaluated LLMs demonstrate a strong preference for mono- lithic structures, with their medians tightly clustered between 1 and 3 files. This indicates a shared behavioral strategy among LLMs: centralizing logic into a single or very few files. From an agentic (a) File count distribution under OpenHands. (b) File count distribution under Mini-SWE-Agent. Figure 5: Distribution of total files generated by the human Oracle and various LLMs. The y-axis is truncated at 20 files. Extreme outliers are annotated with red stars and their max- imum values at the top of the violins. perspective, this monolithic approach seems to be a practical adap- tation to minimize cross-file dependency issues (e.g.,ImportError) and to keep the entire system state easily accessible within the model’s limited context window. Workspace Management and Debugging Behaviors. Be- yond the median values, the maximum file counts (annotated with red stars) reveal diverse workspace management behaviors during the generation process. Models like GPT-5.4 and Claude-Sonnet- 4.6 maintain strictly low file counts, indicating a behavior of edit- ing and overwriting existing files in place when fixing bugs. Con- versely, several other models (e.g., DeepSeek-V3.2, Qwen-3.5-plus, and MiniMax-M2.5) occasionally generate hundreds of files. Rather Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosConference acronym ’X, June 03–05, 2018, Woodstock, NY than implying lower functional capability, this extreme file sprawl reflects a different debugging pattern: when encountering execution errors, these agents tend to create numerous temporary scripts, du- plicate modules, or isolated test files instead of cleaning up or mod- ifying the original code. Furthermore, this behavior is influenced by the environment. For instance, GLM-5 and Kimi-k2.5 maintain compact workspaces under OpenHands (Figure 5 (a)) but exhibit great file sprawl under Mini-SWE-Agent (Figure 5 (b)), suggesting that the underlying framework’s prompt structure and feedback mechanisms impact agents’ file system management. Finding 4: While human developers naturally adopt modular structures, LLMs prefer monolithic designs to simplify context management. Besides, agents exhibit diverse debugging behav- iors, ranging from in-place editing to extensive file sprawl. 5 Discussion 5.1 Beyond Correctness: Runtime Efficiency Robustness to Invalid Inputs. A production-ready CLI tool must not only handle expected inputs but also gracefully manage invalid arguments or edge cases. To evaluate this, we compare the agents’ performance on positive (valid) versus negative (invalid/edge-case) test commands. Runtime Efficiency of Generated Code. To investigate the runtime efficiency of LLM-generated code beyond mere functional correctness, we conduct a controlled per- formance analysis. Specifically, we isolate a subset of 27 reposito- ries where all models successfully built and executed the CLI tools within the Mini-SWE-Agent framework. As presented in Table 4, we observe a counterintuitive phenomenon: the tools generated by all LLMs consistently outperformed the human-written Oracle in terms of raw execution speed. On average, the human Oracle re- quired 270.37 milliseconds per command. In stark contrast, models like GPT-5.4 and Qwen-3.5-plus achieved great runtime reductions, executing in roughly 126-127 milliseconds—effectively halving the execution time (0.47×relative slowdown). Even the relatively slower models in this subset, such as Kimi-k2.5 (0.88×), still maintained a distinct speed advantage over the human baseline. This unexpected efficiency may contribute to the inherent “mini- malism” of LLM-generated solutions compared to human engineer- ing practices. Human developers typically design CLI tools with robust non-functional requirements, incorporating extensive error handling, rich logging mechanisms, and user-friendly formatting (e.g., progress bars or colored terminal outputs). These engineering best practices, while crucial for maintainability and user experience, inevitably introduce runtime overhead. Conversely, LLM agents are highly goal-oriented; they tend to generate the most direct, stripped-down logic necessary to satisfy the immediate functional requirements or pass the provided test cases. They frequently by- pass defensive programming and decorative outputs, resulting in leaner, faster-executing binaries. This finding highlights a fascinat- ing trade-off in agentic software engineering: while agents may currently lack the holistic design foresight of human developers, their hyper-focused, minimalist coding style can inadvertently yield superior raw execution performance. Table 4: Runtime efficiency of the generated CLI tools. SourceAvg. Tool Runtime (ms)Relative Slowdown Human Oracle270.371.00× GPT-5.4126.140.47× Claude-Sonnet-4.6167.410.62× DeepSeek-V3.2201.440.75× Qwen-3.5-plus127.490.47× GLM-5138.850.51× MiniMax-M2.5142.780.53× Kimi-k2.5238.110.88× 5.2 Case Study To better understand the bottlenecks in autonomous software gen- eration, we conducted an in-depth qualitative analysis by manually inspecting a diverse set of failed agent trajectories. Aligning with our evaluation funnel (Section 3.2), we categorize the most preva- lent and representative failure modes into three distinct phases: Installation, Execution, and Behavioral Equivalence. Figure 6 illus- trates these typical failures. To gain deeper insights into the behavioral bottlenecks of LLM agents beyond quantitative metrics, we conducted a qualitative analysis of typical failure modes encountered during CLI tool de- velopment, as illustrated in Figure 6. Figure 6 (a) demonstrates a prevalent "build failure" driven by what we term a text-to-file bias. In this Go project scenario, the agent successfully generated the correct source code logic but attempted to resolve dependen- cies by manually hardcoding them into the “go.mod” file instead of executing standard package management commands like “go mod tidy”. Because LLMs are predominantly trained on static code repositories, they often default to direct text manipulation and lack the “environmental intuition” required to interact with dynamic, command-driven toolchains, ultimately leading to compilation er- rors due to missing dependencies. Beyond a lack of environmental intuition, agents also exhibit significant deficiencies in maintaining a coherent mental model of the workspace over long trajectories, leading to severe execution failures. Figure 6 (b) illustrates a catastrophic workspace misman- agement scenario where the agent, while attempting to build a Python tool, trapped itself in a recursive directory generation loop, creating deeply nested “build/lib/” structures. This chaotic file system state completely broke the package’s entry points, result- ing in a runtime “Command not found” (Exit Code: 127) error during evaluation. This case highlights a critical spatial blindness; when faced with unexpected build errors, the agent blindly retried commands or applied localized patches rather than systematically diagnosing and correcting the corrupted directory structure. Perhaps the most deceptive failure mode is the behavioral mis- match shown in Figure 6 (c), which perfectly illustrates the "illu- sion of success." In this scenario, the agent-generated “deletor” tool executed flawlessly without any runtime crashes, returning a successful Exit Code 0. However, our file system state evaluation revealed that, unlike the human oracle, which successfully removed the target files, the agent’s tool produced absolutely no system-level side effects. If an evaluation framework relied solely on execution status or standard output, this silent failure would be falsely re- warded as a success. This stark contrast underscores a fundamental principle in evaluating CLI tools: "running without crashing" does Conference acronym ’X, June 03–05, 2018, Woodstock, NYRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 module tview go 1.21 // Agent manually wrote dependencies require ( github.com/fatih/color v1.15.0 github.com/spf13/cobra v1.7.0 ) ... go.mod package main import ( "fmt" "os" "strings" "github.com/spf13/cobra" "tview/internal/tview" ) // ... valid logic ... main.go $ go install . main.go:8:2: missing go.sum entry for module providing package github.com/spf13/cobra (imported by tview); to add: go get tview internal/tview/tview.go:11:2: missing go.sum entry for module providing package github.com/fatih/color ... $ tree whattheflag_repo/ ├─ build/ │ └─ lib/ │ └─ build/ │ └─ lib/ │ └─ build/ │ └─ lib/ │ └─ build/ │ └─ lib/ │ └─ whattheflag/ │ ├─ data.py │ └─ __main__.py ├─ whattheflag/ ├─ pyproject.toml └─ README.md $ whattheflag tools -1 sh: 1: whattheflag: not found Exit Code: 127 $deletor -cli -d /test_data --older [duration] -skip-confirm Exit Code: 0 $deletor -cli -d /test_data --older [duration] -skip-confirm Exit Code: 0 ✅ [Oracle Execution] ❌ [Agent Generated] [File System Diff] -/test_data/old_file.tmp -/test_data/file_a.txt -/test_data/file_b.log (Files successfully deleted) [File System Diff] (No changes detected) (a) Build Failure. (b) Execution Failure.(c) Behavioral Mismatch. Figure 6: Representative failure modes of LLM agents in CLI tool development. not equate to task completion, firmly validating the necessity of our multi-dimensional evaluation approach that rigorously verifies actual file system state changes. 5.3 Implication For Researchers: Our findings emphasize the critical need to evolve evaluation methodologies and agent structures. The con- trast between exact match and semantic match scores proves that traditional string-based evaluations are insufficient for agentic tasks, necessitating multi-dimensional, state-aware benchmarks like CLI- Tool-Bench. Besides, the prevalent issues of "thrashing" (high token consumption without task progress) and catastrophic workspace mismanagement highlight a structural gap. Future research could focus on equipping agents with better spatial awareness of the file system, long-horizon planning, and self-reflection mechanisms to break out of unproductive debugging loops. For Developers: For software developers and practitioners, our study reveals actionable strategies for optimally collaborating with current LLM agents. Since agents universally prefer monolithic designs and struggle with cross-file dependency resolutions, de- velopers should provide explicit structural scaffolding or mandate the use of highly standardized CLI frameworks (e.g., Click or Co- bra) to guide the agent’s generation process. Additionally, while agent-generated tools often exhibit impressive raw execution speed due to their goal-oriented, minimalist coding style, they typically lack defensive programming practices. Therefore, developers must treat these outputs as functional prototypes, actively reviewing and reinforcing them with necessary error handling, robust logging, and security measures before production deployment. 5.4 Threats and Limitations First, our dataset focuses on three mainstream languages. Future iterations will expand to include heavily compiled languages. An- other threat arises from the inherent randomness of LLMs. Since we use LLMs during experiments, results may vary across trials. Hence, we conduct multiple runs and report the average results. 6 Related Work 6.1 Agents for Software Engineering The application of LLMs in software engineering has evolved from static code completion to autonomous, environment-interacting agents [3,4,20,27]. Early multi-agent frameworks, such as Chat- Dev [19] and MetaGPT [9], utilized role-playing and Standardized Operating Procedures (SOPs) to orchestrate software design via sim- ulated collaboration. To enable real-world repository interaction, Agent-Computer Interfaces (ACIs) were introduced. Frameworks like SWE-agent [26] and OpenHands [23] allow LLMs to interact directly with command-line terminals, execute tests, and iteratively debug within isolated sandboxes. Recently, minimalist scaffolds like Mini-SWE-agent [22] have demonstrated that state-of-the-art models require remarkably little scaffolding to achieve high success rates, relying primarily on their native reasoning capabilities. 6.2 Software Generation and Agent Evaluation As agentic capabilities advance, evaluation methodologies have shifted to complex systems. Early benchmarks like HumanEval [6] and MBPP [2] focused on static, function-level synthesis but suffer from data contamination and a lack of repository context. Subse- quent efforts addressed this through dynamic problem sourcing (LiveCodeBench [10]) or complex API integration (BigCodeBench [28]). For system-level tasks, SWE-bench [12] established the standard for software maintenance by evaluating patch generation for GitHub is- sues, while OSWorld [25] and Terminal-Bench [17] assess multi-step command execution in live operating systems. However, evaluating zero-to-one software generation remains a critical challenge. While NL2Repo-Bench [7] explores repository-level generation from nat- ural language, it relies heavily on human-written oracles. This rigid white-box approach severely penalizes functionally correct but structurally diverse solutions. CLI-Tool-Bench bridges these paradigms, extending black-box differential testing to the repository level to evaluate end-to-end software generation without imposing structural constraints. 7 Conclusion In this paper, we introduce CLI-Tool-Bench, a benchmark for end- to-end evaluating LLM agents on software generation. Extensive evaluations reveal that the highest overall success rate remains below 43%, indicating substantial room for future improvement in this challenging domain. Furthermore, our analysis uncovers several intriguing phenomena: agent performance exhibits a coun- terintuitive U-shaped trend with repository complexity, and higher Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool ScenariosConference acronym ’X, June 03–05, 2018, Woodstock, NY token consumption does not equate to better task resolution. Fi- nally, we observe that agents universally prefer monolithic designs over modular structures to simplify context management. References [1]Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2024. Advancing re- quirements engineering through generative ai: Assessing the role of llms. In Generative AI for Effective Software Development. Springer, 129–148. [2]Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732 [3]Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [4] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. CoRR abs/2406.01304 (2024). arXiv:2406.01304 doi:10.48550/ARXIV.2406.01304 [5]Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, Wei Ye, and Shikun Zhang. 2024. A Survey on Evaluating Large Language Models in Code Generation Tasks. CoRR abs/2408.16498 (2024). arXiv:2408.16498 doi:10.48550/ARXIV.2408.16498 [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shan- tanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Eval- uating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 [7] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xi- ang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyao Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. 2025. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. CoRR abs/2512.12730 (2025). arXiv:2512.12730 doi:10.48550/ARXIV.2512.12730 [8]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al.2024. A survey on llm-as-a-judge. The Innovation (2024). [9]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum? id=VtmBAGCN7o [10]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Rep- resentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https: //openreview.net/forum?id=chf JJYC3iL [11]Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. CoRR abs/2406.00515 (2024). arXiv:2406.00515 doi:10.48550/ARXIV.2406.00515 [12]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66 [13]Wen Li, Austin Marino, Haoran Yang, Na Meng, Li Li, and Haipeng Cai. 2024. How are multilingual systems constructed: Characterizing language use and selection in open-source multilingual software. ACM Transactions on Software Engineering and Methodology 33, 3 (2024), 1–46. [14]Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. CoRR abs/2409.02977 (2024). arXiv:2409.02977 doi:10. 48550/ARXIV.2409.02977 [15]Stephane H Maes. 2025. The gotchas of ai coding and vibe coding. it’s all about support and maintenance. OSF Preprints (2025). [16]Zhenyy Mao, Jialong Li, Dongming Jin, Munan Li, and Kenji Tei. 2024. Multi-role consensus through llms discussions for vulnerability detection. In 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE, 1318–1319. [17]Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868 (2026). [18]Christian Meske, Tobias Hermanns, Esther Von der Weiden, Kai-Uwe Loser, and Thorsten Berger. 2025. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda. IEEE Access 13 (2025), 213242–213259. [19]Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 15174–15186. doi:10.18653/V1/2024.ACL-LONG.810 [20]Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. 2024. AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 3003–3021. https://aclanthology.org/2024.acl-long.165 [21]Partha Pratim Ray. 2025. A review on vibe coding: Fundamentals, state-of-the-art, challenges and future directions. Authorea Preprints (2025). [22]SWE-Agent. [n. d.]. The 100 line AI agent that solves GitHub issues or helps you in your command line. https://github.com/SWE-agent. [23]Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=OJd3ayDDoF [24]Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Jihong Wang, Fengbin Yin, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2024. Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 4966–4974. [25]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). [26]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). [27]Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, 1592–1604. doi:10.1145/3650212.3680384 [28]Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen- Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, and et al. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.