Paper deep dive

RAT: RunAnyThing via Fully Automated Environment Configuration

Renhong Huang, Dongdong Hua, Yifei Sun, Sitao Ding, Hanyang Yuan, Daixin Wang, Yang Yang

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 79

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%

Last extracted: 6/21/2026, 6:24:08 AM

Summary

The paper introduces RAT (RunAnyThing), a language-agnostic, multi-stage framework designed to automate the configuration of executable environments for repository-level software engineering tasks. RAT utilizes an LLM-driven pipeline consisting of semantic initialization (ImageRetriever), dual-mode planning (Standard and Automated), a specialized toolset (e.g., file manipulation, error recovery, version switching), and a robust sandbox. To evaluate this framework, the authors propose RATBench, a large-scale benchmark containing over 2,000 heterogeneous GitHub repositories across Python, Java, Rust, and JavaScript/TypeScript. Experimental results show that RAT achieves a state-of-the-art Environment Setup Success Rate (ESSR), outperforming existing baselines by an average of 29.6%.

Entities (9)

ESSR · metric · 100%ImageRetriever · module · 100%Java · programming-language · 100%JavaScript/TypeScript · programming-language · 100%Language-Agnostic Abstraction · module · 100%Python · programming-language · 100%RATBench · benchmark · 100%RAT (RunAnyThing) · framework · 100%Rust · programming-language · 100%

Relation Signals (4)

RAT (RunAnyThing) → contains → ImageRetriever

confidence 100% · RAT features a multi-stage pipeline that integrates semantic initialization... ImageRetriever module conducts an LLM-based semantic analysis

RAT (RunAnyThing) → evaluatedon → RATBench

confidence 100% · Furthermore, to enable rigorous evaluation, we propose RATBench... Extensive experiments on RATBench demonstrate that RAT achieves a state-of-the-art...

RATBench → includes → Python

confidence 100% · RATBench encompasses four widely used languages (Python, Java, Rust, and JavaScript/TypeScript)

RAT (RunAnyThing) → uses → ESSR

confidence 100% · Extensive experiments demonstrate that RAT achieves state-of-the-art performance, improving the Environment Setup Success Rate (ESSR)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Automating repository-level software engineering tasks is a foundational challenge for autonomous code agents, largely due to the difficulty of configuring executable environments. However, manual configuration remains a labor-intensive bottleneck, necessitating a transition toward fully automated environment configuration. Existing approaches often rely on pre-defined artifacts or are restricted to specific programming languages, limiting their applicability to real-world repositories. In this paper, we first propose RAT (RunAnyThing), a language-agnostic framework for automated environment configuration on arbitrary repositories. RAT features a multi-stage pipeline that integrates semantic initialization, a planning mechanism, specialized toolset, and a robust sandbox for configuration. Furthermore, to enable rigorous evaluation, we propose RATBench, a benchmark that reflects the the distribution and heterogeneity of real-world repositories. Extensive experiments demonstrate that RAT achieves state-of-the-art performance, improving the Environment Setup Success Rate (ESSR) by an average of 29.6% over strong baselines.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

78,868 characters extracted from source content.

Expand or collapse full text

RAT: RunAnyThing via Fully Automated Environment Configuration Renhong Huang 1 2 Dongdong Hua 1 Yifei Sun 1 Sitao Ding 1 Hanyang Yuan 1 Daixin Wang 2 Yang Yang † 1 Abstract Automating repository-level software engineering tasks is a foundational challenge for autonomous code agents, largely due to the difficulty of con- figuring executable environments. However, man- ual configuration remains a labor-intensive bottle- neck, necessitating a transition toward fully au- tomated environment configuration. Existing ap- proaches often rely on pre-defined artifacts or are restricted to specific programming languages, lim- iting their applicability to real-world repositories. In this paper, we first propose RAT (RunAny- Thing), a language-agnostic framework for au- tomated environment configuration on arbitrary repositories. RAT features a multi-stage pipeline that integrates semantic initialization, a planning mechanism, specialized toolset, and a robust sand- box for configuration. Furthermore, to enable rigorous evaluation, we propose RATBench, a benchmark that reflects the the distribution and heterogeneity of real-world repositories. Exten- sive experiments demonstrate that RAT achieves state-of-the-art performance, improving the Envi- ronment Setup Success Rate (ESSR) by an aver- age of 29.6% over strong baselines. 1. Introduction The evolution of Large Language Models (LLMs) has shifted the frontier of autonomous programming from sim- ple snippet generation (Zhu et al., 2022; Bappon et al., 2024; Coignion et al., 2024) to complex, repository-level engi- neering (Zhang et al., 2023; Jimenez et al., 2023; Shrivas- tava et al., 2023; Wu et al., 2024a). However, unlike code snippets, repository-level tasks demand strict adherence to intricate inter-dependencies and environment-specific configurations. Without an executable environment, even logically correct code remains unverifiable and function- ally invalid. Consequently, environment configuration has 1 Department of Computer Science, Zhejiang University, Hangzhou, China 2 Ant Group, Hangzhou, China. Correspondence to: Yang Yang<yangya@zju.edu.cn>. Preprint. April 28, 2026. emerged as a key bottleneck in the development of au- tonomous agents (Hu et al., 2025). Moreover, environment configuration is not merely a matter of software convenience, but a fundamental requirement for the development of code LLMs (Li et al., 2023; Le et al., 2022; Luo et al., 2023). Generally, automated environ- ment configuration plays three critical roles: (1) Scalable Data Synthesis: It enables scalable benchmark construction by transforming static repositories into verifiable datasets, rather than relying on manually curated benchmarks such as SWE-bench (Jimenez et al., 2023). In addition, t helps synthesize accurate execution traces that are essential for LLM training (Da et al., 2025). (2) Execution-based Rein- forcement: It supports functional feedback loops, allowing reward models to move beyond static or heuristic signals (Le et al., 2022) toward true executability. (3) System Reliability: It ensures deployment reproducibility, overcoming the inher- ent brittleness of CI/CD integration. Overall, automated en- vironment configuration is key to transforming code agents from symbolic generators into reliable autonomous systems. Although several pioneering efforts have attempted to automate environment setup, they often impose con- straints that limit generalizability. For instance, INSTAL- LAMATIC (Milliken et al., 2025) and EXECUTIONA- GENT (Bouzenia & Pradel, 2025) rely heavily on pre- existing artifacts, such as curated Dockerfiles, installation contexts, or CI logs. While Repo2Run (Hu et al., 2025) introduces a dual-environment architecture to decouple con- figuration from monitoring, it still operates within a rigid framework that lacks the flexibility to handle diverse, uncu- rated repositories. Collectively, these methods are ill-suited for scaling to thousands of real-world repositories due to their dependence on specific prior knowledge and limited language support. At a fundamental level, the generalizabil- ity of environment configuration is constrained by informa- tion sparsity and intricate cross-component dependencies that these rule-based frameworks struggle to address. To overcome the limitations of existing approaches, we in- troduce RAT (RunAnyThing), a modular, language-agnostic framework for fully automated environment configuration. RAT employs an LLM-driven multi-stage pipeline starting with ImageRetriever, which semantically analyzes reposito- ries to select optimal base images and reduce configuration 1 arXiv:2604.23190v1 [cs.SE] 25 Apr 2026 RAT: RunAnyThing via Fully Automated Environment Configuration overhead. A dual-mode planning mechanism enables han- dling of complex or previously unseen repositories, while a specialized toolset and long-term expertise accumulation resolve execution ambiguities and support cross-session knowledge retention. Together, these components provide scalable, adaptive, and robust environment configuration for diverse repositories. Furthermore, to rigorously evaluate environment configu- ration methods under realistic repository settings, we in- troduce RATBench, a large-scale multilingual benchmark comprising over 2,000 GitHub repositories. Unlike existing datasets that are limited in language coverage or biased to- ward trivial projects, RATBench is constructed via stratified sampling to capture real-world diversity in project distri- bution, programming languages, and availability, and is validated through a rigorous executability-driven pipeline. Extensive experiments on RATBench demonstrate that RAT achieves a state-of-the-art Environment Setup Success Rate (ESSR) and exhibits environment configuration capabilities that surpass human experts. 2. Preliminary Configuration Artifacts.We refer to files that specify environment setup as configuration artifacts. Typical ar- tifacts include Dockerfiles, CI pipelines, build manifests (e.g.,package.json,pom.xml,Cargo.toml), lock- files (e.g.,poetry.lock), and explicit test scripts. When present, these artifacts partially or fully define the required runtime environment (e.g., language and library versions) and the verification sequence. When artifacts are absent or incomplete, both would be inferred from documentation (e.g., README) and repository structure. Environment Configuration. LetRdenote a repository comprising source code and metadata, and letE(R)denote the set of feasible containerized environments compatible withR. The environment configuration is to construct an environmente∈E(R)along with a verification sequence π(R), whereπ(R)specifies the repository-dependent exe- cution procedure (e.g., tests or build commands). A configuration is considered successful if the execution of π(R)withineterminates without error. The output of the task is either a runnable container image or a Dockerfile that deterministically builds such an image. Execution Trace. An environment configuration session in- duces an interaction traceτ(R) = ((a 1 ,o 1 ),..., (a T ,o T )) , where each actiona t is a tool invocation and each obser- vationo t is the resulting system feedback. The trace repre- sents the agent’s interaction history with the environment and serves as the basis for subsequent decision making. 3. Framework: RAT As shown in Figure 1, we introduce RAT (RunAnyThing), a modular, language-agnostic framework for the auto- mated configuration of complex software environments. To achieve these objectives, RAT employs a multi-stage pipeline that conducts LLM-driven agents within container- ized sandboxes, leveraging an extensive suite of specialized tools. We next describe the framework in detail. Language-Agnostic Abstraction.At the onset of en- vironment configuration, RAT first comprehends the en- tire project and identifies the corresponding programming languages. It encapsulates language-specific heuristics, including dependency patterns (e.g.,pom.xmlfor Java, Cargo.tomlfor Rust), package manager protocols, and specialized test runners. This unified interface allows the model to invoke language-specific toolsets through a unified protocol. As repositories often contain multiple program- ming languages, we currently identify the language with the largest proportion as the primary language. ImageRetriever for Initialization.Environment con- figuration can leverage existing environments to improve success rate, particularly for complex environments that share common architectural patterns with others. Given the repository, RAT first establishes the Language-Agnostic Abstraction module. Then, the ImageRetriever module con- ducts an LLM-based semantic analysis of the repository’s documents and configuration artifacts to determine the opti- mal executable environment, including language versions, OS variants, deep learning framework, critical dependen- cies, etc. Based on the analysis, the ImageRetriever module first infers a standard base image (e.g.,python:3.10, openjdk:17) from the predefined default base image set as a candidate. To further optimize the environment, particularly for projects with complex dependencies (e.g., PyTorch), the module generates search keywords to query Docker Hub for specialized pre-configured images. The module then em- ploys an LLM-based scoring mechanism to evaluate these candidates against the standard image, selecting the most suitable one for initialization. Finally, by leveraging the ImageRetriever module, RAT achieves robust initialization, which reduces certain configuration latency and improves overall success rates. Environment Configuration Planning. The agent’s plan- ning capability is fundamental to the environment config- uration process. RAT offers two distinct planning modes based on varying degrees of flexibility: •Standard Plan Mode: The standard mode utilizes a fixed exploratory workflow for environment configuration. The agent analyzes the repository structure and iteratively 2 RAT: RunAnyThing via Fully Automated Environment Configuration Configuration Artifacts Repository Owner/Repo Source Code PR Issue Input: Language-Agnostic Abstraction 1.Language: 2.Dependency patterns. 3.Package manager protocols. 4.Test runners. Unified interface ImageRetriever : Semantic analysis 1.Infer base image. 2.Generate search keys. “pytorch” “android” 3.Search dockerhub. 4.Score and select images for initialization. Specfic Toolset Environment Configuration Planning Standard Modes: Automated Modes: Read File Edit File View Outline Test Runners Change Version CI/CD Config Error Recovery Test Synthesis Issue Retrieval Fixed workflow for certain language. Tailored strategy with structured Plan.md Thought Observation Action Robust Sandbox Generation Long-term Expertise Accumulation Figure 1. The architecture of RAT (RunAnyThing). The framework consists of several primary modules: (1) Language-Agnostic Abstraction, which identifies project languages and encapsulates domain-specific protocols into a unified interface; (2) ImageRetriever, which performs semantic analysis of the repository to select optimal base images; (3) Agent Planning, featuring both a fixed Standard Plan Mode and an adaptive Automated Plan Mode using a structured document for long-term reasoning; and (4) Specialized Toolset, which provides integrated tools for file manipulation, error recovery, and verification within a robust sandbox. executes configuration commands via a terminal interface to resolve dependencies in a fixed manner corresponding to a certain programming language. •Automated Plan Mode: Recognizing that a fixed con- figuration process is often inadequate for diverse repos- itories, RAT introduces an Automated Mode for high- complexity tasks. Here, the agent autonomously gen- erates a project-specific plan and records progress in a structuredplan.md, serving as an external memory. By retrieving these notes, the agent preserves critical con- text across iterative installation phases, enabling robust, low-overhead management of complex build states. In addition, inspired by Repo2Run (Hu et al., 2025), we adopt the ReAct (Yao et al., 2022) framework forτ(R), which structures interaction into sequences of thoughts, actions (e.g., configuration commands), and observations. This paradigm is uniquely suited for environment configu- ration: The explicit “thought” process allows the agent to reason about the current state of the environment configura- tion, while the real-time “observations” provide immediate feedback from the terminal and ensures that the LLM re- mains synchronized with the evolving system state. Specialized Agent Toolset. To facilitate automated reposi- tory analysis and environment configuration, we developed a comprehensive toolset covering repository analysis, knowl- edge retrieval, environment setup, and validation. Unlike conventional approaches that rely on raw terminal com- mands, these tools are highly integrated and tailored for environment configuration. By abstracting each tool into an interface resembling a terminal command, we improve LLM context management and reduce token overhead through precise control of tool outputs and functionality. This de- sign makes the agent well-suited for complex, multi-step configurations. Specifications of all tools are provided in Appendix E. •Read File: Beyond a standardcatcommand, this tool enables LLM-powered semantic understanding of file’s purpose, dependencies, and key logic patterns, giving compact context for large files. •Edit File: Implements a GitHub-style diff mechanism to ensure precise modifications and supports line-range replacement, regex-based search-and-replace, as well as LLM-guided fuzzy matching. An automatic backup system prevents irreversible errors during iterative edits. •View Outline: Extracts function signatures, class defini- tions, and type annotations while filtering out implemen- tation noise. It supports various programming languages (Python, JS/TS, Rust, Java) via AST-based parsing with regex fallback. • Ls Structure: G senerates a filtered directory tree by pruning irrelevant artifacts, enabling the agent to focus on critical configuration entry points. • Issue Retrieval: When failures are project-specific, RAT queries an internal repository issue pool to reuse prior fixes. We first form a retrieval query from the observed error and an LLM-produced error synopsis, then rank candidates with a hybrid scorer that combines keywords and error types with an LLM reranker that judges semantic relevance and fix usefulness. •Change Version: This tool allows the agent to switch language versions within a container dynamically with- out losing the current workspace state, including in- stalled packages and temporary configurations. By lever- aging Docker commit to capture snapshots and enabling 3 RAT: RunAnyThing via Fully Automated Environment Configuration automated environment rollback, it provides a core util- ity for resolving version conflicts and syntax incompati- bilities. •Error Recovery: Environment configuration is inher- ently error-prone, so an effective error recovery mech- anism is vital for successful configuration. When en- countering failures (e.g.,ModuleNotFoundError), RAT leverages a multi-channel recovery solution: (1) utilizing the LLM’s intrinsic debugging capabilities (Ma- jdoub & Ben Charrada, 2024) to solve, (2) performing semantic search across the repository’s historical issues to identify project-specific solutions, and (3) querying external knowledge information, such as Stack Overflow, to obtain community-documented fixes. • Detect Environment: An automated environment in- spection tool designed to scan and report the configu- ration, capabilities, and available resources of the con- tainer. •CI/CD Config: A CI/CD configuration parser capable of automatically extracting environment configuration steps from GitHub Actions and converting them into executable commands for local containers. •Test Synthesis: For repositories lacking unit tests, RAT automatically generates lightweight smoke tests (Chauhan, 2014) to verify basic executability. It also supports entry-point scripts, validating deployment success through strategic timeouts. •Test Runners: This tool provides language-specific test runners (e.g.,runpytestfor Python) that au- tonomously discover and execute project tests. The runners apply specialized parsing strategies (e.g., JU- nit XML parsing and regex-based analysis) to extract results and categorize failure modes systematically. Robust Sandbox Generation.Based on initialization, RAT leverages a template-based Docker generation mech- anism. This module constructs a tailored Dockerfile that automates the installation of the required runtime, config- ures localized mirrors for robust network connectivity, and injects the RAT utility toolset into the container. To guaran- tee reliability, each environment undergoes pre-flight build validation before deploying the agent into the sandbox. Long-term Expertise Accumulation. As the adage goes, “Practice makes perfect”. Effective environment configura- tion is a knowledge-intensive process that scales with expo- sure to diverse repository structures. To formalize this, we introduce an automated mechanism for agents to synthesize expertise from historical execution trajectories. This accu- mulated experience is structured into a serialized schema (e.g., JSON), facilitating high-precision configuration. 4. Evaluation: RATBench Next, we evaluate the performance of environment con- figuration. While several configuration benchmarks have been proposed, existing benchmarks are insufficient to fully evaluate a method’s capacity for environment configura- tion. For instance, EnvBench (Eliseeva et al., 2025) relies on language-specific static metrics, such as missing import checks in Python or compilation checks in JVM, which often overlook complex runtime dependencies. Repo2Run (Hu et al., 2025) focuses on the validity of generated Dockerfiles rather than the actual execution of tasks, while benchmarks like Beyond Pip (Milliken et al., 2025) rely on small-scale, manually curated samples, a labor-intensive process that inherently limits scalability and prevents comprehensive evaluation across diverse repositories. To enable rigorous evaluation and overcome the above limi- tations, we introduce RATBench, a large-scale benchmark comprising over 2,000 GitHub repositories. Unlike exist- ing benchmarks that focus on limited languages or static settings, RATBench is curated to reflect the complexity of real-world software repositories through diversity in distri- bution, programming languages, and availability. Moreover, RATBench employs a rigorous construction pipeline de- signed to ensure functional validity. Detailed comparison of RATBench with existing benchmarks is provided in Table 1. Diversity in Distribution. Existing benchmarks often suf- fer from bias toward trivial or popular projects. To mitigate this, we employed a two-dimensional stratified sampling strategy, spanning across multiple tiers of project size (rang- ing from lightweight utilities to large-scale systems) and popularity (spanning long-tail projects to top-tier reposito- ries). This grid-based sampling ensures broad coverage of software complexity and prevents the evaluation from being dominated by simple or overly curated examples. Distribu- tion statistics are provided in Appendix C. Diversity in Programming Languages. RATBench en- compasses four widely used languages (Python, Java, Rust, and JavaScript/TypeScript) to capture diverse real-world environment failure modes. These languages span diverse interpreted and compiled toolchains, leverage different de- pendency managers, and present distinct configuration chal- lenges. Specifically: (1) Python requires runtime verifi- cation as import graphs and optional native dependencies are often resolved only during execution, with failures fre- quently caused by missing system libraries. (2) Java exhibits build-lifecycle complexities (e.g., Maven or Gradle) and fre- quent dependency conflicts within multi-module projects. (3) Rust provides strong compiler guarantees but imposes strict toolchain and linker constraints, especially on target triples and native library linking. (4) JavaScript/TypeScript combines rapid runtime evolution, transpilation overhead, 4 RAT: RunAnyThing via Fully Automated Environment Configuration Table 1. Benchmark comparison. # Repos: total number of repositories. Langs.: programming languages covered (P: Python, J: Java, K: Kotlin, R: Rust, JS/TS: JavaScript/TypeScript;). Stratified: whether repositories are sampled to balance repository size and popularity. Auto-Collect: whether repositories are mined from GitHub via automated collection. Exec-Verified: whether repository validity is assessed via by executing builds/tests rather than static analysis. Difficulty Levels: whether explicit difficulty levels are provided.✓/✗ indicate presence or absence of the feature. Benchmark# ReposLangs.StratifiedAuto-CollectExec-VerifiedDifficulty Levels RATBench2,000+P, J, R, JS/TS✓ EnvBench (Eliseeva et al., 2025)994P, J, K✗✓✗ Repo2Run (Hu et al., 2025)420P✗✓✗ ExecutionAgent (Bouzenia & Pradel, 2025) 5014 Langs✗✓✗ Beyond Pip (Milliken et al., 2025)40P✗✓✗ and native modules (e.g.,node-gyp), making configura- tions highly sensitive to Application Binary Interface (ABI) versions and lockfile consistency. Diversity in Availability. To capture varying degrees of environment ambiguity, RATBench categorizes repositories by the availability of (i) functional containerization artifacts and (i) the inclusion of unit tests. These regimes correspond to increasingly open-ended environment inference and are used within § 5. Automated Collection. We searched GitHub for reposito- ries active within the past year, applying a minimum thresh- old of 10 stars to filter out obsolete or low-quality projects. To guarantee executability, we applied language-specific heuristics: repositories were required to contain standard build manifests (e.g.,pom.xmlfor Java,Cargo.toml for Rust,package.jsonfor Node.js) or explicit test di- rectories (e.g.,tests/,test * .py ). For Python, reposi- tories with Dockerfiles or CI/CD configurations were priori- tized as they provide reliable environment ground truth. 5. Experiments In this section, we extensively evaluate the performance of RAT and baseline methods on RATBench, focusing on their capability and efficiency in handling complex configuration challenges in real-world repositories. Additionally, we com- pare performance across various backbones and evaluate RAT against human engineers. Further results, including failure analysis and case studies on execution trajectories, are provided in Appendix F. The code and dataset are avail- able athttps://anonymous.4open.science/r/ RunAnyThing_Anonymous. 5.1. Evaluation Metrics. Environment Setup Success Rate (ESSR). We evaluate the efficacy of environment configuration using the Envi- ronment Setup Success Rate (ESSR), which measures the fraction of successfully passed unit tests within a configured environment. For a repository withNunit tests, a naive def- inition would be:ESSR = N pass /N, whereN pass denotes the number of tests that pass successfully. However, to ac- count for the fact that real-world repositories often contain pre-existing bugs or broken tests, we specifically refine this metric for Python repositories asESSR = N pass /N verified to account for the inherent validity of ground-truth artifacts. We report ESSR under three scenarios: •S1 (Artifact-guided): Repositories provided with unit tests and functional containerization artifacts. Here, N verified is the total number of existing unit tests, which serve as the gold-standard baseline. •S2 (Artifact-free): Repositories containing unit tests but lacking containerization scripts. To isolate failures caused by misconfiguration, N verified excludes tests that fail due to inherent code defects, even when executed in a manually verified environment. •S3 (Test-deficient): Repositories lacking both pre- defined tests and scripts. In this underspecified set- ting, we constructN verified by identifying runnable entry points or synthesizing smoke tests, defining success by the correct execution of these ad-hoc verification targets. As for Java, Rust, and JS/TS repositories, we evaluate repositories with a deterministic build target. Success is defined by completing the corresponding build command without error (e.g.,mvn clean installorgradle clean buildfor Java,cargo buildfor Rust, and npm install or yarn install for JS/TS). Efficiency Metrics. In addition to assessing environment configuration effectiveness, we measure Latency (average execution time per repository) and Tokens (average token usage per repository) to quantify the practical computational overhead and deployment cost of each method. 5 RAT: RunAnyThing via Fully Automated Environment Configuration 5.2. Baselines. We evaluate our approach against five representative base- lines, grouped into three categories: static & prompt-based, software engineering agent and environment configuration agent: For static & prompt-based, we compare with (1) pipreqs 1 , a traditional static analysis tool that generates dependency files by scanning source code imports; (2) Zero- shot LLM, which generates configuration scripts directly from README files without environment feedback; For software engineering agent, we compare with (3) SWE- agent (Yang et al., 2024), a general-purpose software en- gineering agent that handles repository-level tasks via in- teractive shell commands; For environment configuration agent, we compare with (4) Installamatic (Milliken et al., 2025), a specialized agent for Python utilizing curated instal- lation contexts; (5) Repo2Run (Hu et al., 2025), a state-of- the-art agent that iteratively synthesizes Dockerfiles using an adaptive feedback loop and dual-environment execution. 5.3. Experimental Results Main Result.Table 2 reports the Environment Setup Success Rate (ESSR) across multiple programming lan- guages. The results show that RAT consistently outper- forms all baseline methods. On Python repositories, RAT achieves an ESSR of 63.2%, significantly surpassing the traditional static analysis tool pipreqs. Moreover, compared to general-purpose SWE-agent, RAT yields an average im- provement of 29.6% across all evaluated programming lan- guages. These results indicate that the environment con- figuration design of RAT is substantially more robust than general-purpose code agents when handling complex de- pendency structures. Furthermore, RAT consistently outper- forms specialized environment configuration agents, under- scoring the effectiveness of our approach and its advantage as a language-agnostic solution for automated environment configuration. Table 3 compares RAT across three various scenarios (S1–S3). The strong performance in S2 highlights RAT’s ability to leverage project files and documentation even if there is no containerization scripts. Furthermore, S3 re- sults indicate that RAT autonomously infers entry points and generates effective smoke tests without relying on bro- ken configurations, demonstrating robust deployment across diverse settings. Ablation Studies. To assess the contribution of each compo- nent in RAT, we perform ablation studies with the following variants: (1) RAT w/o init, without ImageRetriever for ini- tialization; (2) RAT w/o tool, without specialized toolset; and (3) RAT-auto, with the automatic plan mode instead of 1 Generate requirements.txt file for any project based on imports in https://github.com/bndr/pipreqs Table 2. Environment setup success rate (ESSR, %, higher is better) on RATBench across various programming languages. Bold: best performance in each column. ‘/’ denotes that the method is not applicable to the dataset. Model ConfigurationProgramming Languages FrameworkLLMPython Java Rust JS/TS Static & Prompt-based pipreqsNone35.8/// Zero-shot DeepSeek-V315.20.00.07.3 Software Engineering Agent SWE-agentDeepSeek-V315.529.3 56.751.8 Environment Configuration Agent InstallamaticDeepSeek-V36.7/// Repo2RunDeepSeek-V344.8/// RAT DeepSeek-V363.241.3 98.768.7 Table 3. Performance of RAT on Python repositories in RATBench across different scenarios with the DeepSeek-V3 model. Bold indicates best performance in each column. LevelsESSR (%)Tokens (K)Latency (min) S150.5451.341.6 S269.5455.259.4 S392.0122.214.4 standard plan mode. Detail implementation are included in Appendix D. Table 4. Ablation study of RAT. Performance is reported with the DeepSeek-V3 model on Python repositories. Bold indicates best performance. VariantESSR (%)Tokens (K)Latency (min) RAT w/o init40.5180.818.3 RAT w/o tool55.7351.236.9 RAT-auto40.5752.639.0 RAT63.2421.924.3 As shown in Table 4, removing the ImageRetriever (RAT w/o init) or the specialized toolset (RAT w/o tool) results in drop in ESSR (to 40.5% and 55.7%, respectively), in- dicating that high-quality initial images and precise tool execution are key drivers of success. Additionally, while RAT consumes the most tokens, it maintains a competitive latency of 24.3 minutes, significantly faster than RAT w/o tool variants, demonstrating that RAT achieves an effective balance between high success rates and execution efficiency. Regarding planning mode, the automatic plan mode shows a drop in performance compared to the standard mode, likely due to the additional budget consumed by dynamic plan 6 RAT: RunAnyThing via Fully Automated Environment Configuration adjustments. Nevertheless, it demonstrates potential for handling complex repository environment configuration. Performance under Different Backbones.We evalu- ate the performance of RAT across a range of backbone models. Based on the code capability rankings reported on LMArena (Chiang et al., 2024), we select several rep- resentative LLMs, including relatively weaker backbones such as Qwen3-Coder-30B (Yang et al., 2025), as well as stronger backbones such as Claude Sonnet 4.5 and GPT-5.2, as summarized in Table 5. Table 5. Performance of RAT under various combinations of back- bone models and frameworks. Due to cost constraints, results for stronger backbones are reported on a subset of Python repositories from RATBench. Details are provided in Appendix D. Bold indi- cates the best performance. Model ConfigurationMetrics Framework LLMESSR (%) Tokens (K) Weaker Backbones Repo2RunDeepSeek-V344.8 ̃400 RAT Qwen3-Coder-30B47.2355.6 DeepSeek-V363.2421.9 Stronger Backbones RAT DeepSeek-V376.6351.6 Claude Sonnet 4.581.3275.1 GPT-5.286.8280.2 As shown in Table 5, our framework consistently outper- forms Repo2Run even when equipped with a relatively smaller backbone (e.g., Qwen3-Coder-30B). Furthermore, the performance of RAT scales significantly when paired with more powerful models such as Claude 4.5 Sonnet and GPT-5.2, with the latter achieving the best results. This trend highlights the strong generalizability and robustness of our framework across various backbone models. Comparison with Human Engineers.To evaluate the performance of automated tools, we engaged several senior engineers to manually configure environments for three rep- resentative Python repositories. The experiment followed a standardized protocol: engineers first selected an appropri- ate Python base image guided by the repository’s README. They then cloned the target GitHub repositories and checked out specific commits. The primary goal was to successfully executepytest --collect-only -qand run inter- nal tests viapytest -q. We recorded the total latency and ESSR to quantify manual overhead. Additionally, each task was assessed on a 5 point Likert scale along two dimen- sions of cognitive workload: Difficulty and Effort. Detailed descriptions are provided in Appendix D. As shown in Table 6, although RAT is slower than human en- Table 6. Comparison on environment configuration between senior human engineers and RAT. ‘/’ denotes not applicable. GroupEfficiencyCognitive Load RoleESSR (%)Latency (min)DifficultyEffort Engineers89.4113.732.9 / 5.03.3 / 5 RAT91.5231.26// gineers in configuration time by a factor of 2.25, it achieves a 2.36% higher ESSR than the engineers. This demonstrates the effectiveness of our agent, which can even surpass en- gineers in environment configuration. Moreover, RAT re- quires no human intervention and supports parallelized envi- ronment setup across large-scale repositories, where manual effort would be costly. Thus, additional time overhead is acceptable given its efficiency and superior performance. Scaling Effects in Configuration. We further compare ESSR across varying execution steps. As shown in Figure 2, the consistent improvement in success rate as steps increase suggests a characteristic scaling law within the domain of environment configuration, mirroring the scaling laws ob- served in large-scale model reasoning (Wei et al., 2022; Wu et al., 2024b). This trend indicates that environment con- figuration is not merely a binary task but a search-based optimization process where increased computational budget (steps) consistently yields higher reliability. Furthermore, the growth of average latency decelerates, im- plying that the agent achieves a better balance between exploration and exploitation in later stages; Additional steps lead to more refined solutions without incurring a strictly linear increase in temporal overhead. 6. Related Work 6.1. Code agents. Autonomous software engineering has evolved from static generation to reasoning-driven agents capable of repository- level problem solving. Early works like MetaGPT (Hong et al., 2023) introduced SOP-based multi-agent collabora- tion, while CodeChain (Le et al., 2023) leveraged modu- lar self-revisions. Recent systems like SWE-agent (Yang et al., 2024) optimize the Agent-Computer Interface (ACI) for benchmarks like SWE-bench (Jimenez et al., 2023). Beyond prompt or workflow engineering, training-based frameworks such as Agent-RLVR (Da et al., 2025) employ environment-based rewards and pedagogical guidance to refine software engineering trajectories. However, despite their proficiency in patch generation, these agents typically assume pre-configured environments, leaving autonomous environment configuration largely unaddressed. 7 RAT: RunAnyThing via Fully Automated Environment Configuration 15202530 51 54 57 60 63 ESSR (%) 15202530 240K 320K 400K 480K Tokens 15202530 18.0 19.5 21.0 22.5 24.0 Latency (min) Figure 2. Performance across different execution steps as budget. As the number of steps increases, the ESSR improves significantly, as well as at the cost of average token consumption and average latency. 6.2. Environment configuration. The research focus for repository-level tasks has shifted from isolated code generation to the challenges of environ- ment configuration. While early benchmarks like SWE- Bench (Jimenez et al., 2023) identified real-world resolution difficulties, they were hindered by manual setup require- ments. Recent benchmarks such as EnvBench (Eliseeva et al., 2025), GitTaskBench (Ni et al., 2025), and systematic analyses of the Python ecosystem (Milliken et al., 2025) have addressed this by providing authentic workflows and ground-truth installation processes, establishing environ- ment setup as a cornerstone of autonomous software engi- neering. However, existing benchmarks only offer prelim- inary explorations of configuration and fail to reflect the representative distribution of real-world repositories. To tackle environment configuration, recent studies employ agentic strategies with iterative feedback. For instance, Repo2Run (Hu et al., 2025) and ExecutionAgent (Bouzenia & Pradel, 2025) employ LLM reasoning to synthesize Dock- erfiles and refine scripts across diverse languages based on execution outcomes. Furthermore, agents like SETU- PAGENT (Ni et al., 2025) automate benchmark construc- tion, enabling large-scale datasets. These advancements signify a transition from rule-based installation to dynamic, reasoning-driven agents capable of navigating complex software dependencies. Nevertheless, current methods re- main rely on configuration artifacts or are constrained by language-specific limitations, leaving them unable to handle generalized environment configurations as RAT does. 7. Limitation Although RAT can effectively construct executable environ- ments for many repositories, it still has limitations. First, our benchmark and agent pipeline assume a single-container setting, which simplifies isolation and evaluation but does not cover common multi-service deployments (e.g., via docker-compose) that depend on cross-container net- working, service readiness, shared volumes, and versioned sidecars (e.g., databases or message queues). These factors introduce additional configuration failures beyond the scope of our pipeline, and Docker-in-Docker scenarios further exacerbate this limitation. In addition, hardware-dependent environments remain largely out of scope. Many modern ML and systems work- loads rely on GPU drivers and runtime integration (e.g., NVIDIA Container Toolkit), requiring strict alignment be- tween host drivers, container runtimes, and CUDA/cuDNN libraries. Such dependencies introduce failure modes that are difficult to diagnose and reproduce. Extending envi- ronment configuration agents to hardware-aware container runtimes is an important direction for future work. Finally, environment configuration often requires external information provided by humans, such as API keys, ser- vice credentials, or proprietary datasets. A more intelligent agent should proactively interact with humans to acquire the necessary information and incorporate into the automated configuration. We leave deeper exploration of human-in- the-loop environment configuration to future work. Detailed error analysis of RAT can be found in the Appendix F. 8. Conclusion Environment configuration is a bottleneck for autonomous code agents. To address this problem, we introduce RAT (RunAnyThing), the first language-agnostic framework for automated repository-level environment configuration. By integrating semantic initialization with a dual-mode plan- ning mechanism and a specialized toolset, RAT effectively mitigates information sparsity and resolves complex de- pendencies. To enable comprehensive evaluation, we con- struct RATBench, a multilingual benchmark comprising over 2,000 real-world repositories. Experimental results demonstrate that RAT achieves state-of-the-art performance, substantially outperforming existing baselines and approach- ing the setup success rates of senior human engineers. Fu- ture work will focus on scaling RAT to a wider range of repositories and more complex deployment scenarios. 8 RAT: RunAnyThing via Fully Automated Environment Configuration Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References Bappon, S. D., Mondal, S., and Roy, B. Autogenics: Au- tomated generation of context-aware inline comments for code snippets on programming q&a sites using llm. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), p. 24–35. IEEE, 2024. Bouzenia, I. and Pradel, M. You name it, i run it: An llm agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering, 2(ISSTA):1054–1076, 2025. Chauhan, V. K. Smoke testing. Int. J. Sci. Res. Publ, 4(1): 2250–3153, 2014. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. Chatbot arena: An open platform for evaluat- ing llms by human preference, 2024. URL https://arxiv. org/abs/2403.04132, 2(10), 2024. Coignion, T., Quinton, C., and Rouvoy, R. A performance study of llm-generated code on leetcode. In Proceedings of the 28th international conference on evaluation and assessment in software engineering, p. 79–89, 2024. Da, J., Wang, C., Deng, X., Ma, Y., Barhate, N., and Hendryx, S. Agent-rlvr: Training software engineer- ing agents via guidance and environment rewards. arXiv preprint arXiv:2506.11425, 2025. Eliseeva, A., Kovrigin, A., Kholkin, I., Bogomolov, E., and Zharov, Y. Envbench: A benchmark for automated environment setup. arXiv preprint arXiv:2503.14443, 2025. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Meta programming for a multi-agent collabora- tive framework. In The twelfth international conference on learning representations, 2023. Hu, R., Peng, C., Wang, X., Xu, J., and Gao, C. Repo2run: Automated building executable environment for code repository at scale. arXiv preprint arXiv:2502.13681, 2025. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023. Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Ad- vances in Neural Information Processing Systems, 35: 21314–21328, 2022. Le, H., Chen, H., Saha, A., Gokul, A., Sahoo, D., and Joty, S. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv preprint arXiv:2310.08992, 2023. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. Majdoub, Y. and Ben Charrada, E. Debugging with open- source large language models: An evaluation. In Proceed- ings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, p. 510–516, 2024. Milliken, L., Kang, S., and Yoo, S. Beyond pip install: Eval- uating llm agents for the automated installation of python projects. In 2025 IEEE International Conference on Soft- ware Analysis, Evolution and Reengineering (SANER), p. 1–11. IEEE, 2025. Ni, Z., Wang, H., Zhang, S., Lu, S., He, Z., You, W., Tang, Z., Du, Y., Sun, B., Liu, H., et al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging.arXiv preprint arXiv:2508.18993, 2025. Shrivastava, D., Larochelle, H., and Tarlow, D. Repository- level prompt generation for large language models of code. In International Conference on Machine Learning, p. 31693–31715. PMLR, 2023. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Wu, D., Ahmad, W. U., Zhang, D., Ramanathan, M. K., and Ma, X. Repoformer: Selective retrieval for repository- level code completion. arXiv preprint arXiv:2403.10059, 2024a. 9 RAT: RunAnyThing via Fully Automated Environment Configuration Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024b. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. Repocoder: Repository-level code completion through iterative re- trieval and generation. arXiv preprint arXiv:2303.12570, 2023. Zhu, M., Suresh, K., and Reddy, C. K. Multilingual code snippets training for program translation. In Proceed- ings of the AAAI conference on artificial intelligence, volume 36, p. 11783–11790, 2022. 10 RAT: RunAnyThing via Fully Automated Environment Configuration A. Notations. The main notations are summarized in the following table. Table 7. Description of major notations. NotationDescription RA repository comprising source code and metadata. E(R)The set of feasible containerized environments compatible with reposi- tory R. e∈E(R)A specific environment instance constructed for the task. π(R)A repository-dependent verification sequence (e.g., tests or build com- mands). τ(R) An interaction trace defined as a sequence of action-observation pairs ((a 1 ,o 1 ),..., (a T ,o T )). a t ,o t The tool invocation (action) and resulting system feedback (observation) at time step t. NThe total number of unit tests available in a given repository. N pass The number of unit tests that pass successfully within the configured environment. N verified The number of verified ground-truth tests. B. Framework In this section, we present the pseudo code of the algorithm underlying RAT. Algorithm 1 Automated Environment Construction in RAT (Standard Plan Mode) Require: RepositoryR, Maximum turns T , Model LLM Ensure: Environment e∈E(R), Interaction trace τ(R), Dockerfile F 1: SetupAgent: 2:Extract configuration artifactsC ⊂R and identify primary languageL 3:ImageRetriever: 4:Perform semantic analysis onC and recommend base image I from the default image set 5:if LLM determines to search Docker Hub then 6:Query specialized image setI hub via Docker Hub 7:Execute LLM-based scoring and select best base image I ← score select(I,I hub ) 8:end if 9:Construct Dockerfile F 0 for e using a template based onL and I 10: Image Build and Validation: Validate e in a temporary context with fallback mechanisms 11: Environment Instantiation: Create container from e, inject specialized toolsetS tool (L) 12: for t = 1 to T do 13:ReAct Loop: 14:Thought: LLM reasons over repositoryR and trajectory τ(R) 15:Action: Select and invoke a t ∈S tool ∪ BASH 16:Observation: Capture system feedback o t (stdout, stderr, tool output) 17:Update trajectory τ(R)← τ(R)∪(a t ,o t ) 18:if configuration success or critical failure then 19:break 20:end if 21: end for 22: LLM infers Dockerfile F from F 0 ∪ τ(R) 23: return e,τ(R),F 11 RAT: RunAnyThing via Fully Automated Environment Configuration C. RATBench Details In this section, we report summary statistics of RATBench to characterize the diversity introduced by the benchmark construction procedure. All statistics and visualizations are computed on a balanced core split of 2,000 repositories (500 per language) spanning Python, Java, JavaScript/TypeScript, and Rust. Language Coverage and Verification. RATBench spans four programming languages, each with distinct environment configuration characteristics. For Java, Rust, and JS/TS, we rely on standard build manifests and verify setups via deterministic build or test commands. In contrast, Python environments are validated through unit tests, supplemented by Dockerfiles or CI scripts (if available), to ensure reliable verification. Table 8 summarizes these language-specific inclusion signals and their dominant failure modes. Table 8. Detailed language-specific characteristics, inclusion signals, and dominant failure modes in RATBench. FeaturePythonJavaRustJS/TS Required manifest– pom.xml / build.gradle Cargo.toml package.json Verification pytest mvn install / gradle build cargo build / cargo test npm install / npm test Unique feature Dynamic runtime dependencies Build lifecycle Compiler safety guarantees Transpilation Primary risk Missing system libraries Dependency version conflicts Linker or target mismatch Node.js or native module issues Repository Size Distribution. Repository size is measured by code bytes (excluding non-code assets such as images and documentation) and discretized into three tiers: Small (< 500KB), Medium (500 KB–5 MB), and Large (> 5MB). Figure 3 and Table 9 summarize size-tier compositions by language. The distributions differ substantially across ecosystems: JS/TS is dominated by small repositories, while Java contains a noticeably larger fraction of large repositories. This heterogeneity is important for environment configuration because large projects tend to introduce deeper build-tool stacks (e.g., multi-module builds and native toolchains), whereas smaller packages often stress dependency resolution and versioning behavior in language-specific package managers. PythonJavaJS/TSRust Programming Language 0 100 200 300 400 Number of Repositories Small (< 500 KB) Medium (500 KB - 5 MB) Large (> 5 MB) Figure 3. Repository size distribution across languages in RAT- Bench. Table 9. Repository size distribution by language on the 2,000- repository core split. LanguageSmallMediumLarge Python258 (51.6%) 190 (38.0%) 52 (10.4%) Java242 (48.4%) 178 (35.6%) 80 (16.0%) JS/TS393 (78.6%) 86 (17.2%)21 (4.2%) Rust257 (51.4%) 199 (39.8%) 44 (8.8%) Repository Popularity Distribution. Repository popularity is measured by GitHub stars. Figure 4 and Table 10 summarize the star distributions by language. Popularity is long-tailed (range: 11 to 155,569 stars), with a median 294 and a mean 2,868, consistent with typical open-source ecosystems. While stars are not a direct proxy for configuration complexity, including both long-tail and top-tier projects helps prevent benchmark bias toward either toy repositories (often under-documented) or 12 RAT: RunAnyThing via Fully Automated Environment Configuration highly engineered projects (often with mature CI/CD and containerization). PythonJavaJS/TSRust Programming Language 10 1 10 2 10 3 10 4 10 5 Number of Stars (log scale) Figure 4. Repository popularity (GitHub stars) distribution by language in RATBench. Table 10. GitHub stars summary statistics by language on the 2,000-repository core split. Language MinMaxMedian Mean Python11 155,5693333,163 Java1175,9372051,880 JS/TS11 137,2883163,564 Rust11 109,6323212,864 D. Additional Experimental Setup Implementation Details. We detail the implementation of evaluated models below. For RAT, we set the LLM temperature to 0.0 for reproducibility and limit each session to 30 turns for efficiency. The system employs a multi-layered timeout strategy (600s for commands; 7200s global) and allocates 150K token limit to support resource-intensive builds. Description of Baselines. To ensure fair and consistent evaluation, all baseline methods implement a unified interface that adapts to our evaluation framework. Specifically, each baseline takes a repository name as input and outputs a properly configured container environment. This standardized interface enables us to systematically run the corresponding test runners within each container to evaluate setup success across all methods. Below, we provide descriptions of the implementation of different baselines used in our experiments. •pipreqs. This baseline employs a static analysis approach and generating a container using predefined Dockerfile templates. It represents a deterministic, non-learning baseline that relies solely on manifest-based dependency resolution without dynamic adaptation. The Dockerfile template used is shown below: FROM python:3.10 WORKDIR / RUN pip install pytest pytest-xdist && \\ pip install pipdeptree COPY . /repo WORKDIR /repo # Ensure we use requirements.txt generated by pipreqs RUN if [ -f requirements_pipreqs.txt ]; \\ then pip install -r requirements_pipreqs.txt; fi # Try pytest collect to sanity-check the environment RUN pytest --collect-only -q || true •Zero-Shot. Unlike template-based methods, this baseline directly prompts an LLM to generate a complete Dockerfile 13 RAT: RunAnyThing via Fully Automated Environment Configuration from scratch based on repository analysis. It evaluates the model’s ability to perform environment configuration in a single-shot generation without iterative refinement or tool-assisted interaction. The prompt template is provided below: You are an expert DevOps engineer specializing in containerization. Your task is to generate a Dockerfile that can successfully setup a given repository. Requirements: 1. Use an appropriate base image for the programming language 2. Set up the correct working directory 3. Copy all necessary files 4. Install all dependencies 5. Set appropriate environment variables if needed Output ONLY the Dockerfile content without any explanation or markdown code blocks. Based on the following repository information, generate a production-ready Dockerfile: Repository: repo_name repo_context Generate a Dockerfile that: - Installs all required dependencies - Sets up all variables and configuration correctly - Uses best practices for the detected language/framework Output the Dockerfile content directly: •SWE-agent. We adapt SWE-agent to the environment configuration task by customizing its system prompts, agent workflow, and tool configurations to align with our evaluation framework. The complete configuration is provided below for reproducibility: agent: model: name: deepseek/deepseek-chat per_instance_call_limit: 30 templates: system_template: |- You are an expert DevOps engineer specialized in repository environment setup and configuration. ... instance_template: |- <uploaded_files> working_dir </uploaded_files> I have uploaded a Python repository in working_dir. <setup_requirements> problem_statement 14 RAT: RunAnyThing via Fully Automated Environment Configuration </setup_requirements> Your task is to configure this repository so it can run and pass all tests. ## Workflow 1. Explore Repository: ... 2. Analyze Dependencies: ... 3. Install Dependencies: ... 4. Verify Installation: ... 5. Fix Environment Issues: ... 6. Validate Setup: ... ## Code Modification Policy ... ## Important Notes - Focus on installing dependencies and configuring the environment - ... problem_statement_template: |- This is a language repository that requires environment setup and configuration. ## Objective Configure the repository to be ready for running tests successfully. ## Detailed Steps ### 1. Repository Analysis ... ### 2. Dependency Installation ... ### 3. Environment Configuration ... ### 4. Installation Verification ... ### 5. Troubleshooting ... ## Success Criteria The repository is considered properly configured when: - All dependencies are installed successfully - The environment is properly set up (config files, env vars, etc.) - ... ## Constraints ... 15 RAT: RunAnyThing via Fully Automated Environment Configuration next_step_template: |- Observation: observation next_step_no_output_template: |- Your command ran successfully, but produced no output. tools: env_variables: PAGER: cat MANPAGER: cat LESS: -R PIP_PROGRESS_BAR: ’off’ TQDM_DISABLE: ’1’ GIT_PAGER: cat bundles: - path: tools/registry - path: tools/edit_anthropic - path: tools/review_on_submit_m registry_variables: USE_FILEMAP: ’true’ SUBMIT_REVIEW_MESSAGES: - | ## Pre-Submission Checklist ... ## Review Your Changes ... ## Action Required ... enable_bash_tool: true parse_function: type: function_calling history_processors: - type: cache_control last_n_messages: 2 env: deployment: type: docker image: python:3.10-slim remove_container: false # Do not auto-remove container remove_images: false # Do not remove images repo: type: local path: ./repo •Installamatic. To ensure a consistent evaluation environment, we adapt the original Installamatic repository to run fully locally on Linux, removing the need for a virtual machine and enabling direct Docker-based evaluation. For LLM consistency, we re-implement the LLM inference interface to support the DeepSeek-V3 API. Key changes include local Docker execution, API modification, and minor initialization updates. • Repo2Run. Repo2Run follows an agent-based framework architecturally similar to our evaluation setup. We apply minimal workarounds to adapt its interface to our benchmark, enabling direct integration with our standardized evaluation pipeline without major structural modifications. 16 RAT: RunAnyThing via Fully Automated Environment Configuration Settings for ablation studies. Here, we elaborate in detail on how each ablation study is conducted: •RAT w/o init. The ImageRetriever module is deactivated. Instead of performing semantic analysis to infer and retrieve the most suitable Docker image from Docker Hub, this variant initializes the sandbox with a fixed, default base image corresponding to the identified primary programming language (e.g.,python:3.10-slimfor Python). This variant verifies the necessity of retrieving project-specific runtime environments for robust initialization. •RAT w/o tool. The specialized agent toolset is disabled. The agent is restricted to interacting with the environment solely through basic shell commands (e.g.,grep,sed,cat, andecho) and essential tools (e.g.,STOP), lacking access to the high-level capabilities such as web search or issue retrieval. This variant verifies the contribution of the specialized tool abstractions to the configuration precision and efficiency. • RAT-auto. This variant activates the automated plan mode. Unlike the standard plan mode, the agent in this setting is required to generate and continuously update an external plan (i.e., aplan.mdfile). This enables the agent to decompose complex configuration procedures into manageable steps and track execution state. Settings for Table 5. To ensure a controlled backbone comparison, RAT and Repo2Run use the same hyper-parameters: each session is capped at 30 turns. We also set the context budget to 150K tokens. We evaluate weaker backbones on 150 Python repositories, while stronger backbones are evaluated on a smaller 30 Python repositories subset due to cost considerations. Notably, Repo2Run does not expose a token-accounting interface, so we cannot report its exact token consumption. E. Specifications of Tools RAT supports a suite of specialized tools within the RAT framework. Table 11 outlines the comprehensive inventory of these tools and their functional specifications. Table 11. Functional specifications of the tools provided in RAT. Tool NameFunctional Description Repository Analysis constructtestScans the repository to identify entry points, extracts run commands from READMEs, and locates test modules. The output includes entry point, run commands, and test information. ls structureDisplays the repository directory tree with a configurable depth. Highlights important files such as README, setup.py, and Dockerfile. view outlineExtracts code outlines including classes and function signatures for a given file or directory. Supports recursive scanning and optional line numbers. read filePerforms file reading with optional LLM-guided analysis. Provides a deeper understanding of key components and dependencies compared to standard cat. Knowledge Retrieval search repoConducts global code snippet searches. Supports multiple modes (detailed/simple/LLM) and provides paths, line numbers, and brief contextual notes. search webQueries external sources like StackOverflow, GitHub, and official documentation for error resolution or general “how-to” guidance. retrieve imageInfers project requirements (e.g., package.json) to search Docker Hub and recommend relevant images and tags with pull commands. retrieveissueSearches an issue database for solutions to specific error messages (e.g., ModuleNotFoundError) and suggests potential fixes. (Continued on next page) 17 RAT: RunAnyThing via Fully Automated Environment Configuration Table 11. Functional specifications of the tools provided in RAT. (Continued) Tool NameFunctional Description Environment Setup editfileModifies file content using various modes (replace/insert/search/LLM). Includes regex support and automatic .bak backup creation. detectenvironmentReports basic system information, including GPU availability, system OS, network status, and available mirrors or tools. cicd configAnalyzes GitHub CI/CD workflows (.yml) to generate setup scripts and command lists for environment replication. changepythonversionSwitches the container’s Python version. Note: This action resets the environment and discards all previously installed packages. changejavaversionSwitches the container’s Java version (e.g., 11, 17, 21). Note: Similar to Python versioning, this resets the current environment state. stopTerminates the environment setup flow and ensures the current state is saved and logged. Validation runtestExecutes tests based on results from constructtest. Supports different execution types, including test, run, and collect. runpytestAutomatically runs all pytest tests in the repository and categorizes errors (e.g., ImportError). Saves results to a structured JSON log. runpytestcollectCollects available pytest tests without executing them to detect import-time errors and count test cases. Settings for Comparison with Human Engineers. To quantify the manual overhead beyond time and success rates, we adopted a subjective assessment framework. Upon completion of each environment configuration task, senior engineers were required to rate their experience based on two cognitive dimensions: • Difficulty: Measures the technical complexity and the presence of obstacles (e.g., dependency conflicts, vague documen- tation) encountered during the setup. •Effort: Measures the mental and physical energy required to complete the task, reflecting the intensity of the engineer’s involvement. The evaluation utilized a 5-point Likert scale as Table 12, ranging from 1 (Very Low/Easy) to 5 (Very High/Difficult). This dual-metric approach allows us to distinguish between tasks that are technically complex but routine (High Difficulty, Moderate Effort) and those that are tedious and draining (Moderate Difficulty, High Effort). Table 12. The 5-point Likert scale for Difficulty and Effort assessment. ScoreLevelDifficulty (Technical)Effort (Cognitive) 1Very LowStraightforward; follows README perfectly.Minimal mental energy required. 2LowMinor tweaks or version adjustments needed.Slight focus; routine operations. 3ModerateRequires external search or troubleshooting.Sustained attention; moderate fatigue. 4HighMajor conflicts; requires deep debugging.High mental strain; multiple attempts. 5Very HighSevere blockers; requires manual code fix.Exhausting; requires extreme persistence. Inaddition,therepositorieswetestedincludeprojectsfromdifferentdomains,suchas opengeos/segment-geospatial,dsphper/lanhu-mcp,andpython-escpos/python-escpos. We then report the averages over these repositories. 18 RAT: RunAnyThing via Fully Automated Environment Configuration F. Additional Experimental Results Distributions of Tokens, Latency, and Pass Rates. Figure 5 shows the distributions of token consumption, model latency, and pass rates across the evaluated repositories. Token consumption exhibits an approximately normal distribution. Model latency displays a strong right-skewed distribution. The pass rate distribution is distinctly bimodal, with most repositories achieving either 0% or 100% pass rates. Figure 6 shows the correlation between token consumption and model latency. The Pearson correlation coefficientr = 0.618demonstrates a significant positive correlation between the two variables, indicating that higher token usage generally leads to longer processing delays. 0.000.250.500.751.00 Tokens (×10 6 ) 0 5 10 15 20 25 30 35 Count 01000200030004000 Latency (s) 0 10 20 30 40 Count 0.000.250.500.751.00 Pass Rate 0 10 20 30 40 50 60 70 Count Figure 5. Distributions of tokens, latency, and pass rates across repositories. 0.00.20.40.60.81.0 Tokens (×10 6 ) 0 1000 2000 3000 4000 Latency (s) Pearson r=0.618 Figure 6. Correlation between token consumption and model latency. Case Study on Trajectories. In this section, we take stlehmann/Flask-MQTT as an example repository to illustrate the different trajectory between RAT and Repo2run as shown in Figure 7. Compared to Repo2Run, our agent explicitly aligns its configuration strategy with the repository’s CI workflow and runtime requirements. By inspecting CI scripts, our method correctly identifies system-level service dependencies (e.g., Mosquitto brokers) that are invisible to Python- centric dependency analysis. This enables our agent to provision the execution environment holistically before test execution, whereas Repo2Run repeatedly attempts to resolve failures through requirement-level manipulations, leading to non-convergent behavior. Tool Distribution. As shown in Figure 8, RAT frequently invokesrun-pytest-collectto construct test programs for determining task completion. File-related tools such asread-file,view-outline, andls-structureare also commonly used. In contrast, tools associated with error recovery, such assearch-web,retrieve-issue, 19 RAT: RunAnyThing via Fully Automated Environment Configuration ls-structure --repo /repo --depth 3 read-file README.md read-file setup.py read-file requirements.txt read-file requirements/ development.txt ls .github/workflows read-file .github/workflows/ci.yml 1.Repository Understanding RATRepo2run 1.Repository Understanding Find and figure out the CI file. ls -la /repo cd /repo && cat setup.py cat requirements.txt ls requirements/ ls -la /repo | grep -E ... cat /repo/setup.cfg Did not understand and find CI file. 2.Dependency installation apt-get install -y mosquitto pip install -r requirements.txt pip install pytest coverage pytest-cov mosquitto -d mosquitto -p 1885 -d mosquitto -p 1886 -d 2.Dependency installation grep requirements.txt sed requirements.txt requirements_clean.txt pip install -e . waitinglist addfile ... grep ... sed ... pip install -e . Rightly saw pip wasn't the fix. Blindly fixing requirements. 3.Evaluation mosquitto -d ... run-pytest-collect run-pytest Collect first, then test. 3.Evaluation run-test Poor error recovery capabilities. Fail to collect before test. SuccessFailure Figure 7. Trajectory comparison between RAT and Repo2Run on repository stlehmann/Flask-MQTT. andchange-python-version, are invoked less often, since most issues can be resolved using the LLM’s intrinsic reasoning and debugging capabilities. Overall, the effective utilization of these tools demonstrates the soundness and rationality of our tool design. 0.000.050.100.150.200.25 retrieve-issue search-web detect-environment search-repo change-python-version cicd-config run-test edit-file ls-structure construct-test stop view-outline run-pytest read-file run-pytest-collect 0.2% 0.3% 0.5% 1.0% 1.2% 1.7% 2.0% 4.4% 4.7% 5.8% 9.2% 10.6% 11.9% 21.1% 25.1% Figure 8. Tool calls distribution of RAT across Python repositories in RATBench. Failure Analysis. Figure 9 summarizes the dominant error categories among Python repositories where RAT fails to complete verification. We observe two recurring failure modes. (1) ConnectionError typically occurs in API-reliant repositories when tests fail to reach external services (e.g., APIs, databases, or brokers). In our sandbox, these errors stem from restricted network access, missing credentials, or unprovisioned local services. (2) RuntimeError is a catch-all for execution-time crashes after installation succeeds. Typical causes include missing system-level libraries (e.g.,libGL, tk), binary wheels incompatible with the selected Python/OS ABI, and hardware/driver assumptions (e.g., CUDA-enabled packages executed on CPU-only machines). These failures are harder to repair automatically because they often require non-Python OS packages, platform-specific pinning, or test refactoring to remove unavailable resources. 20 RAT: RunAnyThing via Fully Automated Environment Configuration 0.000.050.100.150.200.25 SyntaxError TimeoutError IndexError AttributeError TypeError NameError AssertionError KeyError OSError ModuleNotFoundError ImportError FileNotFoundError OtherError RuntimeError ConnectionError 0.4% 0.4% 0.4% 0.9% 0.9% 1.2% 1.5% 1.8% 5.5% 7.5% 9.8% 11.0% 11.0% 22.9% 24.8% Figure 9. Breakdown of pytest error types for Python repositories where RAT fails to solve. Action Call Illustration. The Action Call Example for repositoryabrignoni/aleappas shown in Table 13, highlights the agent’s coherence and operational accuracy. It begins with systematic environment awareness by inspecting the repository structure and configuration files, then adapts dynamically (e.g., switching Python versions) to resolve compatibility issues. The end-to-end execution, from setup to validation, achieves a success rate of over 27 steps, demonstrating robust handling of complex, multi-stage software engineering workflows. Table 13. Command Execution Statistics Table. No.Action (Command/Tool)Duration (s) 1ls-structure –repo /repo –depth 31.76 2read-file /repo/README.md2.07 3read-file /repo/requirements.txt2.02 4construct-test –repo /repo22.97 5pip install -q pytest openai -i https://mirrors.aliyun...1.96 6pip install -q -r /repo/requirements.txt -i https://mi...16.54 7run-pytest-collect1.68 8cd /repo1.68 9python aleapp.py –help1.78 10python –version1.69 11change-python-version 3.1012.77 12pip install -q pytest openai -i https://mirrors.aliyun...2.10 13pip install -q -r /repo/requirements.txt -i https://mi...11.10 14pip install -q packaging≥22 -i https://mirrors.aliyun...2.00 15cd /repo1.68 16python aleapp.py –help2.48 17run-pytest-collect1.64 18run-pytest1.49 19find /repo -type f -name "*test*.py" -o -name "*spec*....1.69 20cd /repo1.69 21python aleapp.py -p2.13 22cd /repo1.70 23python -c "import tkinter; print(’tkinter available’)"...1.76 (Continued on next page) 21 RAT: RunAnyThing via Fully Automated Environment Configuration Table 13. Command Execution Statistics Table. (Continued) No.Action (Command/Tool)Duration (s) 24cd /repo1.70 25python -c "import aleappGUI; print(’aleappGUI imports ...1.91 26detect-environment –format text2.50 27stop0.00 Cost for each repository. We report the per-repository cost of running RAT with DeepSeek-V3 on a Python environment setup. Under the standard budget of at most 30 turns per repository, the agent achieves 63.2% ESSR with an average cost of $0.30 per repository. In practical deployments, response caching further reduces repeated-token usage across turns, making the effective cost even lower. Overall, this cost level is within an acceptable range for large-scale benchmark construction and routine use. G. Broader Impacts RAT is expected to play an important role in multiple areas in the future, with broader impacts: (1) Scalable Data Synthesis: By automatically transforming static repositories into verifiable, runnable datasets, the Agent enables large-scale benchmark construction and generates precise execution traces, supporting post-training LLM development. (2) Execution-based Reinforcement: Automated setup allows agents to leverage true functional feedback, moving reward models beyond static or heuristic signals toward execution-aware learning. (3) Executable Analysis: RAT facilitates systematic analysis of code executability. Moreover, in our future plans, given a functional requirement (e.g., “implement real-time translation”), RAT can bypass manual coding by identifying stable repositories, automatically configuring environments, resolving dependencies, extracting essential logic, and integrating across projects. It continuously enables one-click deployment of fully functional, ready-to- run environments. This end-to-end automation accelerates experimentation, ensures reproducible execution, and supports scalable evaluation in both interactive and headless SaaS settings. 22 RAT: RunAnyThing via Fully Automated Environment Configuration H. Prompts Prompt for ImageRetriever in Initialization. Image Retrieving You are a Docker image selection expert. Infer the best Docker base image from the information below. ## Allowed versions (ALLOWED_VERSIONS.json) ‘json json.dumps(allowed_versions, ensure_ascii=False, indent=2) ‘ ## Repository config files config_str if config_str else "(No config files found)" ## README readme_content if readme_content else "(No README found)" ## Requirements 1. ** Identify the primary language ** (...) 2. ** Analyze version requirements ** : ... 3. ** Select the best image ** : - Prefer selecting language/version/variant from ALLOWED_VERSIONS.json - If the project requires a specific version, pick the closest allowed version - If version is unclear, use default_version - For variant (e.g. slim, alpine), use default_variant by default 4. ** Extract dependency info ** (optional): key frameworks and dependencies ## Output format (raw JSON only; no Markdown) "language": ..., "version": ..., "variant": ..., "base_image": ..., "full_image": ..., "reason": ..., "confidence": ..., "frameworks": ..., "dependencies": ... ## Important rules 1. ** Version constraint ** : ... 2. ** Variant constraint ** : ... 3. ... ## Example output ... 23 RAT: RunAnyThing via Fully Automated Environment Configuration Prompt for Standard Plan Mode. Standard Plan Mode You are an expert in environment setup. You may refer to files and structures in the repository such as requirements.txt, setup.py, etc., and use dependency inference tools like pipreqs to install third-party libraries inside the specified Docker image. This ensures the repository can be set up successfully and the specified tests can run. Note: This repository originally has no Dockerfile, or an existing Dockerfile has been removed. Do not attempt to use the repository’s original Dockerfile information. Workflow: 0. Explore the repository to understand its full structure and the image environment. 1. Find the program entry point; create a usable test case; check whether tests pass without extra setup. 2. Read key documentation, including ... 3. Inspect repository directories and read environment-related files ... 4. Collect dependency lists: find dependency files in the repo root ... 5. Install dependencies based on collected files... 6. Check whether tests pass; if so, call stop. CLI tool instructions: All operations run inside Docker container image_name Think about what to do next, then wrap commands with ... Note: Do not make large changes to /repo; only necessary adjustments. Keep commands on a single line when possible using &&. Avoid multi-line commands, backslash continuations, and HERE-DOCs (<<). Available tools (callable tools, not built-in terminal commands): tools_list Important: ... Special note: ... Prompt for Automated Plan Mode. Automated Plan Mode You are an expert in environment setup. You must configure the environment according to a custom plan file ‘plan.md‘. You may use ‘/repo/plan.md‘ as your task plan (it does not exist initially). This file contains the environment setup goal, phases, and progress. The primary language of the repository is language, so configure the environment accordingly. ## Two-phase workflow ### Phase 1: Check and create the plan (if missing) 24 RAT: RunAnyThing via Fully Automated Environment Configuration 1. First check whether ‘/repo/plan.md‘ exists. 2. If it does not exist, you should: - Explore repository structure and understand the project - Create an initial ‘plan.md‘ based on your analysis (make steps concrete; avoid too many generic steps) - Use the ‘edit-file‘ tool to create the file; use the template below: ‘markdown # Task Plan: Environment Configuration for image_name ## Goal [Goal of environment setup, e.g. configure image_name environment so tests pass. You can execute at most max_turn steps, so keep the plan simple.] ### Phase 1: Repository Analysis  - [ ] Explore repository structure - [ ] Identify main entry point - [ ] Read README and documentation - [ ] Find dependency files (requirements.txt, pyproject.toml, etc.) - ** Status ** : pending ### Phase 2: Dependency Installation  - [ ] Install system dependencies (if any) - [ ] Handle version conflicts (if any) - [ ] Install development dependencies - ** Status ** : pending ### Phase 3: Environment Configuration  - [ ] Configure environment variables - [ ] Set up database (if needed) - [ ] Set up paths and permissions - ** Status ** : pending ### Phase 4: Testing & Validation  - [ ] Run basic tests - ** Status ** : pending ## Current Phase Phase 1 ## Notes [Any important notes] ‘ 3. If the file already exists, read and understand the existing plan. ### Phase 2: Execute the plan 1. Read the current plan and understand the current phase and goal. 2. Find the current phase (an unchecked ‘[ ]‘ phase). 25 RAT: RunAnyThing via Fully Automated Environment Configuration 3. Execute tasks in the current phase: - If the phase status is not marked, update it to ‘in_progress‘ using ‘edit-file‘ - Execute tasks (explore/install/configure/test) - After finishing the phase, update ‘[ ]‘ to ‘[x]‘ and set status to ‘complete‘ 4. Continue to the next phase until all phases are complete ## Important rules ... ## Response format requirements ... ** Example: ** ... ## Initial action guide ... Available tools: tools_list Prompt for Calling Different Actions. Action Calling You are an expert proficient in testing. The current environment has been built using the repository’s Dockerfile (image: image_name), and the environment configuration should be complete. Your main tasks are: 1. Verify that the environment is working correctly 2. Create test cases 3. Run tests and ensure they pass Workflow: 1. Quick environment verification: Check if key commands and dependencies are available 2. Use the construct-test tool to create test cases 3. Run tests (run-pytest-collect and run-pytest) 4. ... CLI Tool Usage Instructions: All operations are performed inside Docker container image_name ... for example: ### Thought: ... ### Action: BASH_FENCE[0] ... BASH_FENCE[1] 26 RAT: RunAnyThing via Fully Automated Environment Configuration Available tools (callable but not terminal built-in commands): tools_list Important Notes: 1. Environment has been built via repository Dockerfile, most dependencies should already be installed 2. ... Special Note: ... 27

Instant research discovery