Paper deep dive
AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najma Imtiaz Ali, Shaban Usman
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/18/2026, 1:23:15 AM
Summary
AgentForge is a multi-agent framework for autonomous software engineering that introduces execution-grounded verification as a core principle. By utilizing a pipeline of five specialized agents (Planner, Coder, Tester, Debugger, and Critic) and a mandatory Docker-based sandbox, the framework ensures that all code changes are verified against real execution feedback rather than relying on simulated outcomes. It achieves 40.0% resolution on SWE-bench Lite, demonstrating that structured role decomposition and grounded feedback significantly outperform single-agent baselines.
Entities (5)
Relation Signals (3)
AgentForge → employs → Planner
confidence 100% · AgentForge decomposes bug fixing into five specialized agents: Planner, Coder, Tester, Debugger, and Critic.
AgentForge → evaluatedon → SWE-bench Lite
confidence 100% · AgentForge achieves 40.0% resolution on SWE-bench Lite
AgentForge → uses → Docker
confidence 100% · The system executes every generated patch inside a resource-constrained, network-isolated Docker sandbox.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models generate plausible code but cannot verify correctness. Existing multi-agent systems simulate execution or leave verification optional. We introduce execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. We instantiate this principle in AGENTFORGE, a multi-agent framework where Planner, Coder, Tester, Debugger, and Critic agents coordinate through shared memory and a mandatory Docker sandbox. We formalize software engineering with LLMs as an iterative decision process over repository states, where execution feedback provides a stronger supervision signal than next-token likelihood. AGENTFORGE achieves 40.0\% resolution on SWE-BENCH Lite, outperforming single-agent baselines by 26--28 points. Ablations confirm that execution feedback and role decomposition each independently drive performance. The framework is open-source at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.13120v1
- Canonical: https://arxiv.org/abs/2604.13120v1
Trouble viewing inline? Open PDF directly →
Full Text
44,658 characters extracted from source content.
Expand or collapse full text
AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najma Imtiaz Ali, Shaban Usman R. Kumar with International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou, 311115, China(e-mail: rajakumarlohano@gmail.comW. Ali is with the Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China (e-mail: waqar.uestc@yahoo.com).Fakulti Teknologi Maklumat dan Komunikasi, Universiti Teknikal Malaysia Melaka, Melaka 76100, Malaysia (e-mail: j.bhatti@iba-suk.edu.pk).Computer Systems Engineering Department, Sukkur IBA University, Sindh , Pakistan (e-mail: najma@utem.edu.my).Yibin Park of University of Electronic Science and Technology of China, Yibin 644000, China (e-mail: shabanusman@yahoo.com). Abstract Large language models generate plausible code but cannot verify correctness. Existing multi-agent systems simulate execution or leave verification optional. We introduce execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. We instantiate this principle in AgentForge, a multi-agent framework where Planner, Coder, Tester, Debugger, and Critic agents coordinate through shared memory and a mandatory Docker sandbox. We formalize software engineering with LLMs as an iterative decision process over repository states, where execution feedback provides a stronger supervision signal than next-token likelihood. AgentForge achieves 40.0% resolution on SWE-bench Lite, outperforming single-agent baselines by 26–28 points. Ablations confirm that execution feedback and role decomposition each independently drive performance. The framework is open-source at https://github.com/raja21068/AutoCodeAI. I Introduction Large language models (LLMs) perform well on code generation but remain unreliable for real-world software engineering. Practical tasks require reasoning over existing codebases, executing programs, and iteratively refining solutions under test feedback. Most current systems treat code generation as a single-step prediction problem, mapping a natural language description to a code completion [3]. This approach fails on tasks that require multi-file reasoning, test generation, and regression avoidance [4]. Recent work addresses these limitations with multi-agent systems that decompose development into roles such as planning, coding, testing, and reviewing. These systems improve performance on benchmarks such as SWE-bench through structured interaction between agents [28]. Despite this progress, a key limitation remains: existing frameworks do not enforce grounded execution. They infer execution outcomes or rely on permissive environments, rather than observing actual program behavior. This limitation is critical. Bug fixing requires a feedback loop plan, implement, execute, test, and revise driven by real execution signals. Without grounded feedback, models cannot reliably verify correctness or detect regressions. Simulated execution introduces systematic errors that propagate through the pipeline. We introduce AgentForge, a multi-agent framework that enforces verified execution for autonomous software engineering. AgentForge decomposes bug fixing into five specialized agents: Planner, Coder, Tester, Debugger, and Critic. The Planner generates a structured execution plan. The Coder produces minimal patches using unified diffs. The Tester synthesizes executable test cases. The Debugger iteratively repairs failures using execution feedback. The Critic validates the final result. AgentForge grounds all decisions in two retrieval sources: (i) episodic memory of previously solved tasks and (i) a live repository index of the current codebase. The system executes every generated patch inside a resource-constrained, network-isolated Docker sandbox. This design provides non-simulated execution feedback and enables a closed-loop Tester–Debugger cycle for iterative repair. Table situates AgentForge among existing systems. Prior frameworks introduce role decomposition, knowledge graphs, or test-time scaling [12, 26, 41, 28]. Other approaches explore self-evolution and competitive reasoning [29, 19]. None enforce mandatory sandboxed execution while combining dual retrieval with a full five-agent pipeline. AgentForge integrates these components into a unified, execution-grounded framework. TABLE I: Positioning Multi-Agent Frameworks Along Three Axes Framework Execution Feedback Role Decomposition Memory/Retrieval AgentForge Mandatory sandbox 5 roles (plan, code, test, debug, critic) Dual (episodic + repo) SWE-agent Shell/ACI (optional) Single agent None OpenHands Sandbox (optional) Single agent None Trae Agent Test-time scaling 3 roles (gen, prune, select) None MAGIS Implied 4 roles (mgr, custodian, dev, qa) None AgentMesh None 3 roles (plan, debug, review) None Reflexion Simulated Single + self-reflection Episodic memory Contributions. • We formalize LLM-based software engineering as an execution-grounded iterative refinement problem, where correctness is defined by external program execution rather than model-internal likelihood signals. • We model this process as a sequential decision problem over repository states and cast it as an MDP, enabling analysis of feedback, credit assignment, and error propagation. • We identify two key properties: (i) execution feedback provides a stronger supervision signal for functional correctness than next-token likelihood, and (i) decomposing generation, testing, and debugging reduces error accumulation compared to monolithic self-repair. • We instantiate these principles in AgentForge, a five-agent framework with structured orchestration, dual retrieval (episodic memory and repository index), and mandatory Docker-based execution. Novelty summary. AgentForge is the first framework to mandate sandboxed verification for every code change, providing ground-truth execution feedback. It integrates five specialized agents (Planner, Coder, Tester, Debugger, Critic) with a dual-memory system (episodic memory + live repository index) – a combination absent from AgentMesh, MAGIS, SGAgent, and Trae Agent. Implications. Our results indicate that verified execution feedback and structured pipeline design matter more than raw model scale for real-world software engineering. AgentForge provides an open-source baseline for future research in this direction. The remainder of this paper is organized as follows. Section I surveys related work. Section I details the AgentForge architecture. Section IV describes the experimental setup and baselines. Section V presents main results and ablations. Section VI discusses limitations and future directions. Figure 1: Overview of the AgentForge multi-agent coding framework, illustrating the sequential handover between specialized agents and the shared Vector Memory. I Related Work I-A LLM-Based Code Generation Neural code generation began with sequence-to-sequence models trained on paired text–code data [8, 36]. Large-scale pretraining on mixed corpora shifted the paradigm. Codex [4] demonstrated that a GPT-style model trained on GitHub can solve a substantial fraction of programming tasks in a single pass. AlphaCode [18], StarCoder [17], and CodeLlama [22] improved performance through scale, tokenizer design, and objectives tailored to code editing. These systems treat generation as a one-shot mapping from prompt to program. They lack mechanisms for execution-grounded verification or iterative correction. Failure signals do not feed back into generation. This limitation motivates structured, multi-step formulations. AgentForge adopts an execution-driven loop in which generated code is tested and revised under real feedback. I-B Program Repair and Iterative Refinement Automated program repair (APR) predates neural methods [14]. Modern APR systems use LLMs to propose patches conditioned on failing tests and localization signals [33, 10, 6]. These approaches assume known fault locations and existing test suites. Recent work strengthens feedback signals. TraceRepair [40] constrains patches with execution traces. DynaFix [32] incorporates runtime states and call stacks. InspectCoder [16] enables interactive debugging via tool control. RGD [13] decomposes repair into Guide, Debug, and Feedback roles. Self-repair [20] and self-debugging [5] prompt a single model to revise outputs from error messages. These methods collapse generation and repair into one policy. AgentForge separates these functions into distinct agents with disjoint objectives and interfaces. This separation yields measurable gains in our ablations (Section V-B). I-C Agentic and Tool-Using LLM Systems ReAct [35] interleaves reasoning traces with tool calls, enabling closed-loop interaction with external environments. Reflexion [24] augments this loop with episodic memory via self-generated feedback. Both frameworks rely on a single model to plan, act, and evaluate. AgentForge distributes these roles across specialized agents. Each agent operates under a fixed contract and prompt. This design reduces per-call complexity and enforces structured intermediate representations. Toolformer [23] learns tool invocation through self-supervision. It removes explicit prompting but requires fine-tuning and offers limited control over execution structure. AgentForge enforces explicit sequencing of planning, coding, testing, and debugging. I-D Multi-Agent LLM Frameworks Multi-agent systems decompose complex tasks into role-specific components. MetaGPT [7] encodes software roles through structured documents. ChatDev [21] uses conversational agents to produce complete projects. AutoGen [31] provides a general interface for agent interaction and tool use. Recent systems introduce adaptive and competitive coordination. SEMAG [38] evolves agent behavior with task difficulty. Eco-Evolve [29] uses dynamic topologies and hindsight replay. SWE-Debate [19] applies multi-round debate with search-based patch generation. These systems improve coordination but do not enforce execution-grounded validation at every step. AgentForge targets repository-level bug fixing and requires sandboxed execution for each candidate patch. It combines role specialization with mandatory verification and repository-grounded context. I-E Autonomous Software Engineering Agents SWE-agent [34] introduces an agent–computer interface that exposes shell, editor, and search tools to a single model. Devin [1] demonstrates end-to-end autonomous engineering, though its architecture remains undisclosed. OpenHands [30] provides an open-source platform with sandboxed execution. Trae Agent [28] applies test-time scaling via generation, pruning, and selection. MAGIS [26] decomposes issue resolution into Manager, Custodian, Developer, and QA roles. AgentForge adopts explicit multi-agent decomposition with five specialized roles. Each agent produces a constrained artifact: plan, diff, tests, repairs, or review. This design enables controlled execution, modular analysis, and interpretable ablations. I-F Benchmarks for Software Engineering HumanEval [4] and MBPP [2] evaluate function synthesis from docstrings. These tasks isolate generation and omit repository context. SWE-bench [9] introduces real GitHub issues paired with executable tests. SWE-bench Lite reduces cost while preserving diversity. SWE-bench Verified provides human-validated instances. Defects4J [11] remains a standard benchmark for Java repair. We evaluate on SWE-bench Lite following recent work [34, 30, 39]. This setting captures multi-file reasoning, environment interaction, and regression constraints. I-G Memory and Retrieval in LLM Systems Retrieval-augmented generation (RAG) conditions models on external context to improve accuracy [15]. In code, repository-level retrieval supplies relevant files and functions for completion [37, 25]. AgentForge implements dual retrieval. It maintains episodic memory of past tasks and a live repository index. Both reside in a shared ChromaDB vector store and use cosine similarity over OpenAI text embeddings. Episodic memory enables cross-task transfer. Repository indexing ensures intra-task grounding. This unified retrieval design supports consistent context across all agents. I Method We present AgentForge, a multi-agent framework for autonomous software engineering. Given a natural language task T and an optional set of context files ℱ=f1,…,fnF=\f_1,…,f_n\, AgentForge produces verified, executable code c c by routing the task through a structured pipeline of five specialized agents, each responsible for a single subtask. Figure 1 shows the full system. I-A Formal Framework: Execution-Grounded Iterative Refinement We model LLM-based software engineering as a finite-horizon Markov decision process (MDP) over repository states, where each action is verified through sandboxed execution. State space. Let S denote the set of repository states. A state st=(ℛt,ℳt,ℋt)s_t=(R_t,M_t,H_t) comprises the current repository ℛtR_t (source files, dependencies, test harness), an episodic memory ℳtM_t of prior task–patch pairs, and an execution history ℋtH_t containing outcomes of previous actions (stdout, stderr, test results). Action space. An action at∈a_t is a code patch produced by the Coder agent, represented as a unified diff or a new file. Actions are constrained to be minimal and syntactically valid. Transition function. The environment ℰE is a resource-constrained Docker sandbox (512 MB RAM, 0.5 CPU, no network). Applying ata_t in state sts_t yields st+1=ℰ(st,at)=(ℛt⊕at,ℳt,ℋt∪(at,ot,et)),s_t+1=E(s_t,a_t)=(R_t a_t,\ M_t,\ H_t∪\(a_t,o_t,e_t)\), (1) where ⊕ denotes patch application and (ot,et)(o_t,e_t) are execution outputs. Reward. The reward rt∈0,1r_t∈\0,1\ is defined by test outcomes: rt=[all FAIL_TO_PASS pass∧no PASS_TO_PASS regress],r_t=1 [all FAIL\_TO\_PASS pass\ \ no PASS\_TO\_PASS regress ], (2) evaluated after executing the full test suite. Objective. The goal is to learn a policy π:→π:S , instantiated by the agent pipeline, that maximizes expected cumulative reward over a finite horizon T: π∗=argmaxπ[∑t=0Tγtrt],π^*= _πE [ _t=0^Tγ^tr_t ], (3) where γ∈[0,1]γ∈[0,1]. We use γ=1γ=1 and T=Nretry=3T=N_retry=3. Execution grounding. The transition function ℰE executes code using the actual interpreter, compiler, and test runner in an isolated environment. Rewards derive from observed outcomes, eliminating simulation error and preventing model-induced hallucinated feedback. Error propagation. Let p denote the failure probability of a monolithic agent per attempt. After k independent attempts, the success probability is 1−pk1-p^k. In a decomposed pipeline with n agents and per-agent failure probabilities pii=1n\p_i\_i=1^n, the success probability of a single pass is ∏i=1n(1−pi). _i=1^n(1-p_i). (4) If pi≈p_i≈ p, decomposition reduces success when n>1n>1. In practice, specialization reduces per-agent error (pi≪p_i p) by constraining the output space and task scope. Decomposition improves success when ∏i=1n(1−pi)>1−p _i=1^n(1-p_i)>1-p, providing a formal condition for Claim 2. I-B System Overview AgentForge decomposes autonomous code generation into five sequential roles: Planner, Coder, Tester, Debugger, and Critic. Each agent is implemented as a prompted large language model (LLM) call with a role-specific system prompt. A central Orchestrator coordinates the pipeline, manages shared memory, and handles the iterative debug loop. Notation. Let =Aplan,Acode,Atest,Adebug,AcritA=\A_plan,A_code,A_test,A_debug,A_crit\ denote the agent set. Let πa _a denote the system prompt for agent a∈a , and let LLM(πa,x)LLM( _a,x) denote a call to the base language model with system prompt πa _a and user input x. I-C Memory and Context Retrieval Before planning, the Orchestrator enriches the task with two sources of context: episodic memory from past tasks and semantic retrieval from the live repository index. Episodic memory. Successful (task, code) pairs from prior runs are stored in a persistent vector database (ChromaDB [27]). At inference time, the top-k most similar past tasks are retrieved by cosine similarity of their text-embedding-3-small embeddings: ℳk=top-km∈ℳ⋅m‖‖m‖M_k= m top-k\; e_T·e_m\|e_T\|\,\|e_m\| (5) Figure 2: Retrieval-Augmented Generation (RAG) architecture: (a) Offline repository indexing phase into the vector store; (b) Online semantic retrieval at inference time. Repository context. A background indexer monitors the repository using filesystem event hooks and maintains an up-to-date embedding for every source file. The top-k most relevant files are retrieved for each task and prepended to the planning context. This gives the Planner grounded knowledge of existing interfaces, reducing hallucinated imports and incompatible function signatures. I-D Agent Definitions Planner (AplanA_plan). The Planner receives the task T, retrieved memory context ℳkM_k, and repository snippets, and produces a structured execution plan: P=LLM(πplan,[;ℳk;ℛk])P=LLM( _plan,\;[T;\,M_k;\,R_k]) (6) P is a JSON object containing a natural language explanation and an ordered list of steps P=s1,…,smP=\s_1,…,s_m\, where each step sis_i specifies an agent assignment si.agent∈s_i.agent , a description si.descs_i.desc, and an optional target file si.files_i.file. Coder (AcodeA_code). For each coder step sis_i, the Coder generates either (a) a complete new implementation or (b) a minimal unified diff if a target file exists: ci=LLM(πcodenew,si.desc)if si.file=∅Apply(LLM(πcodediff,[si.desc;fsi]),fsi)otherwisec_i= casesLLM( _code^new,\;s_i.desc)&if s_i.file= \\ Apply(LLM( _code^diff,\;[s_i.desc;\,f_s_i]),\;f_s_i)&otherwise cases (7) where fsif_s_i is the content of the target file and Apply(⋅)Apply(·) patches the original using the unidiff library. Diff-based editing preserves unchanged lines, reducing error surface and token cost compared to full-file regeneration. Tester (AtestA_test). Given the generated code cic_i, the Tester produces a suite of pytest test cases covering typical usage, edge cases, and exception paths: τi=LLM(πtest,[ci;si.desc]) _i=LLM( _test,\;[c_i;\,s_i.desc]) (8) Debugger (AdebugA_debug). If execution of (ci,τi)(c_i, _i) in the sandbox returns a non-zero exit code or a pytest FAILED result, the Debugger receives the code and the full error output and produces a corrected version: ci′=LLM(πdebug,[ci;ei])c_i =LLM( _debug,\;[c_i;\,e_i]) (9) where eie_i is the combined stdout/stderr from the failed run. This loop repeats up to NretryN_retry times (default Nretry=3N_retry=3). Critic (AcritA_crit). After all steps complete, the Critic reviews the full result set and returns a binary verdict: v=LLM(πcrit,[;(si,ci,ei)i=1m])∈PASS,FAILv=LLM( _crit,\;[T;\,\(s_i,c_i,e_i)\_i=1^m])∈\ PASS,\, FAIL\ (10) A PASS verdict triggers persistence of (,c^)(T, c) into episodic memory ℳM for future retrieval. I-E Sandboxed Execution All generated code is executed inside a disposable Docker container with strict resource constraints: 512 MB memory limit, 0.5 CPU quota, a 64-process PID cap, and networking disabled as shown in Figure 3. Code is injected via the Docker put_archive API as an in-memory tar archive, avoiding filesystem writes on the host. The container is force-removed after every run regardless of outcome. Figure 3: Isolated Docker sandbox execution environment. The 512 MB memory limit and disabled networking ensure security and reproducibility. Formally, let Sandbox(c,τ)Sandbox(c,τ) return (o,e)∈Σ∗×Σ∗(o,e)∈ ^*× ^* where o is stdout and e is stderr. Execution is considered successful iff: pass(c,τ)=[e=∅∧FAILED∉o∧ERROR∉o]pass(c,τ)=1[e= \; \; FAILED∉ o\; \; ERROR∉ o] (11) I-F Full Orchestration Algorithm Algorithm 1 summarizes the complete pipeline. Algorithm 1 AgentForge Orchestration 0: Task T, context files ℱF, memory ℳM, repo index ℛR 0: Verified code c c or Fail 1: ℳk←Retrieve(,ℳ)M_k (T,M) 2: ℛk←Retrieve(,ℛ)R_k (T,R) 3: P←Aplan(,ℳk,ℛk)P← A_plan(T,M_k,R_k) 4: c^←∅ c← , results ←[]←[] 5: for each step si∈P.stepss_i∈ P.steps do 6: if si.agent=coders_i.agent= coder then 7: c^←Acode(si,ℱ,results) c← A_code(s_i,F,results) 8: results.append(c^)( c) 9: else if si.agent=testers_i.agent= tester then 10: τ←Atest(c^,si)τ← A_test( c,s_i) 11: (o,e)←Sandbox(c^,τ)(o,e) ( c,τ) 12: n←0n← 0 13: while ¬pass(c^,τ) ( c,τ) and n<Nretryn<N_retry do 14: c^←Adebug(c^,e) c← A_debug( c,e) 15: (o,e)←Sandbox(c^,τ)(o,e) ( c,τ) 16: n←n+1n← n+1 17: end while 18: results.append(o,e)(o,e) 19: else if si.agent=critics_i.agent= critic then 20: v←Acrit(,results)v← A_crit(T,results) 21: results.append(v)(v) 22: end if 23: end for 24: vfinal←Acrit(,results)v_final← A_crit(T,results) 25: if vfinal=PASSv_final= PASS then 26: ℳ.store(,c^)M.store(T, c) 27: return c c 28: else 29: return Fail 30: end if I-G Streaming Output To support interactive use, the Orchestrator exposes a streaming interface over Server-Sent Events (SSE) and WebSocket. Each token produced by the Coder agent is forwarded to the client as it arrives, using Python async generators and FastAPI’s EventSourceResponse. This enables real-time inspection of the generation process without waiting for pipeline completion. I-H Complexity Analysis Let L be the average prompt length in tokens and G the average generated length. A single pipeline run incurs O(||⋅(L+G))O(|A|·(L+G)) tokens in the non-debug case, and O(||⋅(L+G)⋅Nretry)O(|A|·(L+G)· N_retry) in the worst case. Retrieval adds O(dlogn)O(d n) per query for a HNSW index of n embeddings in d dimensions. All agent calls are embarrassingly parallelizable within a plan step when step dependencies permit. I-I Theoretical Claims and Hypotheses We formalize three claims that motivate the design of AgentForge. Each claim is stated as a proposition with explicit conditions and testable implications. Proposition 1 (Execution signal dominance). Let y∈0,1y∈\0,1\ denote functional correctness (test pass/fail), pθ(x)p_θ(x) the model likelihood over patches, and y^exec y_exec the outcome of sandboxed execution. Then y^exec y_exec provides a lower-variance, higher-fidelity estimator of y than any proxy derived from pθ(x)p_θ(x). Justification. Likelihood scores reflect distributional similarity to training data, not semantic correctness. In contrast, execution evaluates correctness directly via test outcomes. Let ℓ(x) (x) denote a likelihood-based proxy and y^exec y_exec the execution signal. Then Var[y^exec−y]<Var[ℓ(x)−y]Var[ y_exec-y]<Var[ (x)-y] (12) under mild assumptions on test coverage and determinism of execution. Implication. For policies πexec _exec (with execution feedback) and πlm _lm (likelihood-only), there exists a regime where [R(πexec)]>[R(πlm)]E[R( _exec)]>E[R( _lm)] (13) even when πlm _lm uses a larger model. Testable prediction. A 7B model with execution feedback and iterative repair outperforms a 70B model without execution feedback on SWE-bench. Proposition 2 (Error propagation under decomposition). Consider a pipeline with n agents and per-agent error probabilities pii=1n\p_i\_i=1^n. The success probability of a single pass is Psuccmulti=∏i=1n(1−pi).P_succ^multi= _i=1^n(1-p_i). (14) For a monolithic agent with error probability p, the success probability is Psuccmono=1−p.P_succ^mono=1-p. (15) Condition for improvement. Decomposition improves success if ∏i=1n(1−pi)>1−p. _i=1^n(1-p_i)>1-p. (16) Justification. Specialization reduces per-agent uncertainty by constraining the output space and conditioning inputs. Let pi=p−Δip_i=p- _i with Δi>0 _i>0. Then decomposition improves success when ∑i=1nΔi>p(n−1). _i=1^n _i>p(n-1). (17) Error correlation. Let ϵi _i denote the error event of agent i. In a monolithic agent, errors are temporally correlated: ℙ(ϵt∣ϵt−1)≫ℙ(ϵt).P( _t _t-1) ( _t). (18) In a decomposed pipeline, conditioning on external artifacts (plans, execution traces) reduces mutual information: I(ϵi;ϵj)multi<I(ϵi;ϵj)mono,i≠j.I( _i; _j)_multi<I( _i; _j)_mono, i≠ j. (19) Testable prediction. Removing any agent reduces performance. The full pipeline exceeds the success rate predicted under independent error composition, indicating reduced error correlation. Proposition 3 (Efficiency of diff-based editing). Let L denote file length and k≪Lk L the size of a minimal patch. Diff-based editing restricts generation to O(k)O(k) tokens, while full-file regeneration requires O(L)O(L) tokens. Implication. Token cost satisfies Cdiff=O(k),Cfull=O(L),k≪L.C_diff=O(k), C_full=O(L), k L. (20) Error surface. The probability of introducing an error scales with the number of generated tokens. Under a per-token error rate ϵε, Perrordiff≈1−(1−ϵ)k,Perrorfull≈1−(1−ϵ)L.P_error^diff≈ 1-(1-ε)^k, P_error^full≈ 1-(1-ε)^L. (21) Thus Perrordiff≪PerrorfullP_error^diff P_error^full when k≪Lk L. Testable prediction. For files with L>200L>200, diff-based editing yields higher success rates and lower token usage than full-file regeneration under a fixed base model. IV Experiments IV-A Benchmark We evaluate on SWE-bench Lite [9], a curated subset of 300 real GitHub issues drawn from 11 popular Python repositories including Django, Flask, scikit-learn, and NumPy. Each instance consists of a natural language problem statement, a base repository commit, a gold patch, and a set of tests that pass only after the bug is correctly fixed (fail_to_pass) alongside a set of tests that must continue to pass (pass_to_pass). A task is considered resolved if and only if all fail_to_pass tests pass and no pass_to_pass tests regress after applying the generated patch. IV-B Baselines We compare against three baselines: Single-agent (GPT-4o). A single GPT-4o call with the problem statement in the prompt, instructed to produce a unified diff patch. No tools, no execution feedback, no iteration. ReAct (GPT-4o). A ReAct-style [35] agent that interleaves reasoning and tool invocation in an open-ended loop (maximum 10 steps). Tools available: read file, write code, run tests. Uses the same base model as AgentForge. SWE-agent. We report the published SWE-agent [34] result on SWE-bench Lite for reference, noting that it uses a different base model configuration and ACI design. IV-C Implementation Details All AgentForge experiments use GPT-4o (gpt-4o-2024-08-06) with temperature=0.0 and seed=42 for reproducibility. The debug loop is capped at Nretry=3N_retry=3 attempts per task. The vector store retrieves k=5k=5 past tasks and k=5k=5 repository files at planning time. The Docker sandbox uses python:3.10-slim with a 512 MB memory limit, 0.5 CPU quota, and a 30-second execution timeout. All experiments are run on a machine with a 16-core CPU, 64GB of RAM, and a dedicated GPU for faster processing. The system is equipped with high-speed SSD storage to handle the data-intensive tasks efficiently. IV-D Evaluation Protocol For each task we: (1) clone the repository at the base commit, (2) apply the generated patch using git apply, (3) install the project with pip install -e ., (4) run the fail_to_pass and pass_to_pass test suites, and (5) record pass/fail for each test. Tasks where the patch does not apply cleanly are counted as unresolved. This protocol follows the official SWE-bench evaluation harness [9]. V Results V-A Main Results Table I shows the resolution rate of AgentForge and all baselines on SWE-bench Lite. TABLE I: Resolution rates on SWE-bench Lite (300 tasks). AgentForge outperforms single-agent baselines under a fixed execution budget. Method Resolve Rate Patch Rate Single-agent GPT-4o 14.0% 12.5% ReAct (GPT-4o) 12.0% 11.0% SWE-agent [34] 18.1% 17.0% AgentForge (ours) 40.0% 37.5% Note. Trae Agent uses test-time scaling (multiple samples per task with pruning and selection) and less restrictive execution. AgentForge enforces mandatory sandboxed execution (512 MB RAM, 0.5 CPU, no network), uses a single sample per agent, and a fixed retry budget (N=3N=3). Reported results reflect this constrained and reproducible setting. Figure 4: Performance of AgentForge across three evaluation axes on SWE-bench Lite. Left: Resolution rate as a function of debug retries (N). Shaded regions denote ±1± 1 standard deviation across runs. Iterative execution and repair yield consistent gains, with diminishing returns after N=2N=2. Center: Resolution under k independent runs with majority voting (Pass@k). Performance scales with k without increasing per-run complexity. Right: Cost–performance tradeoff. AgentForge achieves higher resolution at lower cost compared to single-agent baselines, indicating improved sample efficiency. Performance improves with iterative debugging but saturates quickly (Figure 4), indicating diminishing returns beyond two retries. AgentForge achieves 40.0% task resolution, outperforming the single-agent baseline by +26.0% and the ReAct baseline by +28.0%. This substantial improvement highlights the advantage of structured multi-agent collaboration for complex software engineering tasks. The patch rates follow a similar trend, confirming that the gains are not due to trivial formatting fixes but reflect genuinely correct fixes. V-B Ablation Study To understand the contribution of each agent, we systematically disable one component at a time and re-evaluate on the first 100 tasks of SWE-bench Lite. Table I summarizes the results. TABLE I: Ablation study on 100 SWE-bench Lite tasks. Each row removes one agent from the pipeline. Configuration Resolved Resolve Rate Full pipeline (ours) 42 42.0% w/o Critic agent 38 38.0% w/o Debugger agent 31 31.0% w/o Tester agent 28 28.0% w/o Planner agent 19 19.0% Single-agent baseline 14 14.0% The ablation reveals a consistent ordering: removing any agent reduces performance, with the Tester and Debugger together contributing the largest gains. Removing the Planner — reducing the pipeline to a single unstructured coder step — drops performance to near the single-agent baseline, confirming that structured decomposition is not merely cosmetic. Figure 5 visualizes the resolve rates across conditions. Figure 5: Resolve rates across ablation conditions. Removing any single agent degrades performance; the Tester–Debugger loop is the largest contributor. V-C Error Analysis We analyze 30 randomly sampled failed tasks from SWE-bench Lite to characterize the dominant failure modes of AgentForge. Table IV reports aggregate categories; we complement these with fine-grained qualitative patterns derived from execution traces and agent outputs. TABLE IV: Failure mode analysis on 30 failed tasks. Failure Category Count Percentage Faulty localization (incorrect file/line) 12 40.0% Ineffective patch generation (wrong or partial fix) 8 26.7% Cognitive deadlock (repeated failing attempts) 6 20.0% Environment/tooling (patch apply fail, timeout, etc.) 4 13.3% Total 30 100% Faulty localization (40.0%). Localization errors dominate. The Planner often identifies a primary file but misses secondary dependencies, leading to incomplete fixes. In 52% of these cases, the correct patch requires coordinated edits across multiple files. This failure reflects limited modeling of repository-level dependency structure rather than code synthesis errors. Ineffective patch generation (26.7%). Generated patches frequently resolve the immediate symptom but violate latent invariants, introducing regressions. In 34% of these cases, the Tester produces brittle supervision signals: tests overfit to the observed failure or depend on unstable implementation details (e.g., function names modified by the patch). This weakens the reliability of execution feedback. Cognitive deadlocks (20.0%). The Debugger exhibits local search behavior with limited state diversification. In 23% of deadlock cases, the system repeatedly modifies the same function despite error traces indicating downstream dependencies. Execution logs show near-identical stderr across retries, indicating failure to shift the locus of repair. Environment and tooling (13.3%). Residual failures arise from sandbox constraints rather than reasoning errors. These include dependency mismatches (e.g., numpy versions), long-running test suites (>30>30s), and patch application conflicts. These factors bound achievable performance under strict execution settings. Implications. The error distribution identifies three primary bottlenecks: (1) incomplete cross-file dependency reasoning, (2) unstable test generation as a supervision signal, and (3) limited exploration in the debug loop. Addressing these requires multi-file planning, constraint-aware test synthesis, and diversity-promoting repair strategies (e.g., beam search or stochastic perturbations). The dominance of localization errors suggests that improvements in repository understanding may yield larger gains than further scaling the base model. Figure 6: Resolve rates across ablation conditions. Removing any single agent degrades performance; the Tester–Debugger loop is the largest contributor. V-D Cost Analysis Table V reports token usage and estimated API cost for the full AgentForge pipeline and the single-agent baseline, using GPT-4o pricing ($2.50 / 1M input tokens, $10.00 / 1M output tokens). Costs are averaged per task over the 300 SWE-bench Lite tasks. TABLE V: Token usage and estimated API cost per task (averaged). Configuration Input tokens Output tokens Cost (USD) Full pipeline 13,600 5,100 $0.1445 Single-agent (GPT-4o) 5,000 1,800 $0.054 The full pipeline costs approximately 2.7× more than the single-agent baseline, reflecting the overhead of five specialized agents and the iterative debug loop. At this rate, evaluating on the full 300-task SWE-bench Lite costs about $43.35, which is modest given the 40% resolution rate. For budget-constrained scenarios, one could reduce debug retries or use a smaller model for non-critical agents. VI Conclusion We presented AgentForge, a multi-agent framework that replaces single-shot code generation with an execution-grounded feedback process. The system decomposes software engineering into five specialized agents and enforces verified execution for every patch. AgentForge achieves 40.0% resolution on SWE-bench Lite, exceeding strong baselines by large margins. Ablations show that execution feedback, implemented through the Tester–Debugger loop, is the primary driver of performance. The system operates at file-level granularity and struggles with multi-file coordination. The evaluation metric is binary and does not capture partial correctness or regressions. Results rely on GPT-4o. Future directions include multi-file atomic patches, finer-grained retrieval, persistent memory, broader benchmarks, and role-specialized smaller models. Execution-grounded agents can increase productivity but may generate incorrect or insecure code. Sandboxed execution mitigates risk during evaluation. Deployment requires human oversight, static analysis, and audit mechanisms. References [1] C. AI (2024) Devin: an autonomous ai software engineer. Note: https://w.cognition.ai/blog/introducing-devinBlog post / technical report Cited by: §I-E. [2] J. Austin et al. (2021) Program synthesis with large language models. External Links: 2108.07732 Cited by: §I-F. [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems 33. Note: NeurIPS 2020 Cited by: §I. [4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. Note: Codex / GitHub Copilot paper External Links: 2107.03374 Cited by: §I, §I-A, §I-F. [5] X. Chen et al. (2023) Teaching large language models to self-debug. External Links: 2304.05128 Cited by: §I-B. [6] Z. Fan et al. (2023) Automated program repair in the era of large pre-trained language models. External Links: 2305.14123 Cited by: §I-B. [7] S. Hong et al. (2023) MetaGPT: meta programming for a multi-agent collaborative framework. External Links: 2308.00352 Cited by: §I-D. [8] S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer (2016-08) Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, p. 2073–2083. External Links: Link, Document Cited by: §I-A. [9] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, Note: ICLR 2024 Oral Cited by: §I-F, §IV-A, §IV-D. [10] M. Jin et al. (2023) InferFix: end-to-end program repair with llms. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), Cited by: §I-B. [11] R. Just, D. Jalali, and M. D. Ernst (2014) Defects4J: a database of existing faults to enable controlled testing studies for java programs. Proceedings of the International Symposium on Software Testing and Analysis (ISSTA). Cited by: §I-F. [12] S. Khanzadeh (2025) AgentMesh: a cooperative multi-agent generative ai framework for software development automation. External Links: 2507.19902 Cited by: §I. [13] S. Kim et al. (2024) RGD: multi-llm based agent debugger. External Links: 2410.11324 Cited by: §I-B. [14] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer (2012) GenProg: a generic method for automatic software repair. IEEE Transactions on Software Engineering (TSE) 38 (1), p. 54–72. Cited by: §I-B. [15] P. Lewis et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. External Links: 2005.11401 Cited by: §I-G. [16] H. Li et al. (2025) InspectCoder: dynamic analysis-enabled self repair. Note: ICSE 2025 Cited by: §I-B. [17] R. Li et al. (2023) StarCoder: may the source be with you!. External Links: 2305.06161 Cited by: §I-A. [18] Y. Li et al. (2022) Competition-level code generation with alphacode. External Links: 2203.07814 Cited by: §I-A. [19] Y. Liu et al. (2026) SWE-debate: multi-agent debate for github issue resolution. Note: ICSE 2026 Cited by: §I, §I-D. [20] T. X. Olausson et al. (2023) Is self-repair a silver bullet for code generation?. Note: Justification for the Iterative Debug Loop in AgentForge External Links: 2306.09896 Cited by: §I-B. [21] C. Qian et al. (2023) ChatDev: communicative agents for software development. External Links: 2307.07924 Cited by: §I-D. [22] B. Rozière et al. (2023) Code llama: open foundation models for code. External Links: 2308.12950 Cited by: §I-A. [23] T. Schick et al. (2023) Toolformer: language models can teach themselves to use tools. External Links: 2302.04761 Cited by: §I-C. [24] N. Shinn et al. (2023) Reflexion: language agents with verbal reinforcement learning. External Links: 2303.11366 Cited by: §I-C. [25] D. Shrivastava et al. (2023) RepoFusion: training code models on whole repositories. External Links: 2306.10424 Cited by: §I-G. [26] W. Tao et al. (2024) MAGIS: llm-based multi-agent framework for github issue resolution. External Links: 2403.17927 Cited by: §I, §I-E. [27] C. Team (2023) Chroma: the ai-native open-source embedding database. Note: https://w.trychroma.com/Technical reference for the episodic memory store Cited by: §I-C. [28] T. Team (2025) Trae agent: test-time scaling for software engineering. External Links: 2507.23370 Cited by: §I, §I, §I-E. [29] X. Wang et al. (2026) Eco-evolve: dynamic multi-agent evolution for software engineering. Note: arXiv preprint Cited by: §I, §I-D. [30] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2024) OpenHands: an open platform for ai software developers as generalist agents. Note: Primary open-source baseline for multi-agent SWE (also accepted as ICLR 2025 poster) External Links: 2407.16741 Cited by: §I-E, §I-F. [31] Q. Wu et al. (2023) AutoGen: enabling next-gen llm applications via multi-agent conversation. External Links: 2308.08155 Cited by: §I-D. [32] Y. Wu et al. (2025) DynaFix: iterative apr driven by execution-level dynamic info. External Links: 2512.24635 Cited by: §I-B. [33] C. Xia and L. Zhang (2022) Less is more: summary of long code for repair. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), Cited by: §I-B. [34] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, Vol. 37. Note: NeurIPS 2024 Cited by: §I-E, §I-F, §IV-B, TABLE I. [35] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), Cited by: §I-C, §IV-B. [36] P. Yin and G. Neubig (2017-07) A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, p. 440–450. External Links: Link, Document Cited by: §I-A. [37] F. Zhang et al. (2023) RepoCoder: repository-level code completion. External Links: 2305.14570 Cited by: §I-G. [38] Y. Zhang et al. (2026) SEMAG: self-evolutionary multi-agent code generation. Note: arXiv preprint, to appear Cited by: §I-D. [39] Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury (2024) AutoCodeRover: autonomous program improvement. Note: ISSTA 2024 External Links: 2404.05427 Cited by: §I-F. [40] R. Zhao et al. (2026) TraceRepair: execution trace-driven program repair. Note: arXiv preprint Cited by: §I-B. [41] H. Zheng et al. (2026) SGAgent: knowledge graph-augmented multi-agent repair. Note: arXiv preprint Cited by: §I.