Paper deep dive
CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Shutong Wu, Kevin Calderone, Andy Tsen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/22/2026, 5:56:05 AM
Summary
The paper introduces CraftBench-UE, an evaluation harness and benchmark for assessing coding agents in Unreal Engine. It features 70 tasks across C++, Blueprint, and editor scripting, utilizing deterministic build, asset, and runtime checks without LLM judges. The study evaluates seven models under two MCP configurations, finding that C++ completion rates significantly exceed Blueprint rates (by 30-43 percentage points) and that many Blueprint submissions pass asset checks but fail runtime gameplay assertions.
Entities (16)
Relation Signals (15)
CraftBench-UE → evaluates → C
confidence 100% · spanning C++ source, Blueprint assets, and editor scripting
CraftBench-UE → evaluates → Blueprint
confidence 100% · spanning C++ source, Blueprint assets, and editor scripting
CraftBench-UE → uses → Unreal Engine
confidence 100% · CraftBench-UE, an evaluation harness that runs agents in an isolated Unreal Engine environment
CraftBench-UE → evaluates → Gemini 3.7 Flash
confidence 95% · We evaluate seven models... Gemini 3.7 Flash... under two editor-tool configurations
CraftBench-UE → evaluates → GPT-5.6 Luna
confidence 95% · We evaluate seven models... GPT-5.6 Luna... under two editor-tool configurations
CraftBench-UE → evaluates → Grok 4.6
confidence 95% · We evaluate seven models... Grok 4.6... under two editor-tool configurations
CraftBench-UE → evaluates → GLM-5.3 Flash
confidence 95% · We evaluate seven models... GLM-5.3 Flash... under two editor-tool configurations
CraftBench-UE → evaluates → GPT-5.6 Sol
confidence 95% · We evaluate seven models... GPT-5.6 Sol... under two editor-tool configurations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.
Tags
Links
- Source: https://arxiv.org/abs/2609.23142v1
- Canonical: https://arxiv.org/abs/2609.23142v1
Trouble viewing inline? Open PDF directly →
Full Text
70,641 characters extracted from source content.
Expand or collapse full text
CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine Shutong Wu * University of Rochester Kevin Calderone, Andy Tsen † RamenVR # swu85@ur.rochester.edu§ https://github.com/ramenvr/craftbench-ue ABSTRACT Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We in- troduce CraftBench-UE, an evaluation harness that runs agents in an isolated Un- real Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we build a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These sub- missions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report. 1INTRODUCTION Implementing a gameplay feature involves more than producing code that compiles. In Unreal Engine, a feature can span C++ classes, Blueprint graphs, and other features such as the Niagara VFX system. Blueprint is Unreal’s visual programming system, which uses node graphs saved in binary .uasset files. The evaluation question is therefore whether the saved project implements the requested behavior, not simply whether it compiles. Existing benchmarks evaluate generated functions, repository changes, and tasks in interactive en- vironments (Chen et al., 2021; Jimenez et al., 2024; Merrill et al., 2026; Zhou et al., 2024). Game- development benchmarks extend executable evaluation to game engines such as Godot, Unreal, and Roblox (Chi et al., 2026; La et al., 2026; Zhang et al., 2025). Evaluating projects that combine source code and native assets requires checking both the submitted files and the behavior they produce. The benchmark harness should also differentiate between structural requirements, like a valid asset refer- ence or successful compilation, and behavioral requirements, such as a character decelerating while gliding. We introduce CraftBench-UE, an evaluation harness for coding agents working with Unreal projects. Each task defines the starting state and the goal deliverables. The harness collects the generated work, then reconstructs it in a fresh project and grades it based on the build, asset, and runtime * Work done during internship at RamenVR. † Corresponding author. 1 arXiv:2609.23142v1 [cs.AI] 19 Sep 2026 C++ 33 tasks Source editing allowed Glide while falling Slow a falling character. EXAMPLE CHECKS Descent slows during glide Power drains while active Normal fall resumes at zero Blueprint / assets 25 tasks Saved editor-native assets Mud wading Wade through a mud patch. EXAMPLE CHECKS Speed falls from 5 to 2 m/s Wading animation plays Normal movement returns Editor scripting 12 tasks Saved project state or reports Saved level (diagram) Floor Start Bounds Enemy_1 Enemy_2 Enemy_3 A small test level Create six named objects. EXAMPLE CHECKS All six objects are present Labels and positions match Level is saved to disk Figure 1: Sample benchmark tasks across the three deliverable groups. The screenshots illustrate the glide task (gp-glide-stamina-cpp) and mud-wading task (t1-mud-wade-bp). The editor- scripting panel shows the kp-spawn-level-actors task, which requires creating and saving a level containing six named objects. checks. Runtime checks observe behavior in Play-in-Editor (PIE), Unreal’s mode for running a game inside the editor. Based on the harness, we construct a benchmark of 70 bounded tasks: 33 C++ tasks, 25 Blueprint/native-asset tasks, and 12 editor-scripting tasks. The task suite contains 10 gameplay task pairs, each with both C++ and Blueprint versions. This design enables analysis of how the same agent approaches the same tasks across different deliverable formats. We evaluate seven models under two editor-tool configurations, MCP1 and MCP2 (§4), with a file- and-shell C++ baseline. In the paired analysis, we found that C++ completion exceeds Blueprint completion by 30.0 and 42.9 percentage points under MCP1 and MCP2 within the time budget. Among on-time Blueprint submissions in this set that satisfy asset checks, 42.2% (19/45) under MCP1 and 50.0% (20/40) under MCP2 fail explicit runtime assertions. The comparison and result analyses address complementary questions: whether agents complete the same gameplay under different deliverable requirements, and what passing an asset check tells us about the final completed feature. Our contributions are: 1. An Unreal evaluation harness. CraftBench-UE runs an agent on a task in an isolated Unreal project, reconstructs its submission and grades it through build, asset, and runtime verifiers. 2. A benchmark of 70 tasks. The tasks cover source code, native assets, and editor scripting, including paired gameplay specifications with the same runtime tests. 3. An empirical study of how coding agents work in game development tasks. We share our results on how different models work through the benchmark, and how they perform and verify their work during the benchmarking process. Paired comparisons quantify dif- ferences under two editor configurations; submission and trajectory analyses show where asset construction and authoring feedback fall short of completed gameplay. 2RELATED WORK Execution-based agent evaluation. Tests can reject a valid solution if they impose requirements that the task description does not state. SWE-bench Verified addresses this problem through devel- oper review, screening out unclear tasks and tests that reject valid solutions (OpenAI, 2024). The 2 checks should match what the agent was asked to do, whether the task involves writing a function, modifying a repository, or working in a terminal or desktop environment (Chen et al., 2021; Jimenez et al., 2024; Merrill et al., 2026; Xie et al., 2024). We document our validation process in §3. Game-development benchmarks. GameDevBench uses deterministic tests for Godot scripts, scenes, and resources and examines how image and video feedback affect agent performance (Chi et al., 2026). In GameEngineBench, agents make C++ changes in open source Unreal projects, and the changes are examined by both build and runtime tests; an LLM judge then considers the test results and implementation work (La et al., 2026). OpenGameEval evaluates Roblox Studio tasks with executable tests (Zhang et al., 2025). GameXpert-Bench spans generation, repair, and multi-turn optimization of browser-native JavaScript games (Chen et al., 2026). These benchmarks already evaluate the behavior of changes made by agents. GBQA instead scores autonomous defect discovery in running web games by the share of human-verified bugs found (Jiang et al., 2026). Other work in Unreal covers cinematic assets and complete games. CutsceneBench combines tool and asset checks for Level Sequence assets with evaluation of rendered videos (He et al., 2026). AutoUE focuses on end-to-end Unreal game generation using an LLM judge as the grader (Yin et al., 2026). Our tasks focus on individual gameplay features with outcomes that could be deter- ministically checked. The harness reconstructs each submission in a fresh Unreal project and applies build, asset, and runtime checks. Appendix G provides a detailed comparison of these benchmarks. Interfaces and tool use. MCP exposes application operations as tools (Anthropic, 2024), and MCP benchmarks study tool selection and composition (Luo et al., 2025; Wu et al., 2025). Orak likewise standardizes agent access to games through MCP, but its deliverable is a gameplay trajec- tory scored by game outcomes rather than a saved engine artifact (Park et al., 2026). MCP-Unity connects MCP clients through a Python server to a C# Unity Editor plugin (Wu & Barnett, 2025); Epic’s in-editor Unreal MCP server exposes tools over local HTTP and dispatches calls on the game thread (Epic Games, n.d.). These interfaces provide authoring tool sets; our harness evaluates saved submissions in fresh Unreal projects. Agent-interface studies show that the surrounding configuration can affect coding outcomes (Yang et al., 2024; Zhang et al., 2026). SkillsBench compares matched conditions with and without curated procedural Skills using deterministic verifiers (Li et al., 2026), whereas we vary editor configurations and required deliverables. We therefore describe editor services, editing permissions, and execution settings alongside the model panel. The observed differences compare these configurations; they do not identify the effect of an individual tool operation. 3CRAFTBENCH-UE 3.1EVALUATION SETTING AND TASK SCOPE CraftBench-UE measures whether an agent can implement a bounded gameplay or editor behavior in an existing Unreal project. The unit of evaluation is an atomic mechanism task: an instruction specifies a bounded gameplay or editor behavior, and states which project files or assets the agent may change. Examples include attaching an object to a skeletal socket, enforcing a cooldown, persisting a Blueprint graph, and authoring a mini level (Figure 1). Each task focuses on a small set of engine mechanisms and uses named checks to identify which tested conditions fail. Tasks may still couple subsystems, as a movement behavior can involve input, state, timing, and physics, but they do not combine unrelated features merely to increase difficulty. Each score records whether the submitted project implements that mechanism under the stated task conditions. 3.2CRAFTBENCH-UE EVALUATION HARNESS Container-oriented frameworks such as Harbor package instructions, environments, and tests (Har- bor Framework Team, 2026). Unreal evaluation is comparatively more complex since the framework needs to control a versioned Editor and PIE environment, while preserving binary assets and saved changes. A task package specifies the starting project, prompt, writable area, engine build, verifier, and reference implementation. The harness reconstructs the project and delivers the prompt through 3 1 Task construction Specify observable behavior and deliverable Map requirements to checks Task package Instructions + writable paths Starting project Reference + grader Hidden from agent Admission requirements Reference must PASS Unchanged must FAIL Revise if either is wrong 2 Agent authoring Fresh context; task and starting project Agent Allowed tools File/shell and/or MCP as configured Project files saved assets Tool permissions depend on the configuration. 3 Evaluate the saved submission Fresh project Task-required checks L1 BuildL2 RuntimeL2I Inspect PASS requires every task check; report failing checks. Copy permitted saved changes Figure 2: Task construction, agent authoring, and evaluation of the saved submission in a fresh project. Each task declares its required checks. Reference-pass and unchanged-project-fail are admission criteria; §3.5 describes task authoring and validation. the selected agent configuration. Reference implementations and verifier sources are withheld dur- ing authoring; detected exposure excludes a run from scoring (Appendix B.1). The harness records interactions, timing, and project changes. When authoring ends, it collects the saved changes allowed by the task and applies them to a fresh copy of the starting project. This clean replay separates the submitted artifact from unrelated workspace modifications. It then builds the project, runs behavioral checks, and inspects native assets as declared by the task. Reports include the overall verdict, gate outcomes, and the failing checks identified. Figure 2 summarizes the workflow. 3.3EXECUTION-BASED DETERMINISTIC GRADING Each task specifies which of the following checks its submission must pass: 1. Build (L1). Compile the relevant editor and game targets. Submission-caused compiler or linker errors are failures. 2. Behavior (L2). A fixed-timestep Play-in-Editor (PIE) session will start and evaluate named assertions at certain timesteps using Unreal Functional Tests. 1 Assertions focus on state transitions and variable changes during the session. 3. Artifact (L2I). Load the saved assets in the editor without its graphical interface and check the required graph topology, class hierarchy, typed properties, references, or persistence. For each task, we specify the checks a submission must pass and evaluate them directly in Unreal Engine. The same assertions and checkpoint schedule are applied to every submission for a task. A fixed simulation timestep (1/60 second by default) makes time-dependent checks comparable, while per-task numerical tolerances accommodate the quantities being tested; Appendix B lists the engine switches and the tolerances in force. No LLM reads the submission, trace, screenshot, or test report 1 Unreal’s Functional Testing framework runs scripted checks inside a level; tests can be written in C++ or Blueprint. See Epic’s Functional Testing documentation. 4 to decide whether it passes or fails. Screenshots may be retained for diagnosis, but never to change the final grade. Structural checks establish required asset properties, while runtime checks establish the tested behavior. This excludes creative tasks whose success depends on visual judgment, a limitation we discuss in Sec. 8. 3.4THE BENCHMARK AND REQUIRED DELIVERABLES The task suite contains 33 C++ tasks, 25 Blueprint tasks, and 12 editor-scripting tasks. C++ tasks permit source code changes in a designated folder; BP tasks require saved native assets; scripting tasks are evaluated on their resulting saved project state. The tasks cover domains including game- play logic, character movement, animation, UI, and editor operations. Of the 70 tasks, 50 start from the Unreal Engine 5.8 Third Person template and 20 from a customized template. Table 1 gives their required checks; Appendix F lists the individual tasks. Figure 1 shows the three task types in our benchmark. Tasks usually contain multiple checks: glide, for example, must slow a falling character, consume Power, and restore to normal falling state when Power is drained. Editor-scripting tasks instead check the saved output, such as whether a level contains the requested objects. The harness uses each task’s declared checks since not all check types are needed for each task. Task groupTasksBuildRuntimeAssets C++3333330 BP25251520 Python1212012 Table 1: Required checks in the measured suite. Runtime and asset columns overlap; each paired BP gameplay task requires both. Ten gameplay specifications have C++ and BP versions that share a starting project template and runtime verifiers. They contain the same interface and gameplay instruction; the only difference is the requested deliverable type (BP and C++). BP additionally requires saved assets. We inspect the submitted files when interpreting the comparison. §4 defines the paired analysis. 3.5TASK AUTHORING AND VALIDATION The tasks are designed around common Unreal workflows and informed by the team’s engine exper- tise and public engine documentation. Team members, including artists, designers, and engineers, propose behaviors that represent the work they use agents to perform. The lead author filters pro- posals using three admission criteria: the requested outcome must be (i) specific enough to verify with explicit checks, (i) observable in a clean project without inspecting a solution, and (i) imple- mentable through a declared deliverable. When a gameplay behavior is suitable for both C++ and Blueprint, it can become a matched pair with different artifact requirements. The team refines each accepted proposal into a task package that includes instructions, starting project, grading criteria, and other necessary information for agent to start the implementation. We verify that there are at least one reference solution that passes all required checks and that the un- modified starting project fails. Appendix A follows one task from its instruction to the final grading pass. 4EXPERIMENTAL DESIGN 4.1MODELS AND TOOL CONFIGURATIONS We evaluate Claude Sonnet 5, DeepSeek V4 Pro-0813, Gemini 3.7 Flash, GPT-5.6 Luna, GPT-5.6 Sol, Grok 4.6, and GLM-5.3 Flash. The two editor-tool configurations use Unreal MCP (MCP1) and Aura MCP (MCP2). Each model attempts every task under both configurations; C++ also has a file/shell baseline without editor MCP. All configurations use Claude Code with Unreal Engine 5.8 on Windows. Appendix B records model details. 5 MCP1 exposes catalog discovery and a gateway for editor operations. MCP2 exposes operations directly, including delegated Blueprint and Python tools. The two MCPs provide different editor tools and file-editing permissions, as summarized in Table 2. On BP and Python, their completion differences are therefore not attributable to editor tools alone (Appendix B). ConfigurationC++ tasksBP and Python tasks File/shell baselineRead, write, edit, shell; no editor MCPNot evaluated MCP1File/shell tools and editor MCPFile/shell tools and editor MCP MCP2File/shell tools and editor MCPRead/search and editor MCP; no native write/edit/shell Table 2: Agent tool permissions. Both editor configurations can perform several kinds of project edits; tool availability is part of the evaluated configuration. 4.2TASK RUN AND COMPLETION Each task–model–MCP combination is evaluated with one scored attempt. The agent starts in a fresh project with the task prompt and is prompted to conduct the task. The prompt instructs agents to perform at most one final PIE playtest, which the harness does not enforce. We added this instruction after observing repeated playtest loops in preliminary runs. Agents receive no human feedback during authoring. We report pass@1 with a 40-minute authoring budget. An attempt is successful if the agent finishes within this budget and its final submission passes all checks. Authoring time is measured from an agent launch to its exit, and the project setup and grading time are not included. Sessions normally have a 60-minute execution ceiling; final submissions returned after the 40-minute budget but under the ceiling are retained for analysis. 4.3PAIRED COMPARISONS AND OBSERVED WORKFLOWS For each model, we compare its completion count under the two MCP settings and, on C++, also against the file/shell baseline. We also identify tasks that succeed in one setting and fail in the other, since equal totals can contain successes in different tasks. The ten paired gameplay specifications compare C++ and BP implementations of the same requested behavior. We analyze the interaction logs alongside the final test reports to examine how agents conduct their work. We also measure authoring time and the efficiency of tool calls. We compare completion using only the gameplay tests to see whether Blueprint’s extra asset checks account for its lower score. Appendix C provides the analysis details. 5RESULTS We report task completion for each model and configuration, then examine the saved submissions and tool requests to understand incomplete gameplay and the work performed during authoring. Completion uses the 40-minute budget; trajectory and runtime diagnostics examine the full recorded sessions and final submissions. Appendix E reports authoring time and tool calls. Appendix Figure 5 shows the full task result. C++ completion is 180/231 (77.9%) with file/shell tools, 199/231 (86.1%) with MCP1, and 203/231 (87.9%) with MCP2. Both MCP configurations complete 110/175 BP runs (62.9%). Python completion is 62/84 (73.8%) and 66/84 (78.6%). 5.1C++ AND BLUEPRINT IMPLEMENTATIONS OF THE SAME GAMEPLAY The ten paired gameplay tasks compare the same requested behavior across C++ and BP (Ta- ble 3). MCP1 completes 46/70 C++ runs (65.7%) and 25/70 BP runs (35.7%); MCP2 completes 50/70 (71.4%) and 20/70 (28.6%), respectively. Every model completes more C++ versions under both configurations (Figure 3; Table 7). The C++ file/shell baseline already completes 45/70 runs (64.3%), so the higher C++ completion also occurs without editor MCP on the C++ side; most of 6 ConfigurationC++ completionBP completion File/shell baseline45/70 (64.3%)— MCP146/70 (65.7%)25/70 (35.7%) MCP250/70 (71.4%)20/70 (28.6%) Table 3: Completion within 40 minutes on the same ten gameplay tasks. Seven models run each task, giving 70 runs per evaluated configuration and variant. 0246810 Passed out of 10 tasks Claude Sonnet 5 DeepSeek V4 Pro Gemini 3.7 Flash GPT-5.6 Luna-Xhigh GPT-5.6 Sol-Xhigh Grok 4.6 GLM 5.3 Flash MCP1 0246810 Passed out of 10 tasks MCP2 C++Blueprint Figure 3: C++ and BP completions within 40 minutes on the ten paired gameplay tasks. Each line connects the same model’s totals under one MCP configuration, with one attempt per task variant. each MCP configuration’s advantage over that baseline is on C++ tasks without BP counterparts (Table 8). The C++ completion advantage is 30.0 percentage points under MCP1 and 42.9 under MCP2. Counting submissions that pass the runtime checks without requiring the additional BP asset checks leaves all four completion counts unchanged (Table 9). Within the deadline, no BP submissions pass the runtime checks but fail the asset checks. The BP deficit persists when completion is assessed by gameplay testing. Producing saved native assets remains part of the BP authoring task. The submissions show how the successful implementations were authored. Although some C++ prompts also permit Blueprint assets, every successful matched C++ submission under either MCP configuration changes C++ source files without changing Blueprint assets. Successful BP submis- sions change assets without changing the source. The BP breakdown separates local edits from complete gameplay features (Appendix Figure 6). The three basic graph tasks ask for logging, arithmetic, or delayed movement; two local repair tasks target specified defects in existing behaviors. Both configurations complete 34/35 runs (97.1%) on these five tasks. On ten asset-configuration tasks, completion is 51/70 (72.9%) and 56/70 (80.0%); on the ten gameplay tasks, it falls to 25/70 (35.7%) and 20/70 (28.6%). Agents often complete the included local operations, while fewer submissions implement all parts of a gameplay feature correctly. 5.2WHERE BLUEPRINT IMPLEMENTATIONS FALL SHORT On the ten paired gameplay tasks, seven additional MCP1 BP runs and ten additional MCP2 runs yield passing submissions after the 40-minute deadline. Across all final submissions, those that pass asset checks but fail an explicit runtime assertion account for 24/70 (34.3%) under MCP1 and 28/70 (40.0%) under MCP2. Restricted to submissions whose authoring finished within 40 minutes, they 7 are 19/45 and 20/40 of the on-time submissions that pass asset checks. Appendix D.3 lists the failed checks and the available reports. The runtime reports provide further observations about these incomplete features. Among submis- sions that pass asset checks but fail runtime checks, 13 submissions across four BP tasks successfully activate an ability but fail to produce its required effect or meet the required resource check. Seven of them concern the gliding task: the character must descend more slowly while consuming Power. All seven abilities activate, and six consume Power, but none produce the required glide. For example, in three implementations, Power always activates at the start but does not exhaust after gliding for a set time. The other cases comprise four missing periodic burns, one second jump without its Power cost, and one damage operation that leaves Health unchanged. Nine of these runs finish authoring within the scoring budget. Appendix D.3 identifies the 13 submissions and the runtime observations supporting this breakdown. 5.3HOW AGENTS INSPECT AND VERIFY THEIR WORK We examine how agents respond to editing feedback and check their work under the instruction to perform at most one PIE playtest. Asset readback means retrieving an asset’s current properties or graph from the editor. PIE state readback means retrieving values from a running game, such as character position or remaining Power. Both are actions during authoring; the harness applies the task’s runtime checks to the submitted project afterward. Agents revise both rejected and accepted edits. Of the 175 MCP1 BP runs, 95 directly write Blueprint graphs; 75 read task assets before their first graph write. Among the 58 runs with graph- writing errors, 37 responded in their next tool call by changing the request for the same graph. Agents also revised graphs after requests that returned without an error: this occurred in 30 of the 94 runs with such responses. Before making these later changes, agents often inspected the graph or requested compilation. Appendix D.1 gives the detailed counts and sequence definitions. Asset readbacks are more frequent than PIE state readbacks. Most BP runs contain asset read- back or compilation requests, while PIE state readback appears in 24/175 MCP1 and 30/175 MCP2 runs (Table 4). Agents obtain these values through property queries, scripts that combine actions and measurements, and game-state recordings. C++ agents also use shell commands to request game or engine-test execution, in 57/231 MCP1 and 39/231 MCP2 runs, including unsuccessful attempts. In ten Python MCP2 runs across six tasks and five models, agents find the relevant API or tool, run an editor script that reports success, and read back task-asset state. The analysis follows these different routes through the request arguments and responses (Appendix D.1). Observed method in BP runsMCP1MCP2 Task-asset readback149/175165/175 Compilation request131/175132/175 PIE start request38/17544/175 Image-capture request15/17510/175 PIE state readback24/17530/175 Table 4: Checking actions in BP trajectories, counted over full authoring sessions. Rows overlap. Request counts include unsuccessful attempts; readback counts require returned values. PIE starts include tool calls and starts within scripts. Image requests can capture either the editor or gameplay. Appendix D.1 defines the categories and their coverage. Success revisions do not guarantee correct gameplay. In 39 MCP1 Blueprint runs that finished within 40 minutes, every graph-writing error was followed by a changed request for the same graph that returned without an error. Nevertheless, 16 submissions failed the final evaluation, including 15 that failed runtime assertions across six models and eight tasks. A separate inspection of final submissions found five MCP1 Blueprint submissions with resource references that the engine could not resolve; these submissions also failed runtime checks (Appendix D.2). On the ten paired BP tasks, five MCP1 and ten MCP2 runs contain PIE state readback and pass the asset checks, yet fail the final runtime checks. Reading a value during PIE can leave other required behaviors untested. 8 6DISCUSSION Design tasks with interacting gameplay requirements. A glide feature must slow descent, con- sume Power, and restore normal falling when Power is exhausted. These requirements need to hold together: checking resource consumption alone would not detect an implementation that leaves descent unchanged. The runtime reports and graph-revision results show why individual editing op- erations are not sufficient evidence of completion. For bounded gameplay tasks, evaluation should cover each required behavior and the transitions between them within one gameplay mechanism. Implications for editor-native assets. Our findings motivate evaluating native assets in the engine systems that use them. For animation assets, materials, and scene configuration, the question is not only whether their saved properties are correct, but whether they produce the requested motion, appearance, or interaction. CraftBench-UE examines this connection in Unreal gameplay. Whether the same failure patterns occur in other engines and asset workflows remains a question for future work. Verification inside the game engine. The observed workflows suggest two complementary forms of feedback. Inspecting asset properties and compiling graphs helps agents check what they have built; running the game and measuring its state helps them check what it does. Asset readbacks are more frequent than PIE state readbacks in the retained trajectories, measured under the prompt’s instruction to perform at most one PIE playtest (§4). The runtime reports show several errors that can be identified using this method, such as unresolved references and missing or incorrect resource usage. Because game development offers many verification methods, such as screenshots, console logs, and PIE tests, how agents choose or combine these methods is worth studying with a larger sample size, especially when many methods are hints or stimuli rather than direct test feedback. For example, console logs in major game engines can be helpful but may also include unrelated information that bloats the agent context. For agent design, this suggests connecting each requested expected behavior to a set of verification methods and expected results, with a verification flow that connects these individual parts and automatically play-tests the implementations. 7CONCLUSION CraftBench-UE evaluates coding agents through their saved source and assets and the behavior those deliverables produce in Unreal Engine. Its harness reconstructs submissions in fresh projects and applies task-specific build, asset, and runtime checks. The 70-task benchmark supports comparisons across three deliverable groups. On 10 matched gameplay specifications, C++ completion exceeds BP completion under both editor configurations. Submissions that pass asset checks can still fail runtime assertions, demonstrating the value of evaluating structural requirements and gameplay be- havior together. 8LIMITATIONS Task and check scope. The 70 tasks are selected for explicit, checkable outcomes rather than sam- pled representatively from game development. They do not measure visual appeal or the integra- tion of many features into a complete game. Future work could extend the task set to long-horizon projects and creative requirements with separately validated evaluation criteria. Meanwhile, our cur- rent checks include three necessary types of tests in Unreal Engine, while future work could explore custom tests to verify generated features, potentially providing more coverage than this harness. Experimental scope. Each task–model–configuration run has one attempt, with at most one PIE playtest during authoring. Results do not estimate repeated-attempt variability, long-horizon task performance, or performance with unrestricted testing. Future work could explore more about how agents verify themselves to produce the expected outcome. Environment coverage. All experiments use Unreal Engine 5.8 on Windows through Claude Code. We have not tested whether the observed completion patterns persist across other engine versions, operating systems, or agent runtimes. 9 ACKNOWLEDGMENTS We thank the entire Aura team for their support and contributions to the benchmark tasks. ETHICS STATEMENT The authors’ organization develops MCP2, whose implementation remains proprietary. Appendix B describes its exposed tool surface. The benchmark team authors the tasks and verifiers. Both editor configurations are evaluated on the same task suite using the same verifiers, and their results are reported separately. REPRODUCIBILITY STATEMENT The harness, benchmark materials, and analysis code will be released upon publication at https: //github.com/ramenvr/craftbench-ue. Task instructions, starting-project configurations, ver- ifier, and reference implementations are included as well. Appendix D defines the trajectory analy- ses. Appendix B.2 lists the release contents. Users will need to install Unreal Engine and provide it in the environment; engine binaries and proprietary MCP internals are not redistributed. AI USE STATEMENT Generative-AI assistants supported task ideation, methodological critique, data analysis, and figure brainstorming. The rule-based pipeline in §3 produced benchmark verdicts. The authors remain responsible for the analysis, claims, and final manuscript. REFERENCES Anthropic. Introducing the Model Context Protocol. industry blog, 2024. URL https://w. anthropic.com/news/model-context-protocol. Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, and Zeno Zeng. GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?, 2026. URL https://arxiv.org/abs/2608.21833. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. Evaluating Large Language Models Trained on Code, 2021. URL https://arxiv.org/abs/2107.03374. Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating Agentic Capabilities Through Game Development. In ICML 2026, 2026. URL https://arxiv.org/abs/2602.11103. Epic Games.Unreal MCP in Unreal Editor.Unreal Engine 5.8 official docu- mentation, n.d.URL https://dev.epicgames.com/documentation/unreal-engine/ unreal-mcp-in-unreal-editor. Accessed September 14, 2026. Harbor Framework Team.Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments.Software, 2026.URL https://github.com/ harbor-framework/harbor. Lanshan He, Haozhou Pang, Qi Gan, Xin Shen, Ziwei Zhang, Yibo Liu, Gang Fang, Bo Liu, Kai Sheng, Shengfeng Zeng, Chaofan Li, Zhen Hui, Keer Zhou, Lan Zhou, and Shujun Dai. Cutscene Agent: An LLM Agent Framework for Automated 3D Cutscene Generation, 2026. URL https: //arxiv.org/abs/2604.25318. Shufan Jiang, Chios Chen, and Zhiyang Chen. GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers. ICLR 2026 workshop paper, 2026. URL https://arxiv.org/ abs/2604.02648. 10 Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In ICLR 2024 (oral), 2024. URL https://arxiv.org/abs/2310.06770. Brian La, Sejoon Chang, Ben Kim, Junyoung Bae, Aamish Ahmad Beg, Sei Chang, Gonzalo Gonzalez-Pumariega, and Kanav Goyal. GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments, 2026. URL https://arxiv.org/abs/2607.03525. Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Ky- oung Whan Choe, et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, 2026. URL https://arxiv.org/abs/2602.12670. Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. MCP-Universe: Benchmark- ing Large Language Models with Real-World Model Context Protocol Servers, 2025. URL https://arxiv.org/abs/2508.14704. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. In ICLR 2026 (poster), 2026. URL https: //arxiv.org/abs/2601.11868. OpenAI. Introducing SWE-bench Verified. industry blog, 2024. URL https://openai.com/ index/introducing-swe-bench-verified/. Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, et al. Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games. In ICLR 2026 poster, 2026. URL https://arxiv.org/abs/2506.03610. Shutong Wu and Justin P. Barnett. MCP-Unity: Protocol-Driven Framework for Interactive 3D Au- thoring. In SIGGRAPH Asia 2025 Technical Communications. Association for Computing Ma- chinery, 2025. doi: 10.1145/3757376.3771417. URL https://doi.org/10.1145/3757376. 3771417. Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, et al. MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use, 2025. URL https://arxiv.org/abs/2509.24002. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In NeurIPS 2024 (Datasets and Benchmarks Track), 2024. URL https://arxiv.org/abs/2404.07972. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineer- ing. In NeurIPS 2024 (poster), 2024. URL https://arxiv.org/abs/2405.15793. Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, and Gangyi Ding. AutoUE: Au- tomated Generation of 3D Games in Unreal Engine via Multi-Agent Systems. In Findings of the Association for Computational Linguistics: ACL 2026, p. 2341–2364, 2026.URL https://aclanthology.org/2026.findings-acl.111/. Tiantian Zhang, Kartik Ayyar, Mengsha Sun, and Lynn Gong.Using OpenGameE- val to Benchmark Agentic AI Assistants for Roblox Studio.Roblox news- room,2025.URL https://about.roblox.com/newsroom/2025/12/ opengameeval-benchmark-agentic-ai-assistants-roblox-studio. Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K. Reddy. Stop Comparing LLM Agents Without Disclosing the Harness, 2026. URL https://arxiv.org/ abs/2605.23950. 11 Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. In ICLR 2024, 2024. URL https:// arxiv.org/abs/2307.13854. A WORKED EXAMPLE OF TASK VALIDATION The glide task in Figure 4 asks the agent to slow a falling character while consuming Power, then restore normal falling when Power runs out. The runtime fixture requests ability activation and observes descent and Power at scheduled checkpoints. Checking Power alone would miss an imple- mentation that consumes the resource without slowing the character. 1 Task requirements Slow descent while glide is active Consume Power during the glide Resume falling when Power reaches zero 2 Measured checkpoints PASSFAILafter request Downward speed (cm/s) 0 800 1600 0.51.01.51.92.32.73.1 Checkpoint time (s) Power 0 50 100 0.51.01.51.92.32.73.1 Checkpoint time (s) 3 Task view and grading outcomes Manny: glide task scene PASS: all required behaviors Descent slows, Power drains, falling resumes. At 2.3 s: speed 122 cm/s; Power 14. FAIL: descent never slows Power drains, but the glide has no effect. At 2.3 s: speed 1405 cm/s; Power 26. Figure 4: Archived development/validation traces for the glide task, separate from the study’s agent runs. In the shaded post-request interval, the passing trace slows descent while Power drains; the failing trace drains Power without slowing descent. The plotted values and verdict labels are tran- scribed from a retained development capture. The Manny screenshot is a separate scene illustration, not a frame from either trace. This example illustrates why task authors check behavior after integration into the engine. Figure 1 in the main paper illustrates tasks across the three deliverable groups. 12 BEXECUTION CONFIGURATION AND SUBMISSION HANDLING The experiments use Unreal Engine 5.8 on Windows with Claude Code as the agent driver. Table 5 records the requested outer-agent model identifiers. Provider-qualified identifiers are served via the same Claude Code CLI, with its Anthropic base URL pointing to OpenRouter’s Anthropic-compatible endpoint and the OpenRouter key supplied as the CLI’s bearer token; claude-sonnet-5 is requested directly from Anthropic. The analysis manifest retains available model and driver metadata. Display labelRequested model identifier Claude Sonnet 5 claude-sonnet-5 DeepSeek V4 Pro deepseek/deepseek-v4-pro-0813 Gemini 3.7 Flash google/gemini-3.7-flash GPT-5.6 Luna openai/gpt-5.6-luna GPT-5.6 Sol openai/gpt-5.6-sol Grok 4.6 x-ai/grok-4.6 GLM-5.3 Flash z-ai/glm-5.3-flash Table 5: Requested agent model identifiers. Tool exposure. MCP1 exposes three entry points: catalog discovery, tool description, and a gate- way that calls a named operation. MCP2 exposes 176 entry points, including direct editing and delegated tools. These names include asset and Blueprint inspection, Blueprint graph editing and compilation, asset creation and editing, text and C++ file editing with in-editor Python execution, play-in-editor control, and more. A gateway can access many operations, so entry-point counts do not measure tool coverage. Table 2 gives file and shell permissions. Initialization failures count as failed attempts. Timing and comparison scope. The runner used a 60-minute execution ceiling. Completion re- quired the agent to finish within 40 minutes and its final submission to pass all required checks. The submitted project is graded afterward. The C++ baseline uses two hosts, whereas its MCP compar- isons use one. Both hosts use the same hardware configuration, each with an NVIDIA GeForce RTX 5080 GPU. On BP and Python tasks, MCP1 also retains the native write, edit, and shell tools that MCP2 does not (Table 2), and some MCP1 runs on these surfaces issue native file or shell writes, so this permission difference is exercised in the recorded sessions. Driver versions vary within and across task groups, and some records lack version metadata. These differences limit attribution of completion or timing differences to MCP access alone. Behavior (L2) checks launch the editor with -deterministic -FPS=60, which fixes the simula- tion step at 1/60 second from the first frame; a task may declare further legs at other fixed steps, which are replayed in separate sessions and must all pass. Checkpoints are evaluated against the world’s game time, counted from when play begins rather than from when the test fixture starts. Agent inputs and testing. Each run supplies the task prompt and starting project, specifies where to save changes, and instructs the agent to perform at most one PIE playtest. The limit is prompt- level only: the harness does not intercept or refuse play-start requests during authoring, so additional playtests consume the run’s authoring budget rather than being blocked; however, there is no such request in the current trajectories. Play-start requests are instead recovered from the trajectories after the run (Appendix D.1). The agent receives no human feedback or grading reports during authoring. The harness grades its saved submission afterward. Saved submissions. The harness collects changed or new files accepted by the starting project’s submission manifest and applies them to a fresh starting project. The grader applies the task’s addi- tional file and configuration constraints before evaluating the reconstructed project. This procedure makes the saved submission, rather than unsaved editor state, the object of evaluation. 13 B.1ACCESS TO EVALUATION MATERIAL Before authoring, the harness withholds reference implementations, verifier sources, and repository metadata that could expose them. It also removes other tasks’ scaffolds and replaces runtime test bodies with stubs while retaining the build definitions needed to compile the authoring project. Withheld material is restored for grading in the clean project. The harness monitors the withheld paths and excludes an attempt from scoring if it detects exposed evaluation material. Task instructions and permitted starting content remain available during author- ing. B.2RELEASE MATERIALS The release includes the harness, task instructions, starting-project setup, verifiers, reference imple- mentations, retained submissions, and analysis code. Run identifiers and source hashes connect the reported panel, check results, and trajectory classifications to their underlying records. Public tra- jectories omit credentials and private content while retaining the feedback and submission evidence needed to inspect the analyses. Users provide an Unreal Engine installation; engine binaries and proprietary MCP2 internals are not redistributed. CDETAILED RESULTS AND SCORING CHECKS This appendix gives detailed results for the 70-task suite shown in Figure 5 and its ten paired game- play specifications. Each record is identified by task, model, and configuration. The resulting panel has no missing or duplicate combinations. Completion requires overall=PASS and meets the 40- minute time budget. Build checks are L1, runtime checks of gameplay are L2, and asset checks of saved structure are L2I. BP task groups. We group BP tasks by their instructions: three basic graph tasks (logging, arith- metic, and delayed movement), two local repairs (door hitch and inventory stacking), ten asset- configuration tasks, and ten paired gameplay tasks. Asset-configuration tasks check saved structure; gameplay tasks also check runtime behavior. Counting runtime passes. The ten C++/BP gameplay pairs share runtime fixtures; BP also re- quires asset checks. Counting recorded runtime passes within 40 minutes leaves all four completion counts unchanged (Table 9). Every counted runtime pass also passes the build checks. All 20 BP submissions that fail asset checks have runtime reports, including five MCP1 and seven MCP2 submissions returned within 40 minutes. Five attempts have FAIL_NO_EDITS and no runtime outcome; they remain failures in the denominator. One MCP1 submission passes runtime but fails asset checks after 52.1 minutes. Runtime automation failures remain failures even when individual messages report successful behavior. Python input bindings. The input-binding task checks action types, six key bindings, and their modifiers across four input assets. Scores use the engine’s active DefaultKeyMappings.Mappings array in the retained submissions. These checks assess saved input configuration without simulating keypresses. DTRAJECTORY ANALYSIS D.1INSPECTION AND VERIFICATION IN AGENT TRAJECTORIES The trajectory analysis covers 693 C++ transcripts, 350 BP transcripts, and 168 Python transcripts, using the run membership and reported outcomes in Appendix C. Counts cover full authoring ses- sions unless a time limit is stated. For MCP1, the operation is read from the gateway request’s toolset and operation arguments. Requests are matched to returns by tool-use identifier and oc- currence; seven requests lack retained returns, and six nested calls in the baseline C++ traces are 14 Claude Sonnet 5 DeepSeek V4 Pro Gemini 3.7 Flash GPT-5.6 Luna-Xhigh GPT-5.6 Sol-Xhigh Grok 4.6 GLM 5.3 Flash Baseline MCP1 MCP2 Baseline MCP1 MCP2 Baseline MCP1 MCP2 Baseline MCP1 MCP2 Baseline MCP1 MCP2 Baseline MCP1 MCP2 Baseline MCP1 MCP2 010203040506070 Completed tasks 27/33 30/3319/2511/12 30/3317/259/12 26/33 28/3317/2511/12 27/3315/2512/12 23/33 29/3315/259/12 31/3317/259/12 21/33 29/3313/256/12 27/3316/2511/12 29/33 30/3318/258/12 30/3319/258/12 28/33 30/3317/259/12 30/3318/2512/12 26/33 23/3311/258/12 28/338/255/12 Baseline: file/shell tools on C++ only; MCP bars cover all three surfaces. C++ (33 tasks)Blueprint (25 tasks)Python (12 tasks) Figure 5: Task completion within 40 minutes, by model and configuration. Each model has three stacked bars: Baseline, MCP1, and MCP2. Colored segment lengths indicate the counts of com- pleted C++, Blueprint, and Python tasks. Labels inside each segment give completed tasks out of 33 C++, 25 Blueprint, or 12 Python tasks. The baseline uses file/shell tools and was evaluated only on C++, so its total bar length is not comparable with those of the MCP configurations. Table 6 gives the same counts. 15 C++ ModelFile/shell baselineMCP1MCP2 Claude Sonnet 527/33 (81.8%)30/33 (90.9%)30/33 (90.9%) DeepSeek V4 Pro26/33 (78.8%)28/33 (84.8%)27/33 (81.8%) Gemini 3.7 Flash23/33 (69.7%)29/33 (87.9%)31/33 (93.9%) GPT-5.6 Luna21/33 (63.6%)29/33 (87.9%)27/33 (81.8%) GPT-5.6 Sol29/33 (87.9%)30/33 (90.9%)30/33 (90.9%) Grok 4.628/33 (84.8%)30/33 (90.9%)30/33 (90.9%) GLM 5.3 Flash26/33 (78.8%)23/33 (69.7%)28/33 (84.8%) BP ModelMCP1MCP2 Claude Sonnet 519/25 (76.0%)17/25 (68.0%) DeepSeek V4 Pro17/25 (68.0%)15/25 (60.0%) Gemini 3.7 Flash15/25 (60.0%)17/25 (68.0%) GPT-5.6 Luna13/25 (52.0%)16/25 (64.0%) GPT-5.6 Sol18/25 (72.0%)19/25 (76.0%) Grok 4.617/25 (68.0%)18/25 (72.0%) GLM 5.3 Flash11/25 (44.0%)8/25 (32.0%) Python ModelMCP1MCP2 Claude Sonnet 511/12 (91.7%)9/12 (75.0%) DeepSeek V4 Pro11/12 (91.7%)12/12 (100.0%) Gemini 3.7 Flash9/12 (75.0%)9/12 (75.0%) GPT-5.6 Luna6/12 (50.0%)11/12 (91.7%) GPT-5.6 Sol8/12 (66.7%)8/12 (66.7%) Grok 4.69/12 (75.0%)12/12 (100.0%) GLM 5.3 Flash8/12 (66.7%)5/12 (41.7%) Table 6: Full-task completions within 40 minutes, with counts and percentages shown separately for each surface. The file/shell baseline is evaluated only on C++. ModelMCP1MCP2 C++BPC++BP Claude Sonnet 58/105/108/103/10 DeepSeek V4 Pro6/105/106/102/10 Gemini 3.7 Flash6/105/108/103/10 GPT-5.6 Luna6/101/106/104/10 GPT-5.6 Sol8/106/108/104/10 Grok 4.68/102/108/104/10 GLM 5.3 Flash4/101/106/100/10 Table 7: Per-model completion within 40 minutes on the ten paired gameplay specifications. C++ task subsetBaselineMCP1MCP2 Ten paired gameplay tasks45/7046/7050/70 Other 23 C++ tasks135/161153/161153/161 Table 8: C++ completion within 40 minutes, split by whether the specification also has a BP version. The split follows the task definitions. It is not selected from outcomes. excluded. The counts describe visible agent requests and returns; unobserved work inside tool ser- vices is outside this analysis. Asset and PIE state readback. An asset readback is a returned value from a task-owned asset, excluding errors, search results, schema descriptions, and object handles without state. A PIE state 16 0255075100 Runs completed within 40 minutes (%) Basic graph operations 3 tasks; 21 runs Local edits and fixes 2 tasks; 14 runs Asset configuration 10 tasks; 70 runs Gameplay 10 tasks; 70 runs 100.0% 95.2% 92.9% 100.0% 72.9% 80.0% 35.7% 28.6% MCP1MCP2 Figure 6: BP completion by task type. Row labels give the number of tasks and runs per configura- tion (seven models per task). Groups follow the task instructions and differ in content, starting state, and required checks. ConfigurationC++ fullBP fullBP runtime passGap (p) MCP146/7025/7025/7030.0 MCP250/7020/7020/7042.9 Table 9: Completion on the ten paired gameplay tasks, with and without the BP asset checks. Each task group has 70 attempts per configuration. All counts require authoring to finish within 40 min- utes. Counting recorded BP runtime passes leaves the full-task counts and the C++–BP gap un- changed; the original submissions are used in both counts. Gap is measured in percentage points. readback contains a value measured while the game runs in Play-in-Editor (PIE). It may come from a dedicated getter, a property or transform query targeting a UEDPIE object, an inspected script, a combined action sequence, or a recording. A PIE start request counts the request to start the game, including starts within identified scripts. A start request alone is insufficient for a state readback. Table 4 reports these methods for all BP runs; Table 10 separates them by model. Images count separately and require an image block in the agent’s conversation: seven and eight runs qualify, respectively. Script-based readbacks require returned measurements rather than a printed success statement. Compilation counts requests; a return without an error, including a null return, does not establish successful compilation. On C++, 151 reviewed shell requests ask for game or engine-test execution, covering 57/231 MCP1 and 39/231 MCP2 runs. Two further requests only list available tests and are excluded. The counts include failed starts and background requests whose completion is unconfirmed. These shell opera- tions are separate from PIE start requests. On Python, ten MCP2 runs across six tasks and five models contain tool/API discovery, a task- directed modification script, an explicit bSuccess:true return, and subsequent asset readback in the task’s namespace. The readback need not concern the modified field. MCP1 has a different return protocol, so this sequence is not used to rank script success between configurations. Graph revisions. All 175 MCP1 BP trajectories are screened for direct graph writing. The 95 qualifying runs contain 454 writes with complete request–return matching; 75 read task assets before their first graph write. An accepted graph write returns without a reported tool error. A revision 17 PIE start requestPIE state readbackImage return ModelMCP1MCP2MCP1MCP2MCP1MCP2 Sonnet1718121115 DeepSeek462420 Gemini241300 Luna020100 Sol000000 Grok111100 GLM141381043 Table 10: BP inspection methods by model, out of 25 runs per configuration. PIE start requests include identified scripts. PIE state readbacks require returned measurements from the running game. An image must appear in the agent’s conversation. Rows use the definitions in Appendix D.1. changes the arguments for the same graph after the previous write returns. Six malformed writes have no recoverable graph identity and cannot be paired. Immediate means that the revision is the next agent tool request, rather than a short elapsed time. Table 11 distinguishes revisions after accepted and rejected writes, together with intervening read- backs and compilation returns. Revisions can extend functionality as well as correct errors. Observed sequenceAll (95)Same tasks (49)≤ 40 min (79) Accepted write, then changed same-graph write301620 Readback between those writes271517 Compilation return between those writes211112 Both readback and compilation return18109 Error, then immediate same-graph change372026 Acceptance, then immediate same-graph change414 Table 11: Graph revisions in MCP1 BP runs. Columns cover all direct graph-writing runs, the seven tasks on which every model directly writes graphs, and sessions ending within 40 minutes. Rows overlap. A compilation return need not report successful compilation. Immediate means the next agent tool request after the preceding write returns; 58 runs contain errors, and 30 revise accepted writes. Table 12 links graph-writing errors and subsequent accepted revisions to submission outcomes. Of the 16 on-time failures after every reported error receives an accepted revision, 15 fail explicit run- time checks, spanning six models and eight gameplay tasks. Table 12: Graph-writing errors and subsequent revisions in MCP1 BP runs. Rows successively restrict the 175-run panel. Accepted revisions return without a reported tool error. On-time passes finish authoring within 40 minutes; final passes include later submissions. SequenceRunsOn-time passFinal pass Uses direct graph writing955562 Receives graph-writing error582734 Changed same-graph request accepted552632 Every error has a later accepted change502329 Previous row, session ends within 40 min392323 D.2UNRESOLVED RESOURCE REFERENCES Five MCP1 BP submissions across five tasks and two models contain both an unresolved-resource warning and a failed runtime check. Two omit the resource’s owning class; three use a class-qualified name that the engine cannot resolve. The warnings establish unresolved references, but need not explain every failed check: the healing warning concerns Health, while the failed check concerns MaxHealth initialization. The census covers retained runtime reports from 101/105 MCP1 and 18 103/105 MCP2 runs on the 15 BP tasks that declare runtime checks, including 66/70 and 69/70 paired gameplay runs. UE 5.8 source inspection identifies two supported lookup routes: an owning class plus a member name, or a valid full field path. No submission was modified to test whether resolving its reference would complete the task. D.3RUNTIME FAILURES AFTER PASSING ASSET CHECKS Of the 140 BP attempts on matched gameplay tasks, runtime reports are retained for 66/70 under MCP1 and 69/70 under MCP2. The five attempts without reports have FAIL_NO_EDITS outcomes. We select submissions that pass asset checks and fail an explicit runtime assertion: 24/70 and 28/70. One additional MCP1 automation failure has no such assertion and is excluded from this category. Failure after passing asset checksMCP1MCP2 Initial Health or Power34 Required effect or resource use86 Pose, display, or object indicator58 Restoration or preserved behavior56 Poison damage ratio34 Total attempts24/7028/70 Authoring finished within 40 minutes19/7020/70 Table 13: BP submissions that pass asset checks but fail a runtime check. Each of the 52 selected attempts has one observed failure category; subsequent checks may not execute if a previous check failed. Selected BP submissionsMCP1MCP2 On-time submissions passing asset checks4540 Runtime checks pass2520 Explicit runtime assertion failure19/45 (42.2%)20/40 (50.0%) Runtime automation failure10 Table 14: Runtime outcomes among BP submissions on the ten gameplay specifications that pass asset checks and finish authoring within 40 minutes. The last three rows partition this selected subset; automation failures are separate from failed assertions. Categories group the state measured by failed assertions, with duplicate messages removed within each attempt. Each of these 52 attempts has one observed category; later checks may not execute. The reports evaluate the final saved submissions. Thirteen effect or resource failures outside poison stacking also show the ability got activated suc- cessfully: seven under MCP1 and six under MCP2, across area burn, double jump, glide, and Health operations. Seven concern gliding; six consume Power before any verifier-induced exhaustion. Some reports contain no valid glide sample or no forced-exhaustion stage, so these observations do not establish a common failure mechanism across all seven submissions, but show how agents might fail when implementing features. Eight versioned verifier source objects establish how ac- tivation and resource use were measured. C++ failure selection requires passing build checks; BP selection requires passing asset checks. EAUTHORING TIME AND COST For all seven models, median C++ authoring time is longer with either MCP configuration than with the Baseline (Figure 7). Durations include unsuccessful runs and sessions that exceed the 40-minute deadline; completion still requires finishing within that deadline. Input context and billed cost. Of 231 C++ MCP task–model pairs, 229 have identical retained task prompts and 185 have both first requests linked to billing records. In these 185 pairs, median 19 Claude Sonnet 5 DeepSeek V4 Pro Gemini 3.7 Flash GPT-5.6 Luna GPT-5.6 Sol Grok 4.6 GLM 5.3 Flash 131030 Median duration (minutes; log scale) C++ authoring durationCompletion 27/33 (81.8%) 30/33 (90.9%) 30/33 (90.9%) 26/33 (78.8%) 28/33 (84.8%) 27/33 (81.8%) 23/33 (69.7%) 29/33 (87.9%) 31/33 (93.9%) 21/33 (63.6%) 29/33 (87.9%) 27/33 (81.8%) 29/33 (87.9%) 30/33 (90.9%) 30/33 (90.9%) 28/33 (84.8%) 30/33 (90.9%) 30/33 (90.9%) 26/33 (78.8%) 23/33 (69.7%) 28/33 (84.8%) All 33 C++ runs per model/configuration; duration includes failures. File/shellMCP1MCP2 Figure 7: C++ completion and median authoring time on the same 33 tasks per configuration. Mark- ers show median duration on a logarithmic scale; labels give completion counts. All runs enter the duration calculation. first-request input is 3.99 times as large under MCP2 as under MCP1, a median difference of 82,354 tokens. Each configuration retains its own system instructions and tool descriptions. Full-session billing is available for 162 prompt-matched pairs (Table 15). The median paired MCP2/MCP1 charge ratio is 0.21 for DeepSeek and 1.80–2.69 for the other five covered models. Cache use also differs by model: among 19 DeepSeek pairs with complete token records, the median share of input served from cache is 3.47% under MCP1 and 98.21% under MCP2; for Sol’s 33 pairs, it is 92.12% and 93.17%. ModelPairsMCP1MCP2Charge ratioHigher Claude Sonnet 50/33— DeepSeek V4 Pro20/3318/2019/200.21×6/20 Gemini 3.7 Flash32/3328/3231/321.86×23/32 GPT-5.6 Luna29/3327/2925/292.48×29/29 GPT-5.6 Sol33/3330/3330/332.69×33/33 Grok 4.631/3328/3129/311.80×29/31 GLM-5.3 Flash17/3315/1716/172.09×17/17 C++ linked subset162/231146/162150/1622.21×137/162 Table 15: C++ completion and billed inference on the matched subset. Completion columns count passes within 40 minutes. Charge ratio is the median paired MCP2/MCP1 ratio; Higher counts pairs with larger MCP2 charges. Pair counts give billing coverage out of 33 tasks per model. 20 Accounting. We match generation IDs from CLI summaries and assistant messages to Open- Router billing records, require the requested model and a numeric charge for every observed gener- ation, deduplicate IDs across four ledgers, and sum cost_details.upstream_inference_cost. This excludes unrelated projects in the same ledgers. On fully linked C++ runs, CLI cost esti- mates exceed charges by model-dependent median factors of 1.25–54.85; we use the billed amounts directly, without a conversion factor. Charges cover full agent sessions, including requests after the deadline, and exclude engine compute and unobserved inference inside tools. BP, Python, and direct-Anthropic Sonnet runs have no usable billing linkage and are omitted from cost comparisons. Input totals add the ledger’s separate uncached, cache-read, and cache-write fields; cache shares use that total as the denominator. The result-summary input field already includes cached tokens and is never added to its cache fields. FTASK CATALOG AND PAIRED GAMEPLAY SPECIFICATIONS Table 16 lists every task in the primary study, its starting project, and its required checks. The 50 Third Person tasks use Unreal’s standard template; the other 20 use the CraftBench template. Task names are directory identifiers so that a reader can locate the instructions and verifier directly. Table 17 gives the requested behaviors in the 10 C++/BP pairs. Their task definitions name the same interface and verifier, while BP additionally requires saved BP graphs rather than C++. The task bank also contains 47 additional tasks outside the primary study that were excluded due to the time and budget limit. They do not enter the reported completion rates or trajectory counts. Table 16: The 70 study tasks, with their starting projects and required grading gates. IDs refer to task directories. Build, runtime, and inspect denote L1, L2, and L2I;✓ marks a required check and — marks one not required. The C++, BP, and Python groups contain 33, 25, and 12 tasks. RowTask IDStarting projectBuild Runtime Inspect C++ source tasks C01 gp-crafting-queueCraftBench template ✓— C02 gp-dot-aoe-burn-cppThird Person✓— C03 gp-double-jump-stamina-cppThird Person✓— C04 gp-glide-stamina-cppThird Person✓— C05 gp-harvestable-regrowCraftBench template ✓— C06 gp-heal-over-time-cppThird Person✓— C07 gp-health-attribute-ops-cppThird Person✓— C08 gp-inventory-stackingCraftBench template ✓— C09 gp-poison-dot-stack-cppThird Person✓— C10 gp-spawner-populationCraftBench template ✓— C11 t0-sanity-log-on-beginplayCraftBench template ✓— C12 t1-data-asset-drives-speedCraftBench template ✓— C13 t1-datatable-drives-valueCraftBench template ✓— C14 t1-default-cube-mesh-actorCraftBench template ✓— C15 t1-extraction-volume-per-actor- trigger CraftBench template ✓— C16 t1-gameplay-tag-gateCraftBench template ✓— C17 t1-movement-component-drives-actorCraftBench template ✓— C18 t1-mud-wade-cppThird Person✓— Continued on next page 21 Table 16 (continued) RowTask IDStarting projectBuild Runtime Inspect C19 t1-overlap-logs-onceCraftBench template ✓— C20 t1-overlap-teleport-portalThird Person✓— C21 t1-physics-drop-and-restCraftBench template ✓— C22 t1-screen-tint-cppThird Person✓— C23 t2-gravity-floating-pawn-movementCraftBench template ✓— C24 t2-homing-projectileCraftBench template ✓— C25 t2-hud-layout-and-countdownThird Person✓— C26 t2-ladder-climb-volumeThird Person✓— C27 t2-melee-ability-with-cooldownCraftBench template ✓— C28 t2-npc-follows-playerThird Person✓— C29 t2-race-clock-cppThird Person✓— C30 t2-timeline-color-cycleCraftBench template ✓— C31 t2-weapon-fire-animation-on-triggerThird Person✓— C32 t3-gate-and-door-cppThird Person✓— C33 tp2-sprint-staminaThird Person✓— Blueprint/native-asset tasks B01 gp-additem-stack-fix-bpThird Person✓— B02 gp-door-hitch-fix-bpThird Person✓— B03 gp-dot-aoe-burn-bpThird Person✓ B04 gp-double-jump-stamina-bpThird Person✓ B05 gp-glide-stamina-bpThird Person✓ B06 gp-heal-over-time-bpThird Person✓ B07 gp-health-attribute-ops-bpThird Person✓ B08 gp-poison-dot-stack-bpThird Person✓ B09 t0-sanity-bp-log-on-beginplayCraftBench template ✓— B10 t1-blueprint-event-to-actionCraftBench template ✓— B11 t1-blueprint-graph-on-beginplayCraftBench template ✓— B12 t1-dawn-fog-lighting-rigThird Person✓—✓ B13 t1-hero-blueprint-copy-with- flashlight Third Person✓—✓ B14 t1-mud-wade-bpThird Person✓ B15 t1-playable-level-bootstrapThird Person✓—✓ B16 t1-screen-tint-bpThird Person✓ B17 t1-third-person-chase-cameraThird Person✓—✓ B18 t1-walk-animation-footstep-cuesThird Person✓—✓ B19 t2-consistent-enum-namesThird Person✓—✓ B20 t2-cutscene-camera-push-and-hero-riseThird Person✓—✓ B21 t2-datatable-csv-exportThird Person✓—✓ B22 t2-race-clock-bpThird Person✓ B23 t2-weapon-held-in-right-handThird Person✓—✓ B24 t3-gate-and-door-bpThird Person✓ B25 t3-piercing-projectileThird Person✓—✓ Editor-scripting tasks P01 kp-anim-track-bakeThird Person✓—✓ P02 kp-blueprint-actor-audit-reportThird Person✓—✓ P03 kp-character-boom-and-movementThird Person✓—✓ P04 kp-config-source-auditThird Person✓—✓ P05 kp-derived-class-searchThird Person✓—✓ P06 kp-engine-source-searchThird Person✓—✓ P07 kp-fog-and-postprocess-rigThird Person✓—✓ P08 kp-motion-set-shares-one-rigThird Person✓—✓ Continued on next page 22 Table 16 (continued) RowTask IDStarting projectBuild Runtime Inspect P09 kp-retarget-maps-two-rigsThird Person✓—✓ P10 kp-routine-usage-searchThird Person✓—✓ P11 kp-spawn-level-actorsThird Person✓—✓ P12 kp-typed-input-bindingsThird Person✓—✓ Task familyRequested behaviorCatalog rows Area burnApply periodic damage inside a bounded area and stop after its dura- tion. C02 / B03 Double jumpReverse a fall with an upward impulse; debit Power once and reject insufficient Power. C03 / B04 GlideSlow descent while draining Power and restore normal falling when Power is exhausted. C04 / B05 Healing over timeRestore Health repeatedly for a limited duration without exceeding MaxHealth. C06 / B06 Health operationsInitialize writable Health and provide repeatable damage and healing abilities. C07 / B07 Stacking poisonApply periodic damage with up to three stacks and refresh the duration on reapplication. C09 / B08 Mud wadingSlow movement and change animation in mud, restore both afterward, and preserve the unaffected character. C18 / B14 Speed-driven tintTrack movement speed relative to the current maximum while preserv- ing an unrelated screen setting. C22 / B16 Race clockRun a timed coin-collection round with live readouts, final scoring, and restart behavior. C29 / B22 Gates and doorOpen each new gate for its required pair of named crates while preserv- ing the existing door behavior. C32 / B24 Table 17: The ten gameplay specifications with C++ and Blueprint counterparts. Each pair names the same starting project, runtime map, and test class in its task definitions. Both counterparts require build and runtime checks; the Blueprint counterpart also requires artifact inspection. These summaries describe the requested behavior; the task prompts specify the acceptance tolerances and deliverables. 23 GCOMPARISON WITH GAME-DEVELOPMENT BENCHMARKS Table 18 compares what agents are asked to produce and the evidence used to evaluate it. Different task scopes support different claims: atomic gameplay checks, visual cutscene quality, and complete game generation are not interchangeable outcomes. BenchmarkSetting and task sourceAgent’s deliverableGrading evidence GameDevBench (Chi et al., 2026) Godot; 333 tasks from web/video tutorials Scripts, scenes, resourcesIn-engine tests of behavior and scene state GameEngineBench (La et al., 2026) Unreal; 110 tasks from nine projects C++ changes in existing projects Build and runtime tests, followed by an LLM behavioral judge OpenGameEval (Zhang et al., 2025) Roblox; expert-authored tasks (47 in the initial release) Scripts and changes to existing experiences Executable tests in Roblox Studio GBQA (Jiang et al., 2026) Web games; 124 human-verified injected bugs in 30 agent-built games Bug reports from interactive exploration LLM critic matches reports to ground truth; recall of verified bugs CutsceneBench (He et al., 2026) Unreal; 65 authored cutscene scenarios Level Sequence assetsTool and sequence checks; model-judged quality on 25 sampled videos per model AutoUE (Yin et al., 2026) Unreal; complete game generation Game projects with scenes and gameplay Automated play-testing; LLM-judged game quality GameXpert-Bench (Chen et al., 2026) No prescribed engine, browser JavaScript; 97 generation tasks, 100 repair tasks from 50 human-verified levels, 102 optimization turns All game source files, written from a blank workspace; source patches for repair Deterministic headless-browser probes (repair); evaluation-agent rubric with human rating (generation); hidden-rubric judging (optimization) CraftBench-UEUnreal; 70 bounded gameplay/editor tasks Source changes, native assets, editor-script outputs Task-specific build, runtime, and saved-asset checks; no LLM judge Table 18: Benchmark designs, using the cited releases. The OpenGameEval count refers to its initial release; its task bank has since expanded. CraftBench-UE also includes 10 C++/BP gameplay pairs and evaluates two editor MCP configurations. 24