Paper deep dive
OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine
Eray Turkel, Mengsha Sun, Kartik Ayyar, Sean Dunigan, Jack Lu, Vlad Shcherban, Hsiang-Shun Shih, Xin Wang, Tiantian Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/5/2026, 6:27:44 AM
Summary
The paper introduces OpenGameEval, a benchmark and evaluation framework for agentic game development within the Roblox Studio environment. It evaluates 13 frontier language models on 84 human-curated core tasks, measuring both final task success and the agent's exploration behavior by separating observation tools from editing tools. The study finds that current models struggle with these tasks, with the best model solving only 51.7% on a single attempt. Crucially, the paper demonstrates that an agent's exploration behavior—specifically, inspecting relevant objects before acting—strongly predicts task success, independent of the model's overall capability.
Entities (29)
Relation Signals (25)
OpenGameEval → runson → Roblox Studio
confidence 100% · agentic game development inside Roblox Studio.
GLM 4.7 → achievespass1 → 36.7%
confidence 95% · GLM-4.7 36.7
GPT-5.4 → achievespass1 → 40.0%
confidence 95% · GPT-5.4 40.0
GPT-5.6 Luna → achievespass1 → 40.3%
confidence 95% · GPT-5.6-luna 40.3
GPT-5.6 Terra → achievespass1 → 40.4%
confidence 95% · GPT-5.6-terra 40.4
GLM-5 → achievespass1 → 42.9%
confidence 95% · GLM-5 42.9
Claude Opus 4.7 → achievespass1 → 45.5%
confidence 95% · Claude Opus 4.7 45.5
GPT-5.5 → achievespass1 → 46.1%
confidence 95% · GPT-5.5 46.1
GPT 5.6 Sol → achievespass1 → 47.3%
confidence 95% · GPT-5.6-sol 47.3
Claude-fable-5 → achievespass1 →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in reproducible, stateful game-engine sessions and scores each run with executable checks, both on the edited scene and in a simulated play session. Most agentic coding benchmarks require exploration but score only final task success. OpenGameEval separates observation tools from editing tools in its eight-tool action space, so exploration can be measured directly. We measure the pass rates and exploration behavior of 13 frontier models on 84 human-curated core tasks, with 16 attempts per task. The tasks are hard for current models. The best model solves 51.7% of tasks on a single attempt and 39.4% five times out of five, and no tested model solves six of the tasks. Models at the frontier reach similar pass rates by solving different tasks: splitting tasks by the kind of work they require spreads the top five by 5.0pp on script-authoring tasks and 12.5pp on scene-change tasks. Exploration behavior predicts whether a run succeeds. Holding task and model fixed, a run that inspects every object a reference solution touches before acting on it passes 13.4pp more often than a run that inspects none of them on scene-only tasks, and 9.8pp more often on script-only tasks. We release the task suite, its place files, the per-task annotations, a plugin that runs the tasks inside Roblox Studio, and an updated leaderboard under the MIT license at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.02563v1
- Canonical: https://arxiv.org/abs/2610.02563v1
Trouble viewing inline? Open PDF directly →
Full Text
102,387 characters extracted from source content.
Expand or collapse full text
OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game EngineThanks: Large language model tools assisted with software development, data-analysis implementation, and manuscript editing. The authors directed the research and verified all code, results, and claims. Eray Turkel †thanks: Corresponding author: eturkel@alumni.stanford.edu. Mengsha Sun Kartik Ayyar Sean Dunigan Jack Lu Affiliation: Vlad Shcherban Hsiang-Shun Shih Xin Wang Tiantian Zhang Affiliation: Roblox Abstract We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in reproducible, stateful game-engine sessions and scores each run with executable checks, both on the edited scene and in a simulated play session. Most agentic coding benchmarks require exploration but score only final task success. OpenGameEval separates observation tools from editing tools in its eight-tool action space, so exploration can be measured directly. We measure the pass rates and exploration behavior of 1313 frontier models on 8484 human-curated core tasks, with 1616 attempts per task. The tasks are hard for current models. The best model solves 51.7%51.7\% of tasks on a single attempt and 39.4%39.4\% five times out of five, and no tested model solves six of the tasks. Models at the frontier reach similar pass rates by solving different tasks: splitting tasks by the kind of work they require spreads the top five by 5.05.0p on script-authoring tasks and 12.512.5p on scene-change tasks. Exploration behavior predicts whether a run succeeds. Holding task and model fixed, a run that inspects every object a reference solution touches before acting on it passes 13.413.4p more often than a run that inspects none of them on scene-only tasks, and 9.89.8p more often on script-only tasks. We release the task suite, its place files, the per-task annotations, a plugin that runs the tasks inside Roblox Studio, and an updated leaderboard under the MIT license at https://github.com/Roblox/open-game-eval. 1 Introduction Code-generating language models are mostly evaluated on outcomes: a model receives a problem, and a fixed oracle checks its output. HumanEval and MBPP score function bodies against unit tests (Chen et al., 2021; Austin et al., 2021), and SWE-bench and its verified subset score patches against a repository’s own test suite (Jimenez et al., 2024; Chowdhury et al., 2024). This approach has driven substantial progress, but it assumes that the final outcome is the only quantity worth measuring. The same passing result can come from an agent that understood the system or from one that guessed, and the outcome alone cannot tell them apart (Zhu et al., 2025; Bean et al., 2025). We study a domain where that assumption breaks down, and we focus on exploration: what a model finds out about the world it is working in. Consider an AI coding assistant inside a game engine that is asked to “implement health regeneration that begins two seconds after damage and restores ten health per second” in an existing game. The prompt does not say which scripts manage health, whether damage is server-authoritative, or where in the object hierarchy the player rig lives. The agent has to find these facts itself, by searching the object hierarchy, reading the scripts already in the game, and inspecting the properties of the objects it finds. Correctness is also a play-time property. It can only be observed once the game runs, the damage event fires, and the client sees health climb on the right schedule. We introduce OpenGameEval, a benchmark and evaluation framework for agentic game development that runs an LLM agent inside a reproducible editor session. Four design properties together let us measure exploration, not just outcomes: (i) a stateful world with hidden structure that the agent must discover through tools (Figure 1), (i) an instrumented eight-tool action space that separates observation from action, (i) play-time verification with real physics, networking, and client–server dynamics, and (iv) human-written reference solutions annotated with the objects and scripts a correct run should consult. We evaluate 13 models on the 84-task core suite, with 16 attempts per task, and report three main results. First, the tasks are hard for current models. The strongest model solves 51.7%51.7\% of core tasks on a single attempt and only 39.4%39.4\% five times out of five. No model solves six of the core tasks in any of their 16 attempts, although each task’s human-written reference solution passes its checks. Second, the models at the frontier do not fall into a simple order. Five models finish within 1.11.1p of one another at about 51%51\%, but they get there by solving different tasks. All five solve only 27 of the 84 core tasks, while at least one of them solves 5858 (§6). Scoring each kind of work separately shows where they differ: the spread among the same five models grows from 1.11.1p to 5.05.0p on script-authoring tasks and to 12.512.5p on scene-change tasks (Figure 2). A single average pass rate hides how different these models are. Third, how a run explores predicts whether it succeeds. The first two results concern which tasks a model solves, and this one concerns the trajectory. Motivated by the failure modes we observe in game development tasks, we ask whether an agent’s exploration predicts the outcome, and find that it does on both kinds of work. On tasks whose solutions have annotated dependencies, and holding task and model fixed, inspecting the expected objects before acting on them is associated with a 13.413.4p higher chance of passing on scene-only tasks (95%95\% CI [+1.2,+25.7][+1.2,+25.7]) and a 9.89.8p higher chance ([+3.6,+16.0][+3.6,+16.0]) on script-only tasks, whose work is only to author script text (§7). The measurement scheme is not specific to game development. Any evaluation of tool-using agents that labels its tools as observation or action can compute the label-based features, and one that also annotates each task’s dependencies can compute the coverage signal. We release all task definitions, the place files they run against, the per-task annotations used in §7, a plugin that runs the tasks in Roblox Studio, and the leaderboard under the MIT license at https://github.com/Roblox/open-game-eval. 2 Related Work Agent evaluation has become a broad subfield, with its own taxonomies of capabilities, environments and benchmark dimensions (Yehudai et al., 2026). We know of no prior work that combines three elements: an agent working inside a deployed game engine, correctness judged by how the running game behaves under simulated player input and unit tests, and the agent’s exploration instrumented as a measured signal. §B covers the related literature on code generation and benchmark validity. WebArena, OSWorld and AgentBench (Zhou et al., 2024; Xie et al., 2024; Liu et al., 2024) evaluate agents in live, stateful environments, and GAIA (Mialon et al., 2024) evaluates assistants on real-world questions that need web browsing and tool use. All of them report end-to-end success, and none is built around writing software. Terminal-Bench (Merrill et al., 2026) is, with agents working inside a stateful container, but by design its tests check only the final state, not the agent’s commands. None of these benchmarks annotates what a correct solution has to consult. AgentBoard (Ma et al., 2024) also looks beyond binary success. Across its nine tasks it reports a progress rate that reaches one at completion. This rate is the running maximum, over steps, of how close the agent’s state has come to the goal, so it is a finer-grained measure of outcome. We instead measure a property of the investigation: the fraction of a task’s annotated dependencies the agent consulted. Park et al. (2026) study exploration directly. They build partially observable grid environments over an unknown task DAG, evaluate 1313 frontier models, and find that exploration error strongly predicts success across models (R2=0.95R^2=0.95 for log exploration error). Consistent with our results, they show that models with indistinguishable success rates behave in qualitatively different ways. The level of analysis differs: they report a correlation across models, while we relate exploration to success across repeated runs with task and model held fixed. GameDevBench (Chi et al., 2026) is the closest work to ours. Its verification is also in-engine and deterministic, decided by Godot unit tests rather than by a judge. Its verifiers inspect scene and runtime state. Ours drive a gameplay session with simulated player input and assert on what the client observes, which requires a running server and a driven client (§A). Our focus on exploration and trajectories is the other main difference. GameCraft-Bench (Luo et al., 2026) instead scores the delivered game with a multimodal judge (§B). On the process side, Mehtiyev & Assunção (2026) analyze 9,3749,374 trajectories from 1919 agents on 500500 SWE-bench Verified tasks. After controlling for task difficulty, what remains predictive is trajectory structure: agents that gather context before editing and invest in validation succeed more often. Their behavioral features are the same for every task. Ours are scored against each task’s own reference solution and annotated dependencies, so we can ask whether the agent investigated the objects and scripts that a specific task requires, not just how much it investigated in general. 3 The OpenGameEval Framework OpenGameEval is designed for fidelity (physics, networking, and gameplay behave as a creator or player would experience them), harness agnosticism (any agentic system can be evaluated on identical tasks without modifying the environment), and verifiability (machine-checkable assertions decide success). Each evaluation opens a real placefile, a base game inside the editor, and runs a candidate agent inside it. The framework ships seventeen curated Roblox experiences, ranging from an empty baseplate to a racing game, a laser-tag arena, a suburban town and a modern city. The 113113 scored tasks use eleven of them: 7070 tasks are set in a populated game and 4343 start from a bare placefile (4242 baseplates and one flat terrain). Every task has a human-written reference solution and unit tests. The public repository includes 6767 of the 113113 reference solutions, and the rest are withheld. The agent works through eight tools, which we group into observation and action. The engine calls every node of a place’s object hierarchy (a part, a model, a folder, a script, a service) an Instance. The tool names and our annotations use this term, so from here on we write “instance” for what the earlier sections called an object. Observation (reading hidden state) Action (mutating state) search_game_tree (traverse scene graph) multi_edit (edit script source) read (read a script) execute_luau (run engine code) grep (content search) insert_from_creator_store search (fuzzy name match) inspect_instance (read properties) Separating observation from action is the paper’s main methodological choice. It lets us say where two models with equal pass rates differ (in observation, in action, or in the mix between them) instead of reporting a single tool-call count. §A describes the task lifecycle, the two-phase check structure, the authoring pipeline, and what can be reproduced from the release. Every result in this paper is a property of the model together with one fixed scaffold, which we describe here (§A gives details). Each attempt delivers the task prompt as a single user turn. The model then runs its own tool-calling loop: tool calls are accepted automatically, there is no follow-up prompting, and the whole conversation is carried forward without truncation. A failed tool call is not retried. Its error message is returned to the model as an ordinary tool result. The loop ends when the model stops calling tools, when the 300300 s wall-clock budget runs out, or when the run reaches 200200 tool calls, whichever comes first. Almost every attempt ends the first way: 98.6%98.6\% of core attempts stop because the model stopped calling tools, the remaining 1.4%1.4\% run out of time, and no run reaches the tool-call ceiling. All thirteen models use the same system prompt, which is based on the Roblox assistant’s prompt and cannot be released. Sampling parameters are fixed per model family, not per model or per run, and reasoning effort is set to high for every family that accepts a reasoning level. All attempts were collected between May and July 20262026. Each task’s checks run in two phases. check_scene asserts properties of the edited world, and check_game asserts behavior in a play session started after the edits. A task passes only if every phase passes (§A). 4 The OpenGameEval Benchmark The 113113 scored tasks are what remains of 117117 after the check-validity screens of §8. They form two suites that we score separately: 8484 core tasks and 2929 debugging tasks. §H lists all 113113 with their prompts, place files, and check phases. A core task gives the model a place and a feature request, and the model must decide what to build and where. Most core tasks open on a populated game (6363 of 8484). The other 2121 (2020 baseplate, 11 flat terrain) start the model in an empty world, where the work is construction rather than discovery. A debugging task gives the model a place with a planted defect and asks for a repair, so the target is located and verifiable. The core suite is our primary object of analysis. Every estimate, interval and test is computed within one suite and never pooled across the two. The debugging suite appears next to the core partitions in Figure 2, and Table 4 (§C) reports it per model. No model solves six of the 84 core tasks in any of their 16 attempts. All six are solvable: each task’s own reference solution passes its checks, so no check is impossible to satisfy. We also read each of the six checks against its prompt and found no mismatch. Four more tasks are solved on every attempt. Together, the two groups shrink the usable core suite to the 7474 tasks that carry any between-model information (§E names all ten). Every estimate still runs over all 8484: the ten change no model’s ranking and add nothing to a paired comparison. Figure 1 walks through one core task from start to finish. Figure 1: A worked core task: 080_surburban_school_lights_on. The entire prompt is “Make every light in the school start in the ‘on’ state.” Left: the scene as the agent receives it. Center: the editor’s object explorer at Workspace/School/Lights. The prompt names none of the eight per-area subfolders, the LightsOn boolean each carries, or the switch scripts that read it. Right: the classroom with the lights on. The check reads the resulting instance state (§A). Pass@1 spans 0%0\% to 100%100\% across the 1313 models. Two kinds of work: script authoring and scene change. We also divide the core suite by the kind of work each task’s reference solution requires, using our annotations. A task is script-authoring if its solution requires multi_edit, meaning it must write or rewrite script text. Every other task is scene-change: it is completed without writing script text, by running Luau code that mutates the live game scene. This rule labels every core task and splits the suite 5959 to 2525. The two classes are a requirement and its complement, not two disjoint activities. Of the 5959 script-authoring tasks, 2929 also require execute_luau. All 2525 scene-change tasks require execute_luau, and none requires multi_edit. §7 uses a stricter, non-overlapping split. It sets aside the 2929 tasks that require both tools and scores the 3030 script-only and 2525 scene-only tasks that remain. 5 Metrics Outcome metrics. For each task we draw k=16k=16 independent attempts and estimate four success criteria at a subsample size of 55. Throughout, k is the number of attempts per task, and the number after the @ is the subsample size a criterion is named for. Instead of plugging in raw sample rates, we use unbiased U-statistic estimators. pass@1 is the per-task success rate. pass@5, cons@5 and all@5 are the probabilities that a uniformly random subsample of 55 of a task’s 1616 attempts contains at least one, at least three, and five successes. They are computed in closed form from the 1616 draws, not by extrapolation. These estimators follow the unbiased pass@k estimator of Chen et al. (2021), extended from “at least one success” to the consensus and unanimity criteria. The spread between pass@5 and all@5 is a consistency gap. Our all@5 is the passk reliability metric of τ-bench (Yao et al., 2025), and Figure 3 (§C) traces it. Process metrics. We report per-tool usage, split into observation and action, and a per-call tool-error rate. We define exploration as how much of what a task depends on the agent reaches through its observation tools before acting on it. Observation that only verifies an edit already made therefore does not count as investigation (§7). Inference. Every comparison between two models is paired. We take the mean per-task difference in pass@1 over the matched core tasks and compute a percentile bootstrap interval by resampling tasks. A two-sided Wilcoxon signed-rank test on the same per-task differences serves as a check. A pair counts as resolved when the bootstrap interval excludes zero. To decide whether an individual task flips between two models, we apply Fisher’s exact test to that task’s 2×22× 2 table of passes and failures for the two models over their 1616 attempts each. Pairing is what makes a suite of this size sensitive: it removes the per-task difficulty that both models face, which dominates the spread across tasks. §D gives the resampling settings, the suite’s resolution, and further details. 6 Core-Suite Results Table 1: Core-suite results for all 13 models, each running the same 8484 core tasks at k=16k=16 attempts under identical system prompts and tools, with a 300300 s timeout counted as a failure. “Tools”, “Obs” and “Obs%” are mean tool calls per run, mean observation calls per run, and their ratio, from the transcripts. “Err” is errors per tool call, and “Tmo” is the share of a model’s attempts that exhaust the 300300 s time budget. Pass-rate subscripts are marginal standard errors from the task bootstrap of §D, which describe how precisely each model’s own suite-level rate is estimated. Every model-versus-model claim in the paper rests on paired comparisons. Best per column in bold. §C gives the reliability ladder and the debugging suite. Model Pass@1 Pass@5 Cons@5 All@5 Tools Obs Obs% Err (%) Tmo (%) Claude Fable 5 51.7(4.9)51.7_(4.9) 63.7(4.9)63.7_(4.9) 51.8(5.3)51.8_(5.3) 39.4(5.1)39.4_(5.1) 6.4 3.9 62 1.17 2.31 Gemini 3 Flash 51.6(4.7)51.6_(4.7) 66.7(4.9)66.7_(4.9) 51.8(5.2)51.8_(5.2) 34.5(4.5)34.5_(4.5) 10.8 7.0 65 1.94 0.15 Claude Opus 4.8 51.5(4.7)51.5_(4.7) 65.9(5.0)65.9_(5.0) 52.4(5.1)52.4_(5.1) 35.0(5.0)35.0_(5.0) 7.0 4.8 68 1.31 2.16 Gemini 3.1 Pro 51.3(4.7)51.3_(4.7) 66.6(4.8)66.6_(4.8) 50.9(5.2)50.9_(5.2) 36.7(5.0)36.7_(5.0) 7.8 4.1 52 2.95 0.00 Gemini 3.5 Flash 50.6(4.7)50.6_(4.7) 66.1(4.9)66.1_(4.9) 51.3(5.2)51.3_(5.2) 33.7(4.8)33.7_(4.8) 12.9 8.3 64 2.12 1.64 GPT-5.6-sol 47.3(4.9)47.3_(4.9) 60.5(5.2)60.5_(5.2) 46.5(5.2)46.5_(5.2) 35.4(5.1)35.4_(5.1) 10.4 8.5 82 0.89 2.31 GPT-5.5 46.1(4.7)46.1_(4.7) 61.6(5.0)61.6_(5.0) 46.3(5.1)46.3_(5.1) 30.5(4.9)30.5_(4.9) 8.7 6.9 79 0.74 1.49 Claude Opus 4.7 45.5(4.7)45.5_(4.7) 60.8(5.1)60.8_(5.1) 45.4(5.1)45.4_(5.1) 30.7(4.8)30.7_(4.8) 5.0 3.0 59 2.82 0.82 GLM-5 42.9(4.4)42.9_(4.4) 61.2(5.1)61.2_(5.1) 45.3(5.1)45.3_(5.1) 20.3(3.8)20.3_(3.8) 8.4 5.9 70 2.48 1.04 GPT-5.6-terra 40.4(4.8)40.4_(4.8) 53.5(5.2)53.5_(5.2) 40.3(5.2)40.3_(5.2) 27.5(4.7)27.5_(4.7) 9.7 7.6 78 1.05 1.56 GPT-5.6-luna 40.3(4.6)40.3_(4.6) 56.2(5.1)56.2_(5.1) 40.5(5.1)40.5_(5.1) 24.9(4.6)24.9_(4.6) 12.0 9.7 81 1.44 0.74 GPT-5.4 40.0(4.7)40.0_(4.7) 55.0(5.2)55.0_(5.2) 40.3(5.2)40.3_(5.2) 24.5(4.4)24.5_(4.4) 12.1 10.3 85 1.14 3.57 GLM-4.7 36.7(4.3)36.7_(4.3) 56.0(5.0)56.0_(5.0) 37.3(4.9)37.3_(4.9) 17.0(3.8)17.0_(3.8) 8.6 5.6 64 3.74 0.00 The five leading models are within 1.11.1p of one another, between 50.650.6 and 51.7%51.7\% (Table 1). Their 95%95\% marginal intervals overlap, and so do those of every other model pair. Overlapping marginal intervals are not evidence that two models are equivalent, so we compare models on their paired per-task differences rather than on separate summary statistics (Miller, 2024). A paired test on the shared tasks separates 3434 of the 7878 pairs, and 1515 of them survive a Holm correction. Similar average pass rates do not mean these models behave similarly, and the sets of tasks they solve show this directly. Counting a task as solved when a model passes a majority of its 1616 attempts, each of the top five models solves between 4141 and 4444 of the 8484 core tasks. If they all solved the same half of the suite, the union of their solved sets would be about that size too. Instead, the union contains 5858 tasks and the intersection 2727. So 3131 tasks, over a third of the suite, are solved by at least one of these models and missed by at least one other. These agreement numbers should not be compared against a baseline of zero. “Solved” is a majority verdict on 1616 noisy attempts, so two identical models can land on opposite sides of the threshold whenever a task’s true success rate is near one half. A union larger than the intersection is therefore expected even when nothing distinguishes the models. We estimate the right baseline with a pooled placebo. For each task, we pool the five models’ successes and attempts into a single success rate. This keeps each task’s difficulty but erases every difference between the models. We then redraw all five models from that common rate, 1616 attempts each, apply the same majority threshold, and record the union and intersection. Over 2,0002,000 draws, the union has a median of 4747 tasks (95%95\% interval [45,50][45,50]) and the intersection a median of 3737 ([34,40][34,40]). The observed 5858 and 2727 fall outside both intervals, and none of the 2,0002,000 draws reaches either value. The divergence is not an artifact of thresholding sixteen attempts. Comparing models pair by pair gives a sharper picture. For each of the ten pairs among the top five models, we compare the two models’ 1616-attempt success counts on each task with Fisher’s exact test and count the tasks that separate at p<0.05p<0.05. The median across the ten pairs is 1717 tasks, with a range of 99 to 2121. We set a floor with the same kind of placebo. For each task we pool the pair’s 3232 attempts into a single rate, redraw both models from that rate so that they differ only by chance, and run the same test on the simulated counts, again over 2,0002,000 draws. The placebo flags 0.60.6 tasks on average, and no pair’s 9595th percentile exceeds 22. Across all 7878 pairs in the panel, the median number of tasks separating at p<0.05p<0.05 is 1919. Two models at the top of the leaderboard, a tenth of a point apart in aggregate, therefore disagree about individual tasks nearly as much as two models ten points apart. The disagreements run in both directions in nearly equal numbers, which is why they cancel in the average. Claude Fable 5 and Gemini 3 Flash differ by 0.10.1p in aggregate but on 1717 individual tasks, 88 favoring one model and 99 the other. Splitting by kind of work shows what the models disagree about. For the same five models, the spread between the best and the worst grows from 1.11.1p on the full core suite to 5.05.0p on script-authoring tasks, 12.512.5p on scene-change tasks, and 13.113.1p on the debugging suite. Under the same placebo, five identical models give median spreads of 2.32.3p and 4.04.0p on the two kinds of work. The observed spreads are therefore 2.22.2 and 3.13.1 times their floors, while on the full suite the observed 1.11.1p is below its floor of 2.02.0p. The rankings also change (Figure 2, left). Gemini 3.1 Pro is first of thirteen at script authoring and eighth at scene change. Claude Opus 4.8 is the reverse: first at scene change and fifth at script authoring. What looks like a single frontier at 50%50\% is five differently shaped capability profiles that average to the same number. The five also use tools in visibly different ways: mean tool calls per run range from 6.46.4 to 12.912.9, and the observation share of those calls from 52%52\% to 68%68\% (Table 1). The same pattern holds across model generations (Figure 2, right). Claude, GPT and GLM each gain about 66p in core pass@1 from their older to their newer model, but on different kinds of work. Gemini is flat in aggregate. Figure 2: One pass rate hides what a model is actually good at. Core tasks are split by what their solution must do: author or rewrite script text (5959 tasks) or mutate the game scene (2525). The debugging suite (2929) is scored separately. Left: pass@1 per group, with models ordered by script-authoring rate. Right: four within-family generational steps, broken down by group, with 95%95\% paired-bootstrap intervals over tasks. Filled markers indicate intervals that exclude zero. To ask which tasks separate models and what those tasks have in common, we fit a 2PL item-response model to the 13×8413× 84 response matrix. With only 1313 respondents, the fit is a diagnostic rather than a precise measurement. §E gives the specification, priors and diagnostics. When ability is modeled hierarchically by provider family, 79%79\% of its variance lies between families rather than between versions within a family (interval [0.71,0.87][0.71,0.87]). Seven tasks have a discrimination α whose 95%95\% posterior interval lies entirely above the suite’s median under both priors we fit (§F). They are turning lights on and off by time of day (14%14\% pass rate), making the player bounce when shooting the ground within 55 studs (19%19\%), a gravity well (82%82\%), a flashlight for every joining player (83%83\%), friendly fire in a laser-tag game (79%79\%), a fridge door that opens on touch instead of click (67%67\%), and a pasted error about a missing getDriversInPart member (90%90\%). Among the 2222 mid-difficulty tasks (|β|≤0.5|β|≤ 0.5), discrimination is essentially unrelated to pass rate (Spearman 0.120.12), so a task cannot be made discriminating by aiming it at a 50%50\% pass rate. What they share instead is an expectation the prompt implies but does not state. For instance, the lighting task needs a day–night cycle, the bounce must not fire at walls, ceilings or the sky, friendly fire must reach every blaster, and a flashlight must carry an actual light. Fixing the getDriversInPart error means writing a function that actually returns the drivers in a race, not one that merely makes the error go away. 7 Exploration Behavior as a Predictor of Task Success Much of the difference between models lies in which tasks they solve. We now ask whether the trajectory of a run carries information about whether that run succeeds. For each trajectory on the core panel, we measure whether the run carried out the investigation the task calls for. We use the per-task dependencies annotated in the benchmark: the specific instances the reference solution touches and the specific scripts it reads. Figure 4 in §G shows representative runs. We regress the 0/10/1 outcome of a run on one trajectory feature at a time by least squares, with a dummy for every task and every model. Confidence intervals use a cluster-robust sandwich estimator with tasks as clusters. The fixed effects make every estimate within task and within model: we compare runs of the same model on the same task that went about it differently. Because the outcome is an indicator, each coefficient is directly in percentage points. A reference-solution instance counts as “observed” only when the run reaches it through an observation tool. The observation must also precede the run’s first action on that instance. A run that edits the right object and only then inspects it to confirm the edit gets no credit for investigating before acting. A run’s coverage on a task is the fraction of the task’s annotated dependencies credited this way. It is 00 for a run that acts without inspecting any of them and 11 for a run that reaches all of them before acting. Observing an instance is not the same as modifying it correctly, so coverage does not simply restate the checks that decide success. Table 2 reports the estimates. Coverage is defined only where a task’s dependencies are annotated, and 5454 of the 8484 core tasks carry the annotation. We analyze 3737 of them, split into 2424 scene-only and 1313 script-only tasks. The remaining 1717 require both tools and are reported in §G. On the scene-only tasks (2424 tasks, 1313 models), the bottleneck is finding and acting on the right live objects in the game. There, each standard deviation of coverage predicts a +6.4+6.4p higher pass probability (+13.4+13.4p from none to full, 95%95\% CI [+1.2,+25.7][+1.2,+25.7]). The script-only tasks (1313 tasks, 1313 models) require authoring script text and no scene mutation. On them, coverage shows an association of similar size: +4.3+4.3p per standard deviation and +9.8+9.8p from none to full (CI [+3.6,+16.0][+3.6,+16.0]). Table 2: The grounded success signal is present on both kinds of work, at similar magnitudes. Effects on the probability that a run passes, in percentage points, from a linear probability model with task and model fixed effects over the per-run trajectories. Coverage features count a reference-solution dependency only when the run reaches it through an observation tool before its first action on that dependency. Expected-tool match is the fraction of the observation tools a task is annotated as requiring that the run used at least once, in any order. “/SD” is the effect per standard deviation of the feature, computed within the arm the column reports and over the tasks that row uses. “none→ ” is the same coefficient, scaled to a change from none to full. Brackets are 95%95\% task-cluster-robust intervals. The difference column is the coefficient on the feature’s interaction with a script-arm indicator, from one fit over both arms, which gives the gap its own cluster-robust interval. It need not equal the difference between the two columns, because each per-SD figure is scaled by its own arm’s feature dispersion. The lower block shows two sequencing features that use no annotated dependency information. §G gives cluster counts per row, the feature definitions, and alternative inference schemes. Trajectory feature Scene-only Script-only Difference Observe reference instances (/SD) +6.4[+0.6,+12.3]+6.4[+0.6,+12.3] +4.3[+1.6,+7.0]+4.3[+1.6,+7.0] −1.6[−7.6,+4.3]-1.6[-7.6,+4.3] none→ +13.4[+1.2,+25.7]+13.4[+1.2,+25.7] +9.8[+3.6,+16.0]+9.8[+3.6,+16.0] −3.5[−16.3,+9.3]-3.5[-16.3,+9.3] Read reference scripts (/SD) – +8.3[+1.1,+15.5]+8.3[+1.1,+15.5] – none→ – +17.8[+2.3,+33.3]+17.8[+2.3,+33.3] – Expected-tool match (/SD) +5.3[−2.5,+13.0]+5.3[-2.5,+13.0] +0.5[−3.4,+4.4]+0.5[-3.4,+4.4] −6.8[−17.8,+4.1]-6.8[-17.8,+4.1] Search, then read it (/SD) +8.6[−3.3,+20.5]+8.6[-3.3,+20.5] −0.3[−4.3,+3.7]-0.3[-4.3,+3.7] −8.9[−21.7,+3.9]-8.9[-21.7,+3.9] Reads that were grounded (/SD) +4.0[+0.4,+7.6]+4.0[+0.4,+7.6] +1.6[−2.1,+5.3]+1.6[-2.1,+5.3] +1.4[−3.5,+6.4]+1.4[-3.5,+6.4] Scripts are a second investigation channel. On the 1111 tasks whose reference solution depends on other scripts (77 of them outside the two arms, §G), reading those scripts before making edits predicts a +8.3+8.3p higher pass probability per standard deviation (+17.8+17.8p from none to full, [+2.3,+33.3][+2.3,+33.3]). On script work, then, the signal also runs through the scripts a task depends on. We cannot test whether scene work has a script-side counterpart, because only 22 scene tasks carry that annotation. The other rows of Table 2 do not credit the specific instances a reference solution touches. Expected-tool match rewards using the annotated observation tools in any order, and the two sequencing features in the lower block use no annotation at all (§G). None of them shows an effect on the script-only arm, and on the scene-only arm only grounded reads is significant. This is consistent with the signal lying in which instances a run reaches, not in which tools it calls. The two arms are estimated on disjoint task sets, so comparing their estimates side by side does not show whether they differ. We estimate the gap directly, with one fit over both arms that includes coverage and its interaction with a script-arm indicator. The gap is small and not significant: −1.6-1.6p per standard deviation, cluster-robust interval [−7.6,+4.3][-7.6,+4.3], p=0.59p=0.59 across the 3737 task clusters. We detect no difference between the two kinds of work, and coverage predicts success on both at magnitudes we cannot tell apart. If observation is credited at any point in the run, rather than only before the first action on the dependency, the scene effect is about twice the script effect. That ratio comes from the unrestricted feature. The timing restriction lowers the scene estimate from +7.8+7.8 to +6.4+6.4p per standard deviation and does not affect the script estimate. The amount of exploration does not explain the effect. Controlling for the number of tool calls moves the scene estimate only from +13.4+13.4 to +12.6+12.6p, even though the number of calls predicts success on its own on that arm (+5.6+5.6p per standard deviation). A sharper test uses a feature that measures context-gathering without knowing what the task depends on, which is the form the finding of Mehtiyev & Assunção (2026) takes on SWE-bench. We measure it as the number of distinct instances a run inspects before its first action. On the same runs and tasks, this generic feature predicts +6.2+6.2p per standard deviation on the scene arm ([+1.9,+10.5][+1.9,+10.5]), almost as much as coverage, and +2.7+2.7p on the script-only arm ([−2.4,+7.9][-2.4,+7.9]), where it is not significant. To test whether coverage measures anything the generic feature does not, we enter both in the same fit. On scene-only tasks the two features correlate at +0.55+0.55, and neither stands out: with both in the model, coverage takes +5.0+5.0p [−1.1,+11.1][-1.1,+11.1] and the generic feature +4.1+4.1p [−0.4,+8.6][-0.4,+8.6]. On script-only tasks they come apart. They correlate at only +0.32+0.32, coverage holds at +3.9+3.9p [+0.8,+6.9][+0.8,+6.9], and the generic feature drops to +1.2+1.2p [−4.6,+7.0][-4.6,+7.0]. On script-only work, knowing which scripts and instances a task depends on carries information that a plain count of exploration does not. On scene-only work we cannot separate the two. Task and model fixed effects enter our model additively. Part of the relationship could therefore come from differences between (task, model) cells rather than within them: a model that happened to do well on a particular task may also have happened to observe more there, and additive effects do not absorb that. A stricter specification gives every (task, model) cell its own fixed effect, leaving only the variation among one model’s 1616 attempts at one task. Coverage survives it on both arms, at +5.7+5.7p per standard deviation on the scene arm ([+0.2,+11.3][+0.2,+11.3]) and +2.8+2.8p on the script arm ([+0.8,+4.8][+0.8,+4.8]). Implications for process supervision. Process reward models supervise reasoning at the step level (Lightman et al., 2024), and recent work scales automated process verifiers by asking which intermediate signals predict eventual success (Setlur et al., 2025). A verifier that rewards coverage of the instances a task depends on, credited only when the agent reaches them before acting, would reward a behavior worth +9.8+9.8p of pass probability on script-only work and +13.4+13.4p on scene-only work. On script-only tasks, such a verifier would need to be task-specific: there the signal survives the generic pre-edit exploration feature, so a verifier that rewards only how much an agent explored would miss it. Our results also show that one can measure which process signals carry information about success before training any verifier. These are within-task, within-model associations, so they identify signals worth testing as process rewards, not signals proven to work. 8 Limitations and Threats to Validity OpenGameEval covers a single engine and a single language, so the trajectory signal we find may not transfer unchanged to other engines. We conjecture that the link between task-required investigation and success holds generally in stateful worlds with hidden structure, but we cannot test this here. Our exploration measure is also a behavioral proxy tied to our eight-tool action space. It captures how much a model probes the world through the provided tools, not what the model may infer without a tool call. Coverage also depends on annotating each task’s dependencies from its reference solution, and none of the annotation-free alternatives we test shows a significant association with success on script-only work. Deriving dependencies automatically, for example from the instances and scripts a reference solution reads and writes when executed, would remove the annotation step. Agents in our setting observe the world only through text. Tool outputs describe instances, their properties and script contents, and no tool returns a rendered frame. Visual feedback matters for game development: adding image and video feedback raises model performance on GameDevBench (Chi et al., 2026). Our results therefore describe text-only agents, and we do not test what vision would add on tasks judged by layout or visible behavior. Our checks could be wrong in two directions, and we screen for both. Running each task’s reference solution shows that a check is not too strict. A null arm that makes no edits at all shows whether a check can be passed without doing the work. A third screen reads the definition of every check that no model passes. Four tasks failed these screens and were dropped (§A lists them and the reasons). The null arm also passes 55 of the 113113 scored tasks, four core and one debugging. We keep these: their setup is intact, but their checks also accept an untouched world, so they test whether a model preserves functionality that already works. Three of the four core tasks still separate models, and the fourth separates none. Our play-time checks are scripted rather than model-driven. This makes them deterministic, but far from representative of real player behavior. Model-driven user simulators can approximate real users when conditioned on detailed user profiles (Dou et al., 2025), though a measurable distributional gap to real users remains (Mehri et al., 2026). The tasks, checks, worlds and annotations are public, and a released editor plugin re-runs each task’s setup, reference solution and checks. The scaffold around them is specified closely enough to reimplement (§3). However, the system prompt is part of a live product and cannot be released, and the runtime that executes sessions is hosted rather than open source. Public release creates a risk of contamination. To test whether this risk grows across model generations, we track the pass@1 gap between two cohorts of core tasks, the 4646 October tasks whose solutions are withheld and the 3838 April tasks whose solutions are public, and measure how that gap changes from older to newer models. We average these changes within each of four model families. The comparison assumes that, without contamination, capability gains would affect both cohorts similarly. The family effects split evenly in sign, with a mean of −1.0-1.0p and a 95%95\% interval of [−6.9,+5.0][-6.9,+5.0], so we find no consistent evidence that newer models improved disproportionately on either cohort (§A gives the calculation). The test cannot detect contamination that both generations share. 9 Conclusion We introduced OpenGameEval, a play-time-verified benchmark for agentic software development inside a deployed game engine. The strongest of 1313 frontier models fails 48%48\% of the core suite, its five leaders reach nearly identical aggregates on different tasks, and investigating what a task requires before acting predicts whether a run succeeds. We release the suite, its annotations and a leaderboard as a testbed for studying how agents act in stateful worlds. References Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Balloccu et al. (2024) Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2024. arXiv:2402.03927. Bean et al. (2025) Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, et al. Measuring what matters: Construct validity in large language model benchmarks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025. arXiv:2511.04703. Cameron et al. (2011) A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Robust inference with multiway clustering. Journal of Business & Economic Statistics, 29(2):238–249, 2011. doi: 10.1198/jbes.2010.07136. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Chi et al. (2026) Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating agentic capabilities through game development. In Proceedings of the International Conference on Machine Learning (ICML), 2026. arXiv:2602.11103. Chowdhury et al. (2024) Neil Chowdhury et al. Introducing SWE-bench Verified. OpenAI technical report. https://openai.com/index/introducing-swe-bench-verified/, 2024. Accessed 2026-06-08. Dou et al. (2025) Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. arXiv:2510.05444. Jimenez et al. (2024) Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. Khanal et al. (2026) Aaditya Khanal, Yangyang Tao, and Junxiu Zhou. Beyond pass@1: A reliability science framework for long-horizon LLM agents. arXiv preprint arXiv:2603.29231, 2026. Kipnis et al. (2025) Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench – a sparse benchmark of reasoning and knowledge in large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. arXiv:2407.12844. Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050. Liu et al. (2024) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. arXiv:2308.03688. Luo et al. (2026) Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. GameCraft-Bench: Can agents build playable games end-to-end in a real game engine? arXiv preprint arXiv:2606.17861, 2026. Ma et al. (2024) Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2401.13178. Maia Polo et al. (2024) Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: Evaluating LLMs with fewer examples. In Proceedings of the International Conference on Machine Learning (ICML), 2024. arXiv:2402.14992. Mehri et al. (2026) Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar, Marwa Abdulhai, Sergey Levine, Michel Galley, and Dilek Hakkani-Tür. Measuring and mitigating the distributional gap between real and simulated user behaviors. arXiv preprint arXiv:2605.07847, 2026. Mehtiyev & Assunção (2026) Tural Mehtiyev and Wesley K. G. Assunção. Beyond resolution rates: Behavioral drivers of coding agent success and failure, 2026. arXiv:2604.02547. Merrill et al. (2026) Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In Proceedings of the International Conference on Learning Representations (ICLR), 2026. arXiv:2601.11868. Mialon et al. (2024) Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. arXiv:2311.12983. Miller (2024) Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024. Park et al. (2026) Jaden Park, Jungtaek Kim, Jongwon Jeong, Robert D. Nowak, Kangwook Lee, and Yong Jae Lee. Exploration and exploitation errors are measurable for language model agents. arXiv preprint arXiv:2604.13151, 2026. Rabanser et al. (2026) Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a science of AI agent reliability. In Proceedings of the International Conference on Machine Learning (ICML), 2026. arXiv:2602.16666. Setlur et al. (2025) Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. arXiv:2410.08146. Trivedi et al. (2024) Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2407.18901. Xie et al. (2024) Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. arXiv:2404.07972. Yao et al. (2025) Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. arXiv:2406.12045. Yehudai et al. (2026) Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. A survey on evaluation of LLM-based agents. In Findings of the Association for Computational Linguistics: ACL 2026, 2026. arXiv:2503.16416. Zhou et al. (2024) Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. arXiv:2307.13854. Zhu et al. (2025) Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, et al. Establishing best practices in building rigorous agentic benchmarks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025. arXiv:2507.02825. Appendix A Framework and Benchmark Details Lifecycle and verification. A task file declares a natural-language prompt, a placefile, and four hooks. setup changes the world and selection state before the agent runs. reference is an authoritative solution, which the model never sees during evaluation. check_scene asserts edit-mode (static) properties, and check_game (or client checks) asserts play-mode (runtime) behavior. §H lists every scored task with its prompt, its place file, and which of the two check phases it uses. Separating the two phases separates static correctness from runtime behavior. For example, the health-regeneration task removes the existing health script and installs a server-authoritative damage event. At play time, the check fires that event and asserts that the health the client sees holds at 5050 for 1.41.4 s, reaches exactly 6060 at 2.12.1 s, and reaches 7070 at 3.13.1 s. No static check could capture this timing-dependent, server-to-client property. The check for the worked task of Figure 1 asserts that every LightsOn boolean is true, every spotlight is enabled, and every light part has been changed from SmoothPlastic to Neon. Agent scaffold. Each attempt consists of a single user message. Across all sampled runs, every user-role message after the first is a tool result, so nothing prompts, clarifies or re-plans the agent mid-run. Tool errors are returned to the model rather than retried. Of all tool calls, 1.6%1.6\% return an error, and 9.7%9.7\% of runs contain at least one error. The 200200 tool-call ceiling never binds: the median core attempt makes 77 calls, the 99.999.9th percentile makes 5252, and the largest attempt on the core panel makes 6363. The time budget, tool-call budget, attempt counts, system prompt and tool configuration are identical across all runs. The only settings that differ are the served model identifier, the API family it is called through, and a worker count that affects throughput but not the agent. Sampling is set per family, and Table 3 records every request parameter. Reasoning effort was set explicitly, not left to provider defaults, and at the same nominal level, high, for all four families. An attempt that returned no scoreable result because of an infrastructure failure, such as a rate-limit response or an upstream server error, was relaunched. A timeout is not an infrastructure failure: it is scored as a failure and was never relaunched. No attempt was relaunched because of its output. Table 3: Request parameters, by family. The sampling parameters used by the harness. A dash is a parameter we did not send. Reasoning effort was high for all four families, under each provider’s own parameter name. Family Temperature top-p top-k min-p Output cap Claude (3 models) — — — — 64,00064,000 Gemini (3 models) 1.01.0 0.950.95 — — 32,76832,768 GPT (5 models) — — — — — GLM (2 models) 1.01.0 0.950.95 2020 00 — The eight tools of §3 are a subset of the tools the engine’s assistant exposes, which the engine vendor documents publicly (https://create.roblox.com/docs/studio/mcp). The full set is larger. It also includes asset and material generation, playtest control, simulated input, documentation retrieval and subagent delegation, and some tool names there have changed since our transcripts were recorded. Only the eight listed tools were enabled for the runs reported here, so generation, retrieval and subagent delegation were all disabled. Reproducibility. The engine version is pinned, and checks are written to tolerate benign non-determinism. A fixed 300300 s per-phase wall-clock budget is held constant across all models and treated as part of the quality bar. The framework simulates player interaction programmatically (button clicks, keyboard input, camera manipulation), so tasks that require user action can be verified at play time. The task definitions, their place files, and the per-task annotations that §7 uses are available in https://github.com/Roblox/open-game-eval. The release also includes an editor plugin that runs a task’s setup, reference solution and checks, first in edit mode and then in play mode, so anyone can repeat the screen for over-strict checks. The plugin does not invoke an assistant. The service that drives a model’s session inside Roblox Studio is not open source, so the released artifacts fix what is evaluated but not how it is run (§8). Construction. Tasks are built with input from domain experts and game developers. For each task we build or select a placefile that provides context, and we hand-write the executable checks and an authoritative reference solution. Every scenario was reviewed by people for comprehensiveness, generalizability and stability. The suite is public and grew through dated releases, each recorded in the repository’s release log. The repository was created on October 13, 20252025, and the first 4747 core tasks were released the next day. Another 3030 debugging tasks followed on March 3, 20262026, and a further 4040 core tasks on April 14, 20262026 brought the core suite to 8787. The tasks have therefore been public for different lengths of time, so their exposure to any training corpus is uneven across the suite. The repository held 117117 task files. We collected runs on all of them and then dropped four after data-quality checks: one from the October core release, two from the April core release, and one from the debugging suite. Three of these have since had their checks repaired in the public repository. The check of 001_make_cars_faster_bug_1 crashed on a new version of the engine. The check of 098_pirate_lose_health_underwater relied on a precondition the harness never established. The check of 120_food_spawn_hunger_bar demanded behavior that the prompt did not make clear. The fourth task, 073_homestore_dynamic_pricing, is passed by the null arm (§8) on every attempt. Its setup never planted the hardcoded price that the prompt asks the model to fix, so as run, the task had nothing to repair. We withdrew it while we work on a better replacement. This leaves the 113113 tasks scored in this paper, 8484 core and 2929 debugging. Exposure and contamination. Of the 8484 core tasks, 4646 come from the October 20252025 release and have no public reference solution, and 3838 come from the April 20262026 expansion and have one. The two cohorts differ in difficulty, so we do not interpret their raw pass-rate difference. Instead, for each model we compute the mean task-level pass@1 on each cohort and subtract the published-cohort mean from the withheld-cohort mean. For each of the nine older-to-newer model pairs in our four families, we subtract the older model’s cohort gap from the newer model’s, and we average these changes within each family. The family-level changes are +0.7+0.7p for Claude, +2.9+2.9p for GPT, −1.6-1.6p for GLM, and −5.8-5.8p for Gemini. Their mean is −1.0-1.0p, with a 95%95\% Student-t interval of [−6.9,+5.0][-6.9,+5.0] that treats the four families as independent units. A positive change means that newer models gained more on the October cohort, whose prompts and checks had been public longer. A negative change means that they gained more on the April cohort, whose reference solutions are public. The interval is wide and includes zero, so this test shows no differential improvement between the cohorts. Null-arm validation. The null arm of §8 is a do-nothing agent: the task’s checks run against the world its setup leaves behind, with no edits of any kind. It passes 55 of the 113113 scored tasks. Four are core tasks, which it passes on every attempt: 041_platformer_make_checkpoints, 052_surburban_trampoline_bounce, 059_surburban_merrygoround_max_velocity and 071_surburban_teleport_sandbox. The fifth is the debugging task 010_left_shift_sprint_5s_bug_2. The four core checks each assert a property of a live play session that the untouched world already satisfies, such as a player coming to rest inside a tolerance that the target already meets. We keep all four scored. Models still fail three of them on 99 to 22%22\% of attempts, so these three separate agents that break a world that was already correct. The fourth, 059_surburban_merrygoround_max_velocity, is passed by all 1313 models on all 208208 attempts and separates nothing. Appendix B Extended Related Work HumanEval and MBPP (Chen et al., 2021; Austin et al., 2021) established execution-based scoring against unit tests for code generation. SWE-bench (Jimenez et al., 2024) and its verified subset (Chowdhury et al., 2024) are methodologically closest to our executable verification: they include real repository context, require coordinating changes across files, and come with a reproducible harness. Our setting differs in where the work happens and how it can be judged. A SWE-bench task is text in a file tree, and the repository’s own tests run over it once the agent stops. Our tasks allow two kinds of action with different timing. Executing Luau code in the live engine session changes the game scene immediately, while edited scripts show their behavior only once the game is running and a player acts. In AppWorld (Trivedi et al., 2024), agents write programs against a stateful world of apps and must discover task-relevant facts through API calls. Its agents, however, operate existing applications through a fixed API. Ours write the behavior that is later exercised in the game. Mehtiyev & Assunção (2026) ask whether an agent’s investigative policy adapts to task difficulty, and find that it does not. We ask whether the association between that behavior and the outcome varies by kind of task, and we cannot detect a difference. The two findings are consistent on a point that matters: agents apply a fixed investigative policy, even though in both kinds of work we study, investigation pays off. A growing literature documents contamination and weak construct validity in LLM benchmarks (Bean et al., 2025; Balloccu et al., 2024). The Agentic Benchmark Checklist (ABC) (Zhu et al., 2025) formalizes task validity, outcome validity and benchmark reporting, and asks that tasks be designed so that success cannot come from guessing. OpenGameEval’s checks are hand-written and human-verified against explicit reference solutions, and together with its process metrics they are designed to meet these criteria. §8 reports our known limitations. A related line of work applies item-response theory to benchmarks, treating questions as items with estimated difficulty and discrimination. Kipnis et al. (2025) distill six benchmarks (28,63228,632 items) to under 3%3\% of their combined size, and Maia Polo et al. (2024) estimate full-benchmark scores from 100100-item subsets. Both fit item parameters on far more respondents than we have: tens to hundreds of models for tinyBenchmarks and thousands for metabench. We fit the same class of model in §6 to find which of our tasks separate models. With only 1313 respondents, we report the tasks whose discrimination is identified rather than compressing the suite. Like OpenGameEval, Terminal-Bench (Merrill et al., 2026) has agents write software inside a stateful environment that they must explore and manipulate. In the authors’ words, however, its tests “do not test the agent’s commands or console output,” a deliberate choice for a purely outcome-driven benchmark. In GameCraft-Bench (Luo et al., 2026), agents build a launchable Godot game and submit replayable input traces, which a verifier replays and a multimodal judge scores against a hidden rubric. The score depends only on the delivered game. The authors describe agent behavior, such as tool-call counts and screenshot inspections, but the trajectory does not enter the score. Appendix C Per-Model Results Table 1 in §6 is the core-suite leaderboard behind the main text’s estimates. This appendix gives additional results, and Table 4 reports the debugging suite. The 2929 debugging tasks form a separate suite and are never averaged into the core results (§4). A growing reliability-science literature argues that accuracy alone does not describe how an agent will behave in deployment, and that consistency across repeated attempts, which passk (our all@5) measures, should be reported alongside it (Khanal et al., 2026; Rabanser et al., 2026). Figure 3 traces the consistency gap: as the reliability bar tightens from pass@5 to all@5, success falls by ∼31 \!31 points. Figure 3: The reliability ladder (84-task core suite, k=16k=16). Each line traces one model as the success criterion tightens from pass@5 (≥1≥\!1 of 5) through pass@1 and cons@5 to all@5 (all 5, i.e. pass5). The ∼31 \!31-point drop is the consistency gap. Line crossings show that the reliability ranking is close to the single-attempt ranking but not the same (Spearman 0.900.90 between pass@1 and all@5 ranks in Table 1). Main panel: the latest generation of each family. Small panels: every generation within a family, showing that the crossings persist within families. Table 4: The secondary suite: 29 debugging tasks, scored on its own and never averaged into Table 1. Columns are as there. §6 reports how widely the top five models spread on it. Model Pass@1 Pass@5 Cons@5 All@5 Tools Obs Obs% Err (%) Tmo (%) Claude Fable 5 65.5(7.9)65.5_(7.9) 80.4(8.2)80.4_(8.2) 66.3(8.7)66.3_(8.7) 48.8(9.3)48.8_(9.3) 5.8 4.1 71 2.53 2.37 GPT-5.6-luna 65.1(7.6)65.1_(7.6) 80.3(8.6)80.3_(8.6) 68.2(8.5)68.2_(8.5) 43.2(8.5)43.2_(8.5) 9.5 7.8 82 2.63 1.72 Gemini 3.1 Pro 63.6(7.8)63.6_(7.8) 80.5(9.2)80.5_(9.2) 64.9(8.8)64.9_(8.8) 42.3(8.4)42.3_(8.4) 7.7 4.8 63 4.65 0.00 Claude Opus 4.8 62.7(8.2)62.7_(8.2) 76.6(7.9)76.6_(7.9) 64.2(9.0)64.2_(9.0) 46.1(8.7)46.1_(8.7) 7.1 4.8 68 7.42 1.94 Claude Opus 4.7 59.1(7.9)59.1_(7.9) 78.8(7.4)78.8_(7.4) 57.9(9.0)57.9_(9.0) 41.6(9.1)41.6_(9.1) 5.6 4.0 72 1.95 2.16 GLM-5 59.1(7.8)59.1_(7.8) 75.1(8.5)75.1_(8.5) 63.0(9.3)63.0_(9.3) 35.5(7.9)35.5_(7.9) 6.5 5.0 76 1.89 1.72 GPT-5.6-sol 58.8(8.5)58.8_(8.5) 72.7(8.8)72.7_(8.8) 59.2(9.1)59.2_(9.1) 43.7(9.3)43.7_(9.3) 11.4 9.7 86 8.79 3.45 GPT-5.6-terra 58.8(7.7)58.8_(7.7) 77.0(8.7)77.0_(8.7) 61.1(8.4)61.1_(8.4) 37.0(9.0)37.0_(9.0) 8.3 6.5 79 3.33 2.37 Gemini 3 Flash 57.3(8.3)57.3_(8.3) 74.3(7.5)74.3_(7.5) 56.8(9.4)56.8_(9.4) 41.8(8.8)41.8_(8.8) 12.6 9.3 74 3.75 2.16 GPT-5.5 57.1(8.6)57.1_(8.6) 68.1(9.7)68.1_(9.7) 59.0(9.2)59.0_(9.2) 42.5(9.3)42.5_(9.3) 7.4 6.0 81 1.61 1.29 Gemini 3.5 Flash 52.4(7.4)52.4_(7.4) 73.8(8.5)73.8_(8.5) 54.1(8.6)54.1_(8.6) 26.0(6.7)26.0_(6.7) 8.7 6.0 70 4.13 1.94 GPT-5.4 51.3(7.8)51.3_(7.8) 70.0(8.7)70.0_(8.7) 51.0(8.6)51.0_(8.6) 31.6(8.3)31.6_(8.3) 9.6 8.2 85 2.04 2.16 GLM-4.7 50.2(7.2)50.2_(7.2) 74.7(7.8)74.7_(7.8) 52.4(8.8)52.4_(8.8) 21.3(6.2)21.3_(6.2) 8.7 6.0 69 6.19 2.16 Timeout policy. We treat the time budget as part of the quality bar. Timeout rates are uneven. Gemini 3.1 Pro and GLM-4.7 never time out on the core suite (0%0\%), GPT-5.4 times out on 3.6%3.6\% of attempts (4848 of 1,3441,344), and four of the five leaders time out at least occasionally. The most generous alternative is to drop timeouts instead of counting them as failures, which credits a model with its success rate on the attempts that finished. This moves core pass@1 by at most 1.51.5p for GPT-5.4 and by at most 1.21.2p for every other model. That is enough to change some ranks but none of our claims. The top five remain the same five models and Claude Fable 5 stays first, though their order among themselves changes, as expected for aggregates that do not separate (§6). Further down, GPT-5.4 rises from twelfth to tenth, passing GPT-5.6-terra and GPT-5.6-luna. Rank among nearly tied models therefore partly reflects finishing within the budget, which is what we intend. Appendix D Statistical Procedures The panel. Every (model, task) cell carries exactly k=16k=16 attempts, so every cell is estimated at the same precision: 17,47217,472 core attempts over 1,0921,092 cells. No attempt is dropped from the corpus on the basis of its outcome. Resampling. Suite-level rates are reported with a studentized bootstrap over tasks, which captures both the spread between tasks and the sampling variance within a task. The paired intervals behind every model-versus-model claim use a separate estimator: a percentile bootstrap over tasks on the mean per-task difference (5,0005,000 resamples, seed 00), without studentization. The Wilcoxon signed-rank check of §5 agrees with this bootstrap on 7575 of the 7878 pairs. The three pairs where they disagree all lie between p=0.03p=0.03 and p=0.08p=0.08, so no comparison we report depends on the choice between them. For the paired estimator, the minimum detectable effect at 8484 tasks, α=0.05α=0.05 two-sided and 80%80\% power, is 9.29.2p for the median model pair and ranges from 6.26.2 to 11.711.7p across the 7878 pairs. A marginal comparison, which discards the pairing, needs 18.318.3p, which is why no pair separates marginally. The count of resolved pairs in §6 uses 20,00020,000 resamples with a per-pair seed. The Holm step-down at α=0.05α=0.05 across the 7878 pairs, reported in §6, is the only family-wise correction we make. We apply none within a single paired comparison, because the quantities examined there are strongly dependent and each conclusion rests on an interval rather than on clearing a threshold. Clustering. Resampling tasks treats them as independent, but they are not fully independent. The 8484 core tasks are authored against 1111 shared place files, so tasks in the same place share a scene graph and scripts. We therefore recomputed every suite interval with place files, not tasks, as the resampling unit. The within-place intraclass correlation of per-task pass@1 averages −0.000-0.000 across the 1313 models, which implies a design effect of 1.001.00, and the clustered standard errors average 1.061.06 times the task-level ones. No leaderboard conclusion changes. The trajectory estimates of §7 cluster on tasks rather than resampling them, and §G recomputes each of them in three further ways. Appendix E Item-Response Model Specification This appendix gives the full specification of the 2PL fit reported in §6, its diagnostics, and a prior sensitivity analysis. In short, the model’s scale depends on the prior but its orderings do not, which is why the main text relies on rankings of α and β rather than their magnitudes. Data and likelihood. The response matrix is the 1313 models × 84×\,84 core tasks of the core panel. Cell (m,e)(m,e) contributes nme=16n_me=16 attempts and cmec_me passes. Writing θm _m for model ability, βe _e for task difficulty and αe _e for task discrimination, yme∼Binomial(nme,pme),pme=logit−1(αe(θm−βe)).y_me\; \;Binomial\! (n_me,\,p_me ), p_me\;=\;logit^-1\! ( _e\,( _m- _e) ). Hierarchical structure and priors. Task parameters are partially pooled, and so are model abilities, which are nested in the four provider families f(m)∈f(m)∈\Claude, Gemini, GPT, GLM\. All three levels use a non-centered parameterization, which makes the funnel geometry sampleable: θm _m =ρϕ~f(m)+1−ρθ~m, = ρ\, φ_f(m)+ 1-ρ\, θ_m, ϕ~f,θ~m,β~e,α~e φ_f,\ θ_m,\ β_e,\ α_e ∼(0,1), (0,1), ρ ρ ∼Beta(1,1), (1,1), βe _e =μβ+σββ~e, = _β+ _β\, β_e, μβ _β ∼(0,1.5), (0,1.5), σβ _β ∼HalfNormal(0.5), (0.5), logαe _e =μlogα+σlogαα~e, = _ α+ _ α\, α_e, μlogα _ α ∼(0,0.5), (0,0.5), σlogα _ α ∼HalfNormal(1.0). (1.0). The 1313 models are not independent respondents. They come from four families (three Claude, three Gemini, five GPT and two GLM), and versions within a family share training data, tooling and post-training. Treating them as 1313 exchangeable students would understate how correlated their responses are, and the family level above handles this. Estimating αe _e on the log scale forces αe>0 _e>0, so every item characteristic curve increases with ability, and by construction no task can be easier for a weaker model. This is a modeling assumption, not a finding. Identification. The likelihood depends on the parameters only through αe(θm−βe) _e( _m- _e), so the location and scale of the latent axis and the scale of α are jointly unidentified. The usual 2PL convention anchors them by fixing θm∼(0,1) _m (0,1) with no estimated hyperparameters. The parameterization above keeps that anchor exactly: ϕ~ φ and θ~ θ are standard normal and the weights are ρ ρ and 1−ρ 1-ρ, so θm _m has total variance 11 whatever ρ is. Adding the family level therefore costs nothing in identification, and it lets us estimate the share of ability variance that lies between families rather than within them. We compute that share from the fitted abilities. On each posterior draw, we split each model’s ability into its family mean and its deviation from that mean, and take the variance of the family means as a fraction of the total. This gives 0.7920.792 [0.707,0.871][0.707,0.871]: about four fifths of the spread in ability lies between providers rather than between versions within a provider. Two models from the same family resemble each other far more than two models from different families, which is why we do not treat the 1313 models as independent respondents. The family level does not create this concentration. Refitting with no family structure at all, θm∼(0,1) _m (0,1) independently across models, still leaves 0.7620.762 [0.670,0.844][0.670,0.844] of the ability variance between families. Permuting the family labels at random drops it to 0.2470.247 [0.026,0.593][0.026,0.593] (p=0.002p=0.002). Continuity correction. Observed successes enter as min(n−12,max(12,c)) (n- 12, ( 12,c)). Without this correction, the 54.7%54.7\% of cells sitting exactly at 00 or n would push the corresponding βe _e toward ±∞±∞. This is a deliberate bias toward the interior: it shrinks the most extreme difficulties, and it is one reason we do not interpret β magnitudes. Sampling and diagnostics. NUTS via numpyro, 44 chains, 2,0002,000 tuning iterations and 1,0001,000 retained draws each (4,0004,000 posterior draws), target_accept=0.9=0.9, seed 4242. The fit reports 55 divergent transitions, R^≤1.000 R≤ 1.000 across θ, β and α (1.0101.010 over all parameters including the raw non-centered terms), and minimum bulk/tail effective sample size 1,2371,237. Posterior means (sd) for the hyperparameters are μβ=0.25(0.49) _β=0.25\,(0.49), σβ=1.50(0.31) _β=1.50\,(0.31), μlogα=0.65(0.24) _ α=0.65\,(0.24), σlogα=0.86(0.10) _ α=0.86\,(0.10). Prior sensitivity. σβ _β has a posterior mean of 1.501.50 against a prior median of 0.340.34. The data call for a much wider spread of task difficulties than HalfNormal(0.5)HalfNormal(0.5) expects, and the prior is actively shrinking it. Because only the product αe(θm−βe) _e( _m- _e) is identified, compressing the difficulty scale inflates the discrimination scale to compensate. The panel shows why the data pull so hard: 54.7%54.7\% of the 1,0921,092 model-by-task cells are all-pass or all-fail. Such a cell pushes difficulty toward infinity in one direction, and only the prior keeps β finite. We therefore refit with diffuse hyperpriors on both item parameters, loosening both scales and both locations rather than only the difficulty scale: σβ∼HalfNormal(10) _β (10), σlogα∼HalfNormal(3) _ α (3), μβ∼(0,102) _β (0,10^2) and μlogα∼(0,32) _ α (0,3^2). Ability keeps total variance 11 in both fits, because that fixes the scale of an otherwise unidentified model. The diffuse fit still converges (1616 divergences, R^≤1.010 R≤ 1.010, minimum ESS 1,1611,161). The two item parameters trade off exactly as expected: σβ _β rises to 8.048.04, the range of β widens from [−2.2,+3.1][-2.2,+3.1] to [−11.1,+16.0][-11.1,+16.0] logits, the median α falls from 1.961.96 to 0.420.42, and the maximum α falls from 13.013.0 to 2.82.8. None of our IRT-based claims changes substantially under the refit. The rank order of tasks is preserved on both parameters (Spearman 0.9990.999 for each). The refit selects the same seven tasks that §6 reports, because the criterion is relative to each fit’s own median α rather than an absolute threshold. An absolute cut would select all seven under the reported priors and none under the diffuse ones, only because the scale of α shrinks by a factor of about five. The between-family share of ability is also unchanged: 0.7930.793 [0.712,0.880][0.712,0.880] against 0.7920.792 [0.707,0.871][0.707,0.871]. Six unsolved tasks and α. The six core tasks that no model solves are 019_secret_door_puzzle, 021_gem_orbiting_part, 035_laser_tag_regenerate_health, 057_surburban_no_trespassing, 073_lasertag_crate_drop_disappear and 115_racing_wrong_direction_warning. The four that every model solves on every attempt are 013_inf_cube_fall, 059_surburban_merrygoround_max_velocity, 083_platformer_coin_increment_down and 113_racing_extend_laps. For a task that no model solves, the data only establish that its response probability is near zero for every model, and any sufficiently large βe _e achieves that at any αe _e. The two parameters therefore trade off along a ridge instead of being separately identified, and where the posterior lands on that ridge depends on how far the σβ _β prior lets β travel. Their parameters are therefore not interpretable. Under the reported priors, the six tasks sit at β≈2.3β≈2.3 with α≈6.3α≈6.3. Under the diffuse prior they move to β≈11.8β≈11.8 with α≈1.3α≈1.3. Under both settings, their α credible intervals are about 4.64.6 times wider than those of other tasks, and all six rank between 66th and 1212th of 8484 by α. Whichever prior is used, six of the twelve largest α values belong to tasks that carry no between-model information. This distortion of the α ranking is a property of the design, not of the prior. Appendix F Item Analysis Details Table 5 lists the seven tasks whose discrimination §6 reports as clearly above the suite median. Table 5: Where discrimination is identified. The seven core tasks whose 95%95\% interval for discrimination α lies above the suite’s median α of 1.961.96. Because α’s level is not identified independently of ability, the criterion is relative to each fit’s own α scale. We select these same seven under the diffuse-hyperprior refit of §E, whose α values are roughly five times smaller. Difficulty β is on the ability logit scale, across which the 8484 tasks span β∈[−2.2,+3.1]β∈[-2.2,+3.1]. Task Kind α 95%95\% interval β pass@1 103_city_lights_on_off script 12.9712.97 [5.27,22.91][5.27,22.91] +0.27+0.27 14.4%14.4\% 092_fps_shoot_ground_bounce script 10.6710.67 [4.23,17.50][4.23,17.50] +0.24+0.24 19.2%19.2\% 020_gravity_well script 7.107.10 [3.04,11.92][3.04,11.92] −0.30-0.30 82.2%82.2\% 105_lasertag_friendly_fire script 6.756.75 [3.22,11.28][3.22,11.28] −0.27-0.27 79.3%79.3\% 049_surburban_fridge_door_open script 6.056.05 [3.00,9.70][3.00,9.70] −0.15-0.15 66.8%66.8\% 108_racing_fix_getdriversinpart script 5.475.47 [2.13,9.21][2.13,9.21] −0.46-0.46 89.9%89.9\% 054_surburban_equip_flashlight scene 4.784.78 [1.97,8.10][1.97,8.10] −0.39-0.39 83.2%83.2\% Appendix G Trajectory Analysis Robustness This appendix states which tasks each row of Table 2 is estimated on, defines the features the main text does not, and reports alternatives to the inference scheme and the linear model. Of the 8484 core tasks, 5454 carry the instance annotation that coverage is built from, and coverage is defined on exactly those 5454. The other 3030 are dropped from the coverage rows, although 77 of them carry the script-dependency annotation and feed that channel instead. Of the 5454, the 3737 whose required tools place them clearly in one kind of work form the two arms of §7: 2424 scene-only and 1313 script-only. The other 1717 require both tools and are reported separately below. No annotation is shown to the model, and none is used in grading. Cluster counts for every row are listed under the inference alternatives below. Features in Table 2. The two sequencing features in the table’s lower block are (i) whether the run followed a scene-graph or text search by reading a script and (i) the fraction of script reads whose target had already appeared in an earlier observation result rather than being guessed. Neither uses any annotation. The reference-script rows appear only for the script arm, because just 22 scene tasks carry that annotation, too few to estimate a difference. Tasks that require both tools. The partition excludes these tasks on purpose. Splitting by a requirement and its complement would put the 2929 core tasks that need both execute_luau and multi_edit into the script arm, making it half scene work. We therefore score only the two classes that share no required tool. The excluded group is uninformative rather than contradictory: on its 1717 annotated tasks, coverage gives −2.8-2.8p none→ , with an interval too wide to interpret ([−21.6,+15.9][-21.6,+15.9]). Non-circularity. On the scene arm, per standard deviation, the estimate depends on how strictly a dependency is credited. Crediting a dependency reached through an action as well as an observation gives +14.2+14.2p. Crediting only observation, at any point in the run, gives +7.8+7.8p. Crediting only observation that precedes the run’s first action on that dependency, which is what we report, gives +6.4+6.4p. A still stricter variant requires the observation to precede the run’s first action of any kind. It gives +3.9+3.9p [−0.1,+7.9][-0.1,+7.9] on the scene arm and +3.0+3.0p [+0.3,+5.8][+0.3,+5.8] on the script arm. We do not headline it, because execute_luau is both the graded action on scene tasks and the only way to run engine code that probes the live world. Cutting the window at the first such call discards observation that is legitimately interleaved with probing. Additional controls. We also run several robustness checks. The first two are on the none→ scale and compare against the +13.4+13.4p of §7. Controlling for the number of distinct scripts read leaves the coefficient at +13.4+13.4p. The place file a task is set in does not change the result, though this check is weaker than the others. Two of the 2424 scene tasks are authored against a bare baseplate, and dropping them leaves the estimate essentially unchanged at +13.0+13.0p, but with only 2222 clusters left the interval no longer excludes zero ([−0.9,+26.9][-0.9,+26.9]). Dropping the tasks the benchmark labels as hard leaves the scene estimate unchanged at +6.4+6.4p per standard deviation and the script estimate at +3.3+3.3p [+0.4,+6.1][+0.4,+6.1]. This restriction affects only the script arm, because no scene-change task carries the hard label. Finally, counting timeouts as failures, as the leaderboard does, rather than dropping them moves every estimate by at most 0.10.1p, so the signal does not depend on conditioning on a graded verdict. Alternative inference schemes. Each row of Table 2 uses every task in the relevant class on which its feature is defined, so the sample differs from row to row. The coverage rows are limited to tasks with a dependency annotation and the expected-tool row to tasks with an expected-tool annotation. The two sequencing rows need no annotation, so they run on the full classes of 2525 scene-only and 3030 script-only tasks. The intervals rest on few clusters (scene/script): 24/1324/13 tasks for observing reference instances, 1111 for reading reference scripts, 13/2613/26 for expected-tool match, 25/3025/30 for search-then-read, and 20/3020/30 for grounded reads. With so few clusters, the cluster-robust sandwich of §7, which applies a G/(G−1)G/(G-1) correction for G clusters and uses a normal reference distribution, can under-cover. We therefore recomputed every row in three further ways: the same sandwich against a t(G−1)t(G-1) reference, a score wild cluster bootstrap by task with Rademacher weights (999999 draws), and two-way clustering on task and model (Cameron et al., 2011). No entry changes its verdict under any of them. The headline row’s interval is [+0.6,+12.3][+0.6,+12.3] as reported, [+0.3,+12.6][+0.3,+12.6] under t(23)t(23), [+1.1,+11.7][+1.1,+11.7] under the bootstrap, and [+0.5,+12.3][+0.5,+12.3] under two-way clustering. Figure 4: Representative per-run tool-call trajectories. The vertical axis lists seven of the eight tools (all but insert_from_creator_store), grouped into an observation band (upper, blue) and an action band (lower, orange). The older model’s path is grey and dashed, and the newer model’s is black and solid. Arrows trace the order of calls, so a path that stays high keeps gathering information and a path that drops low is acting. Each panel compares runs of an older and a newer model on a task whose outcome differs between them, with the newer model passing in every case: (1) 075_create_npc_enemy, Opus 4.7 to Fable 5; (2) 085_platformer_rosphere_hover, GPT-5.4 to 5.5; (3) 054_surburban_equip_flashlight, GLM-4.7 to GLM-5; (4) 068_village_collectable_plants, Gemini 3 Flash to 3.1 Pro. Not every flip looks like this. Some pairs trace nearly identical short paths and differ only in outcome, so the trajectory adds a signal but does not fully explain the flip. Appendix H The Scored Tasks Every scored task is listed below with the prompt the agent receives, the place file the session opens, and the check phases the task uses. check_scene inspects the edited world at edit time, and check_game asserts a property of a play session started after the edits. A task passes only if every phase it defines passes. A task with only one phase marked is not weaker: its objective is observable in only one of the two places. Prompts are reproduced verbatim, with whitespace collapsed. One prompt is cut at 400400 characters (marked [...]), and a serialization wrapper is stripped from one other. Table 6: The 84 core tasks. Scene and Play mark which check phases the task defines: check_scene at edit time and check_game in a play session. Task Place Scene Play Prompt 001_make_cars_faster racing ∙ – Make the cars of this game 2x faster 002_emit_white_smoke surburban ∙ – Make white smoke come out of every chimney. 003_make_leaves_fall_colored village ∙ – Make the leaves of trees in this game fall colored 004_reduce_car_friction_enable_sliding racing ∙ – Reduce the friction of the cars that spawn in this game with the ground so I can slide around 007_change_logo_cube_green platformer ∙ – Change the Roblox logo like cube to be green 008_spawn_as_r6 baseplate ∙ ∙ make every player spawn as a R6 test dummy as their character for now. 010_left_shift_sprint_5s baseplate – ∙ Can you make a script where if you hold the left shift button your walkspeed gets 20 added to it and if you hold it for 5 seconds your walkspeed goes back to normal? 011_change_height_to_1 baseplate – ∙ chahge players height to 1 012_remove_legs baseplate – ∙ Make a script so that the game removes your 2 legs 013_inf_cube_fall baseplate – ∙ make it so that a size 1,1,1 cube falls at position 0,0,0 every second 017_make_traffic_light baseplate – ∙ write me a script for a simple 4 way traffic light 018_weather_machine baseplate ∙ ∙ Create a ’WeatherMachine’ part that, when activated, cycles through sunny, rainy, and foggy conditions every 3 seconds, visually affecting the entire game world. 019_secret_door_puzzle baseplate – ∙ Design a puzzle, when stepping on color tile parts in the order of red, blue, green, yellow, the secret door will open. 020_gravity_well baseplate – ∙ Add a ’GravityWell’ part that, when touched, inverts a player’s gravity for 5 seconds or until they leave its spherical influence. 021_gem_orbiting_part baseplate ∙ ∙ Make a gem orbit around this part, and automatically attract towards the player when they are within 10 studs. 022_add_world_time baseplate – ∙ Add a script that adds 10 seconds to the world time in Lighting every second. 023_music_playing_part baseplate – ∙ write me a script to play and stop music when within certain trigger parts. 025_chase_and_damage baseplate – ∙ make an entity that chase and damage the player 026_make_traffic_light_v2 surburban – ∙ write me a script for a simple 4 way traffic light 026_make_traffic_light_v3 surburban – ∙ write me a script for a simple 4 way traffic light 027_firstperson_block baseplate – ∙ hey can you make me a block that makes you firstperson when you touch it? 029_play_idle_animation baseplate – ∙ generate a script that makes you play this animation everytime your idle in 5 seconds after your still idle: 507771019 030_play_idle_animation_v2 platformer – ∙ generate a script that makes you play this animation everytime your idle in 5 seconds after your still idle: 507771019 033_count_coins platformer ∙ – count the coins in the game, and create same number of parts 035_laser_tag_regenerate_health laser_tag – ∙ Implement a health regeneration system that starts 2 seconds after taking damage and regenerates at 10 health per second. 038_platformer_coin_multiple_pickup platformer ∙ ∙ Make it so that the coins can be picked up multiple times. 041_platformer_make_checkpoints platformer – ∙ Make this a checkpoint when player step on it. 043_platformer_bouncing_jumper platformer – ∙ Make all OneJump become BounceJump, players will bounce when step on it. 048_surburban_fountain_insert surburban ∙ – Place the water fountain on backyard deck and fit the size there. 049_surburban_fridge_door_open surburban – ∙ Change the store fridge door so that it can be opened with touch, not click. 050_surburban_gaspump_explode surburban – ∙ Gas pump explode when player is in NearArea. 052_surburban_trampoline_bounce surburban – ∙ Make people keep bouncing when they are on the trampoline. 053_surburban_billboard_change_decal surburban – ∙ Make the billboard change image every 1 second, using the decals starting with ’ad’. 054_surburban_equip_flashlight surburban – ∙ When player join the game, they should have a flashlight. 055_surburban_tree_fallcolor_approach surburban – ∙ Make trees change color to red when player approach it. 057_surburban_no_trespassing surburban – ∙ No trespassing: when player enters the fenced yard, they will receive 10 damage. 058_surburban_merrygoround_no_fling surburban – ∙ Can you make the MerryGoRound not fling the player? 059_surburban_merrygoround_max_velocity surburban – ∙ User’s speed on MerryGoRound should cap at 1. 068_village_collectable_plants village – ∙ The plants in garden cannot be picked up and equipped, can you make it work? 070_village_make_npc_walk village – ∙ Make NPCs walk around randomly 071_rainbow_hexagon platformer – ∙ Make 6 parts in a hexagon in the colors of a rainbox, only the red part should kill the player when they touch it 071_surburban_teleport_sandbox surburban – ∙ When player step on the sandbox, they will be teleported to the spawn location. 073_lasertag_crate_drop_disappear laser_tag – ∙ Make each crate drop an item and disappear after shooting them. 074_red_grass_sway flat_terrain ∙ – Make the grass red and sway from side to side 074_village_fire_pit_damage village – ∙ Make all fire pits in the game damage the player when walked over 075_create_npc_enemy laser_tag ∙ – Create an NPC enemy 075_village_remove_tutorial_assets village ∙ – Remove tutorial assets from the map 076_plane_flyby baseplate – ∙ Add a plane flyby scene 079_platformer_roblonk_blue_raise platformer – ∙ Make the Roblonk Model blue and raise by 5 studs smoothly when touchiung it, and then back to normal when you stop touching it 080_surburban_school_lights_on surburban ∙ – Make every light in the school start in the ’on’ state. 081_racing_car_color_change racing ∙ ∙ Please make the cars of this game constantly change colors when they have an occupied driver. 082_platformer_moving_platform_speed_up platformer ∙ – Make the moving platforms speed up by 5. 083_platformer_coin_increment_down platformer – ∙ Make the coins go down by 1 instead of up by 1 when collecting them. Don’t let it go below 0. 084_platformer_roblonk_rotate platformer ∙ – Can you rotate roblonk by 90 degrees? 085_platformer_rosphere_hover platformer – ∙ Can you make the Rosphere have a hovering effect? 086_racing_car_jump racing – ∙ Make it so that when a player is driving a car and jumps, the car jumps 088_surburban_garage_door_speed_up surburban ∙ – Double the speed of the garage doors opening. 089_fps_box_fling_harder fps_system ∙ – Make the boxes fling harder when you shoot them. 090_fps_display_target_damage_ui fps_system – ∙ Make the targets display the damage with a UI after shooting the targets. 091_surburban_fix_grass_in_house surburban ∙ – Fix the grass in the floor of the house named ’Double Story Urban House’ which shouldnt be there 092_fps_shoot_ground_bounce fps_system – ∙ Make the player bounce in the air when shooting the ground within 5 studs of the player. 093_surburban_jeep_in_every_garage surburban ∙ – Place a copy of the jeep in every garage. 094_village_add_cave village – ∙ Add an enclosed cave to the map. The cave should have a ceiling, floor, and walls forming a room-like structure that a player can walk into. Don’t do it on terrain. 095_racing_nitro_effect_color racing – ∙ Make the nitro effect color the same color as the car. 096_fps_target_overhead_health_ui fps_system ∙ ∙ Make the targets have an overhead UI displaying their current health. 097_ugc_mannequin_match_outfit ugc_homestore – ∙ Create a new mannequin for every player currently in the game and make it match their outfit 099_city_add_cars modern_city ∙ – Add cars on the road 100_obby_add_death_trap classic_obby ∙ ∙ Add a spinning death trap 101_obby_flatten_segments classic_obby ∙ – Flatten all obby segments within a consistent y axis 102_city_spawn_on_tallest_building modern_city ∙ – Set the spawn for the players on top of the tallest building 103_city_lights_on_off modern_city – ∙ Turn the lights on/off based on time of day 104_lasertag_mobile_camera_recoil laser_tag ∙ ∙ My camera recoil is too high when the user is playing from a mobile device. Change the recoil amount based on the user’s platform. 105_lasertag_friendly_fire laser_tag ∙ ∙ Allow players to hit members of their own team / Allow friendly fire in this game. 106_lasertag_weapon_balance laser_tag ∙ ∙ Can you change the behavior of the two weapon types in my game? I want one of them to feel more like a shotgun, easier to hit but with lower damage, and the other one to feel more like a sniper rifle, harder to hit but with higher damage. 107_lasertag_grenade_weapon laser_tag ∙ ∙ Add a realistic grenade weapon as a third weapon option to this game. 108_racing_fix_getdriversinpart racing ∙ ∙ Fix the error: getDriversInPart is not a valid member of ModuleScript "ServerScriptService.Racing.RaceManager" 110_racing_car_offtrack_reset racing – ∙ Add a ’reset’ functionality to the game that detects whether the car is upside down or is off-track for 5s. The reset functionality should teleport the car to the starting position on the track with the player. 113_racing_extend_laps racing ∙ ∙ Make the races last longer. I want them to be 20 laps. 115_racing_wrong_direction_warning racing – ∙ If the player is going in the wrong direction in the race, show a warning on their screen that says ’WRONG DIRECTION’. 117_fnf_enter_game_ui_menu baseplate ∙ ∙ 1. Before entering the game, display a menu with: - "Play" button that leads to the scene. - "Freeplay" button to choose songs (only placeholder). - "Exit" button that closes the menu. 2. Add a logo that says "Friday Night Funkin’ Roblox" 118_weapon_spawn_and_pickup baseplate – ∙ Make the items pistol, axe, and knife spawn in various locations on the map, and ensure that the player can only pick up one of each. 119_lasertag_add_megablaster laser_tag ∙ ∙ Add a MegaBlaster that works in StarterPack. 121_monster_chase_at_night baseplate – ∙ Make ’scary monsters’ spawn at night, they will chase the player 122_animal_item_with_rarity baseplate – ∙ I want 6 items of different rarity to spawn in the middle, each item must have a unique animal shape. Rarity with random income between two values for each rarity: Common 60% $1-5, Rare 20% $6-50, Very Rare 11% $100-1,000, Mythic 7% $1,000-10,000, Legendary 1% $10,000-50,000, Ultra 0.9% $50,000-100,000, God 0.09% $100,000-300,000, Limited 0.01% Random value between $300,000-500,000. 6 items […] Table 7: The 29 debugging tasks. Each is a core task with a defect introduced and a prompt written from the point of view of a developer who has hit it, so the place file is shared with the core task it derives from. Task Place Scene Play Prompt 001_make_cars_faster_bug_2 racing ∙ – I was trying to make the cars faster, but now they don’t spawn in the game at all and I’m getting errors. I think I might have moved the car prefab somewhere by accident. 002_emit_white_smoke_bug_1 surburban ∙ – The chimneys are emitting black smoke instead of white smoke. It looks really weird and pollutes the neighborhood vibe. 003_make_leaves_fall_colored_bug_1 village ∙ – I tried to make the leaves on the trees change to autumn colors, but they all turned bright blue and green instead. That doesn’t look like fall at all. 004_reduce_car_friction_enable_sliding_bug_1 racing ∙ – I’m trying to run a script to make the cars drift more, but it’s not working. The friction feels exactly the same. I think the script might not be finding the car parts correctly. 004_reduce_car_friction_enable_sliding_bug_2 racing ∙ – I tried to make the cars more slippery so they could drift, but now they feel even more stuck to the road and difficult to turn. It feels like the friction went up instead of down. What’s wrong? 007_change_logo_cube_green_bug_1 platformer ∙ – I tried to make the floating Roblox logo cube green, but it turned red instead. Can you fix it? 007_change_logo_cube_green_bug_2 platformer ∙ – I’m trying to change the color of the big floating Roblox logo cube to green, but it keeps turning red right after I change it. It seems like some other script is interfering and overriding the color. 008_spawn_as_r6_bug_1 baseplate ∙ ∙ Players are spawning as the newer R15 models, but we need them to be the classic R6 blocky characters for our game. 010_left_shift_sprint_5s_bug_1 baseplate – ∙ I tried setting up a sprint script, but whenever I hold down the Left Shift key, my character slows down to a crawl instead of speeding up. 010_left_shift_sprint_5s_bug_2 baseplate – ∙ When I hold Left Shift to sprint, it works, but if I let go of the key early, I keep sprinting for the full 5 seconds. I should stop sprinting as soon as I release the key. 010_left_shift_sprint_5s_bug_3 baseplate – ∙ When I hold the sprint key, it works, but it feels like the sprint lasts way too long. It’s supposed to time out after 5 seconds, but it seems to go on for much longer than that before my character gets tired and slows down. 011_change_height_to_1_bug_1 baseplate – ∙ I wrote a script to make players 1 stud tall, but when I test it in Studio, my character is still the normal height. The script doesn’t seem to be doing anything. 011_change_height_to_1_bug_2 baseplate – ∙ I wrote a script to make player characters 1 stud tall when they spawn, but it’s not working. They’re still spawning at their default height. 012_remove_legs_bug_1 baseplate – ∙ When my character spawns, they still have legs. The script is supposed to remove them but it doesn’t seem to be working. 012_remove_legs_bug_2 baseplate – ∙ When I join the game, my character’s legs are supposed to be removed, but they’re still there. The script doesn’t seem to be working. 012_remove_legs_bug_3 baseplate – ∙ I wrote a script to remove player legs when they spawn, but it doesn’t work for my character when I first load into the game. It only seems to work for players who join the server after I’m already there. 013_inf_cube_fall_bug_1 baseplate – ∙ I wrote a script that’s supposed to make a cube fall at the center of the map every second, but nothing is happening. No parts are appearing at all. 013_inf_cube_fall_bug_2 baseplate – ∙ I wrote a script to make a cube fall from the sky every second, but it seems to be happening much slower than that. I’m only seeing a new cube every few seconds. 017_make_traffic_light_bug_1 baseplate – ∙ I’m trying to make a traffic light for my game, but it’s not working. The lights are just stuck on and never change color or cycle. 017_make_traffic_light_bug_2 baseplate – ∙ I added the traffic light model to my game, but it’s just stuck. The lights never turn on or change color. 017_make_traffic_light_bug_3 baseplate – ∙ I added a script to control the traffic light, but it’s not working. The lights are stuck and don’t change color. 020_gravity_well_bug_1 baseplate – ∙ I set up the gravity well part, but when players touch it, they get slammed into the ground instead of floating up. 021_gem_orbiting_part_bug_1 baseplate ∙ ∙ I tried to make a gem that orbits a part, but the gem isn’t showing up in the game at all. It’s completely invisible and doesn’t seem to exist in the Workspace. 022_add_world_time_bug_1 baseplate – ∙ The time of day is stuck and the sun isn’t moving. I thought I added the script for the day-night cycle, but it doesn’t seem to be working. 022_add_world_time_bug_2 baseplate – ∙ The day/night cycle seems to be running way too fast. It feels like time is speeding up more than it should. 022_add_world_time_bug_3 baseplate – ∙ I wrote a script to make the day/night cycle faster, but the time isn’t changing at all. The sun is just stuck in the same spot. 023_music_playing_part_bug_1 baseplate – ∙ I set up a trigger part that’s supposed to play music when a player walks into it, but the music never starts. I’ve checked that the sound and the part exist, but nothing happens when my character touches the part. 023_music_playing_part_bug_2 baseplate – ∙ I created a trigger part that’s supposed to play music when a player walks into it, but the music never starts. The part and the sound are both in the Workspace, but nothing happens. 023_music_playing_part_bug_3 baseplate – ∙ I set up a trigger part to play music when a player enters it, but it seems to be working backwards. The music stops when I enter the zone and only starts playing again when I leave. What’s wrong with my script?