Paper deep dive
LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles
Bingo Zhang, Haochuan Lu, Zongjie Li, Genjian Li, Ari Yu Zhang, Chaozheng Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:GUI agents need long-horizon visual reasoning: they must interpret a changing interface while keeping a multi-step plan viable as earlier actions constrain later ones. Existing benchmarks evaluate grounding, computer use, and game play, but rarely test whether agents stay coherent across long chains of coupled decisions. Long-horizon visual puzzles expose this capability directly: a legal move that looks like progress can make the puzzle unsolvable, and the loss shows only several moves later. We introduce LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced. With Native GUI Actions alone, the strongest agents solve most objectives, but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve nothing harder than Medium, and none completes Bolt Unscrew Hard, which a human solves along with every other objective. Code Execution CUA does not close this gap, and its scores mix visual solving with algorithmic search. Controlled diagnostics trace these failures to one limitation that neither rules, state hints, nor failure memory removes: agents judge each move by the visible progress it makes, not by the future options it leaves.
Tags
Links
- Source: https://arxiv.org/abs/2609.34769v1
- Canonical: https://arxiv.org/abs/2609.34769v1
Trouble viewing inline? Open PDF directly →
Full Text
136,484 characters extracted from source content.
Expand or collapse full text
Preprint LONGPUZZLEBENCH: EVALUATING GUI AGENTS ON LONG-HORIZON VISUAL PUZZLES Bingo Zhang 1 , Haochuan Lu 2 , Zongjie Li 1,3 , Genjian Li 1 , Ari Yu Zhang 4 , Chaozheng Wang 1,5 1 Vera Praxis 2 Tencent 3 The Hong Kong University of Science and Technology 4 Independent Researcher 5 Department of Computer Science and Engineering, The Chinese University of Hong Kong ABSTRACT GUI agents need long-horizon visual reasoning: they must interpret a changing interface while keeping a multi-step plan viable as earlier actions constrain later ones. Existing benchmarks evaluate grounding, computer use, and game play, but rarely test whether agents stay coherent across long chains of coupled deci- sions. Long-horizon visual puzzles expose this capability directly: a legal move that looks like progress can make the puzzle unsolvable, and the loss shows only several moves later. We introduce LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced. With Native GUI Actions alone, the strongest agents solve most objectives, but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve nothing harder than Medium, and none completes Bolt Unscrew Hard, which a hu- man solves along with every other objective. Code Execution CUA does not close this gap, and its scores mix visual solving with algorithmic search. Controlled diagnostics trace these failures to one limitation that neither rules, state hints, nor failure memory removes: agents judge each move by the visible progress it makes, not by the future options it leaves. Code: github.com/AzureStarz/LongPuzzleBench 1INTRODUCTION GUI agents act by reading a rendered interface and choosing the next input (Zheng et al., 2024; Hong et al., 2024; Qin et al., 2025). When a task spans many decisions and earlier actions change what later actions can achieve, reading the current screen is not enough: the agent must track how the state has changed, anticipate what each action rules out, and revise its plan when an option is lost. We call this capability long-horizon visual reasoning. Existing benchmarks measure parts of this capability but rarely isolate it. GUI benchmarks test grounding (Cheng et al., 2024) and multi-step computer use (Koh et al., 2024; Boisvert et al., 2024; Xie et al., 2024; Rawles et al., 2025), visual puzzle benchmarks test reasoning over a single static configuration (Chia et al., 2024; Ghosal et al., 2025; Zhang et al., 2025), and game benchmarks test planning and rule discovery, often through predefined commands or with resets left to the harness (Paglieri et al., 2025; Zhang et al., 2026; ARC Prize Foundation, 2026). A long trajectory alone does not imply a long reasoning horizon: a GUI task can take many steps that are each checked at once (Figure 1, left), and a long real-time game episode mixes planning with perception, timing, and control. Neither isolates whether agents judge each move by the options it leaves. Long-horizon visual puzzles supply this demand. Their horizon is set by decision dependency, not elapsed time or action count: rules tie the current board to the options left after each move, so a legal move that looks like progress can make the puzzle unsolvable (Junghanns & Schaeffer, 2001), with the loss surfacing only several moves later (Figure 1, right). Self-paced play is reasoning-dominant rather than timing-dependent, and reproducible states make each failure replayable. 1 arXiv:2609.34769v1 [cs.CL] 28 Sep 2026 Preprint Typical GUI taskLongPuzzleBench: long-horizon visual puzzleNut and Bolt (schematic) Search Select Submit same start Move A: stack 3 blue nuts looks like progress Move B: shift 1 amber nut little visible gain · 3 more moves · 6 more moves already unsolvable still solvable loss visible only 3 moves later Dead end no move helps Solved next level Short, checked step by step; little carry-over between steps Many moves on a persistent board; an early move decides what stays solvable Figure 1: Early moves decide later solvability. In a typical GUI task (left), each step is checked at once and has little effect on later steps. In a long-horizon visual puzzle (right), two legal moves from the same board diverge: Move A looks like progress but already makes the level unsolvable, which shows only three moves later, whereas Move B keeps it solvable. The Nut and Bolt board is schematic; both outcomes are verified by exhaustive search. We introduce LongPuzzleBench, 114 levels of long-horizon visual puzzles in six games, grouped into 16 objectives that chain up to ten levels under one time budget. Every move persists on the board, and play is long even for people: a human needs a median of 204 GUI actions per objective and 1,106 for a single Nut and Bolt level (Figure 2). When a move makes a level unsolvable, the game never says so; later moves simply stop working. Agents play under self-directed gameplay, acting only through the rendered interface, so each failure traces back to their own decisions. We evaluate agents in two settings: Native GUI Actions, which admits only coordinate-based GUI input, and Code Execution CUA, in which a computer-use agent (CUA) may also write and run code. With Native GUI Actions, the strongest agents solve most objectives (GPT-6-Astra 91.7% OSR), but success falls sharply on harder, longer boards: seven of ten general-purpose agents solve no objective harder than Medium, the four GUI-specialized models complete none, and no agent completes Bolt Unscrew Hard, whereas a human solves all 16 objectives. Code Execution CUA does not close this gap: it raises OSR for six of ten agents, not in proportion to their use of code, and its scores mix visual solving with algorithmic search. Tracing failing trajectories and intervening on reproduced states (Section 5), we find that agents spend the free capacity that keeps a puzzle solvable and cycle through earlier states when no move shows visible progress. At a dead end, they mostly read rejected moves as missed clicks rather than as a lost board. Told that the board is lost, they restart, but most rebuild a dead end, with or without memory of the failed attempt. Stating the rules raises rule knowledge far more than completion. These results point to one limitation: agents judge each move by the visible progress it makes, not by the future options it leaves. Our contributions are: • LongPuzzleBench, long-horizon visual puzzles played through native GUI actions, with repro- ducible states and a validated refresh pipeline (Section 3Arxiv). • An evaluation of 14 agents in two action settings against a human reference: the strongest agents solve most objectives, but success falls sharply on harder, longer boards, and Code Execution CUA does not close the gap (Section 4). • Controlled interventions on reproduced states that trace failures to judging moves by visible progress rather than preserved options, a limitation that rules, state hints, and failure memory do not remove (Section 5). 2RELATED WORK GUI and computer-use benchmarks.GUI benchmarks evaluate multi-step task completion through rendered interfaces, from web navigation (Deng et al., 2023; Zhou et al., 2024; Koh et al., 2024) and enterprise workflows (Drouin et al., 2024; Boisvert et al., 2024) to desktop and mobile environments with executable success checks (Xie et al., 2024; Rawles et al., 2025). Agents either 2 Preprint Table 1: Structural comparison with adjacent benchmarks.✓: yes;×: no;†: in a subset of tasks or one track; –: not applicable or not reported. Property definitions and per-benchmark notes: Appendix A.4. What solving requiresHow the agent must act BenchmarkEnvironment Visual puzzle state Delayed consequences Dead-end states Agent-owned recovery Native GUI actions PuzzleBench (Zhang et al., 2025)Puzzle VQA✓×– VisualWebArena (Koh et al., 2024)Web apps×–× AndroidWorld (Rawles et al., 2025)Mobile apps×–✓ BALROG (Paglieri et al., 2025)Text/grid games✓ † ✓× VideoGameBench (Zhang et al., 2026)Video games✓ † ✓×✓ GameWorld (Ouyang et al., 2026)Browser games✓ † ✓×✓ † VLATIM (Triebel et al., 2026)Physics puzzles✓×✓ † ARC-AGI-3 (ARC Prize Foundation, 2026)Grid games✓× LongPuzzleBenchBoard puzzles✓ † ✓ prompt general-purpose multimodal models (Zheng et al., 2024) or train GUI-specialized models for screen grounding and action (Cheng et al., 2024; Hong et al., 2024; Qin et al., 2025). Their steps are checkable against an instructed goal, and no legal action is designed to make the goal unreachable. LongPuzzleBench instead requires reasoning about what a persistent board still allows. Reasoning in puzzles and games. Visual puzzle benchmarks pose questions about a static puzzle image that no action changes (Chia et al., 2024; Ghosal et al., 2025; Zhang et al., 2025). Interactive puzzle benchmarks require agents to act on their interpretations: VLATIM evaluates physics-puzzle manipulation in single-attempt episodes (Triebel et al., 2026), and ARC-AGI-3, the closest to our setting, tests rule and goal discovery with agent-initiated resets but acts through abstract game ac- tions rather than a rendered interface (ARC Prize Foundation, 2026). Game benchmarks add delayed consequences over extended trajectories: SmartPlay and BALROG evaluate planning across game suites (Wu et al., 2024; Paglieri et al., 2025), VideoGameBench and GameWorld test rendered-frame play, including in real time (Zhang et al., 2026; Ouyang et al., 2026), LMGame-Bench diagnoses game playing through modular agent components (Hu et al., 2026), and EscapeBench studies implicit goals in text-based escape rooms (Qian et al., 2025). These suites contain dead ends, but the harness typically resets failed episodes, and evaluation targets general game play rather than self-paced puzzles whose early moves silently decide solvability. In text-based planning, models given a domain’s full rules still produce invalid plans (Valmeekam et al., 2023), and explicit looka- head search improves planning and puzzle solving (Yao et al., 2023; Hao et al., 2023). Studies of recovery examine verbal feedback across trials (Shinn et al., 2023), intrinsic self-correction (Huang et al., 2024), and policy-induced GUI errors (Bu et al., 2026), but not failures that must be inferred from accumulated visual consequences. LongPuzzleBench combines all five properties in Table 1, and its dead ends are unannounced: the agent must infer from the board that progress is impossible. 3LONGPUZZLEBENCH LongPuzzleBench tests how GUI agents preserve future options on persistent boards. It comprises 114 levels in six puzzle games, grouped into 16 game–difficulty objectives (Figure 2). Moves that constrain later options. Each game pairs a rendered board with rules under which a legal move can use up something that later moves need: free holes in Bolt Unscrew, empty storage in Nut and Bolt, open lanes in Rush Hour, shared space in Color Connect, and a workable departure or- der in Truck Escape (Figure 2a). Maze Paint instead requires full coverage under sliding movement, so the last cells can be reached only through moves that paint nothing new. In Bolt Unscrew and Nut and Bolt, such a move can make the level unsolvable without any warning: later moves simply stop working. These dead ends are common in real play: of the human’s 69 Bolt Unscrew level attempts, 52 reached a certified deadlock, as did 173 of 439 agent attempts; on Nut and Bolt, the human played a median of five transfers on an already-unsolvable board before undoing or restarting (Figure 2c). In the other four games, a poor move cannot make the level unsolvable but can lengthen the path to completion. The games span clicks, drags, and swipes and one to five difficulty tiers (Appendix A.1). Long horizon at two scales. Moves create causal dependencies by changing the options available to later moves. The within-level causal horizon is how long these constraints persist; action counts describe interaction length. Nut and Bolt Nightmare, a single level, took the human 1,106 pointer 3 Preprint (a) Six puzzle games 114 levels, 16 objectives boards can become unsolvable Bolt Unscrew 16 levels· 2 obj. few free holes Rush Hour 30 levels· 3 obj. blocked lanes Nut and Bolt 13 levels· 5 obj. limited storage Truck Escape 5 levels· 1 obj. departure order Maze Paint 30 levels· 3 obj. sliding coverage Color Connect 20 levels· 2 obj. competing routes (b) Human GUI actions per objective >1,000 Bolt Unscrew Rush Hour Nut and Bolt Truck Escape Maze Paint Color Connect median 204 1,106 (one level) 2,500 301003001k3k pointer presses in the human reference episode (log scale) (c) Dead ends during play Attempts that reached a certified dead end Bolt Unscrew human 52/69 (75%) agents 173/439 (39%) Nut and Bolt human 18/40 (45%) agents 82/618 (13%) Nut and Bolt transfers played on an already-lost board human median 5 n = 18 agents median 1 n = 82 1 5 10152025 transfers from the fatal one until undo or restart Figure 2: LongPuzzleBench at benchmark scale. (a) Six games; red marks those whose boards can become unsolvable. (b) Pointer presses in the human reference episode for each of the 16 objectives, all solved; dashed line: median. (c) Level attempts that reached a certified dead end, and the Nut and Bolt transfers played on an already-lost board, counting the fatal one, before an undo or restart. Agent counts cover the 384 evaluation episodes of Table 2. presses. Across levels, 15 of the 16 objectives chain three to ten target levels under one budget; an unsolved level blocks advancement. The human used a median of 204 presses per objective, and 14 of the 16 needed at least 100 (Figure 2b). The first scale tests planning within a board; the second tests sustained solving across boards, carrying mechanics forward and deciding when to restart. Evaluation protocol. An objective o = (g, d, L o ) specifies a game g, a difficulty d, and target levels L o . The agent receives the GUI action specification and a single instruction, “Complete all levels of [game] at [difficulty] difficulty in the Game Suite,” with no manual, solution, or navigation procedure. From the suite home screen, it reads screenshots and uses clicks, drags, key presses, and waits to enter the game, solve each level, advance, and restart through the game’s own controls (Ap- pendix Figure 6), all within one 60-minute budget. This self-directed setting applies one interface to every game and leaves decisions such as abandoning a lost board to the agent. An automatic reset at a dead end would remove the need to infer failure from the board. Agent-owned recovery tests this inference together with the ability to revise a plan, while failed attempts consume the same budget as successful play. Board state and grading signals stay private to the evaluator, and any intermediate board state can be restored exactly, which Section 5 uses as checkpoints for interventions. Construction and validation. Games and levels are reproduced from screenshots or other visual references with a coding model and run in a browser suite with deterministic launch parameters. Before release, GUI agents playtest each candidate to expose defects, which are repaired or excluded (Huang et al., 2026); solvers verify solutions where implemented, including all 30 Maze Paint levels; and manual play confirms that every released level is solvable. Validated levels and games extend the catalogue between snapshots to limit repeated exposure to fixed tasks (Jain et al., 2025), and each comparison fixes the catalogue, runtime, and seed (Appendix A.2). 4HOW WELL DO AGENTS SOLVE LONG-HORIZON VISUAL PUZZLES? We evaluate general-purpose and GUI-specialized agents on all 16 objectives against a human ref- erence. Because an agent that can run code can transcribe a board and search for moves offline, we measure performance in two settings that differ in the action channel. 4.1SETUP Settings. Both settings follow the protocol of Section 3 and share objectives, budget, metrics, and a dedicated Chromium browser at 1280× 900. With Native GUI Actions, the agent sees screenshots and acts only through clicks, drags, key presses, and other coordinate-based inputs; it cannot exe- cute programs of its own, so this setting measures visual solving without computational assistance. With Code Execution CUA, the agent may also write and run code over screenshots and the data 4 Preprint Table 2: Main results. Swatch colors match Figure 3; ⋄ open-weight. Bold: best within each setting. Values in %; H 50 in levels (10+: at least half the episodes clear all ten). –: undefined or not applicable. CompletionHorizonDifficultyBehavior AgentOSR↑LC↑FGP↑ LPR↑ H 50 ↑ LC hard ↑ PER↓ NGAR↑ Human reference100.0100.0100.0100.010+100.00.0– Native GUI Actions general-purpose, Codex GPT-6-Astra91.794.894.897.910+87.50.0– GPT-6-Sol82.891.092.295.410+80.30.0– GPT-6-Luna33.940.043.846.729.068.8– GPT-5.6-Sol71.184.386.091.710+64.76.2– GPT-5.6-Terra48.361.364.778.1532.556.2– GPT-5.6-Luna45.053.356.460.3616.062.5– DeepSeek 4.1 Flash ⋄ 53.972.074.977.510+48.012.5– Qwen3.8 Flash ⋄ 53.967.271.576.010+37.00.0– GLM-5.3-Flash ⋄ 33.951.955.260.6512.712.5– Kimi-k3 ⋄ 56.764.467.273.310+30.70.0– GUI-specialized, autonomous harness UI-TARS-1.5-7B ⋄ 0.05.56.6–00.087.5– Qwen3-VL-32B-Thinking ⋄ 0.01.16.4–00.00.0– Qwen3-VL-30B-A3B-Thinking ⋄ 0.01.02.5–00.018.8– Qwen3-VL-8B-Thinking ⋄ 0.02.23.4–00.00.0– Code Execution CUA general-purpose, Codex GPT-6-Astra91.795.896.798.310+90.00.065.7 GPT-6-Sol80.082.283.682.210+58.318.841.1 GPT-6-Luna36.157.561.271.0632.375.065.4 GPT-5.6-Sol88.391.691.994.710+80.80.08.5 GPT-5.6-Terra45.068.772.775.010+48.762.533.1 GPT-5.6-Luna39.445.649.650.2410.768.852.5 DeepSeek 4.1 Flash ⋄ 67.276.679.282.910+56.86.211.7 Qwen3.8 Flash ⋄ 76.181.082.686.610+64.20.017.5 GLM-5.3-Flash ⋄ 39.452.257.172.3321.20.051.0 Kimi-k3 ⋄ 70.677.678.785.010+59.00.059.3 it derives from them, while every change to the game must still pass through GUI inputs; how it uses computation is analyzed from its trajectory. Game source, hidden state, evaluator outputs, and external retrieval are prohibited in both settings (Appendix Table 5). Agents. Ten general-purpose backbones (six proprietary, four open-weight) run in the Codex scaffold in both settings; with Native GUI Actions, the agent may call only a single GUI-action tool. Four GUI-specialized models (Qin et al., 2025; Bai et al., 2025), trained to map screenshots to coordinate actions, run with Native GUI Actions in an autonomous harness that exposes the same action set in each model’s own output format. A human player provides one reference episode per objective (Appendix B.2). Metrics. The primary metric, objective success rate (OSR), is the percentage of objectives whose target levels are all solved within an episode, averaged over objectives within each game and then equally over the six games; all completion metrics use this game-macro weighting. Level completion (LC) is the percentage of target levels solved, and fine-grained progress (FGP) also credits the largest fraction of the goal reached on the level where an episode stops (e.g., cells painted or pairs connected). LC hard restricts LC to tiers above each game’s easiest. The level pass rate (LPR) is the share of reached levels solved, the 50% level horizon (H 50 ; cf. Kwa et al., 2025) is the number of consecutive levels that at least half of the episodes clear on the ten-level objectives, and the premature exit rate (PER) counts episodes the agent abandons. Under Code Execution CUA, the native GUI action ratio (NGAR) is the share of steps that consist only of screen observation or native GUI input (Appendix B.3). 4.2NATIVE GUI PLAY FALLS SHORT ON HARDER TIERS With Native GUI Actions, most agents solve mainly the easiest puzzles, whereas the human solves all 16 objectives (Table 2). Of the 84 successful episodes of the general-purpose backbones, 52 fall 5 Preprint on each game’s easiest tier, 20 on Medium, and 12 on harder tiers, all by GPT-6-Astra (6), GPT- 6-Sol (4), and GPT-5.6-Sol (2). The other seven solve 70.0–100.0% of easiest-tier levels but only 9.0–48.0% of harder-tier levels, and GPT-6-Astra solves every objective except Bolt Unscrew Hard. These successes suggest that operating the interface is not the main barrier; the barrier is carrying solving to boards that demand longer chains of coupled moves. GUI-specialized models do not close this gap: the four clear eight levels in 64 episodes, complete no objective, and reach an FGP of only 2.5–6.6, so screenshot-grounding training does not carry over to sustained puzzle play. 4.3CODE EXECUTION CUA MIXES VISUAL SOLVING WITH SEARCH Under Code Execution CUA, OSR ranges from 36.1% to 91.7% (Table 2), but agents reach these scores in two ways. GPT-6-Astra acts natively in 65.7% of steps and reaches the same OSR as with Native GUI Actions (91.7%). GPT-5.6-Sol, Qwen3.8 Flash, and DeepSeek 4.1 Flash, second, fourth, and sixth in OSR, do so in only 8.5%, 17.5%, and 11.7% of steps; their other steps search for moves over transcribed boards, parse screenshots pixel by pixel, or query page geometry. None accesses game source or evaluator state (Appendix B.2), so these steps are permitted, but, as in program-aided reasoning (Gao et al., 2023), they move the solving outside the model’s visual reasoning. An agent that transcribes a board can search it offline and replay the solution in one call; on levels solved in both settings, such play takes 60% of the time needed with Native GUI Actions. The saving raises completion only where play with Native GUI Actions is still solving levels at the deadline, as in the 11 timeouts with Native GUI Actions that Code Execution CUA completes (Appendix C.2). Removing code lowers OSR for six of the ten backbones, but not in proportion to their reliance on computation. DeepSeek 4.1 Flash, with an NGAR of 11.7%, falls from 67.2% to 53.9% OSR, and Kimi-k3, with an NGAR of 59.3%, falls by a similar margin, from 70.6% to 56.7% (Appendix C.1). GPT-6-Sol, GPT-5.6-Terra, and GPT-5.6-Luna even score higher with Native GUI Actions (by 2.8, 3.3, and 5.6 points), partly because each abandons more episodes under Code Execution CUA. The settings also differ in tool integration (Appendix B.2), so these gaps characterize each setting as a whole rather than the action channel alone. We therefore treat Native GUI Actions results as the measure of visual solving and read every Code Execution CUA score together with its NGAR. 4.4WHERE PERFORMANCE BREAKS DOWN Harder tiers separate agents that tie on easy ones. Under Code Execution CUA, six agents solve every easiest-tier level, yet their LC hard spans 56.8–90.0% (Figure 3a). Maze Paint ties this spread to solution length: solver-optimal solutions grow from 8.8 to 21.7 slides per level from Easy to Hard, all ten agents clear every Easy level under Code Execution CUA, and Hard-level LC ranges from 10% to 100% (Figure 3b). Easy tiers saturate; the harder tiers measure long-horizon solving. Small per-level gaps become large objective gaps. Levels are won in catalogue order in every valid episode, so one unsolved level forfeits all later ones. With Native GUI Actions, GPT-6-Sol exceeds GPT-5.6-Sol by 3.7 points in LPR but by 11.7 points in OSR, and Kimi-k3 passes 92.6% of the levels it reaches on the ten-level objectives, yet only four of its eight episodes clear all ten. Across agents, only 65% of episodes under Code Execution CUA and 52% with Native GUI Actions clear all ten levels (Figure 3c), so scoring levels in isolation would understate how far agents remain from sustained solving. Some agents stop early. GPT-5.6-Luna, GPT-6-Luna, and GPT-5.6-Terra, the three weakest propri- etary agents in OSR, voluntarily end 62.5–75.0% of their episodes under Code Execution CUA and 56.2–68.8% with Native GUI Actions; since progression stays with the agent, the decision to keep playing is part of the measured capability. 4.5HUMANS AND AGENTS SUCCEED ON DIFFERENT PUZZLES No episode in either setting clears more than four of the eight Bolt Unscrew Hard levels, all of which the human completes, yet GPT-6-Astra solves Nut and Bolt far more directly than the human. We compare the human with Astra’s evaluation episodes at shared puzzle states, mainly those with Native GUI Actions, which match the human’s interface (Appendix C.3). From the same Nut and Bolt Nightmare state, Astra needs 65 stack-state changes (67 under Code Execution CUA) and never breaks a solved stack, whereas the human needs 431 changes and 17 resets; Astra also uses fewer moves on all six Maze Paint and Rush Hour objectives, though more gestures in Color Connect. Bolt 6 Preprint 255075100 Easiest-tier LC (%) 0 50 100 Harder-tier LC (%) Human 6 code agents at 100% easiest (a) Harder tiers separate agents Easy 8.8 Medium 14.0 Hard 21.7 Maze Paint: optimal slides per level 0 50 100 (b) Longer mazes split agents 0246810 First k levels cleared 0 50 100 Episodes (%) 65% 52% (c) Failures compound Figure 3: Performance falls as the required horizon grows (all ten general-purpose agents in both settings; values as in Table 2). Filled blue: Code Execution CUA; hollow orange: Native GUI Actions; darker shades: higher LC hard ; star and dashed lines: human. (a) Level completion on each game’s easiest tier against its harder tiers. (b) Maze Paint level completion per tier, labeled with mean solver-optimal slides per level. (c) Episodes that clear the first k levels of the eight ten-level objectives; thin: agents, bold: mean over agents. 051015 Move 0 1 2 3 Free holes same board agent, 10/12 retries: dead end, 4/24 boards verified solution: 24/24 (a) Same start, different spending 3 n=61 2 n=73 1 n=73 Free holes before the move 0 50 100 Agent moves (%) frees a board fills a hole dead end (b) The last hole gets filled 050100 Board painted (%) 0 50 100 Valid moves (%) unsolved solved revisit new cells (c) No new cells, then loops Figure 4: Moves that look like progress spend solvability; moves without visible progress loop (RQ1). (a) Free holes along Bolt Unscrew Hard level 1 from one start: a verified solution and the agent path repeated in 10 of 12 retries. (b) Effect of agent moves by free holes before the move, over all Bolt Unscrew retries. (c) Maze Paint, 260 attempts with Native GUI Actions: share of valid moves that revisit an exact earlier state or paint new cells, by board coverage. Unscrew Hard reverses the ranking. On level 4, the human and both Astra episodes first fill the four free holes and deadlock before removing a board. The human then changes the opening, removes 12 boards on the next attempt, and clears the level on the third; Astra under Code Execution CUA repeats the zero-board dead end in its first five attempts and clears the level on the eighth, and Astra with Native GUI Actions reaches the level with two minutes left (Appendix Figure 8). The reversal follows what each puzzle demands. Nut and Bolt rewards bookkeeping over a visible state in which progress accumulates as solved stacks; Bolt Unscrew rewards preserving capacity, and a free hole is not visible progress. Current agents thus match or exceed the human at maintaining explicit state but fall short when a move must be judged by the options it preserves. This contrast rests on one human trajectory per setting; Section 5.1 tests the capacity mechanism across agents and retries. 5FAILURE DIAGNOSIS: WHY DO AGENTS FAIL? The gap lies on harder tiers and long level sequences (Section 4). With Native GUI Actions, 66– 80% of each general-purpose agent’s missing level completion lies in levels it never reached (Ap- pendix D.2), so an objective is lost inside the few levels where progress stops. This section follows such a level from the move that loses it to the attempt that follows: how agents lose a solvable board (RQ1), whether they notice (RQ2), whether recovery and memory produce a better attempt (RQ3), and whether knowing the rules would have prevented the loss (RQ4). Each diagnostic restores a reproducible puzzle state and varies one factor while the agent, task, and starting board stay fixed (Appendix D.1). Mechanism evidence comes mainly from Bolt Unscrew and Maze Paint, where the hidden cost of a move can be counted. Controlled studies use GPT-5.6-Sol and repeat the key contrasts with GPT-5.6-Terra and GPT-5.6-Luna; in this section, Sol, Terra, and Luna denote these three GPT-5.6 models. 7 Preprint 5.1MOVES THAT LOOK LIKE PROGRESS SPEND SOLVABILITY RQ1: How do agents lose a solvable board? Two games make the hidden cost of a move mea- surable. In Bolt Unscrew, a screw moved without freeing a board occupies one of a few free holes, and a level with no free hole left is a dead end. In Maze Paint, the last cells can be reached only by sliding across painted ones, so late moves rarely show progress. Finding 1. Agents choose moves by what they change now, rarely by the options they leave. Visible progress spends capacity. From the same start of Bolt Unscrew Hard level 1, the agent and a verified solution both remove four boards in three moves and keep two free holes. The solution’s next move frees three more boards and reopens a hole. The agent instead fills the remaining holes and deadlocks with 20 of 24 boards still attached, along the same path in 10 of 12 retries (Figure 4a). The pattern holds across retries: with one free hole left, three quarters of agent moves fill it and end the level at once (Figure 4b), as do 15 of 17 first moves from solvable level starts with a single free hole. Many are misjudgments, not slips: in 28 of the 55 fatal moves, the agent’s stated reason predicts that the move will release a board. Without visible progress, agents loop. Early in a Maze Paint level, nearly every move paints new cells. As the board fills, solved attempts slow down but keep reaching new states. Unsolved attempts cycle instead: in the last fifth of the board, only 15% of their valid moves paint new cells and 30% return to an exact earlier state, against 50% and 6% in solved attempts (Figure 4c). This contrast is descriptive, since solved and unsolved attempts are not matched for difficulty; per-agent rates appear in Table 11. Replacing one costly move with a shortest-path move from the same state barely changes completion (3/27 versus 4/27), which suggests that the loss accumulates over many moves rather than arising from one error. Both games point to the same gap. Agents respond to what the screen shows, such as empty holes and unpainted cells, but not to the quantities that decide whether the level can still be finished: the capacity a move consumes and the states already tried. Neither is drawn on the board, so a more accurate reading of the current screen is unlikely to repair the failure on its own. 5.2DEAD ENDS LOOK LIKE MISSED CLICKS RQ2: Do agents notice that a board is lost? We restore 24 certified Bolt Unscrew dead ends and let Sol continue with no hint (A), with a truthful statement that the board requires recovery (B), or with B plus the name of the restart control (C). Solvable boards serve as controls. Finding 2. Agents tend to blame a lost board on their own execution, not on the puzzle state. On a dead board, every move is rejected, just as a mis-aimed click would be. Without a hint, two thirds of runs explain the rejection as an execution error (“the screw did not move; try the other hole”) and retry other targets. Only 8 of 24 runs recover, and only one restarts promptly (Figure 5a). The diagnosis is within reach: 7 of those 8 runs state the cause correctly (“both storage holes are occupied without releasing a plank”). One sentence of state information reverses the attribution. Click-error explanations fall to 3 of 24 runs and recovery rises to 23 of 24, and from 7 to 16 of 18 across three models, while no solvable control triggers a restart. Naming the restart control adds nothing, and in Nut and Bolt no model recovers under any condition (Appendix D.3). In Bolt Unscrew, the bottleneck is therefore the inference behind the restart, not the restart action: whether an unexpected outcome is attributed to the world or to one’s own execution. Blaming execution is a reasonable default in GUI control, where a rejected action often means a missed click (Cheng et al., 2024; Xie et al., 2024), but in a puzzle that never announces a dead end, it leaves most dead ends unrecognized. 5.3RECOVERY RESETS THE BOARD, NOT THE PLAN RQ3: Does recovery lead to a better attempt? A restart returns the agent to a solvable board, and the restarted agent can keep direct evidence of how its last attempt failed. We examine the recovered RQ2 runs and 72 matched retries that branch each agent-owned Bolt Unscrew failure 8 Preprint NoneState +Label 0 50 100 Dead-end runs (%) recovers blames the click solves (a) State hint ClearedHistory Summary 0 50 100 Retries (%) same dead end solves 0/24 summaries propose a new plan (b) Failure memory None Card 0 50 100 Sessions, mean (%) rules stated levels completed (c) Rule card Figure 5: Each support moves the behavior it targets far more than it moves completion (RQ2–RQ4). (a) Sol on 24 certified Bolt Unscrew dead ends with no hint, state information, or state information plus the restart label. (b) Retries from the restarted board with the prior attempt cleared, kept as history, or replaced by an agent-written summary. (c) Sol sessions without and with the rule card (P0), means over 16 objectives. Solid: GPT-5.6-Sol; dashed: GPT-5.6-Sol, -Terra, and -Luna (18 units per condition). from the same restarted board. The prior attempt is cleared (M0), kept as native history (M1), or replaced by an agent-written failure summary (Shinn et al., 2023) of equal size (M2). Finding 3. Agents remember what they tried, not why it failed, so memory does not keep them from rebuilding the same dead end. None of the 153 dead-end continuations in RQ2 completes the level. Of the 45 runs that recover under B or C, 34 build a new certified dead end, at a median of 129 seconds after recovery, and 29 of these make visible progress first. Memory does not change completion: no retry completes the level under any memory condition (0/126, including the three-model repeat; Figure 5b). The memory variants choose different first moves in 20 of 24 sources, yet 79% of retries end in the exact dead end of their source, and agents holding the full history return to it at least as often as agents without it. The summaries suggest why. All 24 attribute the failure to execution or missing observations (“the click did not register”), none identifies that the free holes ran out, and none proposes a different plan; 15 instead direct the next attempt to resume the failed one (Figure 10b). A failure record helps only if it encodes a causal lesson that constrains the next plan. These records encode actions and open questions: the attribution that hides a dead end (Finding 2) also keeps it out of memory, so a restart restores the capacity without a reason to spend it differently. 5.4KNOWING THE RULES IS NOT PLANNING WITH THEM RQ4: Would knowing the rules have prevented these failures? Findings 1–3 could reflect rules the agent never inferred, and full sessions add a second burden: entering games and advancing levels. RQ4 crosses a mechanics card that states each game’s rules (K1 versus K0) with a deterministic controller for entry and advancement (P1 versus P0) on all 16 objectives, leaving every puzzle move to the agent. Periodic probes measure which rules the agent can state. Finding 4. Rule knowledge helps, but agents can state a puzzle’s rules well before they can plan with them. Agents do discover rules through play: without the card, most rules that Sol ever states correctly are first stated during play (Appendix D.5). The card doubles rule knowledge, from 43% to 88% of rules stated correctly, yet raises level completion only from 50% to 60% (Figure 5c), and 7 of the 16 sessions whose final probe states every rule correctly still leave levels unsolved. Across three mod- els, the card adds about nine LC points on average, with large variation across objectives; workflow control removes 19–29 model actions per session but does not reliably add levels (Figure 10a). Rules describe what one move does; solving requires predicting what a sequence of moves does to future options. Knowing that a screw needs an empty hole does not tell an agent that filling the last one ends the level, the misjudgment behind the dead ends of RQ1. Before committing to the move, the agent must predict whether it will release a board and reopen capacity for subsequent moves. 9 Preprint 6CONCLUSION LongPuzzleBench evaluates long-horizon visual reasoning through self-paced GUI puzzles in which early decisions constrain later moves and agents must recognize and recover from dead ends. Per- formance falls on harder, longer boards, and computational assistance does not close the gap. Con- trolled diagnostics trace failures to a common limitation: agents judge moves by immediate visible progress while overlooking the future options they consume. State hints, failure memory, and rule cards change the targeted behaviors far more than completion. These findings make preserving fu- ture options a central challenge for visual puzzle agents. Evaluating this capability requires tasks where decisions remain consequential over many moves and success depends on keeping a plan viable through completion. AI USE STATEMENT AI tools were used only to polish the language, grammar, and phrasing of this paper. They were not used to generate research ideas, experimental conclusions, or scientific findings. The authors take full responsibility for the content of the manuscript. ETHICS STATEMENT This work evaluates AI agents on puzzle games in a controlled, locally run browser environment. It does not involve human subjects, sensitive personal data, or real-world high-risk deployment; the human reference consists only of in-game actions and board states. We therefore identify no ethical concerns that require further discussion. REPRODUCIBILITY STATEMENT Sections 3 and 4.1 summarize the benchmark, the two action settings, and the evaluation proto- col. Appendices A.1 and A.2 specify the games, objectives, progress potentials, and deterministic launch parameters; Appendices B.1–B.4 give the episode protocol, agent endpoints and inference settings, metric definitions, and token usage; and Appendix D.1 gives the shared design of the con- trolled diagnostics. Each comparison fixes the catalogue, runtime, and seed. The benchmark code, game suite, and evaluation harness are available at https://anonymous.4open.science/ r/LongPuzzleBench-CD1A. REFERENCES ARC Prize Foundation. ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence, 2026. URL https://arxiv.org/abs/2603.24621. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL Technical Report, 2025. URL https://arxiv.org/abs/2511.21631. L ́ eo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin.WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks.In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, p. 5996–6051. Curran Associates, Inc., 2024.doi: 10.52202/079017-0195. URL https://proceedings.neurips.c/paper_files/paper/2024/file/ 0b82662b6c32e887b252a74d8cb2d5e-Paper-Datasets_and_Benchmarks_ Track.pdf. 10 Preprint Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu, Bin Yang, and Minying Zhang. Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents, 2026. URL https://arxiv.org/abs/2605.29447. ICML 2026 Spotlight. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9313–9332, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.505. URL https://aclanthology.org/2024.acl-long.505/. Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, p. 16259–16273, Bangkok, Thailand, August 2024. Association for Computational Linguistics.doi: 10.18653/v1/2024.findings-acl.962.URL https://aclanthology.org/2024.findings-acl.962/. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 28091–28114. Curran Associates, Inc., 2023. doi: 10.52202/075280-1220. URL https://proceedings.neurips.c/paper_files/paper/2023/file/ 5950bf290a1570ea401bf98882128160-Paper-Datasets_and_Benchmarks. pdf. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capa- ble are web agents at solving common knowledge work tasks? In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 11642–11662. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/drouin24a.html. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.PAL: Program-aided language models.In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, p. 10764–10799. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/gao23f.html. Deepanway Ghosal, Vernon Toh, Yew Ken Chia, and Soujanya Poria. AlgoPuzzleVQA: Diagnos- ing multimodal reasoning challenges of language models with algorithmic multimodal puzzles. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), p. 9615–9632, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.naacl-long.486. URL https://aclanthology.org/2025.naacl-long.486/. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 8154–8173, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023.emnlp-main.507. URL https://aclanthology.org/2023. emnlp-main.507/. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: A Visual Language Model for GUI Agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), p. 14281–14290. IEEE, 2024. doi: 10.1109/CVPR52733.2024.01354. 11 Preprint Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang.lmgame-Bench: How Good are LLMs at Play- ing Games?In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, p. 81308–81356, 2026. URL https://proceedings.iclr.c/paper_files/paper/2026/file/ 83a4ea71b13bc86308a2bd0b5e07fb61-Paper-Conference.pdf. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou.Large Language Models Cannot Self-Correct Reasoning Yet.In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, p. 32808–32824, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/ 8b4add8b0a8749d80a34ca5d941c355-Paper-Conference.pdf. Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, and Hongcheng Guo. GUI Agents for Continual Game Generation, 2026. URL https://arxiv.org/abs/2605.28258. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, p. 58791–58831, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/ 94074d5a072d28f75a76dabed43767-Paper-Conference.pdf. Andreas Junghanns and Jonathan Schaeffer. Sokoban: Enhancing General Single-Agent Search Methods Using Domain Knowledge. Artificial Intelligence, 129(1–2):219–251, 2001. doi: 10. 1016/S0004-3702(01)00109-6. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried.VisualWebArena: Evaluat- ing Multimodal Agents on Realistic Visual Web Tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), p. 881–905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.50. URL https://aclanthology.org/2024.acl-long.50/. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel Ziegler, Eliz- abeth Barnes, and Lawrence Chan.Measuring AI Ability to Complete Long Soft- ware Tasks.In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghas- semi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, p. 92213–92266. Curran Associates, Inc., 2025.doi: 10.52202/ 085713-3086.URL https://proceedings.neurips.c/paper_files/paper/ 2025/file/85069585133c4c168c865e65d72e9775-Paper-Conference.pdf. Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, and Mike Zheng Shou. Game- World: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents, 2026. URL https://arxiv.org/abs/2604.07429. Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wołczyk, Ak- bir Khan, Eduardo Pignatelli, Łukasz Kuci ́ nski, Lerrel Pinto, Rob Fergus, Jakob Foer- ster, Jack Parker-Holder, and Tim Rockt ̈ aschel.BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games.In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, p. 96666–96702, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/ f0b1515be276f6ba82b4f2b25e50bef0-Paper-Conference.pdf. 12 Preprint Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, and Heng Ji. EscapeBench: Towards Advanc- ing Creative Intelligence of Language Model Agents. In Wanxiang Che, Joyce Nabende, Ekate- rina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 798–820, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.39. URL https://aclanthology.org/2025.acl-long. 39/. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jia- hao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering Automated GUI Interaction with Native Agents, 2025. URL https://arxiv.org/abs/2501.12326. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Bench- marking Environment for Autonomous Agents. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, p. 406–441, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/ 01a83bc2f2732a58e6a731e659e7101-Paper-Conference.pdf. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning.In Advances in Neural Information Processing Systems, volume 36, p. 8634–8652, 2023. doi: 10.52202/075280-0377. URL https://proceedings.neurips.c/paper_files/paper/2023/hash/ 1b44b878b782e6954cd888628510e90-Abstract-Conference.html. Maximilian Triebel, Marco Menner, and Dominik Helfenstein. Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?, 2026. URL https://arxiv.org/abs/2605.11223. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kamb- hampati. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, vol- ume 36, p. 38975–38987. Curran Associates, Inc., 2023.doi: 10.52202/075280-1693. URL https://proceedings.neurips.c/paper_files/paper/2023/file/ 7a92bcdede88c7afd108072faf5485c8-Paper-Datasets_and_Benchmarks. pdf. Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. SmartPlay: A Benchmark for LLMs as Intelligent Agents. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, p. 1543–1561, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/ 063d1fadbde557466bf5bcffe814a64e-Paper-Conference.pdf. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Sil- vio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multi- modal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neu- ral Information Processing Systems 37, NeurIPS 2024, p. 52040–52094. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024. doi: 10.52202/079017-1650. URL https://doi.org/10.52202/079017-1650. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.Tree of Thoughts:Deliberate Problem Solving with Large Lan- guage Models.In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 13 Preprint 11809–11822. Curran Associates, Inc., 2023.doi:10.52202/075280-0517.URL https://proceedings.neurips.c/paper_files/paper/2023/file/ 271db9922b8d1f4d7aaef84ed5ac703-Paper-Conference.pdf. Lance Ying, Ryan Truong, Prafull Sharma, Kaiya Ivy Zhao, Nathan Cloos, Kelsey R. Allen, Thomas L. Griffiths, Katherine M. Collins, Jos ́ e Hern ́ andez-Orallo, Phillip Isola, Samuel J. Ger- shman, and Joshua B. Tenenbaum. AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games, 2026. URL https://arxiv.org/abs/2602. 17594. Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench: Can Vision-Language Models Complete Popular Video Games?, 2026. URL https://arxiv. org/abs/2505.18134. Zeyu Zhang, Zijian Chen, Zicheng Zhang, Yuze Sun, Yuan Tian, Ziheng Jia, Chunyi Li, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai. PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving, 2025. URL https://arxiv.org/abs/ 2504.10885. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 61349–61385. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/ v235/zheng24e.html. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xi- anyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neu- big.WebArena: A Realistic Web Environment for Building Autonomous Agents.In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), In- ternational Conference on Learning Representations, volume 2024, p. 15585–15606, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/ 4410c0711e9154a7a2d26f9b3816d1ef-Paper-Conference.pdf. 14 Preprint Appendix Contents A Benchmark Details15 A.1 Games, objectives, and progress potentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Construction, validation, and refresh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3 Benchmark characterization statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.4 Criteria for the benchmark comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B Evaluation Protocol and Metrics18 B.1 Episode protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2 Action settings and agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.3 Metric definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.4 Compute and cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C Extended Main Results21 C.1 Per-objective results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.2 Behavioral sources of the gap between settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3 Human–agent trajectory comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 D Diagnostic Studies25 D.1 Common design and validity rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 D.2 RQ1: progression and move-level measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 D.3 RQ2: certified dead-end takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 D.4 RQ3: matched retries with memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 D.5 RQ4: rule card and workflow control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 E Complete Per-Objective Results32 ABENCHMARK DETAILS A.1GAMES, OBJECTIVES, AND PROGRESS POTENTIALS Table 3 lists the games and objectives. Table 3: Games and objectives. Each game–difficulty pair is one objective; level counts follow the listed difficulty order. The completion condition applies to each level. GameDifficultiesLevels Level completion Bolt UnscrewEasy / Hard8 / 8 Remove every board. Rush HourEasy / Medium / Hard10 / 10 / 10 Move the target vehicle through the exit. Nut and BoltEasy / Medium / Hard / Extreme / Nightmare 3 / 3 / 3 / 3 / 1 Make every nonempty stack full and monochromatic. Truck EscapeDefault5 Remove all vehicles. Maze PaintEasy / Medium / Hard10 / 10 / 10 Paint every traversable cell. Color ConnectEasy / Hard10 / 10 Connect every matching color pair with non-overlapping paths. Total16 objectives114 Controls. Bolt Unscrew uses clicks to select and relocate screws. Rush Hour uses drags to move vehicles along their axes. Nut and Bolt uses a source click and a destination click to transfer the top run of same-colored nuts, subject to capacity and color. Truck Escape uses clicks to request vehicle departures, Maze Paint uses directional swipes, and Color Connect uses clicks to construct paths. Every game provides native restart and menu controls, and Nut and Bolt also provides undo. The agent observes the full browser viewport, including game menus, and the game’s own instructions and feedback stay visible throughout. Progress potentials. FGP (Appendix B.3) credits partial progress through a game-specific potential P g (s) ∈ [0, 1] that the evaluator computes privately for each observed state s. Bolt Unscrew uses (b exit + 0.5 b released )/b total , where released boards have no remaining supports but have not exited. Rush Hour combines estimated distance and structural progress with weights 0.8 and 0.2 when a canonical reference path is available, and blocker clearance and target advance with weights 0.7 and 0.3 otherwise; unfinished values are capped at 0.99. Nut and Bolt sums, over colors, the largest homogeneous stack of each color and divides by stack capacity times the number of colors. Truck 15 Preprint Escape uses the fraction of vehicles removed, Maze Paint (n painted − 1)/(n paintable − 1), which excludes the initially painted start cell, and Color Connect the fraction of completed pairs. A solved level scores one. A.2CONSTRUCTION, VALIDATION, AND REFRESH Construction. A coding model (GPT-5.6-Sol with xhigh reasoning) reproduces executable games and levels from screenshots or other visual references. Part of the suite builds on an existing game implementation, to which we add task data, private evaluation, and fixed launch parameters; Maze Paint and Color Connect are new. Every game implements a browser evaluation bridge with private completion and progress signals and supports deterministic selection of game, difficulty, level, and seed. Validation. GUI agents play each candidate build to check controls, state transitions, and game behavior; a defect leads to repair and a new playtest, or to exclusion of the candidate. Solvers and reference replays add checks where implemented, and manual play verifies that every released level is solvable. For Maze Paint, a solver solves all 30 levels, replays each solution to completion, and checks the exact minimum number of slides: 88, 140, and 217 in total for Easy, Medium, and Hard (7–12, 13–16, and 18–25 per level). These counts exclude game entry, transitions, exploration, and recovery. Refresh. Validated games and levels are added between evaluation snapshots, and comparisons within a snapshot fix its catalogue, runtime, and seed. Refresh limits repeated exposure to a fixed catalogue but does not by itself establish freedom from contamination: a game reproduced from screenshots can retain familiar mechanics or layouts, and a post-training evaluation would also re- quire the release dates of the added instances and the training cutoff of each model. A.3BENCHMARK CHARACTERIZATION STATISTICS These definitions underlie Figure 2b–c and the corresponding statistics in Section 3. Human reference. One human player completed each of the 16 objectives under the evaluation protocol (same start screen, seed, viewport, native GUI actions, and 60-minute budget), without private state, solver output, or external computation. Actions count pointer presses, one per click, drag, or swipe; pointer-move samples are excluded (Table 4). The human issued 99 restarts and 55 undos, on 9 of the 16 objectives. Table 4: Human reference episodes: target levels, pointer presses, and recovery actions per objective. ObjectiveLevelsPointer pressesRestartsUndos Bolt Unscrew / Easy810010 Bolt Unscrew / Hard82,500520 Rush Hour / Easy108900 Rush Hour / Medium1010200 Rush Hour / Hard10442120 Nut and Bolt / Easy34500 Nut and Bolt / Medium316911 Nut and Bolt / Hard340523 Nut and Bolt / Extreme353888 Nut and Bolt / Nightmare11,1061943 Truck Escape / Default513400 Maze Paint / Easy1012800 Maze Paint / Medium1023800 Maze Paint / Hard1047610 Color Connect / Easy1011800 Color Connect / Hard1031630 Dead-endcertification.InBoltUnscrew,theevaluatorcertifiesadeadlock (noavailablehole) when the board is stable, no operation is pending, and no admissi- ble source–target screw relocation remains; the game does not display this state. In Nut and Bolt, every distinct settled board of a level attempt is classified by exhaustive search over the states reachable under the transfer rule, in which the top run of same-colored nuts moves onto an empty bolt or a matching top nut, as many as fit; every visited board was classified within the search limit. 16 Preprint An attempt reaches a dead end if it reaches an unsolvable board. Its delay counts transfers from the first unsolvable board, including the transfer that produced it, until an undo restores a solvable board or the attempt ends. Snapshots record settled boards, so transfers between two snapshots can be missed and delays are lower bounds. Of the 18 human dead-end attempts in Nut and Bolt, 16 ended in a restart and 2 in an undo; of the 83 agent attempts, 51 ended in a restart, 29 in an undo, and 3 at the end of the episode. The other four games have no certificate, and their states are not annotated as dead ends. Agent statistics. Agent counts pool the 384 episodes of Table 2 (both settings, all 14 agents), in which agents issued 996 restarts and 384 undos. A.4CRITERIA FOR THE BENCHMARK COMPARISON Scope. Table 1 includes representative benchmarks from four paradigms adjacent to long-horizon visual reasoning: static visual puzzles, GUI task completion, interactive games, and interactive puzzles. Text-only game and puzzle benchmarks discussed in Section 2, such as SmartPlay and EscapeBench, are omitted because they do not require interpreting a rendered state (Wu et al., 2024; Qian et al., 2025). Because its play is self-paced, LongPuzzleBench does not test the latency- sensitive control that real-time play in VideoGameBench and GameWorld also evaluates (Zhang et al., 2026; Ouyang et al., 2026). Each entry follows the published task definition and default protocol of the cited benchmark; † marks a property present only in a subset of tasks or in one evaluation track, and – marks a property that does not apply or is not reported. Property definitions. Visual puzzle state: success is defined by rule constraints on a rendered envi- ronment state, and the action sequence must be derived from those constraints rather than read from the task instruction. Delayed consequences: actions update a persistent state over many steps, and whether the task succeeds depends on changes made many steps earlier, not only on the next few decisions. Dead-end states: the task rules admit legal actions after which the goal is unreachable without undo or restart; irreversible side effects that occur incidentally in application tasks do not qualify. In LongPuzzleBench, this holds in Bolt Unscrew and Nut and Bolt (7 of 16 objectives); in the other four games, moves are reversible or only remove pieces. Agent-owned recovery: after a failure, the agent itself decides whether and when to undo, restart, or revise within the episode, with- out a scripted runtime reset; single-attempt episodes are marked×, and settings without dead ends or attempts are marked –. Native GUI actions: the environment changes only through coordinate-based mouse, touch, or key inputs on a rendered application, excluding element-ID commands, abstract game actions, and programmatic state access. Per-benchmark notes. PuzzleBench poses questions about a presented configuration, so no action changes the state and the protocol columns do not apply. VisualWebArena and AndroidWorld tasks involve dependent steps on persistent application state, but they are specified by instructions rather than puzzle constraints, and their definitions do not target dependencies that surface many steps later. VisualWebArena acts through element-ID commands on set-of-marks annotations. BALROG provides visual observations in its vision-language track; its puzzle entry reflects BabaIsAI, one of its environments. VideoGameBench and GameWorld contain puzzle games within broader game suites. Some VideoGameBench games use scripted startup, and GameWorld continues after terminal failure through a runtime reset; both therefore take part of the session lifecycle out of the agent’s control. GameWorld supports native inputs in a subset of its settings. VLATIM evaluates puzzles designed for a single attempt. ARC-AGI-3 acts through abstract game actions; its agents restart a level after GAME OVER through a RESET action, so recovery is agent-owned. AI GameStore is omitted because its generated games with short episodes add no evaluation regime beyond the game benchmarks listed (Ying et al., 2026). Instance renewal. Renewal concerns evaluation maintenance rather than the demands of the task, so Table 1 omits it. PuzzleBench, AndroidWorld, and BALROG generate new instances from fixed task families; AI GameStore adds games through synthesis and human refinement; the other bench- marks use fixed task sets, and ARC-AGI-3 reports public and private sets without a refresh protocol. LongPuzzleBench adds validated levels and mechanics between snapshots (Appendix A.2). 17 Preprint (a) GUI task completion Find a matching item Plan and act Interface feedback User-specified task (b) Rule-guided gameplay Rules + goal Choose movesGame feedback Play using supplied mechanics (c) Self-directed gameplayLongPuzzleBench Complete all levels of a game at a specified difficulty Select difficulty Easy Medium Hard Enter gameExplore and solveAdvance level Infer mechanics from visual feedback Next puzzle Continue or recover? Plan next move Restart and revise Agent-directed progression within one episode Figure 6: Task guidance and gameplay responsibility in representative settings. (a) An agent uses interface feedback to complete a user-specified application task. (b) Games such as SmartPlay and BALROG supply rules and action descriptions to guide play. (c) LongPuzzleBench supplies a game–difficulty objective; the agent discovers mechanics through visual interaction and controls entry, puzzle solving, recovery, and advancement within one episode. BEVALUATION PROTOCOL AND METRICS B.1EPISODE PROTOCOL Start and instruction. Each episode starts at the suite home screen. The agent receives the GUI action specification, the coordinate convention, and the objective instruction of Section 3; no man- ual, solution, navigation procedure, or evaluator signal is supplied. The agent enters the game and decides when to advance, retry, restart, or re-enter (Figure 6). The player instructions require the agent to act only through the visible interface, to keep playing until the goal is visibly complete, and to end only with an explicit completion declaration. Budget and termination. Each objective has a 3,600-second wall-clock budget and no action limit. The clock runs across levels, retries, and re-entry and includes navigation, play, waits, model infer- ence, and transient request retries. An episode ends at full completion, at an explicit final response by the agent, or at the deadline. Success requires evaluator snapshots that show a win on every target level; a completion declaration by the agent is not sufficient. For the GUI-specialized models, the environment answers an unverified completion claim once with a continuation prompt, and a second claim ends the episode. Scored episodes. Completions, deadline terminations, and voluntary exits are valid outcomes; a voluntary exit before completion counts as a failure and toward PER. Episodes stopped by provider, browser, environment, logging, or configuration failures, or stopped externally, are not scored. The main results comprise 384 episodes: 160 under Code Execution CUA and 224 with Native GUI Actions (160 from the Codex backbones and 64 from the GUI-specialized models), covering all 16 objectives for every agent in each of its settings. B.2ACTION SETTINGS AND AGENTS Table 5 summarizes the two settings of Section 4.1; the human reference plays with native GUI actions under the same protocol (Appendix A.3). Shared constraints. Both settings use the same objectives, seed-0 layouts, browser and viewport, budget, termination and validity rules, and aggregation. The agent may not read game source or bundled assets, level catalogues, answer files, DOM or accessibility data, page variables, hidden game state, or evaluator outputs, and may not retrieve game information externally. The player instructions state this prohibition in both settings, and every scored trajectory is checked for source or answer access; no episode shows such access. 18 Preprint Table 5: The two evaluation settings. Both play the same game in a dedicated Chromium browser at 1280×900 and change it only through coordinate-based GUI input (bottom row); they differ in what the agent may do between inputs. Native GUI ActionsCode Execution CUA Action space: one GUI-action toolAction space: code runtime and GUI input click(x,y)primary clickRun JavaScript in a persistent runtime doubleclick(x,y)double clickover screenshots and derived data drag(x0,y0,x1,y1)drag pointerRequest a screenshot press key(key)game keyIssue the same coordinate GUI inputs, typetext(text)text entrydirectly or from code scroll(direction)scrollRead and write own files wait()idle(when file tools are available) batch([...])ordered actions Screenshot returned after every callScreenshots on request Backbones: ten general-purpose (Codex);Backbones: ten general-purpose (Codex) four GUI-specialized (autonomous harness) Metrics: OSR, LC, FGP, LPR, H 50 , LC hard , PERMetrics: as left, plus NGAR Shared in both settings. Game state changes only through pointer and key input on the rendered page, recorded by the game; one 3,600-second episode clock; no game source, bundled assets, level catalogues, DOM or accessibility data, page variables, hidden state, evaluator outputs, or external retrieval. Native GUI Actions: Codex backbones. The only action channel is one tool whose calls name the page URL, an action, and its parameters (Table 5). Coordinates are integer screenshot pixels. Keys are restricted to letters and game keys (Enter, Tab, Escape, Space, Backspace, and arrows), which excludes browser shortcuts and developer tools, and browser navigation is not exposed. A fixed dispatcher validates each call, applies it to the page, and returns a fresh screenshot; a batch executes its actions in order and returns one screenshot after the last, and each executed action counts separately. The first action must be wait, which returns the initial screenshot and starts the clock. The tool accepts no code, selectors, file paths, or page-state queries. A watchdog checks every call against the schema and the run’s URL, and any other tool call or schema violation stops and invalidates the run. The player prompt lists the actions and forbids JavaScript, shell, Python, browser-automation code, DOM manipulation, and any other substitute channel. Native GUI Actions: GUI-specialized models. UI-TARS-1.5-7B and the three Qwen3-VL Think- ing models run in an autonomous harness with the same action set and validation as the Codex tool, plus a terminal answer. Their serving endpoints do not support this tool, so each model acts in its own output format: Qwen3-VL as text function calls with coordinates on a 0–1000 grid, and UI-TARS-1.5 as Thought/Action text with coordinates on its resized input image. The harness converts both to screenshot pixels, validates them, and dispatches and records one primitive at a time. No code execution, DOM or accessibility data, or browser navigation is available. Code Execution CUA. The agent has a persistent JavaScript runtime, exposed through an MCP service, in addition to coordinate GUI input, and may read and write its own files when file tools are available. The instructions permit code execution and leave its use to the agent; they describe initialization, screenshots, and the GUI API without recommending a solving algorithm. Game information must come from returned screenshots. Programs may process these screenshots and any data the agent derives from them, but every change to the game must use the prescribed GUI inputs. Such programs are permitted computation, not privileged access: they lower NGAR but do not invalidate the episode. Prompts. Each Codex agent runs as a player with fixed developer instructions for its setting, added to the scaffold’s default system prompt, and receives one task message stating the goal, the game and difficulty with their on-screen labels, the page URL, and the setting. The game interface is in Chinese, and so are the Codex prompts; the GUI-specialized models receive the English instruction of Section 3 with the game’s on-screen name, plus a system prompt in their own action format. No prompt contains game rules or strategies. All prompts specify (i) the permitted information: what is visible in the returned screenshots, with the prohibited sources listed above; (i) the action interface: the tool schema, coordinate convention and bounds, permitted keys, and batch format for Native GUI Actions, and the JavaScript GUI API for Code Execution CUA; (i) the procedure: select the game and difficulty through the visible interface and check the selection before starting, and use the 19 Preprint visible undo or restart controls when needed, treating neither repeated failure nor elapsed time as a reason to stop; and (iv) the ending: once the page shows that the whole objective is complete, send one message that starts with BENCHMARK COMPLETED and names the visible evidence. Eighteen episodes (five under Code Execution CUA and 13 with Native GUI Actions, all from GPT-5.6-Luna and the open-weight backbones) used an earlier player prompt whose context also contained text from a since-removed orchestration layer; they pass the same access check. Table 6: Model access and inference settings. Effort: requested reasoning effort, identical in both settings and matched by the recorded value in every episode. Limit: context window configured in the Codex scaffold (general-purpose) or maximum output tokens per request (GUI-specialized), in thousands of tokens. Price: list price in USD per million uncached input, cached input, and output tokens, retrieved in September 2026 from each model’s first-party API or the listed endpoint; DeepSeek 4.1 Flash at peak-hour rates. AgentEndpointEffortLimitPrice (in / cached / out) General-purpose, Codex scaffold GPT-6-AstraOpenAImedium 25810.00 / 1.00 / 50.00 GPT-6-SolOpenAImedium 2582.00 / 0.20 / 10.00 GPT-6-LunaOpenAImedium 2580.10 / 0.01 / 0.50 GPT-5.6-SolOpenAImedium 2584.00 / 0.40 / 20.00 GPT-5.6-TerraOpenAImedium 2582.00 / 0.20 / 12.00 GPT-5.6-LunaOpenAImedium 2580.20 / 0.02 / 1.20 DeepSeek 4.1 FlashOpenAI-compatible high9960.30 / 0.006 / 1.20 Qwen3.8 FlashOpenAI-compatible xhigh996 a / 122 b 0.15 / 0.016 / 0.47 GLM-5.3-FlashOpenAI-compatible max950–9960.15 / 0.03 / 0.50 Kimi-k3OpenAI-compatible max9963.00 / 0.30 / 15.00 GUI-specialized, autonomous harness UI-TARS-1.5-7BOpenRouter–20.10 / 0.10 / 0.20 Qwen3-VL-32B-ThinkingSiliconFlow–80.20 / – / 1.50 Qwen3-VL-30B-A3B-Thinking SiliconFlow–80.29 / – / 1.00 Qwen3-VL-8B-ThinkingSiliconFlow–80.18 / – / 2.00 a Native GUI Actions. b Code Execution CUA. Inference and history. Table 6 lists the endpoint and settings of each agent. The Codex scaffold (Codex CLI 0.154–0.156) sets only the reasoning effort; other sampling parameters and output lim- its are provider defaults. Screenshots are 1280 × 900 PNG images. Codex agents keep the full interaction history until it fills the configured context window; the scaffold then replaces earlier turns with a model-written summary and continues. This compaction occurs in 42 of the 320 Codex episodes, 15 of them Qwen3.8 Flash episodes under Code Execution CUA, whose configured win- dow is the smallest. The GUI-specialized models are queried through the Chat Completions API without streaming and with provider-default sampling; each request replays the full text history with the two most recent screenshots. No auxiliary learned model is used. The game suite and browsers run locally on one workstation (Apple M5 Pro), and all models are queried through remote APIs. The episodes of Table 2 were collected between 16 and 26 September 2026. Comparability. Beyond the action channel, the two settings differ in tool-integration latency, and the GUI-specialized models also run in a different harness. Differences between settings, or between general-purpose and GUI-specialized agents, therefore characterize each agent system as a whole (Section 4.3). B.3METRIC DEFINITIONS For objective o with target levels L o , let c oℓ ∈ 0, 1 indicate an observed win on level ℓ in the episode. A won level counts once and keeps its credit across retries; an unvisited level counts as zero. Unless noted otherwise, every metric is computed per objective, averaged equally over the objectives of each game, and then equally over the six games. Completion. Objective success and level completion are s o = 1 P ℓ∈L o c oℓ =|L o | ,LC o = 1 |L o | X ℓ∈L o c oℓ ,(1) and OSR averages 100 s o . Fine-grained progress also credits partial solutions. Let b oℓ be the max- imum potential P g observed on level ℓ (Appendix A.1), set to one for a won level and zero for an 20 Preprint unvisited one; then FGP o = |L o | −1 P ℓ∈L o b oℓ . Because levels are won in catalogue order, FGP exceeds LC only through progress on the level where the episode stops. Hard-tier level completion (LC hard ) restricts LC to the objectives above each game’s easiest tier: Bolt Unscrew Hard, Rush Hour Medium and Hard, Nut and Bolt Medium through Nightmare, Maze Paint Medium and Hard, and Color Connect Hard. Truck Escape has a single tier and is excluded. Horizon. Let v eℓ indicate that level ℓ was reached in episode e, meaning that a ready gameplay state for the requested game, difficulty, and level was observed. Pooling the episodes E o of objective o, the level pass rate is LPR o = P e∈E o P ℓ∈L o c eℓ P e∈E o P ℓ∈L o v eℓ ,(2) which is undefined, for the objective and overall, when no target level is reached. The 50% level horizon uses the eight ten-level objectives O 10 (Rush Hour, Maze Paint, and Color Connect). With d e the number of levels solved in episode e, S(k) = 1 |O 10 | X o∈O 10 1 |E o | X e∈E o 1[d e ≥ k],H 50 = maxk ∈0, . . . , 10 : S(k)≥ 0.5, (3) reported as 10+ when S(10)≥ 0.5. Behavior. The premature exit rate (PER) is the percentage of scored episodes that end, without objective success, through a final response or stop action by the agent while budget remains and the interface is usable. Both an incorrect completion declaration and an explicit decision to abandon count; timeouts and evaluator or environment stops do not. The setting mismatch rate (SMR) is the percentage of episodes whose evaluator snapshots show gameplay in a game or difficulty other than the requested one before completion or termination; such an episode stays in every denominator, and only progress on the requested levels earns credit. PER and SMR pool episodes within each displayed scope. Native GUI action ratio. NGAR is 100 N native /(N native + N other ), pooled over all model action steps of the Code Execution CUA episodes in scope; a step is one tool call or an explicit terminal answer. A step is native when it only observes the screen or issues coordinate-based mouse, touch, or key input, including fixed coordinate batches; a call counts once even if it contains several inputs or an input fails. Offline solving, programmatic image analysis, helper or module authoring, file and coordination tools, standalone waits, terminal answers, semantic element targeting, synthetic DOM events, and direct DOM, accessibility, canvas, or game-state access are non-native, as is any call that combines computation with GUI input. Internal reasoning, commentary, tool results, and automatic recorder events are not steps. NGAR is not reported with Native GUI Actions, where every accepted action is native by construction. B.4COMPUTE AND COST Table 7 reports the tokens consumed by the 384 episodes of Table 2 and their cost at the list prices of Table 6. The episodes issue 45,997 model requests with 5.65 billion input tokens, 97.7% of them cache hits, and 30.7 million output tokens, for an estimated USD 868 (571 with Native GUI Actions and 297 under Code Execution CUA). Input dominates because every request resends the interaction history, including screenshots, and prompt caching bills most of it at the cached rate. The controlled diagnostics of Section 5 are not included. CEXTENDED MAIN RESULTS C.1PER-OBJECTIVE RESULTS Figure 7 gives level completion per objective for every general-purpose agent in both settings; Ap- pendix E lists every episode’s outcome and the denominators behind Table 2. Difficulty tiers are comparable within a game but not across games, whose layouts differ. The OSR losses of DeepSeek 4.1 Flash and Kimi-k3 without code (Section 4.3) fall on different objectives. Kimi-k3 solves Maze Paint Medium and Color Connect Hard only under Code Execution CUA (LC 100% against 30% and 40% with Native GUI Actions), and its Rush Hour Hard LC falls from 60% to 0% without code. 21 Preprint Table 7: Token usage and estimated API cost of the 384 episodes of Table 2. Input counts all prompt tokens, including cache hits; output includes reasoning tokens. Cost applies the list prices of Table 6 to uncached input, cached input, and output; it is an estimate, not a billed amount. AgentRequestsInput (M)Cached (%)Output (M)Cost (USD) Native GUI Actions GPT-6-Astra1,286100.797.90.19129.70 GPT-6-Sol2,592237.598.60.4858.41 GPT-6-Luna2,713288.498.20.373.53 GPT-5.6-Sol3,077327.598.30.55161.87 GPT-5.6-Terra2,132218.698.10.4456.32 GPT-5.6-Luna2,411233.297.80.325.97 DeepSeek 4.1 Flash3,4311,125.999.55.3314.85 Qwen3.8 Flash1,503233.197.42.165.56 GLM-5.3-Flash2,519508.898.11.0116.94 Kimi-k31,415213.797.81.3296.91 UI-TARS-1.5-7B7065.053.90.090.51 Qwen3-VL-32B-Thinking1,0006.00.01.573.55 Qwen3-VL-30B-A3B-Thinking2,18418.10.03.138.38 Qwen3-VL-8B-Thinking1,46510.10.03.378.56 Subtotal28,4343,526.597.520.32571.07 Code Execution CUA GPT-6-Astra1,13154.897.20.1073.58 GPT-6-Sol1,21368.797.40.2219.16 GPT-6-Luna2,501261.398.20.553.30 GPT-5.6-Sol1,468116.897.90.4063.51 GPT-5.6-Terra1,612134.197.60.4037.41 GPT-5.6-Luna1,21786.396.40.242.57 DeepSeek 4.1 Flash2,624714.199.44.0410.32 Qwen3.8 Flash1,65397.692.02.193.64 GLM-5.3-Flash2,617445.398.21.1114.90 Kimi-k31,527142.997.71.1368.74 Subtotal17,5632,121.998.110.38297.13 Total45,9975,648.497.730.70868.19 DeepSeek 4.1 Flash solves Bolt Unscrew Easy and Color Connect Hard only with code (100% against 87.5% and 70%) but Nut and Bolt Medium only without it (100% against 66.7%). EHEMHEMHXNDEMHEH Human GPT-6-Astra GPT-5.6-Sol Qwen3.8 Flash Kimi-k3 GPT-6-Sol DeepSeek 4.1 Flash GPT-5.6-Terra GPT-6-Luna GLM-5.3-Flash GPT-5.6-Luna GPT-6-Astra, native GPT-6-Sol, native GPT-5.6-Sol, native DeepSeek 4.1 Flash, native Qwen3.8 Flash, native GPT-5.6-Terra, native Kimi-k3, native GPT-5.6-Luna, native GLM-5.3-Flash, native GPT-6-Luna, native 100100100100100100100100100100100100100100100100 10050100100100100100100100100100100100100100100 1001210010010010010067100100100100100100100100 10012100100100100100033010010010050100100 100251001006010010000010010010030100100 1000100100100100100671001001001001001001000 10012100100701006700010010010040100100 8801009050100330001001001003070100 10001004060033330010010090205040 1001210030301003300010010060102020 10001004001003300010010005000 10038100100100100100100100100100100100100100100 100251001001001001006710010010010010070100100 1002510010010010010067671001001001009010020 880100100401001000001001001005010070 100010010030100100000100100100309030 100121001004010010000010010040305020 100010010001001003300100100303010040 1000100402010000001001001000600 100010040010033000100605020900 100020400100000010010050000 BoltRushNutTruckMazeColor Figure 7: Level completion (%) per objective for the human reference and the ten general-purpose backbones under Code Execution CUA (black labels) and with Native GUI Actions (orange labels), each group ordered by LC hard . E/M/H: Easy/Medium/Hard; X: Extreme; N: Nightmare; D: Default. 22 Preprint C.2BEHAVIORAL SOURCES OF THE GAP BETWEEN SETTINGS This analysis pairs the Native GUI Actions and Code Execution CUA episodes of the ten Codex backbones by backbone and objective (160 pairs). It uses every tool call with its arguments and timestamps, the pointer and key inputs and puzzle states that the game records in both settings, and per-level win times from evaluator snapshots; reasoning content is encrypted and is not analyzed. A Code Execution CUA call is a computation call when its code does more than list GUI inputs, screenshot requests, and waits: it searches for moves, analyzes data, or generates inputs through loops, computed coordinates, or helpers defined in earlier calls. This classification differs from NGAR, which also counts standalone waits and terminal answers as non-native. Matched levels are the second and later levels that one backbone solves in both settings; levels are solved in order, so matching by level is exact. Table 8 summarizes the measurements. Table 8: Paired behavioral comparison of the two settings over the ten Codex backbones (160 episodes per setting). Inputs are pointer and key inputs recorded by the game. Matched-level ratios are medians of per-level ratios of Code Execution CUA to Native GUI Actions for time, calls, screenshots, and executed inputs. Drag slips are non-adjacent jumps and wrong-endpoint releases. Native GUI ActionsCode Execution CUA Outcomes Episodes solved / timed out / abandoned84 / 41 / 3592 / 31 / 37 Timeouts that solve a level in the final 15 min15 of 417 of 31 Level restarts per episode (mean)2.43.6 Interaction Tool calls per episode (median)100.080.5 Executed inputs per call1.912.66 Screenshots returned per call1.000.81 Observation-only calls (%)18.011.9 Computation calls (% of calls / % of inputs)–20.1 / 56.3 Solved levels with≥80% of inputs from one call (%)4.917.4 Model / tool time per call, median (s)6.2 / 0.306.9 / 0.63 Levels solved per hour9.412.0 Matched levels: Code Execution CUA / Native GUI Actions ratio (time / calls / screenshots / inputs) All 591 matched levels0.81 / 0.83 / 0.67 / 0.93 279 levels with computation calls0.60 / 0.74 / 0.50 / 0.92 312 levels with GUI calls only0.94 / 1.00 / 0.80 / 1.00 Operation errors recorded by the game (%) Maze Paint: invalid / zero-progress / repeated swipes17.7 / 25.9 / 4.112.9 / 29.8 / 9.7 Color Connect: invalid gestures / drag slips5.3 / 5.36.1 / 4.4 Tool outputs flagged as errors0.11.3 Where the gap arises. Pooled over the ten backbones, Code Execution CUA solves 92 of 160 objectives and Native GUI Actions 84. Pair by pair, Code Execution CUA solves more levels in 38 pairs and fewer in 17. Its gains come from episodes that end unsolved with Native GUI Actions: of the 41 timeouts, it solves more levels in 20 and completes 11; of the 35 voluntary exits, it solves more levels in 18 and completes 4. Its losses come mainly from its own exits: in 6 of the 7 pairs where only the Native GUI Actions episode succeeds, the Code Execution CUA agent stops voluntarily. Efficiency comes from computation, not from the interface. On the 591 matched levels, Code Execution CUA takes a median 68.0 s per level against 94.0 s, with 7 against 8 calls and 6 against 8 screenshots, but a similar number of executed inputs (14 against 17). On the 279 levels where the Code Execution CUA agent computes, the median ratio is 0.60 for time, 0.74 for calls, and 0.50 for screenshots; on the 312 levels with GUI calls only, calls and inputs are unchanged (1.00) and time falls only slightly (0.94), so the JavaScript interface alone brings little speedup. With computation, time per level falls for all nine backbones that have such levels (median ratios 0.13–0.79). One call carries a computed sub-plan. Computation calls are 20.1% of Code Execution CUA calls but issue 56.3% of its executed inputs; a call that generates inputs by program issues 13.3 on average, against 5.8 for a batch with Native GUI Actions. One call issues at least 80% of a solved level’s inputs in 17.4% of Code Execution CUA levels, against 4.9% with Native GUI Actions (30.4% against 18.0% at a 50% threshold). With Native GUI Actions, 22.5% of calls batch literal coordinates, but 59.5% are single inputs and 18.0% only observe. Code Execution CUA agents turn 23 Preprint a transcribed board into inputs in three recurring ways: a breadth- or depth-first search over the board (in Maze Paint, Nut and Bolt, and Color Connect), a replay loop that converts a move list to pixels through a cell-to-coordinate map, and move lists or helpers kept in the persistent runtime across calls. On Maze Paint Hard, for example, GPT-5.6-Sol with Native GUI Actions spends 47–84 calls per level, half of them observation only, and reaches the deadline on level 10 with nine solved; under Code Execution CUA, most levels after the first take three or four calls (transcribe and search, replay the 18–44 swipes, advance), and it solves all ten in 1,068 s with 56 calls. Efficiency becomes completion under the deadline. Faster levels change the outcome only when the Native GUI Actions episode runs out of time while still solving. In 10 of the 11 such timeouts that Code Execution CUA completes, the agent with Native GUI Actions solved its last level within 13.2 minutes of the deadline (29.4 minutes in the remaining episode) and had solved 60% of the target levels on average; the Code Execution CUA episodes finish in a median 1,093 s (666–2,685 s) with a median 94 calls against 268, and 9 of the 11 spend at least 20% of their calls on computation. Across backbones, Kimi-k3 and Qwen3.8 Flash each time out on eight objectives with Native GUI Actions and compute in 15.6% and 40.0% of their Code Execution CUA calls; all 20 and 19 levels they add come from those objectives. GLM-5.3-Flash times out on ten objectives but computes in only 9.2% of calls; it adds two levels overall but loses one on those objectives. GPT-6-Astra and GPT-6-Sol compute often (34.7% and 38.7% of calls) but rarely time out with Native GUI Actions; Code Execution CUA raises their pace from 15.1 to 24.9 and from 14.9 to 27.4 levels per hour without raising completion. Exits and restarts. With Native GUI Actions, failures divide into 41 timeouts, 32 of them from the four open-weight backbones, and 35 voluntary exits. Under Code Execution CUA, timeouts fall to 31 while exits rise to 37, 33 of them from GPT-6-Luna, GPT-5.6-Luna, and GPT-5.6-Terra, which compute in 3.3–15.8% of their calls; GPT-5.6-Luna exits in 11 of 16 episodes and solves 12 fewer levels than with Native GUI Actions. Exit timing also accounts for the 28 levels that GPT-6-Luna adds under Code Execution CUA while computing in 3.3% of its calls: its Native GUI Actions episodes stop earlier (on Color Connect Easy, after 30 calls with no level solved, against 171 calls and five levels). Cheap replay also makes restarts cheap, and restarts rise from 2.4 to 3.6 per episode; on Nut and Bolt Medium, which DeepSeek 4.1 Flash completes in 1,585 s with Native GUI Actions, its Code Execution CUA agent restarts 22 times and times out on the third level. Alternative explanations. Tool latency does not explain the gain: tool time per call is higher under Code Execution CUA (median 0.63 s against 0.30 s), as is model time per call (6.9 s against 6.2 s). Operation errors do not favor either setting consistently. In Color Connect, invalid gestures change little (5.3% and 6.1%) and drag slips fall from 5.3% to 4.4%; in Maze Paint, invalid swipes fall from 17.7% to 12.9%, but zero-progress swipes rise from 25.9% to 29.8% and repeated swipes from 4.1% to 9.7%. Qwen3.8 Flash, which adds 19 levels, already makes few invalid operations with Native GUI Actions (1.4–3.8% in both games), and tool errors are rare (1.3% of calls). Kimi-k3 is the exception: its invalid Maze Paint swipes fall from 20.7% to 2.5%, so cleaner execution may contribute to its gain in that game. Interpretation and limits. Code Execution CUA changes the unit of decision. With Native GUI Actions, each move costs a model turn and usually a screenshot; an agent that has transcribed a board can instead search it and execute a level’s solution within one turn. Under a budget shared by all levels, fewer turns per level convert episodes that are still progressing into completions, with- out stronger visual reasoning: the model must still read the board, and the search runs outside its reasoning (Section 4.3). Screenshots requested only when needed and replay through coordinate maps contribute secondarily. For Kimi-k3 and Qwen3.8 Flash, model time per call on matched lev- els is also lower under Code Execution CUA (18.2 s against 25.9 s, and 20.6 s against 23.3 s), and encrypted reasoning does not allow separating shorter deliberation from provider latency. Because the settings also differ in tool integration, these are associations within the evaluated episodes, not controlled estimates of the effect of the action channel. C.3HUMAN–AGENT TRAJECTORY COMPARISON Section 4.5 compares the human reference with the GPT-6-Astra episodes of Table 2 in both set- tings; with Native GUI Actions, the two share objective, seed, viewport, target levels, native action space, and deadline. Because the human and agent recorders differ, the comparison uses puzzle-state 24 Preprint variables that both record, not interface-specific pointer counts, and reports a metric only when both recorders expose its field. In Bolt Unscrew Hard, it follows every attempt on visible level 4, the first level that Astra with Native GUI Actions does not clear, and records boards removed, free holes, and whether the attempt deadlocks or clears the level. Attempts split where the board returns to its initial layout, and snapshots stay in event order because a deadlocked and a restarted state can share a recorder step. Nut and Bolt keeps the last settled snapshot per recorder step and counts changes of the canonical stack configuration, revisits, solved-stack regressions, and resets. Maze Paint and Rush Hour sum the final cumulative moves of each completed level, and Color Connect counts ges- tures. Screenshots, waits, pointer motions, and unchanged snapshots do not count. Figure 8 and Table 9 report the results. 0.10.250.512 ← Astra fewer Astra / human transitions human fewer → Nut and Bolt Maze Paint Rush Hour Color Connect N E M H E M H E H (a) Astra is more economical on most puzzles 12345678 Attempt on Bolt Unscrew Hard level 4 0 7 14 Boards removed HumanAstra, code Astra, native (b) Astra repeats a losing opening Figure 8: Human and GPT-6-Astra succeed on different puzzles (Astra episodes of Table 2; hollow orange: Native GUI Actions, filled blue: Code Execution CUA). (a) Ratio of Astra’s to the human’s state transitions per matched objective (E/M/H: Easy/Medium/Hard; N: Nightmare). (b) Boards removed in each attempt on Bolt Unscrew Hard level 4 (×: dead end; star: level cleared); Astra with Native GUI Actions reaches this level two minutes before its deadline. One human trajectory per objective. Table 9: Trajectory diagnostics for the human reference and the GPT-6-Astra episodes of Table 2 in both settings (Native: Native GUI Actions; Code: Code Execution CUA). Bolt Unscrew values describe visible level 4 (14 boards, four free holes at the start), the first level Astra does not clear with Native GUI Actions, which it reaches near the deadline. Maze Paint and Rush Hour values sum moves over ten levels per objective; E/M/H: Easy/Medium/Hard. PuzzleDiagnosticHuman Astra, NativeAstra, Code Bolt Unscrew Hard L4Dead ends with no board removed115 Attempt that clears the level3–8 Most boards in one attempt14014 Nut and Bolt Nightmare Stack-state changes4316567 Revisited states11540 Solved-stack regressions700 Resets1700 Maze PaintMoves, E / M / H104 / 204 / 381 97 / 187 / 273 88 / 191 / 288 Rush HourMoves, E / M / H55 / 67 / 15543 / 62 / 11244 / 44 / 109 Color ConnectGestures, E / H105 / 223190 / 315176 / 354 DDIAGNOSTIC STUDIES D.1COMMON DESIGN AND VALIDITY RULES Agent and harness. The controlled diagnostics use a screenshot-to-action harness with the GUI action set of Native GUI Actions, which differs from the Codex scaffold of Table 2. At takeover, the agent receives the public action prefix as text, up to four preceding screenshot–action pairs, and the current screenshot; afterwards, it keeps up to 12 recent action turns and four images, with a summary every eight turns. The viewport is 1280× 900, and the runner waits two seconds after each dispatched action. The studies evaluate GPT-5.6-Sol and repeat key contrasts with GPT-5.6-Terra and GPT-5.6-Luna (Sol, Terra, and Luna in Section 5 and below); the dashed series of Figure 5 are these three-model follow-ups. 25 Preprint Matched conditions. Within a matched block, conditions share the restored state, objective, screen- shot, public trajectory prefix, action interface, and continuation limit; only the intervention differs. Checkpoints and source failures were fixed before any continuation outcome was inspected. Validity. Agent stops, timeouts, ineffective actions, repeated failures, and noncompletion are valid outcomes. API transport failures, browser failures, corrupted restoration, and incomplete model responses are infrastructure failures; such an attempt is replaced under the same assignment. When a cell has several valid attempts, the first in a fixed order is the primary observation and the others serve only to test stability. Counts in Section 5 include valid runs only. Reporting. Analyses report paired counts, state- or source-level effects, ranges, and standard de- viations, and do not treat repeated actions within one trajectory as independent samples. Original and follow-up panels are reported separately when they differ in model or unit coverage, claims are scoped to the tested states and games, and null or reversed paired effects are reported as observed. D.2RQ1: PROGRESSION AND MOVE-LEVEL MEASURES The observational measures use the Native GUI Actions episodes of the ten general-purpose back- bones in Table 2, 16 per backbone. Reached and unreached levels. A target level is reached when the evaluator observes its ready gameplay state; each episode contributes one observation per target level. With r oℓ indicating reach and R o =|L o | −1 P ℓ r oℓ , the LC deficit decomposes as 1− LC o = (1− R o ) + (R o − LC o ), with the same game-macro weights for every term. The unreached share is the aggregated 1− R o divided by the aggregated 1− LC o . Across the ten backbones, unreached levels account for 63.7– 80.0% of the deficit (Figure 9a), and completion conditional on reaching a level ranges from 75.5% (GPT-6-Luna, 37/49) to 99.1% (GPT-6-Astra, 109/110; Table 10). Because levels are won in order, stalling on one level also forfeits every later one, so a large coverage deficit can originate in a few unresolved puzzles. 0255075100 Target levels (%) GPT-6-Astra GPT-6-Sol GPT-6-Luna GPT-5.6-Sol GPT-5.6-Terra GPT-5.6-Luna DeepSeek 4.1 Flash Qwen3.8 Flash GLM-5.3-Flash Kimi-k3 94.8 91.0 40.0 84.3 61.3 53.3 72.0 67.2 51.9 64.4 12.2 9.9 11.0 9.6 9.4 12.2 9.1 47.8 11.0 28.8 35.6 18.4 23.4 35.9 26.5 (a) Where completion is lost CompletedReached, unsolvedUnreached 015304560 Elapsed time (minutes) 0 25 50 75 100 Cumulative LC (%) (b) When completion accumulates GPT-6-Astra GPT-6-Sol GPT-6-Luna GPT-5.6-Sol GPT-5.6-Terra GPT-5.6-Luna DeepSeek 4.1 Flash Qwen3.8 Flash GLM-5.3-Flash Kimi-k3 Figure 9: Progression of the ten general-purpose backbones with Native GUI Actions (episodes of Table 2). (a) Each target level is completed, reached but unsolved, or unreached at episode end. Bars average objectives within games and games equally. (b) Cumulative LC uses first-win timestamps and the same weights. Termi- nated episodes retain their final completion through 60 minutes. Stopping. GPT-6-Astra and GPT-6-Sol never stop early; their four unsuccessful episodes reach the deadline. GPT-5.6-Terra and GPT-5.6-Luna end every unsuccessful episode voluntarily (9 of 9 and 10 of 10), at a median of 17.2 and 7.0 minutes, and GPT-6-Luna ends 11 of its 12 at a median of 19.5 minutes. Stopping forfeits later gains: GPT-5.6-Terra leads Qwen3.8 Flash after 15 minutes (53.6% against 31.5% LC), but Qwen3.8 Flash keeps playing and finishes higher (67.2% against 61.3%; Figure 9b). Maze Paint counters. Maze Paint counters are cumulative within an attempt; level changes and counter resets delimit attempts, and each attempt contributes its terminal record, at the first win or at 26 Preprint Table 10: RQ1 denominators and stopping times with Native GUI Actions (Table 2 episodes, 16 per agent). Reached and won levels are pooled episode–level observations. Conditional completion is wins/reached and differs from the macro-weighted LC in Figure 9. Unreached share: fraction of the LC deficit in levels never reached. Exits/failures counts premature exits among unsuccessful episodes. Exit time: minutes, median [first quartile, third quartile], conditional on premature exit. AgentWins/reached Conditional (%) Unreached (%) Exits/failuresExit time GPT-6-Astra109/11099.180.00/1N/A GPT-6-Sol104/10797.270.00/3N/A GPT-6-Luna37/4975.579.711/1219.5 [5.4, 33.7] GPT-5.6-Sol97/10295.170.41/520.6 GPT-5.6-Terra68/7788.374.39/9 17.2 [13.7, 21.0] GPT-5.6-Luna58/6885.376.310/107.0 [5.5, 8.6] DeepSeek 4.1 Flash84/9291.365.82/8 35.2 [27.1, 43.4] Qwen3.8 Flash77/8590.671.40/8N/A GLM-5.3-Flash53/6581.574.72/12 27.3 [18.8, 35.9] Kimi-k370/7889.774.40/8N/A Table 11: Maze Paint move diagnostics with Native GUI Actions (three episodes per agent, 260 attempts in total). Productive moves paint at least one new cell. State observations comprise the initial state and the states after game moves, pooled within attempts; revisits repeat an already observed position and painted-cell configuration. Counters include retries and stop at each level win or attempt end. AgentMaze LC (%) Valid/attempted Productive/valid (%) Revisited/observed (%) GPT-6-Astra100.0557/557441/557 (79.2)6/587 (1.0) GPT-6-Sol90.0505/508386/505 (76.4)7/537 (1.3) GPT-6-Luna50.0276/385169/276 (61.2)143/408 (35.0) GPT-5.6-Sol96.7687/714446/687 (64.9)64/745 (8.6) GPT-5.6-Terra56.7361/541245/361 (67.9)192/566 (33.9) GPT-5.6-Luna66.7451/863256/451 (56.8)466/890 (52.4) DeepSeek 4.1 Flash83.3700/833422/700 (60.3)203/868 (23.4) Qwen3.8 Flash76.7381/396299/381 (78.5)25/420 (6.0) GLM-5.3-Flash43.3266/277187/266 (70.3)21/295 (7.1) Kimi-k353.3256/323192/256 (75.0)69/341 (20.2) the last state before a reset or the episode end. Productive moves paint at least one new cell and, with zero-progress moves, partition valid moves; invalid moves form a separate category. A state is the ball position together with the painted-cell configuration, and the revisit denominator pools unique and revisited states, including each attempt’s initial state. The 30 episodes contribute 260 attempts, all with complete counters (Table 11). GPT-6-Astra and GPT-6-Sol revisit an earlier state in 1.0% and 1.3% of observations; the three agents that revisit most (33.9–52.4%) complete 50.0–66.7% of Maze Paint levels. GLM-5.3-Flash revisits rarely (7.1%) yet completes 43.3%, because its slower play reaches the deadline. For Figure 4c, counter increments between consecutive snapshots are assigned to the coverage bin (fifths of paintable cells) of the earlier snapshot and pooled within bin and outcome; unsolved attempts contribute 460 valid moves and solved attempts 3,980. Bolt Unscrew capacity. Figure 4a–b uses the RQ3 retries, which start from a verified restart of a level’s solvable initial board. For each model decision, we take the last stable evaluator state before it: free holes, boards exited, and the deadlock certificate. Consecutive distinct states define moves. A move frees a board if boards exited increases, fills a hole if free holes decrease without a board exiting, and deadlocks if the next state carries the certificate. The reference path is the certified 16- move solution of Hard level 1 that also supplies the RQ2 solvable controls. Of 73 moves made with one free hole, 55 deadlock the board, as do 15 of 17 first moves from one-hole level starts. Stated reasons are the Thought field of each model response, coded with fixed regular expressions; a fatal move predicts a release if its rationale mentions releasing, freeing, dropping, or clearing a piece (28 of 55). Same-state action intervention. For action a in state s, the extra cost is e(s, a) = 1 + d(T (s, a))− d(s), where d is the exact remaining path length and T the deterministic transition; a shortest-path move has e = 0. Three Maze Paint states are restored, and from each, one branch executes a previously observed legal action with positive extra cost and the other a legal shortest-path action. Both successors are verified to be solvable, so the intervention tests how one costly move changes 27 Preprint the continuation under a finite budget, not whether it destroys solvability. Branch order is balanced, and each state is tested three times with each of Sol, Terra, and Luna, giving 27 paired units (54 continuations) compared on same-level completion. D.3RQ2: CERTIFIED DEAD-END TAKEOVER Design. The original study evaluates GPT-5.6-Sol (medium reasoning effort, temperature 0) on Bolt Unscrew Hard at seed 0. It contains 24 certified dead-end checkpoints (D) from seven source levels (Table 14), each under conditions A, B, and C, and 12 solvable checkpoints (N), distinct states along one verified successful trajectory on level 1, each under A and B: 96 continuations, one per checkpoint and condition. A gives the ordinary objective and visual context, B adds truthful state information, and C adds the label of the restart button to B on D (Table 12). The agent chooses every action; the controller only prepares the checkpoint. A-to-B thus tests an explicit state diagnosis and B-to-C the added naming of the recovery control. D and N are separate cohorts, so differences between them describe the selected states rather than a paired intervention. Checkpoints. A D checkpoint is an unfinished, stable board carrying the no availablehole certificate; independent replays verify the board and screenshot, and a native restart must restore a playable board on the same level. An N checkpoint has a verified normal-play completion that needs fewer GUI actions than restarting and replaying the solution, and passes two independent legal re- plays to same-level completion. Certificates and solutions stay private. The controller replays the legal prefix with fixed simulation steps and holds the game until the initial screenshot is captured, so paired continuations start from identical boards and pixels. The intervention text enters the instruc- tion at takeover and stays in context, including after a restart. Conditions follow a fixed interleaved schedule within checkpoint blocks. Table 12: RQ2 intervention text. A appends no message. The restart button carries a Chinese label, given here by its English translation. Condition/stateAppended information B/DState information: The current puzzle state cannot be completed through normal forward play. Recovery is required before progress can continue. B/NState information: The current puzzle state remains solvable through normal forward play. C/DB’s D message, followed by: Recovery information: Use the visible button labelled Restart to restart the current level. Endpoints. The takeover clock includes inference, waits, and GUI operations. A timely restart requires a recovery action within the first ten actions and 120 seconds. Verified recovery requires a ready, non-deadlocked board on the original level within 300 seconds, and a same-level win requires the success event within the same limit. Every D continuation stays in the denominator, including runs that never recover. The 600-action ceiling never binds (the longest continuation has 24 actions); all 72 D continuations end at the time limit, and N has 19 completions and five timeouts, with no restart or voluntary stop. Estimation. Table 13 reports the outcomes. Rates weight checkpoints equally within each state class, and paired effects compare outcomes at the same checkpoint. The 95% percentile intervals resample the seven D source levels (10,000 bootstrap draws), keeping all checkpoints and conditions of each sampled level; they describe variation across sources, not repeat variability. Two serving endpoints with the same model and settings served the runs; on the 21 A/B pairs that share an endpoint, timely restarts rise from 1/21 to 19/21 and verified recovery from 8/21 to 20/21, and on the 23 B/C pairs that share one, verified recovery is 22/23 in both conditions. Paired effects. Timely restarts improve on 21 of the 24 paired D checkpoints and decline on none; verified recovery improves on 15 and declines on none. Seven of the eight recoveries under A occur after the response window, against one of 23 under B and none of 22 under C. From B to C, timely restarts improve on one checkpoint and decline on one (net zero; 95% interval [−7.9, 15.0] points), and verified recovery differs on one checkpoint, the B/C pair that spans endpoints (−4.2 points; [−7.9, 0.0]). On N, eight checkpoints finish under both A and B, one only under B, two only under A, and one under neither, and no run restarts. 28 Preprint Table 13: RQ2 outcomes for GPT-5.6-Sol. Each cell gives a count and percentage over all continuations in that state and condition. Timely restart must occur within 120 seconds and the first ten actions; verified recovery and same-level wins use the 300-second limit. No N continuation restarts, including after the response window. State ConditionTimely restart Verified recovery Same-level win DA1/24 (4.2%)8/24 (33.3%)0/24 (0.0%) DB22/24 (91.7%)23/24 (95.8%)0/24 (0.0%) DC22/24 (91.7%)22/24 (91.7%)0/24 (0.0%) NA0/12 (0.0%)—10/12 (83.3%) NB0/12 (0.0%)—9/12 (75.0%) Table 14: Within-level effects of state information on D, in percentage points. Every source level improves on timely selection and verified recovery. Checkpoints sharing a level remain together in the paired bootstrap. Win counts use the listed checkpoint count as the denominator for each condition. Hard levelD checkpointsSelection B− ARecovery B− AWins (A/B/C) 110+80.0+50.00/0/0 22+100.0+100.00/0/0 32+100.0+50.00/0/0 52+100.0+50.00/0/0 64+75.0+75.00/0/0 72+100.0+50.00/0/0 82+100.0+100.00/0/0 Attribution coding. A D run blames the click if any rationale describes a move as not taking effect or proposes retrying another target (A: 16/24, B: 3/24, C: 5/24). An unaided recovery names the cause if a rationale before the restart states that the holding holes are full or that no destination remains (7 of the 8 recoveries under A). After recovery. A recurrence is a new stable certificate on the target level after the first verified return to play and within 300 seconds. Of the 45 recovered B/C continuations, 34 recur (17 under each condition), 29 of them after positive progress from the restored board. The median recurrence comes 128.8 seconds after recovery, after eight accepted actions, with 148.5 seconds remaining. The 11 continuations without a new certificate also remain unfinished, despite a median 290.9 seconds available after recovery. Follow-up across models and games. The follow-up repeats A/B/C with Sol, Terra, and Luna on six certified Bolt Unscrew dead ends and three solvable controls per model, and on three certified Nut and Bolt dead ends per model: 72 Bolt Unscrew and 27 Nut and Bolt trials (Table 15). Pooled over Bolt Unscrew dead ends, verified recovery rises from 7/18 under A to 16/18 under B and stays at 16/18 under C; the effect is largest for Luna and nearly absent for Terra, which already recovers in 5/6 baseline trials. None of the 18 solvable controls selects recovery. In Nut and Bolt, recovery selection, verified recovery, and completion are all 0/27. All 81 follow-up dead-end continuations remain incomplete; with the original 72, this gives 0/153 same-level completions after dead-end takeover. Table 15: Verified recovery in the RQ2 follow-up. Each Bolt Unscrew cell has six certified dead ends per condition; Nut and Bolt pools three dead ends per model, nine trials per condition. Same-level completion is zero in every cell. GameModelABC Bolt Unscrew Luna0/6 5/6 5/6 Bolt Unscrew Sol2/6 5/6 6/6 Bolt Unscrew Terra5/6 6/6 5/6 Nut and BoltAll three 0/9 0/9 0/9 D.4RQ3: MATCHED RETRIES WITH MEMORY Design. RQ3 branches from agent-owned Bolt Unscrew Hard failures after a verified native restart. Every branch starts from the same ready board, screenshot, objective, action interface, and budget. M0 clears the pre-restart history; M1 keeps the native history and its restart summary; M2 has 29 Preprint the same capacity as M1 but replaces the restart summary with a structured, agent-written account of prior actions and outcomes, suspected failure causes, the restored state, and the proposed next strategy. After the restart, every condition accumulates history normally. The original Sol panel crosses 24 source failures with M0–M2 (72 retries); the follow-up crosses six fixed source failures with the three conditions and Sol, Terra, and Luna (54 retries). Fifteen model–source–condition units have one extra valid retry, used only to test endpoint stability. Endpoints. A retry ends at same-level completion, recurrence of the source’s certified failure, a different certified failure, another restart, an explicit stop, or the 300-second limit. Same-level com- pletion is the primary outcome. Recurrence requires the same certified mechanism, not merely a similar board. Results. The original panel yields 0/24 completions in every condition, and the follow-up 0/18 in M0, M1, and M2. In the follow-up, source-failure recurrence is 12/18, 16/18, and 15/18, and mean best progress is 0.124, 0.089, and 0.130. Paired within model and source, M2−M1 progress averages +0.041 but ranges from −0.091 to +0.409, and M2−M0 averages +0.006. Of the 15 repeated units, seven keep their endpoint and eight change it. The data thus support a null effect on completion and unstable endpoints, not a ranking of the memory conditions. Failure summaries. Each of the 24 M2 summaries of the original panel has a “Failure Experience” and a “Next Attempt” section. A summary blames execution or observation if the former cites unconfirmed outcomes, missing screenshots, or rejected placements (24/24); it resumes the failed plan if the latter directs continuing or re-testing the previous intention (15/24); and it proposes a different plan if the latter calls for an alternative or safer order (0/24). A manual reading confirms that no summary identifies exhaustion of the free holes as the cause. Source histories often end before the final dead-end frame, so a summary does not always observe the deadlock itself, although earlier frames in several histories already show the holding holes occupied. D.5RQ4: RULE CARD AND WORKFLOW CONTROL Design. RQ4 varies two burdens inside complete sessions. Under K0, the agent must infer the me- chanics through play; K1 adds a fixed game-level card with the legal interactions and constraints but no level-specific solution, coordinates, or action sequence. Under P0, the agent handles game entry, level advancement, and puzzle moves; P1 uses a deterministic controller for entry and evaluator- verified advancement and leaves every puzzle move to the agent. The original Sol study has 68 sessions: the four K/P cells on all 16 objectives, plus two sessions per Bolt Unscrew objective with a controller that restarts after a certified deadlock. Five isolated rule probes per session give 340 probe outcomes; probe responses never enter the acting history. The follow-up runs the four K/P cells with Sol, Terra, and Luna on six objectives (Bolt Unscrew Easy and Hard, Color Connect Easy, Maze Paint Easy and Hard, and Rush Hour Easy), for 72 sessions. Contrasts. LC is the primary outcome. For metric Y , the matched effects are ∆ Y K|P0 = Y K1P0 − Y K0P0 ,∆ Y K|P1 = Y K1P1 − Y K0P1 ,(4) ∆ Y P|K0 = Y K0P1 − Y K0P0 ,∆ Y P|K1 = Y K1P1 − Y K1P0 ,(5) ∆ Y K×P = ∆ Y K|P1 − ∆ Y K|P0 .(6) The original study pairs conditions within objective and reports an objective-within-game bootstrap interval; the follow-up computes each contrast within model and objective and reports the mean, standard deviation, and range over the 18 matched units (Table 16, Figure 10a). Rule knowledge. In the original study, the first probe states 13.9% and 16.1% of rules correctly under K0P0 and K0P1, against 77.1% and 86.3% under K1P0 and K1P1. Figure 5c uses the final probe and LC of each P0 session, averaged over the 16 objectives within a K cell; under P1, the card raises final rule correctness from 52.3% to 81.3% and LC from 62.5% to 65.4%. These unweighted session means differ from the bootstrap contrasts of Figure 10a. Without the card, 29 of 36 (P0) and 40 of 48 (P1) rules that are ever stated correctly are first stated correctly after the initial probe. Of the 40 sessions that state every rule correctly at some probe, 34 subsequently increase LC, and 16 still state every rule correctly at the final probe. The follow-up measures completion only. Workflow control. P0 involves no controller actions. Across the 36 P1 follow-up sessions, the controller executes 111 entry and 164 advancement actions without error. Relative to P0, P1 reduces 30 Preprint Table 16: Cross-model RQ4 outcomes over 18 matched model–objective units per K/P cell. LC and FGP entries are mean paired effects in percentage points. Contrast or cellCompletion ∆LC ∆FGP K0P05/18– K0P16/18– K1P08/18– K1P18/18– K1–K0 under P0–+8.6+7.8 K1–K0 under P1–+9.6+9.7 P1–P0 under K0– −2.5 −2.5 P1–P0 under K1– −1.5 −0.6 model actions by 29.4 per session under K0 and 19.1 under K1, and total actions, including the controller’s, by 22.4 and 10.8. The K×P interaction on LC is +1.0 points (standard deviation 19.3, range [−40, +40]), so the controller transfers workflow without a uniform gain in completion. −20020 Paired ΔLC (p) Rule card, P0 Rule card, P1 Workflow, K0 Workflow, K1 (a) Paired effects of rule and workflow support Sol, 16 obj. (95% CI)3 models (mean ± SD) 050100 Failure summaries (%) Blames click or missing frame Resumes the failed plan Proposes a different plan 24/24 15/24 0/24 (b) What failure summaries say Figure 10: Supplementary views for RQ3–RQ4. (a) Paired LC effects of the rule card and workflow control at each level of the other factor: original Sol factorial over 16 objectives (bootstrap 95% interval) and the three-model follow-up over 18 model–objective units (mean± SD). (b) Content of the 24 agent-written failure summaries (M2). 31 Preprint ECOMPLETE PER-OBJECTIVE RESULTS Table 17 gives the counts behind the aggregate metrics of Table 2, and Table 18 lists the outcome of every objective for every agent and setting. Every completion metric in Table 2 follows from Table 18 by the game-macro average of Appendix B.3; unvisited target levels contribute zero to LC and FGP. Values are rounded to one decimal place. Table 17: Metric denominators by evaluation setting. PER and SMR use all n episodes. S is the number of successful episodes. Native counts visual-observation and GUI-input steps and Other all remaining model action steps (Code Execution CUA only). Cells: objectives with a valid episode. ConfigurationnS PER SMR Native Other Cells GPT-6-Astra / Code Execution CUA16 1500729381 16/16 GPT-6-Sol / Code Execution CUA16 1330486696 16/16 GPT-6-Luna / Code Execution CUA1641201620858 16/16 GPT-5.6-Sol / Code Execution CUA16 1400122 1316 16/16 GPT-5.6-Terra / Code Execution CUA166100528 1067 16/16 GPT-5.6-Luna / Code Execution CUA165110629568 16/16 DeepSeek 4.1 Flash / Code Execution CUA16911305 2303 16/16 Qwen3.8 Flash / Code Execution CUA16 1100275 1295 16/16 GLM-5.3-Flash / Code Execution CUA165001324 1273 16/16 Kimi-k3 / Code Execution CUA16 1000895614 16/16 GPT-6-Astra / Native GUI Actions16 1500N/AN/A 16/16 GPT-6-Sol / Native GUI Actions16 1300N/AN/A 16/16 GPT-6-Luna / Native GUI Actions164110N/AN/A 16/16 GPT-5.6-Sol / Native GUI Actions16 1110N/AN/A 16/16 GPT-5.6-Terra / Native GUI Actions16790N/AN/A 16/16 GPT-5.6-Luna / Native GUI Actions166100N/AN/A 16/16 DeepSeek 4.1 Flash / Native GUI Actions16820N/AN/A 16/16 Qwen3.8 Flash / Native GUI Actions16800N/AN/A 16/16 GLM-5.3-Flash / Native GUI Actions16420N/AN/A 16/16 Kimi-k3 / Native GUI Actions16800N/AN/A 16/16 UI-TARS-1.5-7B / Native GUI Actions (autonomous harness)160140N/AN/A 16/16 Qwen3-VL-32B-Thinking / Native GUI Actions (autonomous harness)16000N/AN/A 16/16 Qwen3-VL-30B-A3B-Thinking / Native GUI Actions (autonomous harness) 16030N/AN/A 16/16 Qwen3-VL-8B-Thinking / Native GUI Actions (autonomous harness)16000N/AN/A 16/16 Table 18: Per-objective results for every agent and setting. S: objective solved; P: premature exit; M: setting mismatch. LC, FGP, and NGAR are percentages; NGAR applies to Code Execution CUA only (–). |L o |: number of target levels. ObjectiveSLC FGP P M NGAR|L o | GPT-6-Astra / Code Execution CUA Bolt / easy1 100.0 100.0 0 0100.08 Bolt / hard050.060.7 0 065.28 Color / easy1 100.0 100.0 0 038.610 Color / hard1 100.0 100.0 0 025.010 Maze / easy1 100.0 100.0 0 092.010 Maze / hard1 100.0 100.0 0 020.010 Maze / medium 1 100.0 100.0 0 027.010 Nut / easy1 100.0 100.0 0 0100.03 Nut / extreme1 100.0 100.0 0 028.93 Nut / hard1 100.0 100.0 0 026.33 Nut / medium1 100.0 100.0 0 041.23 Nut / nightmare 1 100.0 100.0 0 048.01 Rush / easy1 100.0 100.0 0 098.110 Rush / hard1 100.0 100.0 0 058.610 Rush / medium 1 100.0 100.0 0 085.710 Truck / default1 100.0 100.0 0 0100.05 GPT-6-Sol / Code Execution CUA Bolt / easy1 100.0 100.0 0 0100.08 Bolt / hard00.03.6 1 038.58 Color / easy1 100.0 100.0 0 079.010 Color / hard00.02.5 1 080.610 Maze / easy1 100.0 100.0 0 030.410 Maze / hard1 100.0 100.0 0 021.010 Maze / medium 1 100.0 100.0 0 08.810 Nut / easy1 100.0 100.0 0 018.93 Nut / extreme1 100.0 100.0 0 08.13 Nut / hard066.791.4 1 024.63 Nut / medium1 100.0 100.0 0 020.83 Nut / nightmare 1 100.0 100.0 0 011.61 Rush / easy1 100.0 100.0 0 069.810 32 Preprint Table 18 continued ObjectiveSLC FGP P M NGAR|L o | Rush / hard1 100.0 100.0 0 078.310 Rush / medium 1 100.0 100.0 0 098.410 Truck / default1 100.0 100.0 0 016.75 GPT-6-Luna / Code Execution CUA Bolt / easy1 100.0 100.0 0 098.58 Bolt / hard00.03.6 1 092.98 Color / easy050.056.7 1 081.410 Color / hard040.040.0 1 089.310 Maze / easy1 100.0 100.0 0 09.710 Maze / hard020.025.8 1 03.810 Maze / medium 090.098.3 1 029.010 Nut / easy00.025.0 1 098.83 Nut / extreme00.03.2 1 027.33 Nut / hard033.340.5 1 029.93 Nut / medium033.333.3 1 031.73 Nut / nightmare 00.012.5 1 086.01 Rush / easy1 100.0 100.0 0 092.910 Rush / hard060.062.3 1 095.210 Rush / medium 040.046.7 1 097.210 Truck / default1 100.0 100.0 0 019.75 GPT-5.6-Sol / Code Execution CUA Bolt / easy1 100.0 100.0 0 010.98 Bolt / hard012.515.3 0 06.88 Color / easy1 100.0 100.0 0 04.810 Color / hard1 100.0 100.0 0 016.710 Maze / easy1 100.0 100.0 0 02.410 Maze / hard1 100.0 100.0 0 05.410 Maze / medium 1 100.0 100.0 0 03.210 Nut / easy1 100.0 100.0 0 021.43 Nut / extreme1 100.0 100.0 0 02.93 Nut / hard066.769.8 0 01.13 Nut / medium1 100.0 100.0 0 08.33 Nut / nightmare 1 100.0 100.0 0 06.51 Rush / easy1 100.0 100.0 0 029.810 Rush / hard1 100.0 100.0 0 050.010 Rush / medium 1 100.0 100.0 0 02.710 Truck / default1 100.0 100.0 0 02.65 GPT-5.6-Terra / Code Execution CUA Bolt / easy087.598.6 1 018.68 Bolt / hard00.05.2 1 018.28 Color / easy070.078.3 1 077.310 Color / hard1 100.0 100.0 0 023.210 Maze / easy1 100.0 100.0 0 00.710 Maze / hard030.039.5 1 03.910 Maze / medium 1 100.0 100.0 0 03.310 Nut / easy1 100.0 100.0 0 02.63 Nut / extreme00.05.1 1 010.83 Nut / hard00.05.3 1 010.33 Nut / medium033.338.7 1 02.53 Nut / nightmare 00.016.7 1 03.41 Rush / easy1 100.0 100.0 0 098.010 Rush / hard050.055.0 1 085.310 Rush / medium 090.091.4 1 099.010 Truck / default1 100.0 100.0 0 04.45 GPT-5.6-Luna / Code Execution CUA Bolt / easy1 100.0 100.0 0 02.88 Bolt / hard00.03.6 1 088.58 Color / easy00.06.7 1 056.010 Color / hard00.08.8 1 077.810 Maze / easy1 100.0 100.0 0 095.210 Maze / hard050.058.6 1 09.210 Maze / medium 00.09.3 1 012.010 Nut / easy1 100.0 100.0 0 010.63 Nut / extreme00.01.9 1 036.63 Nut / hard00.02.3 1 043.53 Nut / medium033.337.3 1 033.33 Nut / nightmare 00.015.3 1 045.21 Rush / easy1 100.0 100.0 0 063.010 Rush / hard00.06.4 1 088.510 Rush / medium 040.045.5 1 074.310 Truck / default1 100.0 100.0 0 00.95 DeepSeek 4.1 Flash / Code Execution CUA Bolt / easy1 100.0 100.0 0 03.08 Bolt / hard012.517.6 1 166.48 Color / easy1 100.0 100.0 0 02.710 Color / hard1 100.0 100.0 0 05.710 Maze / easy1 100.0 100.0 0 01.610 33 Preprint Table 18 continued ObjectiveSLC FGP P M NGAR|L o | Maze / hard040.049.8 0 013.210 Maze / medium 1 100.0 100.0 0 01.910 Nut / easy1 100.0 100.0 0 04.23 Nut / extreme00.05.1 0 02.13 Nut / hard00.09.1 0 01.23 Nut / medium066.773.0 0 00.93 Nut / nightmare 00.018.1 0 05.21 Rush / easy1 100.0 100.0 0 01.910 Rush / hard070.076.5 0 00.710 Rush / medium 1 100.0 100.0 0 064.810 Truck / default1 100.0 100.0 0 01.95 Qwen3.8 Flash / Code Execution CUA Bolt / easy1 100.0 100.0 0 03.38 Bolt / hard012.515.1 0 03.98 Color / easy1 100.0 100.0 0 017.910 Color / hard1 100.0 100.0 0 014.910 Maze / easy1 100.0 100.0 0 06.010 Maze / hard050.058.8 0 05.310 Maze / medium 1 100.0 100.0 0 03.510 Nut / easy1 100.0 100.0 0 042.53 Nut / extreme033.337.8 0 07.93 Nut / hard00.06.1 0 010.23 Nut / medium1 100.0 100.0 0 013.13 Nut / nightmare 00.015.3 0 028.61 Rush / easy1 100.0 100.0 0 035.110 Rush / hard1 100.0 100.0 0 037.910 Rush / medium 1 100.0 100.0 0 097.010 Truck / default1 100.0 100.0 0 011.85 GLM-5.3-Flash / Code Execution CUA Bolt / easy1 100.0 100.0 0 01.98 Bolt / hard012.524.4 0 03.08 Color / easy020.029.1 0 033.310 Color / hard020.020.0 0 06.410 Maze / easy1 100.0 100.0 0 04.310 Maze / hard010.018.3 0 0100.010 Maze / medium 060.068.4 0 00.810 Nut / easy1 100.0 100.0 0 098.93 Nut / extreme00.06.4 0 07.83 Nut / hard00.07.6 0 03.03 Nut / medium033.356.0 0 01.03 Nut / nightmare 00.013.9 0 0100.01 Rush / easy1 100.0 100.0 0 088.710 Rush / hard030.037.6 0 00.610 Rush / medium 030.032.7 0 0100.010 Truck / default1 100.0 100.0 0 099.65 Kimi-k3 / Code Execution CUA Bolt / easy1 100.0 100.0 0 098.78 Bolt / hard025.026.1 0 0100.08 Color / easy1 100.0 100.0 0 095.110 Color / hard1 100.0 100.0 0 02.310 Maze / easy1 100.0 100.0 0 091.810 Maze / hard030.033.1 0 011.710 Maze / medium 1 100.0 100.0 0 017.610 Nut / easy1 100.0 100.0 0 014.83 Nut / extreme00.05.1 0 014.93 Nut / hard00.06.1 0 017.13 Nut / medium1 100.0 100.0 0 02.43 Nut / nightmare 00.013.9 0 050.91 Rush / easy1 100.0 100.0 0 099.210 Rush / hard060.060.0 0 010.010 Rush / medium 1 100.0 100.0 0 016.110 Truck / default1 100.0 100.0 0 076.55 GPT-6-Astra / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard037.537.9 0 0–8 Color / easy1 100.0 100.0 0 0–10 Color / hard1 100.0 100.0 0 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard1 100.0 100.0 0 0–10 Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme1 100.0 100.0 0 0–3 Nut / hard1 100.0 100.0 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 1 100.0 100.0 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard1 100.0 100.0 0 0–10 34 Preprint Table 18 continued ObjectiveSLC FGP P M NGAR|L o | Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 GPT-6-Sol / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard025.029.8 0 0–8 Color / easy1 100.0 100.0 0 0–10 Color / hard1 100.0 100.0 0 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard070.074.2 0 0–10 Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme1 100.0 100.0 0 0–3 Nut / hard066.784.0 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 1 100.0 100.0 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard1 100.0 100.0 0 0–10 Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 GPT-6-Luna / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard00.04.4 1 0–8 Color / easy00.01.7 1 0–10 Color / hard00.07.5 1 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard00.01.8 1 0–10 Maze / medium 050.057.5 1 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.01.3 1 0–3 Nut / hard00.06.1 0 0–3 Nut / medium00.025.0 1 0–3 Nut / nightmare 00.011.1 1 0–1 Rush / easy020.022.0 1 0–10 Rush / hard00.06.2 1 0–10 Rush / medium 040.045.1 1 0–10 Truck / default1 100.0 100.0 0 0–5 GPT-5.6-Sol / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard025.026.8 0 0–8 Color / easy1 100.0 100.0 0 0–10 Color / hard020.029.0 1 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard090.098.4 0 0–10 Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme066.769.4 0 0–3 Nut / hard066.774.1 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 1 100.0 100.0 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard1 100.0 100.0 0 0–10 Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 GPT-5.6-Terra / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard012.517.9 1 0–8 Color / easy050.056.7 1 0–10 Color / hard020.028.0 1 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard030.034.8 1 0–10 Maze / medium 040.046.9 1 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.09.0 1 0–3 Nut / hard00.06.1 1 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 00.013.9 1 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard040.042.2 1 0–10 Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 GPT-5.6-Luna / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard00.03.6 1 0–8 Color / easy060.060.0 1 0–10 Color / hard00.05.0 1 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard00.06.4 1 0–10 35 Preprint Table 18 continued ObjectiveSLC FGP P M NGAR|L o | Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.05.1 1 0–3 Nut / hard00.06.1 1 0–3 Nut / medium00.023.6 1 0–3 Nut / nightmare 00.09.7 1 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard020.022.7 1 0–10 Rush / medium 040.045.9 1 0–10 Truck / default1 100.0 100.0 0 0–5 DeepSeek 4.1 Flash / Native GUI Actions Bolt / easy087.595.8 0 0–8 Bolt / hard00.04.2 1 0–8 Color / easy1 100.0 100.0 0 0–10 Color / hard070.072.7 0 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard050.059.3 0 0–10 Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.06.4 0 0–3 Nut / hard00.06.1 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 00.015.3 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard040.042.9 1 0–10 Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 Qwen3.8 Flash / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard00.05.7 0 0–8 Color / easy090.098.0 0 0–10 Color / hard030.038.3 0 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard030.036.7 0 0–10 Maze / medium 1 100.0 100.0 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.023.1 0 0–3 Nut / hard00.014.4 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 00.019.4 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard030.033.3 0 0–10 Rush / medium 1 100.0 100.0 0 0–10 Truck / default1 100.0 100.0 0 0–5 GLM-5.3-Flash / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard00.00.5 1 0–8 Color / easy090.090.0 0 0–10 Color / hard00.05.0 1 0–10 Maze / easy060.067.0 0 0–10 Maze / hard020.028.3 0 0–10 Maze / medium 050.052.2 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.05.1 0 0–3 Nut / hard00.06.8 0 0–3 Nut / medium033.340.0 0 0–3 Nut / nightmare 00.016.7 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard00.06.4 0 0–10 Rush / medium 040.045.9 0 0–10 Truck / default1 100.0 100.0 0 0–5 Kimi-k3 / Native GUI Actions Bolt / easy1 100.0 100.0 0 0–8 Bolt / hard00.05.7 0 0–8 Color / easy1 100.0 100.0 0 0–10 Color / hard040.048.3 0 0–10 Maze / easy1 100.0 100.0 0 0–10 Maze / hard030.030.0 0 0–10 Maze / medium 030.037.9 0 0–10 Nut / easy1 100.0 100.0 0 0–3 Nut / extreme00.05.8 0 0–3 Nut / hard033.333.3 0 0–3 Nut / medium1 100.0 100.0 0 0–3 Nut / nightmare 00.019.4 0 0–1 Rush / easy1 100.0 100.0 0 0–10 Rush / hard00.06.2 0 0–10 Rush / medium 1 100.0 100.0 0 0–10 36 Preprint Table 18 continued ObjectiveSLC FGP P M NGAR|L o | Truck / default1 100.0 100.0 0 0–5 UI-TARS-1.5-7B / Native GUI Actions (autonomous harness) Bolt / easy012.512.5 1 0–8 Bolt / hard00.00.0 1 0–8 Color / easy00.00.0 1 0–10 Color / hard00.00.0 1 0–10 Maze / easy00.00.0 1 0–10 Maze / hard00.00.0 1 0–10 Maze / medium 00.07.1 1 0–10 Nut / easy033.351.9 1 0–3 Nut / extreme00.00.0 0 0–3 Nut / hard00.00.0 0 0–3 Nut / medium00.04.2 1 0–3 Nut / nightmare 00.00.0 1 0–1 Rush / easy00.00.0 1 0–10 Rush / hard00.00.0 1 0–10 Rush / medium 00.00.0 1 0–10 Truck / default020.020.0 1 0–5 Qwen3-VL-32B-Thinking / Native GUI Actions (autonomous harness) Bolt / easy00.00.0 0 0–8 Bolt / hard00.00.0 0 0–8 Color / easy00.00.0 0 0–10 Color / hard00.03.8 0 0–10 Maze / easy020.023.2 0 0–10 Maze / hard00.06.4 0 0–10 Maze / medium 00.00.0 0 0–10 Nut / easy00.025.0 0 0–3 Nut / extreme00.00.0 0 0–3 Nut / hard00.00.0 0 0–3 Nut / medium00.00.0 0 0–3 Nut / nightmare 00.00.0 0 0–1 Rush / easy00.05.4 0 0–10 Rush / hard00.04.9 0 0–10 Rush / medium 00.00.3 0 0–10 Truck / default00.018.0 0 0–5 Qwen3-VL-30B-A3B-Thinking / Native GUI Actions (autonomous harness) Bolt / easy012.512.5 0 0–8 Bolt / hard00.00.0 0 0–8 Color / easy00.00.0 0 0–10 Color / hard00.03.8 1 0–10 Maze / easy00.00.0 0 0–10 Maze / hard00.01.4 1 0–10 Maze / medium 00.01.4 0 0–10 Nut / easy00.00.0 0 0–3 Nut / extreme00.00.0 0 0–3 Nut / hard00.00.8 1 0–3 Nut / medium00.00.0 0 0–3 Nut / nightmare 00.04.2 0 0–1 Rush / easy00.02.5 0 0–10 Rush / hard00.06.2 0 0–10 Rush / medium 00.00.3 0 0–10 Truck / default00.02.0 0 0–5 Qwen3-VL-8B-Thinking / Native GUI Actions (autonomous harness) Bolt / easy012.512.5 0 0–8 Bolt / hard00.00.0 0 0–8 Color / easy00.01.7 0 0–10 Color / hard00.02.5 0 0–10 Maze / easy00.00.0 0 0–10 Maze / hard00.00.0 0 0–10 Maze / medium 00.07.1 0 0–10 Nut / easy033.344.4 0 0–3 Nut / extreme00.00.6 0 0–3 Nut / hard00.03.0 0 0–3 Nut / medium00.00.0 0 0–3 Nut / nightmare 00.00.0 0 0–1 Rush / easy00.00.0 0 0–10 Rush / hard00.00.0 0 0–10 Rush / medium 00.00.0 0 0–10 Truck / default00.00.0 0 0–5 37