Paper deep dive
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
Scott C. Lowe, Sander W. Keemink
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/12/2026, 6:07:47 AM
Summary
The paper introduces SpireBench, a benchmark for evaluating long-horizon strategic reasoning in agentic large language models (LLMs) using the commercial deck-builder game Slay the Spire. It evaluates 20 LLMs at the highest difficulty (Ascension 20), finding that while frontier models outperform heuristic baselines and average humans, they lag significantly behind experts, particularly in long-term risk management. The study highlights that models rely heavily on prior knowledge (demonstrated via a reskinning experiment) and fail to improve through self-generated memory across runs.
Entities (9)
Relation Signals (7)
SpireBench → uses → Slay the Spire
confidence 98% · We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS)
SpireBench → evaluates → LLMs
confidence 95% · Across 20 large language models (LLMs) at the game's highest difficulty
LLMs → underperformsvs → Human Experts
confidence 93% · remain far behind experts
LLMs → playsatdifficulty → Ascension 20
confidence 92% · We evaluate 20 LLMs at Ascension 20 on the same ten held-out seeds
LLMs → outperforms → Bottled AI
confidence 90% · frontier models beat heuristic baselines
Reskinning → testsdependencyon → Prior Knowledge
confidence 88% · A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge
SpireBench → implementsvia → CommunicationModPlus
confidence 85% · Agents play the released game throughCommunicationModPlus
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success. Across 20 large language models (LLMs) at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts. This gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger. A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious. Our harness is the first to give a model the complete original game, and we compare it with existing harnesses. Overall, SpireBench offers an unsaturated long-horizon benchmark with tools to locate where and why agents fail.
Tags
Links
- Source: https://arxiv.org/abs/2610.12596v1
- Canonical: https://arxiv.org/abs/2610.12596v1
Trouble viewing inline? Open PDF directly →
Full Text
220,651 characters extracted from source content.
Expand or collapse full text
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire Scott C. Lowe Vector Institute Toronto, Canada scott.lowe@vectorinstitute.ai Sander W. Keemink Department of Machine Learning and Neural Computing Donders Institute for Brain, Cognition, and Behaviour Radboud University Nijmegen, Netherlands sander.keemink@donders.ru.nl Abstract Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success. Across 20 large language models (LLMs) at the game’s highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts. This gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger. A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious. Our harness is the first to give a model the complete original game, and we compare it with existing harnesses. Overall, SpireBench offers an unsaturated long-horizon benchmark with tools to locate where and why agents fail. 1 Introduction Large language model (LLM) progress has been rapid, with high scores on tests of a wide range of general- intelligence-aligned skills (Sajadieh et al., 2026), and LLMs increasingly act as autonomous agents over extended periods (Brohi et al., 2025). However, single-step competence does not necessarily translate into reliable perfor- mance over many sequential decisions, a shortfall termed the horizon gap (Chen et al., 2026). Performance on long tasks is also hard to dissect: does a model rely on stored knowledge or on reasoning (Lowe, 2024), and can it store and effectively use long-term memories? How is it impacted by stochasticity? Existing evaluations leave these questions open. Agentic benchmarks (Yehudai et al., 2026) are mostly short- horizon, with little carried between episodes; game benchmarks (Paglieri et al., 2025) rarely hand a model every decision a player makes, or ask whether performance reflects prior knowledge, reasoning or luck; and reported memory gains (Shinn et al., 2023; Zhao et al., 2024) usually come with an external check on what was learned. We introduce SpireBench, built on the commercial game Slay the Spire (StS), which combines tactical combat, deck construction and route planning over 56 floors (Fig. 1). It is turn-based, fully describable in text, and its choices involve weighing short-term gains over long term success at multiple time horizons, making it well suited for a long-horizon benchmark (Appx. A). Unlike previous work on benchmarking StS (Bateni & Whitehead, 2024; Park et al., 2026), SpireBench interacts directly with all aspects of the game, and allows for separating knowledge from reasoning. Our contributions are as follows: • A harness for the complete original game. Agents play the unmodified commercial game through our harness, making every decision a human player makes (a first) and seeing the same information as a human player would. We allow the model to return a whole turn’s actions as multiple tool calls in one response, and the harness reports the effect of each action back to the model. Against its other variants on the same models and seeds, it plays further and costs less per floor (Table 2). 1 arXiv:2610.12596v1 [cs.AI] 8 Oct 2026 Scale From a cohort of runs down to one turn Consideration Cohort ~10 h 10 runs, in order Learn across runs(meta) What is worth remembering from this run, and what should be forgotten? Only the memory bank carries over; the next run starts with nothing else. 12345678910 Run ~1 h 4 acts Scale ahead of the difficulty (macro) Is the deck getting stronger as fast as the acts get harder? HP, gold, deck, relics and potions carry over for the rest of the run. Act 1Act 2Act 3Act 4 HP B Act ~20 min, 17 floors Route for risk and reward(macro) Go through elites for better rewards, or around them to save HP? The whole act map is shown; the model plans a route at the start of every act. M fight E elite ? event $ shop R rest B boss M M M ? M M M $ ? M E M R M E ? M M M E ? R R R B Floor/Combat ~1 min several turns Spend HP to win(macro) How much HP to spend to end thefight sooner, and which card joins the deck? Enemy intents shown every turn; a log attributes each HP change to the card or enemy behind it. Turn 1 you 80 Nob 86 Turn 2 you 66 Nob 70 Turn 3 you 60 Nob 54 Turn 4 you 52 Nob 32 Turn 5 you 48 Nob 0 Turn ~10 s several actions with one call Sequence the turn(micro) Which cards to play, in what order and on which target, within this turn's energy? One response can hold several tool calls: drink a potion, play cards, end the turn. 1 Strike Attack 1 Pommel Strike Attack 1 Defend Skill 1 Shrug It Off Skill 2 Bash Attack Tool calls: use_potion(Strength Potion) play_card(Bash) play_card(Strike) end_turn() Longer horizons 3 energy per turn Potion Potion memory bank: rewritten after each run, read by the next Figure 1: Multiple time horizons at play in Slay the Spire. Cohort: the model plays a series of runs on different seeds, carrying memories it writes between them. Run: four acts of escalating difficulty, each ending in a boss; the player must grow stronger fast enough to keep up. Act: about 17 floors (Act 3 ends in two bosses at this difficulty; Act 4 has 4) through which the player plans a route, trading safer floors against riskier ones with larger rewards. Floor/Combat: most floors are fights of several turns, each followed by a deck choice that persists for the run; a card weak now may matter later, and a generally good one may be wrong for what lies ahead. Turn: the player draws part of the deck; the model plans the turn and plays it in one response of several tool calls (purple). Sometimes the best play takes more damage now to end the fight sooner. Magenta: choices made during the example Turn, and the part zoomed into for the other rows. •A benchmark at the game’s highest difficulty, calibrated against human data. We evaluate 20 LLMs at Ascension 20 on the same ten held-out seeds, with cost and reasoning spend. For a human performance reference we use Mega Crit’s released telemetry (206 k A20 Ironclad runs) and the run histories of 4 expert streamers. Frontier models beat rule-based and symbolic baselines and the average human player, yet remain far from expert play. The benchmark tracks the Artificial Analysis Intelligence Index but is among the least predictable from other evaluations. •Knowledge ablation with reskin. We reskin the entire game with a holistically consistent theme, separating prior knowledge of the game from reasoning about its rules. Hiding the real names costs performance, concen- trated in a few models. Random and permuted control mappings show that the cost comes from the knowledge attached to familiar names, not from reading unfamiliar ones. • Memory mechanisms, and what the models write. We allow models to write notes and memories within their run and between runs with various methods. No memory formation between runs is able to improve play. Reading the banks shows why: 17% of memories are false, advice to be cautious outnumbers advice to take risks seven to one, reinforcing existing tendencies, and nothing prunes errors. • Analysis of model understanding and failures. We probe the models to predict future game states to evaluate their understanding of play, demonstrating that more performant models are better able to predict near-term game states, but not capable of predicting long-term success. Comparing model in-game choices against humans, the best models are too conservative, failing to amortize their risk profile across the run into earlier sections so they do not take the risks needed to outpace the game’s difficulty curve. 2 Background Slay the Spire is a seminal roguelike deck-builder game, in which players have to balance decision making over several horizons (Fig. 1). In combat turns the player draws a hand from a deck of cards and spends a limited pool of energy to attack, block, or build effects that scale over turns; between fights they add and sometimes remove cards from a deck, and pick a route through a map of nodes (consisting of fights, elite fights, events, shops, and 2 rests — Fig. 1 middle row). Decisions are taken under uncertainty and compound: a card once taken usually stays in the deck across the run, and threats met very late in the game must be prepared for early. A core component is balancing risk, since riskier options early yield the growth that the rising difficulty curve demands (Fig. 1). These are properties shared with other long tasks agents face (Appx. A). 2.1 Related work Slay the Spire. Analyses of the game measure its mix of luck and skill (Andersson et al., 2023; Nam & Rerkjirattikal, 2023) and the complexities of its map uncertainty (Bazzaz & Cooper, 2025). MiniStS (Bateni & Whitehead, 2024) tests LLMs on 8 cards and 5 isolated encounters, evaluating micro-play but no macro-level deck-building. The twelve-game benchmark Orak (Park et al., 2026) drives the real game through the CommunicationMod harness (ForgottenArbiter et al., 2019), but its LLM plays only combat and card rewards, at Ascension 0 on one seed, with a rule-based bot doing the rest. In another approach, Bottled AI, a symbolic AI was handcrafted to play the game (Bottled AI contributors, 2022). While it does not support every card or event, it does report high performance. Finally, in concurrent and most comparable work, AgenticSTS (Cheng et al., 2026), plays complete runs of the sequel Slay the Spire 2. However, the sequel is still in early access with fortnightly balance patches, making it unsuited for benchmarking purposes, and AgenticSTS only plays the game at its lowest difficulty, and was mainly tested with one model. Overall, no existing evaluation has an LLM make every decision in the released game, tests it at the highest difficulty, studies more than a few models, compares against human play, or controls for the models’ prior knowledge of the game. SpireBench plays Slay the Spire at its highest difficulty (A20) across many more models (20). We evaluated model performance against extensive human telemetry, with a controlled reskin; to our knowledge, ours is the first harness giving a model the complete original game. Games as benchmarks. Deep reinforcement learning reached superhuman play in Atari, Go, chess and shogi (Mnih et al., 2015; Silver et al., 2016, 2018) and hard-exploration games (Ecoffet et al., 2021) by train- ing on the game. LLM agents are evaluated untrained, in NetHack, Crafter and Minecraft (Küttler et al., 2020; Hafner, 2022; Wang et al., 2024), benchmark suites (Paglieri et al., 2025; Wu et al., 2024a) and commercial games from frames (Zhang et al., 2025), mostly on simplified states or partial games and rarely with learning across runs. The nearest deck-builder benchmark, BalatroBench (Coder, 2026), has shorter horizons and only the basic difficulty; earlier deck-construction work searches whole decks before play (García-Sánchez et al., 2016). Long-horizon benchmarks. Tool-use, web and agentic benchmarks (Jimenez et al., 2024; Mialon et al., 2024; Yao et al., 2025; Zhou et al., 2024; Koh et al., 2024; Liu et al., 2024b) are mostly tens of steps long; others lengthen the horizon by task duration (Kwa et al., 2025), execution length (Sinha et al., 2026) or a simulated business over tens of millions of tokens (Backlund & Petersson, 2025). Trading (Li et al., 2025a) is long but has a fixed action space. In contrast, a Slay the Spire agent builds the action space it gets to act in later. Memory across episodes. Agents with persistent memory (Park et al., 2023; Packer et al., 2023) have previously been found to improve across episodes (Shinn et al., 2023; Wang et al., 2024; Majumder et al., 2024; Zhao et al., 2024; Wang et al., 2025) given an external check on what is stored; without one, models struggle to self-correct (Huang et al., 2024) and propagate errors in stored experience (Xiong et al., 2026). We test memory on a long, stochastic task with no such check. Reskinning. Renaming entities but not mechanics probes memorization against reasoning: nonsense predi- cates (Valmeekam et al., 2023), changed rules (Wu et al., 2024b), isomorphic worlds (Gu et al., 2026), masked names (Hu et al., 2026), renamed security tasks (Honarvar et al., 2026) and re-textured game objects (Dubey et al., 2018); other benchmarks generate environments a model cannot have seen (Costarelli et al., 2024; Li et al., 2025b). Our reskin covers the whole game with names evoking each mechanic, with random and permuted controls (Sec. 3), paired within seed. 3 Methods Agent harness. Agents play the released game throughCommunicationModPlus, our fork of Communication- Mod (ForgottenArbiter et al., 2019), which exchanges JSON states and text commands and adds snapshots around each enemy’s turn. The harness holds one conversation per floor, shared by a combat, its rewards, and the post-floor reflection. See Fig. 5 and Appx. B in the appendix for an overview and example floor. A cached system prompt holds the task instructions, a compendium of the game’s rules and items, and the memory bank (Sec. 3); each floor adds the act map, the run’s notes and the player’s state. The model acts only through tool calls, interleaving reasoning with action (Yao et al., 2023), and may return a whole turn’s actions in one response 3 Table 1: Benchmark results. Ascension 20 with the Ironclad. Each LLM is run at its highest reasoning setting (grey), played under the themed reskin, with model note-taking across floors within a run, and between runs within a cohort. Runs: number of runs used. Act푘win rate: share of runs that cleared Act푘. Acts cleared: mean number of acts cleared per run. Floor: final floor reached. BottledAI: symbolic agent (Bottled AI contributors, 2022). Human experts: public run histories of Twitch streamers who play Slay the Spire professionally (see Appx. D for details). An author: non-professional player. All humans: Mega Crit released run telemetry (Oct–Nov 2020, final content build) (Appx. D). Best andsecond bestover LLMs. Among the streamers, bold marks the best value on each column among the averages. † Self-hosted; priced at Artificial Analysis headline rates. ‡ Peak-rate list price (upper bound). Act win rate (%) Acts cleared Floor Thinking (k/floor) Cost ($) AgentRuns1234meanmedian[min–max]/floor/run Baseline policies Pick first option1000000.007.16.0[ 1–16]– Pick random option1000000.007.57.0[ 4–14]– Symbolic agent (Bottled AI) BottledAI w/o Heart10200000.2011.39.0[ 5–25]– BottledAI w/ Heart10200000.2011.39.0[ 5–25]– All human players (Mega Crit telemetry, 2020; real game) All human players206,0824715740.7322.116.0[ 1–57]– 1000+ hours played8,28754201170.9224.721.0[ 1–57]– After an earlier Heart kill3,641582921161.2427.923.0[ 1–57]– Human players: averages over a period (real game, own seeds) Baalorlord (2020–2026)687897770632.9847.757.0[ 1–57]– panacea108 (Sep–Nov 2022)51 92 78 71 653.06 48.357.0[ 7–57]– vmService (Sep–Dec 2022)50887464582.8446.857.0[ 7–57]– cheg0 (Feb 2024)46856556462.5243.156.0[ 7–57]– An author (all runs, 2020–2024)50542418141.1026.521.5[ 6–57]– An author (first 10 runs)10502020101.0024.420.0[ 6–57]– LLM agents (thematic reskin) Qwen3.8 27B (xhigh)10200000.2014.916.0[ 5–29]18.10.091 † 1.45 † Qwen3.6 35B-A3B (xhigh)1000000.0010.28.0[ 6–16]4.20.078 † 0.87 † Qwen3.5 122B-A10B (xhigh)1000000.009.89.0[ 7–16]3.70.072 † 0.77 † Claude Fable 5.1 (max)10 90 50 2001.6036.239.5[ 7–52]17.71.66461.91 Claude Opus 5 (max)10 9040001.3032.533.0[ 7–51]21.31.74360.62 Claude Sonnet 5 (max)104010000.5019.316.0[ 7–36]23.20.50910.70 Claude Haiku 4.5 (12k)10100000.1011.911.5[ 4–22]7.60.0941.23 DeepSeek V4.1 Flash (max)10300000.3016.916.0[ 8–22]11.70.019 ‡ 0.35 ‡ Gemini 3.8 Flash (high)105010000.6022.918.0[16–47]13.70.1172.89 Gemma 4 E4B1000000.007.76.5[ 1–16]5.3 0.004 † 0.03 † Muse Spark 1.3 (xhigh)106010000.7021.922.5[ 5–44]12.90.1322.93 Nemotron 3 Nano Omni1000000.006.86.0[ 2–12]34.00.072 † 0.54 † GPT-6 Astra (max)10 90 501001.5037.040.0[ 5–55]3.30.83433.09 GPT-5.6 Sol (max)10702010101.1027.826.0[ 7–57]4.00.2728.39 GPT-5.6 Terra (max)10500000.5018.218.0[10–25]5.00.1492.91 GPT-5.6 Luna (max)10100000.1014.216.0[ 7–24]5.00.0150.24 gpt-oss-120b (high)10100000.1010.48.0[ 5–21]17.00.085 † 0.91 † Grok 4.6 (xhigh)10300000.3015.416.0[ 5–28]11.70.3465.68 GLM-5.3 (max)10600000.6022.122.0[ 5–33]13.40.1673.92 GLM-5.3 Flash (max)10400000.4015.616.0[ 5–27]14.40.0240.40 (multi-action); they run in order up to the first that fails, and each state change is returned as the result of the call that caused it. Our multi-action implementation asks the model to plan the turn before acting and needs fewer calls: a turn costs one response and one pass over the context, where one action per call pays for both again at every card. This is non-trivial to work with since the action space can shift during a turn; we ask models to internalize these changes to make correct tool calls, and interrupt the action queue if an unpredictable event occurs. MiniStS makes one call per action (Bateni & Whitehead, 2024); Orak's agent may return several actions but spends three calls on each decision (reflection, planning, action), shows only the state they leave, re-plans only once they are spent, and leaves every mid-fight card interaction selection to its rule bot (Park et al., 2026); AgenticSTS submits combat as a plan and rebuilds the prompt for every decision by retrieval (Cheng et al., 2026). Table 2 compares the designs and measures ours against its variants and Orak on the same models and seeds. Ours sees what a human player sees, without strategic steering. It plans a route each act, forecasts at the end of each floor its chances of surviving the act and winning the run, and may predict its HP when it ends a turn. Reasoning is kept across turns wherever the 4 Table 2: Comparison of Slay the Spire harness designs. Actions/call: how many game actions one model response may issue; Effects: whether each action's result is returned to the model; Context: what the model sees of earlier decisions; Choices: what resolves card choices opened mid-fight (e.g. select a card to put in your hand, on top of your draw pile, or upgrade). Measured rows are paired with our harness on the same models and seeds without the reskin or a cross-run memory bank for consistency with Orak which does not support either. Not run here: MiniStS (Bateni & Whitehead, 2024) (a reimplementation; one action per call, one fight) and AgenticSTS (Cheng et al., 2026) (the sequel; combat as a plan, a prompt rebuilt by retrieval, routine choices to a fast model tier). Calls are per floor on floors both arms played; cost is the run's API cost per floor it reached, priced as in Table 1 (cached input at its own rate), so the Orak row is comparable with the rest. † Orak keeps no per-floor record: calls and cost per floor reached over whole runs, not floor-matched, and over the 40 of its 51 runs with a usage record, since a run Orak's harness ended records none; a run it ended counts as a loss at the floor reached; its rule bot plays non-combat floors with no model call. For further comparisons, see Table 25. Per floor Acts cleared Tool errors /100 calls HarnessModel playsActions/callEffectsContextChoicesCallsCost ($)Floor Ours, multiactionwhole gamea turnper actionfloormodel8.90.0570.2013.10.8 single actionwhole gameoneper actionfloormodel+3.0+23% −0.07 −1.5+0.4 single action, by indexwhole gameoneper actionfloormodel+2.9+24%+0.00 −0.3+0.1 continuouswhole gamea turnper actionrunmodel −0.4+22% −0.06 −1.2+1.4 Orak (Park et al., 2026)combat, card rewardsa turn, 3 callsend statereflectionrule bot+3.3 † −31% † −0.15 −3.3– provider allows, and no response reached the output ceiling (Appx. B.2). Notes and memory. After each floor the model may write, overwrite or delete free-form run notes under keys of its choosing. After each run it consolidates them into its persistent memory bank one operation at a time (save, delete or finish); the bank enters every later run’s system prompt, so a read-write cohort is a chain played in seed order. We conduct memory ablation experiments to investigate the utility of this schema (Appx. B.3). Knowledge reskin. Unless stated otherwise every run is reskinned: each game-specific name the model can see is replaced by a name from a hand-written internally-consistent mapping themed around tabletop RPGs. We do this so we can evaluate the model’s ability to reason about the game, ablating as much prior information about rules of thumb for how to play StS which it may have seen in pretraining as possible. The model’s own text is not renamed, so a real game name it writes or thinks is evidence of recognition of the base game. We conduct ablation experiments to evaluate the effect of this thematic renaming compared to the original or a randomized rename vocabulary (Appx. B.4). Models and reasoning settings. We evaluate hosted models from Anthropic, DeepSeek, Google, Meta, OpenAI, xAI and Z.ai, and open-weight Qwen, Nemotron and gpt-oss models on our own hardware. Table 1 runs each at its highest reasoning setting (grey, beside its name), resolved to the nearest level the provider supports at or above the request (Appx. C.1). Evaluation protocol.Every main result is on the Ironclad at Ascension 20, the hardest difficulty, on the same ten seeds—freshly minted both to prevent hyperparameter optimization against model/seed pairs used during preliminary experiments, and so that model providers cannot have trained on our earlier results on these seeds (Sainz et al., 2023). A cohort is one model under one configuration on these seeds; each table cell is one cohort, validated by the analysis code (Appx. C.2). We score a run by the acts it clears (Act 4 ends with the Heart) and also report the floor it reaches. A run clears Acts 1 and 2 by beating their (randomly selected) bosses; Act 3 by beating two random bosses; and Act 4 by killing the Corrupt Heart. The act푘win rate is the share of runs that cleared Act푘; mean acts cleared (0–4), is our headline summary. Floor is the final floor reached. A frontier cohort can cost hundreds of dollars (Table 33), so ten seeds is a budget decision, too few to reliably order neighbouring models but sufficient for broader trends. Baselines and cost. Three non-LLM policies play the same seeds without seeing names: pick first option and pick random option bracket the models from below; BottledAI (Bottled AI contributors, 2022) is a hand-written symbolic player (Appx. C.5). Human references at the same setting, scored alike, are the run histories of four expert streamers (Table 1) and 206,082 finished runs of the game’s telemetry, released by Mega Crit. Human games are not matched cohorts against the LLM seeds (Appx. D). 5 $0.02$0.05$0.1$0.2$0.5$1$2$5$10$20$50$100 Mean cost per run (USD, log scale) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Mean acts cleared per run (a) Acts cleared against cost per run 1020304050 Artificial Analysis Intelligence Index 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Mean acts cleared per run Both rungs: Pearson r = 0.82, Spearman ρ = 0.89 (n = 30) Highest effort only: r = 0.80 (n = 20); low effort only: r = 0.91 (n = 10) (b) Acts cleared against the Intelligence Index Figure 2: Mean acts cleared per run for the LLM agents of Table 1. (a) Against mean dollar cost per run (log scale), across reasoning-effort levels. Only points on the Pareto front (blue step line) are named; see Fig. 20 for other names. Dashed lines: the non-LLM baselines (no cost). (b) Against the Artificial Analysis Intelligence Index, at their highest effort and at low effort (low). Dashed line: least-squares fit. Only models on the cost Pareto front of (a) are named. For correlations with each Artificial Analysis evaluation, see Table 18. 4 Results Performance and cost. Our benchmark results are shown in Table 1. We will first consider baseline, symbolic, and human comparisons. Always picking the first option or picking a random option. Neither baseline reaches very far even into the first act. The BottledAI symbolic model Bottled AI contributors (2022) can beat act 1, but goes no further. In contrast, even the average human player in a 200,000 runs human dataset reaches and wins the final act 4% of the time. Top level players comfortably reach 50+% winrates. This illustrates that the game is non-trivial, and has a very high skill ceiling. In our LLM cohort, the strongest models are Claude Fable 5.1, GPT-6 Astra, Claude Opus 5 and GPT-5.6 Sol, and only Sol kills the Heart (the final boss), once; they are also among the most expensive, per run and per floor (Fig. 2a). Mid-tier models outperform Bottled AI but not the average human, and four models never clear an act, no better than the random-choice policy. Performance tracks the Artificial Analysis Intelligence Index (Fig. 2b), but reasoning spent does not buy play: GPT-6 Astra thinks least per floor (3.3k tokens) and clears 1.50 acts, while Nemotron 3 Nano Omni thinks most (34.0k) and clears none. Comparison with human players. The strongest models clear more acts than the average Ascension 20 Ironclad run in Mega Crit's telemetry (0.73) and than players with over 1000 hours (0.92), but far fewer than the four streamers (2.5–3.1 acts). The gap opens as the run lengthens (Table 1 and Fig. 4): the three strongest models win 90% of their Act 1 runs, as the streamers do (85–92%), but 40–50% of Act 2 against 65–78%, and 0–20% of Act 3 against 56–71%. Consequently, the models have a drastically lower win rate for the full game (Act 4) at 0–10% against 46–65%. These results are a demonstration of the horizon gap of Chen et al. (2026), exhibited within a single task. How the models spend a run. The models play Act 1 cautiously. They fight 0.9–1.4 elites, against 2.0 for the telemetry population and 2.6 for the streamers, rest at 57–75% of rest sites against 23% and 9%, and rest at 31% of the rest sites they enter with at least 70% HP, where the population does so at 2% and the streamers never, upgrading a card instead, a permanent boon for the rest of the game. At the Act 1 boss the top four models hold a median 90% of their maximum HP, the streamers 53% (Fig. 3; Appx. E.1). Since the transition to the next act restores 75% of missing HP, HP carried to the end of an act keeps only a quarter of its value; the scarce resource is floors in which to outgrow enemies that scale as the game progresses. Human runs that play Act 1 as the models do clear Act 2 in 15% of cases, against 38% for runs that fight more elites and rest less (Table 13), and in the models' own runs upgrading rather than resting predicts clearing Act 2 (푟=0.56, 49 runs). Experience does not change this: over a chain, rest and elite rates move no differently from no-memory runs; the banks advise caution seven times as often as risk; and the four strongest models record the end-of-act heal correctly yet still rest in 21 of the 32 late-act decisions that weigh it. The route explains only about a fifth of the gap to the streamers; the rest is how the fights are played. 6 under 40% 40–60%60–80% 80% or more HP on arrival (% of max) 0 25 50 75 100 Chose to rest (%) 934 1340 1426 2698 43 61 66 68 114 104 69 77 (a) Resting at rest sites Twitch streamers (6,398 visits) Top 4 LLMs (238 visits) Other LLMs (364 visits) 1173450 Floor 0 20 40 60 80 100 HP at end of floor (% of max) Act 1 36 LLM, 788 human runs Act 2 22 LLM, 671 human runs Act 3 10 LLM, 603 human runs (b) HP left after each floor, over runs that reached the act boss Twitch streamers (833 runs), median and interquartile range Top 4 LLMs (40 runs): Fable 5.1, GPT-6 Astra, Opus 5, GPT-5.6 Sol Figure 3: Do models use HP as a resource? (a) Rest sites can be used to heal or upgrade a card; both top (blue) and other (grey) tier LLMs nearly always rest if they are missing any HP, whereas Twitch streamers (magenta) favour upgrading much more heavily. Bars are labelled with their number of visits. (b) Relative HP left at the end of each floor. Each act is drawn over the runs that reached the act boss (runs that die to the boss count as 0 HP for the boss floor); run counts are above each act. Lines: medians; bands: interquartile range. Streamers spend their HP in Act 1 to get stronger, reducing the amount of damage they take in future floors as a result. Strong models fight better. What separates the models is neither the route (Appx. E.1) nor, much, the deck. Agreement with human card picks barely rises with model strength (Spearman휌=0.16), no model builds its deck for the act boss the map shows, and the one deck-building signal is taking area-damage cards (46% of Act 1 offers for the top four models, 20% for the bottom tier;휌=0.76). The fights separate them. The top four take 8.2 damage per Act 1 hallway fight, the bottom tier 14.0 (휌=−0.87, 19 models; Table 14). Every tier spends the same energy and blocks the same share of incoming damage; the strong models turn the energy into damage (7.5 against 5.8 per energy spent on attacks,휌=0.91), rarely block on a turn when no enemy attacks (6% against 31%) and never miss a lethal play in hand (0% against 8.7%), so they kill a turn sooner and are hit less. The difference holds on the same encounter of the same seed, including the first six floors, before the decks diverge, and about three quarters of the top–bottom gap in Act 1 boss wins (94% against 19%) is not explained by the state in which the models arrive. Relation to public LLM benchmarks. Across 30 model and effort points, mean acts cleared tracks the Artificial Analysis Intelligence Index closely (Pearson푟=0.82, Spearman휌=0.89; Fig. 2b), but the public evaluations behind it, among them GPQA Diamond (Rein et al., 2024) and SciCode (Tian et al., 2024), are strongly correlated with each other too (Burnell et al., 2023; Ruan et al., 2024). Among them SpireBench is one of the least substitutable (Table 18 in Appx. H): the other evaluations predict 50% of its variance for a held-out model, against 85% for Humanity's Last Exam (Center for AI Safety et al., 2026), consistent with lmgame-Bench, where a game probes a blend of capabilities that other benchmarks test in isolation (Hu et al., 2026). Of the agentic evaluations, only Terminal-Bench 4.0 adds information beyond knowledge and reasoning scores (Table 19). The harness moves scores as much as neighbouring models differ. On the same models and seeds (Table 2), Orak's harness, whose rule bot plays every non-combat floor, reaches 3.3 fewer floors than ours and clears 0.15 fewer acts, a gap as large as that between many neighbouring models, and its own failures ended 11 of the 70 seeds. The comparison uses eight mostly weaker models, which clear 0.20 acts on those seeds with our harness and 0.04 with Orak's; its effect on frontier models is untested. Orak spends 85% fewer tokens per floor but costs only 31% less: most of our input is cached. One action per response needs 3.0 more calls per floor, costs 23% more and loses 0.07 acts, a loss that disappears when choices are also made by number (+0.00 acts), so cost is the robust advantage of multi-action. One conversation for the whole run costs 22% more per floor, loses 0.06 acts and raises tool-call errors from 0.8 to 2.2 per 100 calls (Liu et al., 2024a; Sinha et al., 2026). Memory does not compound. Agents that accumulate experience across episodes are reported to improve with it (Shinn et al., 2023; Wang et al., 2024; Majumder et al., 2024; Zhao et al., 2024; Wang et al., 2025; Suzgun et al., 2026); ours do not, nor did the episodic store of Cheng et al. (2026) in StS2 (6/10 wins with and without it). 7 0102030405057 Floor 0 20 40 60 80 100 Runs reaching the floor (%) Act 1Act 2Act 3Act 4 Other LLMs (table rung) Random choices, not an LLM (10 runs) Twitch streamers, average play (4 players) All human players, after an earlier Heart kill (3,641 runs) Gemma 4 E4B GPT-5.6 Luna (high) GPT-5.6 Luna (medium) GLM-5.3 Flash (max) Gemini 3.8 Flash (low) Fable 5.1 (max) Fable 5.1 (low) GPT-5.6 Sol (max) GPT-6 Astra (max) GPT-6 Astra (low) Figure 4: Survival over a run. The share of each agent’s runs that reached each floor. Every LLM agent in Table 1 with all 10 seeds is shown; models on the cost Pareto front (Fig. 2a) are shown. Floor 57 indicates a win of the full game (defeating the Heart at the end of Act 4); a win at the Act 3 boss without the keys to unlock Act 4 stops at floor 52. Dark grey dash-dot: the random-choice policy. Purple: the mean of four Twitch streamers (band spans the four, weighted equally). Black: the 3,641 A20 Ironclad telemetry runs that follow an earlier A20 Heart kill in the same session chain (Appx. D). Shading marks the acts. Table 3: Real game names against our themed reskin, for models with all 10 seeds under both;Δis themed minus real. Control map- pings are in Table 29. Mean acts clearedMean floor reached AgentRealReskinΔRealReskinΔ Qwen3.8 27B (xhigh)0.200.200.0014.014.9+0.9 Qwen3.6 35B-A3B (xhigh)0.000.000.009.710.2+0.5 Qwen3.5 122B-A10B (xhigh)0.100.00 −0.1010.99.8 −1.1 Claude Haiku 4.5 (12k)0.100.100.0011.411.9+0.5 DeepSeek V4.1 Flash (max)0.800.30 −0.5022.016.9 −5.1 Gemini 3.8 Flash (low)0.500.70+0.2020.222.6+2.4 Nemotron 3 Nano Omni0.000.000.005.36.8+1.5 GPT-5.6 Luna (max)0.600.10 −0.5018.514.2 −4.3 gpt-oss-120b (high)0.000.10+0.1010.410.40.0 GLM-5.3 (max)0.700.60 −0.1024.422.1 −2.3 GLM-5.3 Flash (max)0.700.40 −0.3023.115.6 −7.5 Mean (11 models)0.340.23 −0.1115.414.1 −1.3 Consolidating a run's notes into a memory bank or writing a compact summary has no impact on the number of acts won (see Fig. 17 and Table 27). Carrying the notes from during the run forward verbatim without a memory consolidation or compaction phase does slightly better (+0.06 acts): the rewriting adds nothing, much as iterative rewriting of an agent's context erodes its detail (Zhang et al., 2026). Without an external check on what is stored, models do not reliably correct their own reasoning (Huang et al., 2024), and agents propagate the errors they store (Xiong et al., 2026). A run gives one delayed, noisy outcome for hundreds of decisions, so nothing verifies a consolidated memory, and the bank grows run on run (Fig. 19). Moreover, in the final banks 17% of memories are false, mostly generalizations from a single run, and models whose banks hold more errors of their own clear fewer acts (Spearman휌=−0.67, permutation푝=0.002,푛=20; Table 20). Nothing prunes them: over the 200 runs the models opt to delete only 30 memories. Prior knowledge is part of the score. Renaming or altering a familiar task has exposed reliance on memorized content (Valmeekam et al., 2023; Wu et al., 2024b; McCoy et al., 2024), while capture-the-flag agents are largely robust to renaming (Honarvar et al., 2026). Our result lies between. Hiding the real names costs 0.14 acts and 11 points of Act 1 win rate, pooled over 12 model configurations (11 models) and every paired seed (Table 29; per model with all 10 seeds, Table 3), concentrated in a few: DeepSeek V4.1 Flash falls from 0.80 acts to 0.30 and GPT-5.6 Luna from 0.60 to 0.10 (Table 30). Nonsense names cost least (0.01 acts), so unfamiliar words are not the difficulty; the two permuted mappings, which give each entity the name (real or themed) of another, cost about as much as the themed one (0.10 and 0.12). A familiar name pulls a model towards what it usually means although the prompt states the rules (McCoy et al., 2024): the signature of knowledge acquired in training, which contamination studies seek in static benchmarks (Golchin & Surdeanu, 2024) and we show by intervention. The reskin hides names, not the game: in 125 of the 190 themed runs of Table 1 with full logs (Gemma 4 E4B has none), 8 from 18 of 19 models, the model recognizes Slay the Spire or a real entity the reskin replaced, yet recognition does not buy back the cost (Spearman 휌= 0.18, permutation 푝= 0.62; Table 21). Models forecast the next turn, not the consequences for the rest of the run. Larger models are reported to be well calibrated on questions they can answer (Kadavath et al., 2022), while confidence stated in words tends to be overconfident (Xiong et al., 2024); here the horizon decides which holds. Models that forecast the next turn's HP well clear more acts (Spearman휌=−0.84 between error and acts; Table 16). This forecast measures the fight-reading skill above: on turns where the intents shown determine the next HP exactly, the bottom tier still mispredicts it 31% of the time and the top four 0.5%. Forecasts of surviving the act are poorly calibrated: the four models that never clear an act give themselves a 55–76% chance, about as much as the four strongest (59–74%), whose forecasts are followed by surviving the act 63–76% of the time (Fig. 14 in Appx. G), and they barely sharpen as the act nears its end (Fig. 15). Card-reward policy, failures in the reasoning traces and tool-call errors are analysed in Appx. H (Tables 22 to 24). 5 Discussion Our findings consistently demonstrate that agentic models fail to calibrate risk correctly over the long-term. Where the outcome is a turn or a fight away, the strongest models are competent: they win the first act as often as streamers do, win almost every ordinary fight and forecast the next turn well, though they still lose more HP in each fight than humans do. At the time horizon of an act or a run away, they fall behind: the gap to the streamers opens from Act 2, boss fights decide where each model stops, forecasts of surviving an act barely sharpen; moreover, what a model writes down for later runs does not compound. Our findings are indicative of the horizon gap of Chen et al. (2026), but within one task whose time scales share a single pool of resources. Measured as a time horizon (Kwa et al., 2025), the fall is a step: every model succeeds on at least 87% of its turns and falls below 50% between a fight and a run, for most models already at the act (Table 17). The difficulty is not how long a model can keep going: it is fighting well enough to afford risk, and spending early floors on growth whose payoff comes an act later, which no model learns from its own runs. Any agent planning over a long, stochastic task with one delayed outcome needs memories checked against outcomes and forgotten when wrong, forecasts that sharpen as the stakes approach, and a harness that plays the whole task (Sec. 5). Four findings confirm mechanisms that prior work demonstrated outside games. The harness is part of the score, as for other agents (Kapoor et al., 2025; Yang et al., 2024). Unverified memory does not compound wherever the outcome is delayed and noisy over many decisions, as in most open-ended deployments; where a verifier exists, such as tests for code, the reported gains should hold. Long contexts cost performance (Liu et al., 2024a; Sinha et al., 2026). Calibration degrades with the horizon in any task whose outcomes resolve at several distances. Limitations. The ordering of models is measured on one character in a turn-based game played from parsed text; other characters, genres, real-time control or perception would weight the decisions differently. The size of the reskin cost depends on how much of this game each model saw in training, and the reskin hides names but not familiarity with deck-builders, so it is a lower bound. The human comparison is not seed-matched. Ten seeds per cohort cannot order neighbouring models, and providers can change the model behind an identifier without notice. Acknowledgments Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. Thanks to Sam Laing and Marvin da Silva for useful discussions and for providing feedback on the paper. References Johan Andersson, Mikolaj Trojanowski, and Alberto Alvarez.Are you lucky or skilled?An analy- sis of elements of randomness in Slay the Spire.In IEEE Conference on Games (CoG), p. 1–8, 2023. doi:10.1109/CoG57401.2023.10333231. Axel Backlund and Lukas Petersson. Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv preprint arXiv:2502.15840, 2025. doi:10.48550/arXiv.2502.15840. Bahar Bateni and Jim Whitehead. Language-driven play: Large language models as game-playing agents in Slay 9 the Spire. In Proceedings of the 19th International Conference on the Foundations of Digital Games (FDG), p. 1–10. ACM, 2024. doi:10.1145/3649921.3650013. Mahsa Bazzaz and Seth Cooper. Analysis of uncertainty in procedural maps in Slay the Spire. In Proceed- ings of the 20th International Conference on the Foundations of Digital Games (FDG), p. 1–10. ACM, 2025. doi:10.1145/3723498.3723846. Bottled AI contributors. Bottled AI: Slay the Spire – in an automated way.https://github.com/xaved88/ bottled_ai, 2022. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2. Sarfraz Brohi, Qurat-ul-ain Mastoi, N. Z. Jhanjhi, and Thulasyammal Ramiah Pillai. A research landscape of agentic AI and large language models: Applications, challenges and future directions. Algorithms, 18(8):499, 2025. doi:10.3390/a18080499. Ryan Burnell, Han Hao, Andrew R. A. Conway, and José Hernández-Orallo. Revealing the structure of language model capabilities. arXiv preprint arXiv:2306.10062, 2023. doi:10.48550/arXiv.2306.10062. Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649(8099):1139–1146, 2026. doi:10.1038/s41586-025-09962-4. Humanity’s Last Exam. Mingguang Chen, Licheng Wang, and Bo Qu. The horizon gap: Planning, memory, execution, training, and evaluation for long-horizon LLM agents. arXiv preprint arXiv:2608.06663, 2026. doi:10.48550/arXiv.2608.06663. Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, and Kaipeng Zhang. AgenticSTS: A bounded-memory testbed for long-horizon LLM agents. arXiv preprint arXiv:2607.02255, 2026. doi:10.48550/arXiv.2607.02255. Coder. BalatroBench: Benchmark LLMs’ strategic performance in Balatro.https://github.com/coder/ balatrobench, leaderboard at https://balatrobench.com, 2026. Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. GameBench: Evaluating strategic reasoning abilities of LLM agents. arXiv preprint arXiv:2406.06613, 2024. doi:10.48550/arXiv.2406.06613. Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L. Griffiths, and Alexei A. Efros. Investigating human priors for playing video games. In International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, p. 1349–1357. PMLR, 2018. URLhttps://proceedings.mlr.press/v80/ dubey18a.html. Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580–586, 2021. doi:10.1038/s41586-020-03157-9. Ronald A. Fisher. The Design of Experiments. Oliver and Boyd, Edinburgh, 1935. ForgottenArbiter et al. CommunicationMod: Slay the Spire mod that provides a protocol for allowing another process to control the game. https://github.com/ForgottenArbiter/CommunicationMod, 2019. Pablo García-Sánchez, Alberto Tonda, Giovanni Squillero, Antonio Mora, and Juan J. Merelo. Evolutionary deckbuilding in HearthStone. In IEEE Conference on Computational Intelligence and Games (CIG), p. 1–8, 2016. doi:10.1109/CIG.2016.7860426. Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum? id=2Rwq6c3tvr. Ken Gu, Advait Bhat, Mike A. Merrill, Robert West, Xin Liu, Daniel McDuff, and Tim Althoff. SynthWorlds: Controlled parallel worlds for disentangling reasoning and knowledge in language models. In International Conference on Learning Representations (ICLR), 2026. URLhttps://openreview.net/forum?id=46AQ4qaWqQ. Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=1W0z96MFEoH. Shahin Honarvar, Amber Gorzynski, James Lee-Jones, Harry Coppock, Marek Rei, Joseph Ryan, and Alastair F. Don- aldson. Capture the flags: Family-based evaluation of agentic LLMs via semantics-preserving transformations. 10 arXiv preprint arXiv:2602.05523, 2026. doi:10.48550/arXiv.2602.05523. Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. lmgame-Bench: How good are LLMs at playing games? In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=qeziG97WUZ. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=IkmD3fKBPQ. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernan- dez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. doi:10.48550/arXiv.2207.05221. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=Zy4uFzMviZ. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Annual Meeting of the Association for Computational Linguistics (ACL), p. 881–905, 2024. doi:10.18653/v1/2024.acl-long.50. Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefen- stette, and Tim Rocktäschel. The NetHack learning environment. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2020. URLhttps://proceedings.neurips.c/paper_files/paper/2020/hash/ 569f987c643b4bedf504efda8f786c2-Abstract.html. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, p. 102382–102435, 2025. doi:10.52202/085713-3086. Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, K.P. Subbalakshmi, Jimin Huang, Lingfei Qian, Xueqing Peng, Jordan W. Suchow, and Qianqian Xie. INVESTORBENCH: A benchmark for financial decision-making tasks with LLM-based agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, p. 2509–2525, 2025a. doi:10.18653/v1/2025.acl-long.126. Yuchen Li, Cong Lin, Muhammad Umair Nasir, Philip Bontrager, Jialin Liu, and Julian Togelius. GVGAI- LLM: Evaluating large language model agents with infinite games. arXiv preprint arXiv:2508.08501, 2025b. doi:10.48550/arXiv.2508.08501. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024a. doi:10.1162/tacl_a_00638. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), 2024b. URLhttps://openreview.net/forum? id=zAdUB0aCTQ. Scott C. Lowe. System 2 reasoning capabilities are nigh. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, 2024. URL https://openreview.net/forum?id=ZxaipE6f62. 11 Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Taf jord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. In Conference on Language Modeling (COLM), 2024. URLhttps://openreview.net/forum? id=xS6zx1aBI9. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. Embers of autoregres- sion show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121, 2024. doi:10.1073/pnas.2322420121. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=fibxvahvs3. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. doi:10.1038/nature14236. Allan H. Murphy. A new vector partition of the probability score. Journal of Applied Meteorology, 12(4):595–600, 1973. doi:10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2. SangGyu Nam and Pavinee Rerkjirattikal. Exploring the impact of randomness in roguelike deck-building games: A case study of Slay the Spire. In Integrated Uncertainty in Knowledge Modelling and Decision Making (IUKM), Lecture Notes in Computer Science, p. 97–108. Springer, 2023. doi:10.1007/978-3-031-46775-2_9. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. doi:10.48550/arXiv.2310.08560. Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wołczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Foerster, Jack Parker-Holder, and Tim Rocktäschel. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=fp6t3F669F. Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong- Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, Ameya S. Mahabaleshwarkar, Bilal Kartal, Pritam Biswas, Yoshi Suhara, Kangwook Lee, and Jaewoong Cho. Orak: A foundational benchmark for training and evaluating LLM agents on diverse video games. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=H1ncX6O6Yh. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium on User Interface Software and Technology (UIST), p. 1–22, 2023. doi:10.1145/3586183.3606763. Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, p. 48371–48392, 2025. URLhttps://proceedings.mlr.press/v267/patil25a. html. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling (COLM), 2024. URL https://openreview.net/forum?id=Ti67584b98. Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, p. 15841–15892, 2024. doi:10.52202/079017-0506. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 10776–10787, 2023. doi:10.18653/v1/2023.findings- emnlp.722. Sha Sajadieh, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Lapo Santarlasci, Juan Pava, Nestor Maslej, Russ Altman, Erik Brynjolfsson, Carla Brodley, Jack Clark, Virginia Dignum, Vipin Kumar, 12 James Landay, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Elham Tabassi, Russell Wald, Toby Walsh, and Dan Weld. Artificial intelligence index report 2026. arXiv preprint arXiv:2606.15708, 2026. doi:10.48550/arXiv.2606.15708. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, p. 8634–8652, 2023. doi:10.52202/075280-0377. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. doi:10.1038/nature16961. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362 (6419):1140–1144, 2018. doi:10.1126/science.aar6404. Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The illusion of diminishing returns: Measuring long horizon execution in LLMs. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=3lm8lWYxiq. Charlie V. Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=4FWAwZtd2n. Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7080–7106, 2026. doi:10.18653/v1/2026.eacl-long.333. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. SciCode: A research coding benchmark curated by scientists. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, p. 30624–30650, 2024. doi:10.52202/079017-0963. Mikolaj Trojanowski and Johan Andersson. Are you lucky or skilled in Slay The Spire?: An analysis of random- ness. Bachelor’s thesis, Malmö University, 2021. URLhttps://urn.kb.se/resolve?urn=urn:nbn:se:mau: diva-43090. Typhi. Slay the heart (pull request 16 to Bottled AI).https://github.com/xaved88/bottled_ai/pull/16, 2024. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 36, p. 38975–38987, 2023. doi:10.52202/075280-1693. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research (TMLR), 2024. URL https://openreview.net/forum?id=ehfRiF0R3a. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, p. 63897–63911, 2025. URL https://proceedings.mlr.press/v267/wang25bx.html. Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. SmartPlay: A benchmark for LLMs as intelligent agents. In International Conference on Learning Representations (ICLR), 2024a. URLhttps://openreview.net/forum? id=S2oTVrlcp3. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Volume 1: Long Papers, p. 1819–1862, 2024b. doi:10.18653/v1/2024.naacl-long.102. 13 Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=gjeQKFxFpZ. Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Zirui Liu, Jiliang Tang, Himabindu Lakkaraju, and Zhen Xiang. How memory management impacts LLM agents: An empirical study of experience-following behavior. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, p. 623–645, 2026. doi:10.18653/v1/2026.acl-long.27. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, p. 50528–50652, 2024. doi:10.52202/079017-1601. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.휏-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=roNSXZpUDN. Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli- Scheuer. A survey on evaluation of LLM-based agents. In Findings of the Association for Computational Linguistics: ACL 2026, p. 26690–26714, 2026. doi:10.18653/v1/2026.findings-acl.1330. Lu Yihe, Scott C. Lowe, Penelope A. Lewis, and Mark C. W. van Rossum. Program synthesis performance constrained by non-linear spatial relations in Synthetic Visual Reasoning Test. arXiv preprint arXiv:1911.07721, 2019. doi:10.48550/arXiv.1911.07721. Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench: Can vision-language models complete popular video games? arXiv preprint arXiv:2505.18134, 2025. doi:10.48550/arXiv.2505.18134. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Y. Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=eC4ygDs02R. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 38, p. 19632–19642, 2024. doi:10.1609/aaai.v38i17.29936. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=oKn9c6ytLx. 14 Appendices A Why Slay the Spire17 B Harness18 B.1Game interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2 Agent harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.3Notes and memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.4Knowledge reskin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.5Model-facing features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B.6 What the model receives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 B.7Coverage compared with Orak and Bottled AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 C Experimental details28 C.1 Models and reasoning settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 C.2 Seeds and cohort selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 C.3 Metrics and statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 C.4 Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 C.5 Non-LLM baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 C.6 Ablations and analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 D Human baselines32 E Strategy and play quality34 E.1Where agents die: agents against humans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 E.2Conservative Act 1 play among humans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 E.3Risk in the models' own runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 E.4Learning over the chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 E.5Deck building and boss preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 E.6Combat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 F Difficulty and where runs end42 G Calibration44 H Further analyses48 H.1 Time horizon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 H.2 Public benchmarks and reasoning effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 H.3 Agentic residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 H.4 Runs ended by the harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 H.5 One conversation per floor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 H.6 What the memory banks hold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 15 H.7 Reskin cost by model family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 H.8 Recognition of the game under the reskin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 H.9 Deck building does not fail by accumulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 H.10 Where models go wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 I Ablation results in full56 I.1Harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 I.2Reasoning effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 I.3Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 I.4Knowledge reskin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 I.5Prompt material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 J Cost and reasoning64 J.1Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 J.2Time and cost of a run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 16 A Why Slay the Spire Slay the Spire is a seminal game that helped establish the roguelike deck-builder genre, and the properties that make it hard are those that make any long task hard for an agent. Every choice recurs. A card added to the deck on the second floor is drawn again in every later fight, so it has to be useful not only against the next enemy but against the act bosses an hour later, and a bad addition cannot simply be undone: card removals are limited opportunities and the deck starts with cards (Strikes) which the player would already benefit from removing, so adding a bad card puts the player down a card removal for the rest of the game. The consequences of a decision are therefore spread thinly over hundreds of later ones—this is the credit-assignment problem of long-horizon execution, where small per-step errors compound over the length of a task (Sinha et al., 2026). The only verdict on the run as a whole, a win or a loss, comes at its end. An archetype is not a deck. Players learn named strategies, such as building around Strength or around exhausting cards, but each run offers a random array of cards and relics based on its seed. A plan must be devised as the run unfolds rather than determined in advance, and the agent has to judge when to commit to a plan and when to abandon it. A card's worth also depends on the cards and relics already held: a card that scales with Strength is weak in a deck with no source of Strength and strong in one with several, so cards cannot be valued one at a time. The agent builds its own action space. In combat, the actions open to the player on a turn are the cards in hand, and the hand is drawn from a deck the player assembled. Every card reward, purchase, removal and upgrade therefore decides which actions will exist in later fights: the agent chooses not only its moves but the set of moves it will later choose among. This is a sharper form of what an agent does when it builds a library of skills or tools for later use (Wang et al., 2024), with the difference that here the library is also a liability, since every card added is one more card that may be drawn in place of the one that was needed. Even a single turn is a small planning problem, because the order of plays matters: an attack does half as much again to an enemy made Vulnerable first, so Bash is played before the Strikes it sets up. Each fight tests the deck differently. A deck is not good or bad in general but against a particular fight. Groups of enemies call for damage that hits all of them; long fights against enemies that grow stronger call for cards that scale, while short ones call for damage up front; and some enemies punish a kind of play outright: Gremlin Nob gains Strength whenever a Skill is played, the Awakened One whenever a Power is played, and the Time Eater takes over the turn once a set number of cards has been played. A card bought to pass one of these checks can weaken the deck against another, and every card added dilutes the draws for all of them, so the player must decide which checks the deck has to pass, which it can afford to scrape through, and in what order the run will set them. Risk is amortized over the run. HP, gold, potions and the deck form one irreversible pool that carries from floor to floor. Taking an elite fight costs HP now for a relic that pays out over every later fight; resting at a campfire recovers HP at the price of an upgrade. The right choice depends on what the rest of the run will demand, so the agent must price present resources against a future it can only estimate, and how well it does so shows in how well calibrated those estimates are. The challenges ahead are known in kind but not in detail. Each act draws its fights from a fixed pool, and the map shows the types of room ahead, so a player who remembers what the act's elites and bosses do can build the deck to meet them; one who does not meets each as a surprise. Doing well therefore needs memory across runs as well as within one, which is the setting in which memory mechanisms for LLM agents are proposed (Park et al., 2023; Packer et al., 2023; Shinn et al., 2023) but seldom tested over long, stochastic episodes. Some of the game is fixed and some is dealt. Within a fight, each enemy shows its intended action for the coming turn, and enemies follow move patterns that can be learned, so the threat on the current turn is known exactly and a turn can be planned as a deterministic puzzle. What the player will draw next turn, which cards a reward will offer, and what an unknown room holds are random; the player knows which cards are in the draw pile but not their order. The map is visible from the start of each act, but not what will be on offer along a route. A good player therefore treats different parts of the same decision differently, solving what is fixed and pricing what is dealt, and an agent has to tell the two apart, neither gambling on a turn whose outcome it could have computed nor planning the run as though the draws were known. The models we test divide along this line: they forecast the next turn's HP far better than their chance of surviving the act (Sec. 4). These properties are the parts of the horizon gap (Chen et al., 2026) that short benchmarks cannot reach, and a single run of the game, about an hour of human play, sits at the task lengths on which agent progress is 17 now charted (Kwa et al., 2025). The game also has three practical properties that make it a good testbed. It is turn-based and fully described in text, so what is tested is decision-making rather than perception or reaction time. Its outcomes mix skill with luck in a measured proportion (Trojanowski & Andersson, 2021), and its maps pose genuine route decisions under uncertainty (Bazzaz & Cooper, 2025), so a fixed set of seeds gives a paired, repeatable comparison. And it has a large, well-documented population of human players, from general game telemetry to expert streamers, against which models can be placed. B Harness Figure 5 gives an overview of the components of the harness. Figure 6 follows the messages exchanged between the game, the harness and the model through the four points at which the harness calls the model: the start of an act, each combat turn, the end of a floor and the end of a run (Sec. 3). The subsections then describe the game interface, the agent harness, notes and memory and the knowledge reskin, and Appx. B.5 lists every feature of the harness that the model can see. GameHarnessPrompt (one per floor)LLM Slay the Spire The released game, rules unmodified. Every screen is played: • combat • map routing • events, Neow • shops, rest sites • card / relic rewards CommunicationModPlus JSON state out, text commands in; snapshots around each enemy's turn Parse & diff State → text. Each action's effect comes back as the result of the call that caused it; enemy turns attributed enemy by enemy. Control router combat · rewards · events · path → the LLM, or a symbolic player (Bottled AI, left, random) Reskin every game name, exactly once: Bash → Trip Attack Validate & map back Names mapped back to the game; an illegal action is refused with an explanation, never sent; actions run in order until one fails. state cmds cmds Instructions + compendium system prompt, cached across runs Memory bank cached within a run Act map · run notes cached within an act Floor history · HP, gold, deck per floor Turn messages (grow each turn) ## Player Actions - Played Trip Attack. [0] Goblin Berserker HP -8 (86→78); Exposed +2 ## Enemy Actions - [0] Goblin Berserker — Player HP -14 (80→66) ## Turn 3 Hand: [Slash] [Dodge] ... Any provider Reasons, then acts only through typed tool calls (18, all sent on every call). Reasoning is kept across turns wherever the provider allows. whole turn, one reply: play_card( "Slash", enemy_index=0) play_card( "Lunging Slash", enemy_index=0) end_turn(...) Also: choose_path, plan_route, use_potion, choose_by_name, save_note, predict, ... call tool calls results Boundaries: what the model writes down, and whennotes feed the next floor's prompt; the bank, every later run's Act start plan_route over the whole act map End of turn forecast HP next turn and at end of fight End offloor P(survive act), P(win); save / delete run notes End of run consolidate notes into the memory bank Next run bank loaded into the system prompt Figure 5: Our harness. Game states become text, each action’s effect attributed to its tool call; a router sends each part of the game to the model or a symbolic player; in-game names are (optionally) reskinned; tool calls are mapped back and validated. The prompt runs from cached layers to turn messages. Bottom: when the model plans, forecasts and writes notes. B.1 Game interface Our fork of CommunicationMod (ForgottenArbiter et al., 2019),CommunicationModPlus, adds a file-based channel that survives restarts of the game’s Java process, tracking of the three keys that unlock Act 4, and intermediate state snapshots around each enemy’s turn, from which the harness attributes damage, block and status changes to the enemy that caused them. Every run loads the same mod list:BaseMod,StSLib,CommunicationModPlus, rngFix,permNeow(a Neow blessing on every run),SuperFastMode(animation speed only),RelicStatsand fast_multicast. None changes the rules a human plays by. Each run executes in an isolated game profile, and every act boss is marked as seen, so each run’s seed, and not the profile’s history, picks the act bosses. B.2 Agent harness Each call carries a cached system prompt in two parts. The first is stable across runs: task instructions and a compendium of the game’s rules, keywords, status effects, cards, relics and potions with their effects. The second is stable within a run: the memory bank (Sec. 3). The first user message of a floor adds the act map, the run’s notes, 18 Act start once per act Each combat turn repeats until thefight ends End of floor every floor End of run once per run Slay the Spire real game + mod Harness parse · reskin · validate LLM any provider map screen (JSON) whole act map, reskinned (memory bank already in system prompt) plan_route(...) → kept as a run note route over all 17 floors plan_route( route="Monster(1,3)→ Unknown(2,2)→...→Boss", notes="one early elite while HP is high") turn state: hand, energy, enemy intents several tool calls in one reply — the whole turn un-reskin; refuse an illegal action with an explanation play 3 0 new state one command at a time, diffed after each each call's own result, then each enemy's action, then the next turn reply, then tool results play_card("Slash", enemy_index=0) play_card("Lunging Slash", enemy_index=0) end_turn(predicted_hp_after_turn=58) ## Player Actions - Played Slash. [0] Goblin Berserker HP -6 (54→48) ## Enemy Actions - [0] Goblin Berserker — Player HP -8 (66→58) rewards screen: card, gold, potion choose_by_name("Crippling Gash") floor summary + current notes predict(...) · save_note(key, note) predict( act_win_probability=0.7, run_win_probability=0.1) save_note("elites", "Goblin Berserker gains Strength when I play a Technique") outcome, all notes, current memory bank save_memory / delete_memory, one at a time, until finish_consolidation memory bank → next run's system prompt save_memory( "elite:Goblin Berserker", "Open with Attacks only; save Techniques until it is below 30 HP") finish_consolidation() One conversation perfloor. The system prompt (instructions, compendium, memory bank) and the act map are cached; everything the model sees is reskinned exactly once, and its own notes and memories are never renamed. Figure 6: The harness as a sequence of messages, with time running downwards. In combat the model returns a whole turn as several tool calls in one reply; the harness sends them to the game one command at a time, diffs the state after each, and returns each call’s own effect as that call’s result, followed by each enemy’s action and the next turn’s state. At the end of each floor the model forecasts its chances and may update its run notes; at the end of the run it consolidates them into the memory bank that the next run’s system prompt carries. Code on the right is the shape of what the model sends and receives, with names as it sees them under the reskin. a history of the floors played so far and the player’s current state (HP, gold, deck, relics, potions and keys). In combat, each turn’s message reports what each of the model’s actions did, what each enemy did, and the full state at the start of the new turn. Outside combat the message describes the current screen and its legal choices. The tools are: playing a card or a sequence of cards, ending the turn, using or discarding a potion, choosing an option by index or by name, choosing a map node, planning a route, taking notes, predicting the floor’s outcome, and the end-of-run memory operations. The full tool set is sent on every call, so that the cached prefix never changes, and an action illegal on the current screen is refused with an explanation. By default the model may return several actions in one response (multi-action), and the harness executes them in order, stopping at the first that fails. At the start of each act the model is asked for a route plan over the whole act map. At the end of each floor it is asked for its chance of surviving the current act and of winning the run, and may reflect and update its notes; when it ends a turn it may also predict its HP at the start of the next turn and at the end of the fight. Two principles shaped the harness. The model is shown what a human player sees and nothing more: the prompts describe the task on the current screen without steering towards a strategy (they never say, for instance, to check first whether an enemy can be killed this turn), and the system prompt holds only the documentation a player has in the game itself, namely its tutorial, tips and compendium. And the model is asked to do as much as possible per call, to keep cost down. Many combat actions have a deterministic effect, so the model plans the whole turn and returns every action it wants in one response, up to the first whose outcome is random. MiniStS makes one call per action and re-prompts with the new state (Bateni & Whitehead, 2024). Orak’s agent may also return several actions at once, but spends three calls on each decision (reflection, planning, then the actions) and shows the 19 model only the state its actions leave (Park et al., 2026). Multi-action play asks the model to hold the intervening states in mind, which is harder. Batching also helped play in our preliminary runs. Because the model had to plan the whole turn before acting, it committed to a plan, and it did not spend reasoning deriving the same plan again after each action when that action had revealed nothing new. On the main seeds, one action per response needs 3.0 more calls per floor, costs 23% more and clears 0.07 fewer acts (Table 2). So that it can still see what each action did, the harness diffs the game state between actions and returns each change as the result of the tool call that caused it, with enemies’ turns attributed enemy by enemy. A call that names a card imprecisely is resolved where the intended card is unambiguous, and otherwise answered with a “did you mean” suggestion rather than executed. Names, not indices. The model refers to what it acts on by name: a card by the name it is shown under (play_card(card_name="Bash", enemy_index=0)), and an event option, rest-site action, shop item or card on a selection screen by its label (choose_by_name(name="Rest")). Arguments are passed by name too, never by position. Only enemies, card rewards and the reward list are chosen by index, which is shown beside each entry. There are two reasons for this. The first is batching. A batch is written against the state at the start of the turn, but the hand changes while the batch runs: every card played moves the positions of the others, and cards are drawn, created, upgraded and moved by the plays themselves. A position in the hand goes stale after the first change, but a name still identifies the same card. A batch can therefore play a card that an earlier call in it creates, and it can answer, in the same call, a choice that a card opens (the card that Armaments upgrades, the card that Headbutt puts back), through target_card_names. This is what lets the model play a whole turn in one response without giving up any of the game’s mechanics. How Orak handles the same cases is in Appx. B.7. The second is reliability. In preliminary runs, weaker models sometimes reasoned their way to one option and then gave the index of another. A name states the intended option in the call itself, so it cannot pick a different option by mistake. A name that matches nothing is refused with a suggestion (Appx. B.5). On the main seeds this concern did not show up: with one action per response, the models tested chose no worse by index than by name (Table 2). B.3 Notes and memory Run notes are rendered back to the model as<note key=". . . ">tags. The memory ablation (Table 27) varies the mechanism while holding everything else fixed. Consolidate (xml) is the default of Sec. 3. Keep notes copies each run’s notes into the bank verbatim, with no consolidation call, so it keeps persistence but removes the rewriting. None discards the notes at the end of each run. In the compact arms the model instead writes a single prose summary through a one-argumentcompacttool, overwriting it at the end of every floor and, when consolidating, at the end of every run. Every prompt that asks the model to write something down includes the same generic guidance: distil insights rather than log events, and record what went well, what went wrong and what the model wishes it had known. It names nothing specific to the game. A cohort with a read-write bank is a chain. Seed푘starts from the bank that seed푘−1 left behind, so the seeds are played in order and a cohort’s bank is never shared with any other configuration: each bank is partitioned by model, character, the full configuration tag and the seed list. B.4 Knowledge reskin The names replaced are those of cards, relics, potions, enemies, events, powers and keywords, the character, and the map’s node labels. Names are replaced through placeholders, so that one replacement can never trigger another, and each replacement evokes the original’s effect rather than its name. The harness’s own text (the system prompt, state descriptions and tool results) is renamed; the model’s own text is not, so a real game name in a note is evidence of recognized prior knowledge rather than something to be laundered. The five mappings of the reskin ablation (Table 29) are all generated in the same format, so that only the mapping file changes between arms. None shows the real names. Themed is the default. Random replaces every word with a pronounceable nonsense word, consistently, so that multiword names keep their shared structure. Permuted gives each entity the real name of a different entity of the same kind (cards within their colour and type), which is coherent but actively misleading. Permuted and themed applies the same permutation in the themed vocabulary. The permuted mappings keep every rule word that the themed mapping leaves unrenamed, so the arms differ only in the names. Table 4 gives examples. 20 Table 4: Example entries of the five reskin mappings (Table 29). Under none each entity keeps the name in the second column. Card names are shown verbatim. N.B. There are some in-game effects which act on “cards containing Strike”; a permutation would make that rule name a family the model cannot see, so in both permuted mappings the Strike family is marked with a dagger and the rule refers to it. In the thematic reskin we map all instances of Strike to Slash, avoiding the need for a separate marker. KindNone (real)ThemedRandomPermutedPermuted and themed CardBashTrip AttackBekReckless ChargeReckless Blow CardStrikeSlashDrakae†Defend†Dodge MechanicBlockGuardTaedoastChannelSummon StatusVulnerableExposedBaevimiNo DrawDraw Lock RelicBurning BloodBloodied ChaliceVriastild TousornFossilized HelixWinged Statuette EnemyGremlin NobGoblin BerserkerSlagloath SkialCenturionGuardian OrbLightning OrbWater SpiritSofoa SemRuby OrbOpal Spirit B.5 Model-facing features This subsection lists everything the harness does that the model can see, beyond the outline in Appx. B.2. The harness never makes a strategic choice for the model. When it acts by itself, there is only one legal action, or the action has no strategic content. When it corrects a call, it either resolves the call where the intent is unambiguous or refuses it with an explanation. Quoted messages are given as the model receives them, before renaming. What the model is shown. •Map. The act map is drawn with labelled nodes (Monster(5,2): floor 5, column 2). Burning elites, which carry the Emerald Key, are marked, and the act boss is named. The current floor is marked>>and the reachable next nodes *. On map screens the model may revise its route plan before choosing. • Player overview. HP, gold, keys, the deck (identical copies merged as3x Strike), relics with their descriptions and counters (e.g. (2/3), (USED)), and potions. It is omitted from a message when nothing in it has changed. •Combat turn. Each enemy is shown with its index, HP, Block, intent (e.g. Attack for 6x2) and powers with their current descriptions. Dead enemies are removed and the rest renumbered. The player’s HP, Block, energy, powers, stance and orbs are shown, and the hand lists one card per line with cost, type, whether it targets and why it cannot be played. The draw, discard and exhaust piles are listed in full when they are relevant (on a new turn, after a change or a reshuffle, or when a card in hand refers to that pile) and as counts otherwise. •Card names. Cards are referred to by name, not hand position. Copies that differ get a qualifier, e.g.[Strike (0 cost)] when an effect (e.g. Snecko Eye, Mummified Hand) has changed a cost, and so does a card whose value has grown past its base (Ritual Dagger, Genetic Algorithm). •Effects. The result of each tool call is the change in game state that call caused, e.g. “Played Bash. Energy -2 (3→1). Enemy [0] Jaw Worm HP -8 (40→32); Vulnerable +2.” Cards drawn, created and exhausted are named. Between turns, the next message reports the end of the player’s turn, then one line per acting enemy, then the end of the round (powers/statuses decaying or expiring) and the start of the new turn (Block kept, start-of-turn relics, reshuffles). These are split using game-state snapshots from the mod (Appx. B.1). Where several effects land together (the end of a fight, a room, a selection screen), the harness reports the net change without naming a cause, leaving the attribution to the model, which can see its relics. •Non-combat screens. Each screen lists its legal choices with what the model needs to choose: shop prices with (can’t afford) and the card-removal price; the HP a rest would heal; card, relic and potion rewards as one numbered list; event options with (DISABLED) marked; and the Match and Keep board. Each choice is confirmed with what it changed, e.g. “Chose to rest. HP +24 (...)”. •History. One line per floor played so far (room type, encounter, turns taken, HP and gold change, cards added, upgraded and removed, relics gained), with the act’s route plan and the run notes. Reading the model’s calls. • Lenient matching. Card, potion, relic and option names are matched case-insensitively, with brackets, stray quotes and parentheses, and non-breaking spaces removed. A qualified name matches if its qualifiers match 21 the card’s current values. A bare name resolves to the upgraded copy when only that copy is in hand (Bash→ Bash+), but not the reverse. Numbers given as strings are converted. Under the reskin, names are mapped back to the game’s names and checked against what is on screen. •Copies. If the copy of a card that a call names has left the hand by the time the call runs, the harness plays an identical copy that is still in hand, and refuses the call only if there is none. •Deferred plays. A card named before it exists (one an earlier call in the batch will create or upgrade) is looked up again when its call comes up, and refused only if it is still missing then. •Targets. An attack or potion that needs a target gets the only living enemy when there is one, and is refused otherwise. Enemy indices refer to the list the model was shown, not the game’s internal one. If a targeted enemy has died, the refusal says that the remaining enemies have been renumbered. •Pre-targeted selections. A card that makes the player choose another card (Armaments, Headbutt, a discard) can name its target in the same call (target_card_names). The harness then makes the choice on the selection screen that opens and reports what was picked, so the batch continues without a further call to the model. Similarly,choose_by_name(name, target_name)buys a card removal, or picks Smith or Toke at a rest site, and then names the card in a single tool call. •Aliases.choose_by_name("skip")or("leave")means skip, and("proceed")means proceed. A number given as a name is treated as an index. Phrases like “remove” map to the shop’s card removal. An event option can also be named by its effect text when exactly one option matches. •Calls written as text. When a response contains no tool call, the harness reads calls written in its text: function- call notation (play_card(card_name="Strike")), JSON objects in the shapes models commonly use, a bare argument object that fits exactly one expected tool, orkey: valuelines when one tool is required. It reads the reasoning only when there is no visible text, and then only the final draft of the turn. It does not read a response that was cut off. A recovered call is executed like any other and counted as an error in Table 23. Corrections and refusals. A refused call costs nothing in the game. Its result says why, and usually what would be accepted: •Did you mean. A card not in hand is refused with the cards that are, plus the closest name in hand if one is close enough (fuzzy match, similarity at least 0.6), with an example call: “Did you mean ‘Strike (1 cost)’? Example: play_card(card_name="Strike (1 cost)", enemy_index=0)”. Names that match more than one card are refused with the candidates. In preliminary runs, these suggestions stopped models from repeating the same refused call. •Wrong tool or screen. A tool that does nothing on the current screen is refused with the tools that do: “not available on SCREEN. Valid tools here: [...]”. The same applies to an action the game reports as unavailable (e.g. too little energy), to a disabled event option, and to an unaffordable shop item. • Energy check.end_turntakes, as a required argument, the energy the model expects to have left. If the model ends its turn with more energy than it declared while a card in hand is still playable, the end of turn is refused: “Rejected end_turn: declared expected_energy_remaining=퐸, but you actually have퐴energy. Your turn plan didn’t match what happened...”. The model can then play more cards or end the turn again. This catches a model that has lost track of its own turn. For example, the relic Nunchaku gives an energy for every tenth Attack played; a model that does not notice the tenth Attack coming would end its turn with energy it could have spent. These miscounts are reported as tool errors (Table 23). •Actions the game would ignore. The game silently ignores some actions, which would leave the harness waiting for a state change that never comes. These are refused before they are sent: buying or picking up a potion with every potion slot full, drinking a potion the game marks as unusable here, taking a potion under Sozu, and buying a second card removal in one shop visit. The refusal names what is possible instead. •Event fallback. If a named choice matches no option on an event, the event is shown again with both indices and labels, andchoose_by_indexis offered. Events whose options share a label (Memory Stone) are numbered from the start. • Malformed arguments. A required argument left as a. . .placeholder, a missing argument, or a text argument under the wrong name is refused with an example of the correct call. A note given under a sibling field name is accepted, and the confirmation says so. 22 •No tool call. A response with no executable call gets one retry: “Your reply contained no tool call. Text describing a call, whether JSON or play_card(...) notation, is not a call. Call one or more of these tools now: ...”. The model’s own reasoning is quoted back above this message. If the response was cut off at the length limit, the message says that nothing in it was executed. • Game errors. An error from the mod is passed on as “Action failed: error. Choose a different action.”, with the player’s remaining energy added for an unplayable card. The harness does not retry the action itself. • Repeated failure. After two batches in a row on the same screen with every call refused, the full screen state is sent again with the available tools. Batches and interruptions.The model’s calls are executed one at a time, and the state is read after each. The rest of the batch is dropped, with a result for every dropped call, when: • a call is refused (“Skipped: an earlier action in your batch was rejected... Your planned sequence is stale...” ); •cards are drawn or created mid-turn that the plan could not have foreseen (cards that fetch a named card do not count); • an enemy’s intent changes to a different move; • a Lightning orb hits a random enemy while more than one is alive; • a selection screen opens that the plan did not account for; • the fight or the floor ends. Calls afterend_turnare dropped as stale. Note writes always run. A batch with a single card play gets a reminder to send the whole turn in one response, unless that card draws cards or creates random ones, which the model cannot plan past. A card that makes the player choose another card does not excuse the batch, since the choice can be named in the same call. The exemption is decided from the card’s text: a card whose text says it draws, is random or has the player choose is exempt, which among Ironclad and colourless choice cards exempts Dual Wield and Discovery. Actions the harness takes by itself. • Ending the turn when no card is playable and no potion usable (“No more playable options. Turn ended automati- cally.” ). •Single legal choices. The harness takes an event window with one option, a single reachable map node, and a selection screen where the number of eligible cards equals the number to pick. The skipped event windows are shown to the model at the start of its next call, with what they changed. • Screens with no choice. The harness proceeds when proceeding is the only action, confirms a selection once the model has picked, opens chests (except with Cursed Key), and enters a shop it has walked into. It leaves a shop when nothing is affordable, or after 30 actions in it. • Rewards with no trade-off. Gold, stolen gold and the Emerald Key are collected automatically, as are potions when they all fit. Each card reward is opened in the background so that its cards appear in the one reward list the model chooses from. •Waiting for the game to deal the opening hand, finish a transform, or show a purchase before reporting it, so a reported effect is never one the game has not yet applied. Guards.Ten consecutive failed model calls, or ten consecutive responses with no executable game action, end the run as a harness failure, and it is resumed from its autosave. Over the main runs (190 runs and 36,486 model calls) the first fired three times, each when an API account ran out of credit, and the second 16 times, all for Grok 4.6 on five of its seeds, each after ten responses that chose an option with an empty name. Transient API errors are retried with exponential backoff (five retries by default); 48 calls (0.13%) still failed after their retries. Authentication, billing and quota errors stop the whole seed queue rather than skipping the seed, because a skipped seed would change the memory every later seed in a read-write chain starts from; this happened twice. play_comborepeats a card sequence for an infinite combo, up to 1,000 times or until a pass changes nothing; it was called 149 times, and neither limit was reached. Reflection is capped at 20 turns and consolidation at 200; the 23 longest took 13 and 47. When a conversation outgrows the model’s context window, the oldest turns of the floor are dropped (always keeping the floor’s first message and the latest turn) and a note tells the model how many were removed. This happened on 154 calls (0.4%), all from two models with 128k-token windows: Nemotron 3 Nano Omni (139 of its 708 calls, in every run) and gpt-oss-120b (15 calls, in three runs). B.6 What the model receives The excerpts below are quoted verbatim from Claude Opus 5's first run of the main cohort, so names appear as the model saw them under the themed reskin (“room” for floor, Guard for Block, Artifacts for relics). System prompt. A short preamble states the task, followed by the compendium: the game's rules and tips (42 points adapted from the game's own tutorial and loading-screen tips), its keywords and status effects, and the character's cards, the colourless, status and curse cards, and every relic and potion with its effect. On the first seed, before any memory exists, this is 30,715 characters, and with the tool definitions about 18,000 tokens. The memory bank is appended as it grows; by the tenth seed this model's system prompt had reached 227,089 characters. Tools. Seventeen tools are sent on every call (required arguments marked *): play_card(card_name*, enemy_index, target_card_names), play_combo(card_names*, enemy_index), end_turn(expected_energy_remaining*, predicted_hp_after_turn, predicted_hp_after_fight), use_potion(potion_name*, enemy_index), discard_potion(potion_name*), choose_by_index(choice_index*), choose_by_name(name*, target_name), choose_path(choice_index*), skip(), plan_route(route*, notes*), predict(act_win_probability*, run_win_probability*), save_note(key*, note*), delete_note(key*), finish_reflection(), save_memory(key*, memory*), delete_memory(key*), finish_consolidation(). A combat turn.The message that opens a turn, here turn 3 of an elite fight on floor 8. The effect of each card played on the previous turn has already been returned as the result of its own tool call (e.g. “Played Trip Attack. Energy -2 (4→2). Enemy [2] Bolt Sprite HP -8 (41→33); Ward cleared.” ). 24 Lines longer than the page are wrapped with an indent; the text is otherwise as sent. End of a floor. After the floor's last screen the model is asked to reflect, then to forecast: 25 End of the run. The consolidation message carries the run's result and final state, each act's full map, route plan and path taken, the history of every floor and the run notes, and ends: B.7 Coverage compared with Orak and Bottled AI Orak's own rule bot plays the map, events, shops, rest sites, chests, boss relics, Neow and potions, so its model decides combat and card rewards only. Even within a fight its model decides less than ours. Orak asks the model only when a card can be played, so every card choice that a card or potion opens mid-fight (Armaments, True Grit+, Dual Wield, Headbutt, Exhume, an Attack Potion) goes to the rule bot, which picks from a fixed card ranking rather than the fight at hand: the best-ranked card to upgrade or copy, the worst-ranked to exhaust, and, since the flag that would select the best is never set, the worst-ranked card for Headbutt and Exhume. The model is not told which card was chosen, and the rest of its planned turn is dropped. Orak's agent returns several actions at once, as ours does, but plays each card by its position in the hand as shown when the model decided, so a response cannot play a card that arrives mid-turn. Queued actions keep executing through draws and random effects, re-resolved by card identity so a shifted hand still plays the intended card, but the model sees what was drawn only once its queue is spent; Orak's paper describes the state as returned after each action, where its code returns it after the queue. Ours puts every such choice to the model, lets a card-targeting card name its target in the same call, and interrupts the queue when cards are drawn or created, orbs evoke at random or an enemy's intent changes, naming what changed (Park et al., 2026). Game coverage. Table 5 goes through the game by feature. It refines the Model plays and Choices columns of Table 2, and adds Bottled AI. Orak’s model sees no orbs, stance, potions or card piles, so a Defect under Orak plays its orb cards without seeing the orbs they make. Orak builds on spirecomm, a Python client for CommunicationMod whose last commit predates the Watcher, and so cannot play the Watcher at all; our harness adds the Watcher to spirecomm itself. Orak’s rule bot never chooses Recall at a rest site, so it never collects the Ruby Key and can never reach Act 4. Card coverage.Table 6 estimates how much of the game each harness lets its player use in full. For Orak, a card counts as unsupported if playing it opens a choice (either version, so Armaments counts although Armaments+ upgrades every card without asking). Orak’s rule bot makes those choices, not the model. 33 cards open a choice when played, among them Headbutt, Warcry, Dual Wield and Exhume, and every card that has the player pick a card to discard, scry or keep, or pick one of three. Bottled AI plays only cards its combat simulator models, and treats any other card as one it can never play. Its gaps are mostly cards that open a choice or have a random effect (Armaments, Headbutt, Discovery, Infernal Blade), plus a few powers (Brutality, Combust, Rupture). Our harness passes every selection a card opens to the model, so every card is supported. The count covers playing a card, not how well a player uses it. 26 Table 5: Which parts of the game each harness covers.✓: covered (for our harness and Orak, the LLM sees/decides it) rule bot: Orak’s rule-based bot decides it, without the LLM.✗: not covered. For Bottled AI, a symbolic player with no model,✓means it handles it, and a percentage is the share it handles. OursOrakBottled AI Characters Ironclad, Silent, Defect✓ Watcher✓✗✓ Combat: what is seen Hand, energy, HP, Block, relics✓ Enemy intents and powers✓ Draw, discard and exhaust piles✓✗✓ Orbs (Defect)✓✗✓ Stance (Watcher)✓✗✓ Potions✓✗✓ Combat: what is decided Cards to play, and their targets✓91% 푎 85% 푏 Choices a card opens✓rule bot42% 푏 Potions to drink✓rule bot 푐 88% 푑 Outside combat Card rewards after a fight✓ Relic, potion and other rewards✓rule bot✓ Route on the map✓rule bot✓ Neow and events✓rule bot 푒 ✓ Shops✓rule bot✓ Rest sites✓rule bot✓ Boss relics and chests✓rule bot✓ Act 4 Keys (needed to reach Act 4)✓✗w/ Heart 푓 푎 The model plays every card, but 33 of 351 target another card and that selection goes to the rule bot (Table 6). 푏 Share of the 351 cards that its combat simulator models (Table 6). 푐 Only deployed at the start of boss fights. 푑 Share of the Ironclad’s 32 drinkable potions that it can drink (it skips Smoke Bomb, Elixir, Liquid Memories and Snecko Oil). Only deployed in elite, boss, and event-triggered fights. 푒 Context-free policy: always the first or last option, depending on the event. 푓 Only with the Heart strategy (Appx. C.5). Table 6: Cards each harness supports in full, by card pool. A card counts if a player of that character could use it in full; which characters each harness can play, and which parts of the game, are in Table 5. IroncladSilentDefectWatcherColourlessAll cards (75)(75)(75)(75)(51)(351) Ours75 (100%)75 (100%)75 (100%)75 (100%)51 (100%)351 (100%) Bottled AI63 (84%)66 (88%)68 (91%)70 (93%)33 (65%)300 (85%) Orak68 (91%)66 (88%)72 (96%)67 (89%)45 (88%)318 (91%) 27 C Experimental details This appendix gives the details of the experimental setup summarized in Sec. 3. The harness itself is described in Appx. B. C.1 Models and reasoning settings Reasoning intensity is set on one scale (low,medium,high,xhigh,max) and resolved for each provider to the nearest level it supports at or above the one asked for, so a model is never run below its label. Claude Haiku 4.5 takes a thinking budget instead of a level and is run at 12,288 tokens; Nemotron and Gemma 4 E4B take no reasoning setting and run at their server’s default. Test-time compute can matter as much as model size (Snell et al., 2025), so the effort ablation (Table 26) reruns the same configuration at every other level each model was run at. A model name that the provider points at successive versions is always recorded as the version that answered. The models, how each was reached and the setting each ran at are listed in Table 7. Reasoning, temperature and response length. Reasoning is kept wherever the provider allows. Claude's signed thinking blocks are returned unchanged; OpenAI's reasoning models keep their reasoning server side through the Responses API; Gemini's thought signatures are round-tripped; and models that return their reasoning as text (DeepSeek, Z.ai GLM, gpt-oss, Nemotron) have it sent back in later turns. Qwen's reasoning is sent back within a chain of tool calls and dropped at each new user message, as its own chat template would drop it. Temperature is left at each provider's default, and a response may use up to 65,536 tokens, reduced to the model’s own output limit or a quarter of its context window where either is smaller. No response reached this ceiling: none of the 274,306 model calls in the 1,908 finished runs behind this paper was truncated. C.2 Seeds and cohort selection Every main result, ablation and difficulty level is played on the same ten seeds, minted for this paper. We replaced the seed list the harness had used during preliminary development since the opening states of a seed (the Neow choice, the first map and the first card reward) are identical in every run of it, and some providers’ terms permit training on API traffic, which is the route by which static benchmarks become contaminated (Sainz et al., 2023; Golchin & Surdeanu, 2024). New seeds close that route to memorization, though not the possibility that a model has seen the harness’s prompt format and learned about the domain. Runs are recorded in Weights & Biases, and each table cell is built from exactly one cohort, selected and then checked by the analysis code: (i) every run shares one run-name stem, which encodes the model, character, full configuration tag and seed list, so that runs from other configurations or seed lists cannot pool; (i) only runs that finished and recorded a final floor are used; (i) a seed played more than once contributes its latest run; (iv) every recorded configuration value other than the seed is identical across the cohort; and (v) for a read-write memory cohort, the runs form one unbroken chain: the first seed starts from an empty bank and each seed starts from the bank the previous one left. When a restart left a stray run newer than the chain it would be spliced into, the chain is rebuilt backwards from the last seed by matching each run’s starting memory count to the previous run’s final one; this changed one cohort. C.3 Metrics and statistics Act 1 ends at floor 16, Act 2 at floor 33, and Ascension 20’s two Act 3 bosses are at floors 50 and 51. Our main metric is mean acts cleared, which rewards reaching the next act and is not moved by how far into an act a run died. Floor is reported as mean, median and range. Per-floor figures divide by the final floor plus one, since Neow (“floor 0”) counts as a floor for this. Our reason for preferring mean acts cleared as a metric over the mean floor reached is that a model can clear more floors within an act if it takes an easier path (e.g. avoiding elites), but conservative pathing is contrary to the overall objective of winning the run. The player needs to scale their deck’s strength during the run and harder paths yield larger rewards—taking an increased risk of death now for higher odds at eventual success. The end of each act is a challenging boss fight which can not be avoided, creating an inflection point of elevated difficulty. This per-act structure is like a smaller version of the overall game, since the agent must scale well enough within the act to beat the boss that must come at the end of the act. While we do find this metric is somewhat misaligned with the overall objective of winning the game—models play the first act more cautiously than average humans (Appx. E) and top models beat it more often, but all models win runs less often than humans—the models do not win the game often enough for us to rank them on Act 4 win rate alone. 28 Table 7: The models of Table 1: developer, how each was reached, the model ID sent, and the reasoning setting it ran at in the main table (an effort level, or a thinking budget for Claude Haiku 4.5). Self-hosted models were served through an OpenAI-compatible endpoint under the deployment name given. Every main-table run was played between 19 and 25 September 2026. N/A: No reasoning control available; the model runs at its default. ModelDeveloperReached throughModel IDReasoning Qwen3.8 27BAlibabaself-hosted Qwen3_8-27Bxhigh Qwen3.6 35B-A3BAlibabaself-hosted Qwen3_6-35B-A3Bxhigh Qwen3.5 122B-A10BAlibabaself-hosted Qwen3_5-122B-A10Bxhigh Claude Fable 5.1AnthropicAnthropic API claude-fable-5-1max Claude Opus 5AnthropicAnthropic API claude-opus-5max Claude Sonnet 5AnthropicAnthropic API claude-sonnet-5max Claude Haiku 4.5AnthropicAnthropic API claude-haiku-4-5-2025100112,288 tokens DeepSeek V4.1 FlashDeepSeekDeepSeek API deepseek-flashmax Gemini 3.8 FlashGoogleGemini API gemini-3.8-flashhigh Gemma 4 E4BGoogleself-hosted gemma4:e4b-highcontextN/A Muse Spark 1.3MetaMeta Model API muse-spark-1.3xhigh Nemotron 3 Nano OmniNVIDIAself-hosted Nemotron-3-Nano-Omni-30B-A3BN/A GPT-6 AstraOpenAIResponses API gpt-6-astramax GPT-5.6 SolOpenAIResponses API gpt-5.6-solmax GPT-5.6 TerraOpenAIResponses API gpt-5.6-terramax GPT-5.6 LunaOpenAIResponses API gpt-5.6-lunamax gpt-oss-120bOpenAIself-hosted gpt-oss-120bhigh Grok 4.6xAIxAI API grok-4.6xhigh GLM-5.3Z.aiZ.ai API glm-5.3max GLM-5.3 FlashZ.aiZ.ai API glm-5.3-flashmax Thinking tokens are the reasoning tokens a provider reports; where a self-hosted server does not separate them, the count is apportioned by the reasoning text’s share of the response’s characters. Paired comparisons across models, such as an ablation against its baseline, use exact sign-flip permutation tests (Fisher, 1935); correlations across models are tested by permutation (Spearman unless stated otherwise), and within-run contrasts by exact sign tests. Several analyses group the 20 models of Table 1 into three tiers by acts cleared: the top four (Claude Fable 5.1, GPT-6 Astra, Claude Opus 5 and GPT-5.6 Sol), the next eight and the bottom eight. Gemma 4 E4B has no full logs, only its Weights & Biases record, which keeps 200 characters of each call, so analyses read from the logs cover 19 models and the bottom tier there has seven. C.4 Cost Hosted models are priced at each provider’s list price for input, cached input, cache writes and output, with reasoning billed as output. Two conventions apply. OpenAI reports cache reads but not cache writes, so the written share is inferred from the growth of the cached prefix between calls, and can only be underestimated. DeepSeek’s prices vary with the time of day; we charge the peak rate, so its figures are upper bounds. Z.ai’s GLM models were run on a flat-rate subscription and are priced at the list price of the tokens they used. Self-hosted models have no bill, so we price their recorded tokens at the headline prices published by Artificial Analysis (retrieved 22 September 2026). These are the model’s first-party API price where one exists, otherwise the median across providers. Input, cached input and output are each charged at their listed rate; where no cache-hit price is listed, cached input is charged in full. The per-million-token prices are: gpt-oss-120b $0.15 / $0.125 / $0.59; Qwen3.8 27B $0.50 / $0.05 / $3.00; Qwen3.6 35B-A3B $0.38 / $0.38 / $2.25; Qwen3.5 122B-A10B $0.40 / $0.40 / $3.20; Nemotron 3 Nano Omni $0.20 / $0.195 / $1.09; Gemma 4 E4B $0.02 / $0.02 / $0.10 (input / cache hit / output). Each reproduces the blended price that Artificial Analysis publishes, except Gemma 4 E4B's, for which our snapshot lists no blended price. C.5 Non-LLM baselines Two policies bracket the models from below. Pick first option always takes the leftmost valid option, including the leftmost enemy as its target, and is deterministic. Pick random option draws uniformly among the valid options, from a random stream seeded by the game seed. Both drink every usable potion at the first opportunity, since 29 hoarding potions is a known failure of the models. In shops, the first policy buys only relics (the leftmost it can afford), so the stock of slot zero does not decide the run, while the random policy buys random affordable items until it can afford none. Above them, BottledAI (Bottled AI contributors, 2022) is a hand-written symbolic Ironclad player, run with and without its Heart strategy. All three play the same seeds, ascension and game build as the models, but see no names at all, so the reskin does not apply to them. Bottled AI with the Heart. The latest released Bottled AI implementation stops at Act 3. It does not collect the three keys that unlock Act 4, so it can never meet the final boss (the Corrupted Heart of the Spire). The Heart variant comes from an open, unmerged pull request to Bottled AI, Slay the heart, by the GitHub user Typhi (Typhi, 2024). It adds a slay_heart switch to any strategy. With the switch on, the bot: • takes the Emerald Key from a burning elite’s rewards; • takes the Sapphire Key instead of the relic from the Act 3 chest on floor 43; • chooses Recall at a rest site to take the Ruby Key. It does this whenever resting, removing a curse or a high-priority upgrade does not come first, and always at the last rest site before the Act 3 boss; • adds a reward to map paths through the burning elite. The same pull request also adds combat knowledge that applies with the switch on or off: Act 4’s enemy powers (the Heart’s Beat of Death and Invincible), a preference for killing the Spire Spear before the Spire Shield, and a guard against playing a pointless infinite loop of cards. We run Bottled AI’s Ironclad strategy, Requested Strike, with the switch off (w/o Heart) and on (w/ Heart). The code is the pull request’s final commit, with the commits added to Bottled AI’s main branch since then applied on top, plus three fixes of our own: •two handlers added on main after the pull request still used the handler signature the pull request had changed, and failed when called; •the Match and Keep event handler always chose the first card. On the second flip of a pair, that card is already face up, so the game ignored the choice and the bot waited forever; • the Match and Keep event’s name was misspelled (it lacks the final “!”), so the fix above was never reached. Since neither variant reaches Act 3 on the ten main seeds at A20 difficulty (Table 1), in practice they play identical games on all ten. As shown in Fig. 12, there is a difference at lower ascension levels, with the Heart-seeking variant performing worse as it has to forego strictly positive boons (a relic and a card upgrade) to pick up two of the three keys, leaving it in a worse state to face the Act 3 boss. C.6 Ablations and analyses Each ablation changes a single factor from the main table’s configuration and keeps each model at its main-table reasoning setting, unless stated otherwise. Gemini 3.8 Flash is also run at low effort, where it lies on the cost Pareto front (Fig. 2a), and is pooled as a separate entry. Harness. Table 2 compares our default harness with four alternatives. Single action allows one action per response. Single action, by index does the same and also makes every choice by its number: the tool that chooses an option by name is withdrawn, and every choice screen lists its options as numbered entries, so the two single- action arms differ only in how a choice is named. How a problem is represented can bound what a reasoning system can do with it (Yihe et al., 2019), and this arm isolates that factor in the interface. Continuous keeps one conversation for the whole run instead of one per floor. Orak hands the whole run to the reflection-and-planning agent of the Orak benchmark (Park et al., 2026), run from its pinned upstream release (commitc14e5a0) with its own prompts and a budget of 200 decisions, and passed the reasoning level each model runs at in our harness, so that the two harnesses are compared at the same setting. Orak’s own rule bot plays the map, events, shops, rest sites, chests, boss relics, Neow and potions, so the comparison covers combat and card rewards only; what its model decides within a fight is set out in Appx. B.7. Orak predates the current effort-based reasoning models, so we made minor changes to its provider code, none of which touch its prompts, agent or decisions. It sends each model the reasoning level, in the form that model’s API takes (for OpenAI models, through the Responses API, whose scale reaches the highest levels). It lets each model use the context window and response ceiling our harness gives it, where Orak’s defaults would cut a reasoning model’s reply short or truncate its prompt. It 30 reaches Gemini with an API key rather than Orak’s authors’ own cloud project. And it plays through Orak’s own build of CommunicationMod rather than ours, whose choice lists differ on some event screens. These variants were run without the reskin or a memory bank, and so the default arm uses that configuration too. Because each harness was run on a different set of models, the table pools over models and reports each harness’s difference from the default, which is what the comparison is about, on matched pairs of runs (the same model on the same seed). Besides performance it compares efficiency: model calls and tokens per floor. Those are compared on like-for-like floors only, meaning the same model, seed, act, room type and occurrence of that room type within the act (the second elite of Act 1, say), and only where every harness run on that model and seed completed the floor. A harness that dies early would otherwise be credited with the cheap early floors, and one that goes deeper charged for the expensive late ones. Orak keeps no per-floor record, so its usage can only be compared per floor reached over whole runs. Appx. H.4 reports the runs each harness ended itself, through a crash or a loop, since a run stopped by the harness rather than by a death counts against the model in every per-run figure. The memory ablation (Table 27) is reported the same way, against the arm without memory for the same kind of notes. Per-model results are in Appx. I. Prompt material.Tables 31 and 32 add one kind of material to the cached system prompt, on top of the themed reskin. Bestiary extends the compendium with the normal enemies, elites and bosses of each act: their HP, intents and starting powers. The Reddit guide is “Slay the Spire: High Ascension Guide”, posted to r/slaythespire by the user screwyioux in January 2020 (about 7,000 words), included verbatim apart from putting game names in their canonical spelling so that the reskin renames them. Jorbs is a set of transcripts of videos by the streamer Jorbs: his seven-part tutorial series from 2018 (basic resources, health, damage, block, cards, energy and floors; 67 minutes) and “Talking about Ironclad for 6 hours” (December 2025), 7 h 10 min and about 393,000 characters in all. The guide and the transcripts are renamed by the reskin like the rest of the prompt, the streamer’s name included. Reskin. Table 29 compares each reskin with the real game names on the same model and seed, pooled over models, in the same way as the harness and memory comparisons above. The permuted and nonsense mappings separate the two things the themed reskin does at once, hiding each real name and supplying a coherent replacement for it (Sec. 3). Per-model results are in Table 30. Difficulty. The difficulty sweep is described with its results in Appx. F. Cost and reasoning frontiers.Figures 2a, 18 and 21 plot mean acts cleared against mean cost per run, cost per floor and thinking tokens per floor, with one point for each model at each reasoning level it was run at, since an agent's accuracy means little without its cost (Kapoor et al., 2025). A point lies on the Pareto front when no other point clears at least as many acts for no more spend; the front is computed over cohorts with all ten seeds. Public benchmarks. Figure 2b and Table 18 compare mean acts cleared with the scores Artificial Analysis publishes for the same models, taken from a snapshot of its leaderboard (the retrieval date is in the table caption), for the LLM agents of Table 1 with all ten seeds at their highest reasoning effort, and again at low effort where that cohort has all ten seeds and Artificial Analysis lists the setting (it does not for Qwen3.5 122B-A10B, Qwen3.6 35B-A3B or either GLM-5.3 model). Each point is matched to the Artificial Analysis entry for the reasoning setting it ran at here; two matches are approximate, since Artificial Analysis lists Claude Haiku 4.5 without its thinking budget and GLM-5.3 Flash under a single setting. We use the evaluations reported for nearly every one of these models, among them Humanity's Last Exam (Center for AI Safety et al., 2026), GPQA Diamond (Rein et al., 2024) and SciCode (Tian et al., 2024). A correlation with the public suite says little on its own, because the suite's evaluations are correlated with each other (Burnell et al., 2023; Ruan et al., 2024); so we also add SpireBench to the suite as one more row and ask, for every row, how well the rest stand in for it: its strongest correlation with any other single evaluation, and how much of it the first principal component of the others predicts for a model left out of the fit. The Intelligence Index is a composite of evaluations like these and is left out of that part. The two rungs of one model are not independent points, so a model is held out with both its rungs, and we also report the correlations within each rung. Learning from memory. Described with its results in Appx. E.4. Memory growth.Every model call logs the length of its system prompt, and within a cohort the memory bank is the only part of the system prompt that changes between runs. The first call of run푘+1 therefore measures 31 the bank left by run푘, relative to run 1’s empty bank (Fig. 19). This is measured in characters so that sizes are comparable across tokenizers; the bank left after the last run is never read, so the series ends at run 9. Tool-call errors.The harness counts four kinds of tool-use failure, reported per 100 model calls in Table 23 so that models making more calls are not charged for it: a response in which no call could be executed, because every call was malformed or refused or none could be parsed; a single call refused before it reached the game, as illegal on the current screen or with invalid arguments; a turn ended with a different amount of energy left than the model declared, which marks a model that has lost track of its own turn; and a call written as prose rather than through the tool interface, which the harness recovers. The model is told about the first three and the harness asks again or carries on, so none of them ends a run; they measure how precisely a model acts, separately from how well it plays, as function-calling benchmarks measure tool use on its own (Patil et al., 2025). The harness comparison (Table 2) reports the same rate. Calibration. How the forecasts are scored is described with the results in Appx. G. D Human baselines We compare the models with three kinds of human player, described in turn below: four expert streamers (Table 8), the whole population of players at Ascension 20, from the telemetry the game uploads (Tables 9 and 10), and one of the authors, a non-professional player, over all 50 of their A20 Ironclad runs (2020–2024) and over the first ten. The four Twitch streamers, Baalorlord, panacea108, vmService and cheg0, have per-run files with a record of every floor; these are compared floor by floor with the models in Appx. E. Human baselines. We compare against published A20 Heart Ironclad results from four top players, with the same act-clearing definitions as our harness. They differ in how the data were collected, and this determines what each figure means. Only Baalorlord’s is an automatically logged long-run average: the Spireblight client uploads every run file from his game directory, abandoned runs included, and we use all 687 Ironclad A20 runs on his main profiles from 2020 to 2026 (https://baalorlord.tv/runs) 1 . Three further samples of per-run files (panacea108, vmService, cheg0) come from a public archive (https://github.com/MaT1g3R/Slay-the-Spire-data). They are contiguous blocks of one to three months, and the two rotating samples cycle through the characters without a break, consistent with no runs missing within them. Population baseline. For the general player population we use Mega Crit’s own run telemetry, which the game uploads automatically at the end of every run; the developers published 7.05M runs from October to mid- November 2020, all on the final content build (2020-07-30), athttps://drive.google.com/drive/folders/ 1c7MwTdLxnPgvmPbBEfNWa45YAUU53H0l. Unlike self-uploaded run archives it is not selected by the player, so it reflects everyone playing Ironclad at Ascension 20, a level that is itself only unlocked by winning at Ascension 19 with the Ironclad. We keep Ironclad A20 runs (290,280), drop daily, endless, seeded, trial and modded runs and any on another build or the beta branch (249,362 left), and remove runs abandoned before a death or victory, nearly all quit at Neow, leaving 206,082. Act clears use the harness’s definitions, with floors checked against where each boss’s victims died. The record has no player identifier, but each run carries its player’s cumulative Slay the Spire play time (the in-game statistic, not a cross-game or Steam figure), so we also stratify by hours played. By requiring each run’s cumulative play time to equal the previous run’s plus that run’s length, with the run starting within an hour of the previous one ending, we reconstruct chains of consecutive runs (62% of runs link, against 1.0% with a deliberately wrong offset). Chains break at longer pauses, so they are play sessions rather than players, but they let us count only runs made after an A20 Heart kill earlier in the same chain, avoiding the bias of selecting on the outcome being measured. The population clears Act 1 in 47% of runs and kills the Heart in 3.5%. Experience and performance.Each telemetry record carries the player’s cumulative Slay the Spire play time when the run ended, so we can stratify the 206,082 A20 Ironclad runs of Table 9 by hours played (Table 10). Performance rises with experience, but slowly: runs by players with 50–100 h clear 0.67 acts on average and kill the Heart in 2.4% of runs, against 0.97 acts and 6.8% at 1500–2000 h, the best bin. Play time is a lower bound on experience: it lives in the save profile and resets with a new one, so the lowest bin (329 runs under 25 h at A20, 1 This includes Baalorlord’s “mastery” runs, which are abnormal as he tries to win a run specifically with two copies of a card, but we don’t find a significant difference in his overall performance when including these. 32 Table 8: How each human baseline was collected. Complete block: every run over the period, logged automatically. Player- selected: a contiguous block shared by or for the player, chosen by unstated means. Record set: a player-kept set known because it set a record. Summary: a figure copied from the player’s own summary, with no per-run data. PlayerSourceRunsPeriodCoverageNotes Baalorlordsite log of .run files687 2020-04 to 2026-02 complete block abandons kept; older modded profile excluded panacea108 archive, .run files51 2022-09 to 2022-11 player-selected Ironclad-only sample vmServicearchive, .run files50 2022-09 to 2022-12 player-selected Ironclad runs of a 200-run rotation cheg0archive, .run files462024-02player-selected Ironclad runs of a 184-run rotation Table 9: Population baseline for A20 Heart Ironclad, from Mega Crit’s run telemetry (build2020-07-30, 2020-09-30 to 2020-11- 12). Hours is the median cumulative Slay the Spire play time of the player at the run. Acts is the mean number of acts cleared per run. FloorAct cleared (%) PopulationRunsHoursmeanmedian1234Acts All A20 Ironclad runs206,08230022.11647.215.36.93.50.73 All A20 Ironclad runs, incl. abandoned249,36229818.61639.712.85.72.90.61 StS hours < 100 h13,6157721.11643.914.97.02.40.68 StS hours 100–300 h89,47620121.61645.614.46.32.80.69 StS hours 300–1000 h94,70446622.51648.615.87.23.90.76 StS hours≥ 1000 h8,287125324.72153.520.410.97.10.92 Session chains with an A20 Heart kill (biased)16,74937136.33568.850.045.142.82.07 Runs after an A20 Heart kill earlier in the chain3,64143327.92358.029.220.915.81.24 which takes far longer than that to unlock) likely holds experienced players on a fresh profile. It also measures time, not skill. Bins with fewer than 200 runs were merged: 2000–3000 h with 3000 h and above. Figure 7 plots Table 10 against hours played. Between 25 and 1,000 hours performance barely moves (0.67–0.80 acts per run, 2.4–4.4% of runs killing the Heart); it rises only past 1,000 hours (0.92–0.97 acts, 6.8–7.6%). The lowest bin is high because play time resets with a new save profile, so it likely holds experienced players on fresh profiles. 33 Table 10: A20 Heart Ironclad by hours of Slay the Spire played, from Mega Crit’s run telemetry (build2020-07-30). Hours is the player’s cumulative play time when the run ended (bin and bin median). Acts is the mean number of acts cleared per run. FloorAct cleared (%) HoursRunsMedian hmeanmedian1234Acts 0–253291524.91951.122.815.54.00.93 25–501,5444221.21642.916.79.12.90.72 50–10011,7428121.01643.814.56.42.40.67 100–15018,81212821.41644.614.36.32.50.68 150–20025,43317621.31645.214.26.02.70.68 200–30045,23124521.81646.214.66.43.00.70 300–40031,90634622.11646.915.36.93.70.73 400–50021,94544722.41648.115.97.13.70.75 500–75030,07759822.81850.115.97.54.20.78 750–100010,77683923.11850.816.47.84.40.80 1000–15005,962116224.72153.120.311.37.60.92 1500–20001,433171525.62257.221.311.46.80.97 ≥2000892215323.81850.719.17.44.00.81 205010020050010002000 Hours of the game played (median of bin) 0.70 0.75 0.80 0.85 0.90 0.95 Mean acts cleared per run 329 1544 1433 892 205010020050010002000 Hours of the game played (median of bin) 3 4 5 6 7 Heart kills (% of runs) 329 1544 1433 892 Figure 7: Human performance at A20 with the Ironclad against hours of the game played, from the telemetry in Table 10: mean acts cleared per run (left) and the share of runs that kill the Heart (right). Each point is a bin of Table 10 at its median hours; bins with fewer than 2,000 runs are labelled with their count. Play time resets with a new save profile, so the lowest bin likely holds experienced players on fresh profiles. E Strategy and play quality This appendix asks two questions of the runs of Table 1: why the models fall behind human experts, and why the strong models beat the weak ones. Tiers, the models read from the logs and the tests are as in Appx. C.3. E.1 Where agents die: agents against humans We compare the decks and in-game choices of our agents with those of human players at the floors where the agents die. The agents are the 200 main-table runs (20 LLMs, A20 Ironclad on the held-out seeds, themed reskin, xml notes and memory, generic guidance), with per-floor state from each run’s logged history. The humans are the 206,082 finished A20 Ironclad runs of the game’s final content build (2020-07-30) in Mega Crit’s released run telemetry for October and November 2020 (https://github.com/rrenaud/slay_analysis), with beta, daily, seeded, trial, modded, anomalous and abandoned runs removed. Of the agent runs that die, 27% die at the Act 1 boss (floor 16), 11% on floor 7, 7% on floor 22 and 6% at the Act 2 boss (floor 33). On reaching these floors the agents die more often than humans: 41% against 32% at the Act 1 boss, 12% against 4% on floor 7 and 20% against 6% on floor 22 (Table 12). The agents reach the Act 1 boss healthier than humans but built differently (Table 11). They arrive with 82% 34 Table 11: State and Act 1 choices on entering the Act 1 boss (floor 16), for the runs that reached it, split by whether the run then died at the boss. Campfire, fight and potion rates are pooled over campfires, fights and potions on floors 1–15 (potions 0–15); HP, elites, deck size and potions used at the boss are means over runs. AgentsHumans All Died PassedAllDied Passed HP (% of max)827586625665 Campfires spent resting (%)687563232622 excluding Recall (%)728167232722 Elites fought1.251.211.281.961.852.01 Fights with no card taken (%)8126232621 Deck size19.318.419.916.816.517.0 Potions used / obtained0.500.550.460.270.400.21 Potions used at the boss1.081.191.000.560.820.44 Gold (median)10211280152160149 Runs1275275 143,146 45,86597,281 Table 12: Where runs end: the share of all deaths that fall on a floor, and the death rate among the runs that reached it. Agents: 196 deaths in 200 runs (the rest cleared Act 3, one of them going on to kill the Heart). Humans: 195,533 deaths in 206,082 runs. Share of deaths (%)Death rate on reaching (%) FloorAgentsHumansAgentsHumans 7114124 16 (Act 1 boss)27234132 2273206 33 (Act 2 boss)683734 of their maximum HP against 62%, having spent 68% of their campfires resting where humans rest at 23% and upgrade a card at the rest. They fight fewer elites (1.25 against 1.96), and they take a card after almost every fight (none after 8% of fights, against 23%), so their decks are larger (19.3 cards against 16.8). They also drink more of their potions before the boss (0.50 of those obtained, against 0.27); among humans, those who then die at the boss had drunk 0.40 of theirs and those who pass it 0.21. At death, against humans who died on the same floor, the agents’ decks hold 1.4 fewer upgraded cards and 2.5 more cards, and they carry 1.9 fewer relics (1.8 fewer upgrades and 2.2 fewer relics at the Act 1 boss). These are descriptive differences, not matched comparisons: the agents play a reskinned game on at most ten held-out seeds, while the humans play the real game on many different seeds, and every human at A20 has, by the game’s unlock rule, won at A19 with the Ironclad. Four measurements are approximate. Human deck size on entering floor 16 is reconstructed backwards from the final deck, since the record lists only the final deck and does not itemize Neow’s card effects; it agrees with an independent forward count for 79% of runs. Agent potion use countsuse_potioncalls, and the potions obtained are inferred from the potion count, which misses discards, so the agent rate is an upper bound. The agent skip rate counts fights after which no card entered the deck. Some human records carry implausible gold values (0.25% reach 5,000 or more), so gold is reported as a median. Health as a resource. The telemetry above shows the agents reaching the Act 1 boss with more HP than humans because they rest where humans upgrade. Figure 3 follows HP over the whole run against the four Twitch streamers of Appx. D (833 runs with per-floor records). The streamers spend HP through Act 1: after the floor-15 rest site, the last before the boss, their median is 53% of maximum HP, against 90% for the four models that clear the most acts. At a rest site the models rest unless they are nearly full, 76–98% of the time when they arrive below 80% of maximum HP, where the streamers rest 6–48% of the time and upgrade a card otherwise; over all visits the streamers rest at 15% (6,398 visits), the top four models at 66% (238) and the other models at 75% (364). The strongest models manage HP no differently from the weakest: none treats it as a resource to trade for a stronger deck. The streamers play their own seeds, so this is a reference rather than a matched comparison. Routes, relics and deck.Figures 8 and 9 compare the run-level choices of the LLM agents of Table 1, in three groups by acts cleared, with the telemetry population and the four Twitch streamers. In Act 1 the models fight fewer elites (1.4 for the top four models, 1.3 and 0.9 for the next eight and the bottom eight, against 2.0 for the 35 Act 1Act 2Act 3 0.0 0.5 1.0 1.5 2.0 2.5 Elites fought All human players (telemetry)Twitch streamersTop 4 LLMsNext 8 LLMsBottom 8 LLMs Act 1Act 2Act 3 0.0 0.5 1.0 1.5 Shops entered Act 1Act 2Act 3 0 1 2 3 4 Unknown (?) rooms entered Act 1Act 2Act 3 0 1 2 3 4 5 Hallway fights chosen Act 1Act 2Act 3 0 20 40 60 80 Rest sites: rested (%) Act 1Act 2Act 3 0 10 20 30 40 50 Fights with no card taken (%) Figure 8: What runs did on each act's 15 map floors, over the runs that reached that act's boss (floors 16, 33 and 50): the telemetry population (143,146, 47,619 and 22,729 runs), the four Twitch streamers (788, 671 and 603) and the LLM agents of Table 1 in three groups by mean acts cleared: the top 4 (36, 23 and 10 runs), the next 8 (65 and 7) and the bottom 8 (26). A group is drawn for an act only when at least five of its runs reached that boss. Shops and unknown rooms count the map nodes entered; elites fought and hallway fights the rooms entered. Rest-site and card-reward shares pool the visits and fights of all runs in the group. population and 2.6 for the streamers; 56–77% of model runs reaching the boss fought at most one, against 29% and 4%), enter more shops (1.3–1.6 against 1.0 and 0.8), rest at 57–75% of rest sites against 23% and 9%, and take a card after almost every fight (3–13% of fights with no card taken, against 23% and 15%). They therefore fall behind in relics from the first elites and never catch up (9 against the streamers’ 12 at floor 32), carry fewer upgraded cards, and hold a larger deck. These choices do not separate the stronger models from the weaker ones (Fig. 10). Across the 16 models with at least three runs reaching the Act 1 boss, Spearman's휌with acts cleared is+0.37 for elites fought (푝=0.16),−0.25 for shops,−0.20 for resting and−0.21 for skipped card rewards (permutation tests), whereas the net HP change over a hallway fight has휌=+0.62 (푝=0.01), from−2.4 HP for the top four to−6.5 for the bottom eight (−0.2 for both human groups). A logistic model of beating the Act 1 boss from the state on reaching it (HP, maximum HP, relics, deck size and upgrades), fitted on the population, rates the top four's positions at 79% and the next eight's at 70%; they actually won 94% and 55% of those fights. The streamers arrive in positions the model rates at 69%, having spent HP for relics and upgrades, and win 94%. The stronger models are stronger mainly in the fights, not in the route they take to them. The state model explains little of the outcome on its own (McFadden푅 2 0.05), and the human deck, relic and upgrade counts are reconstructed backward from each run's final state, since the telemetry records only that. E.2 Conservative Act 1 play among humans In Mega Crit's A20 Ironclad telemetry we took the human runs that beat the Act 1 boss and compared those whose Act 1 looks like the models' (at most one elite, resting at half or more of the rest sites, at least 70% HP at the boss) with those that fought two or more elites and rested at fewer than half (Table 13). The model-like profile covers 3.2% of human runs and 36% of the models'. It clears Act 2 in 15.5% of cases against 38.0%, and Act 3 in 36 0 5 10 15 20 Relics Act 1Act 2Act 3Act 4 All human players (telemetry) Twitch streamers Top 4 LLMs Other 16 LLMs 15 20 25 30 35 Cards in deck 117345256 Floor 0 5 10 15 Upgraded cards Figure 9: Relics, cards in the deck and upgraded cards at the end of each floor survived: median (lines) and interquartile range (bands), for the telemetry population (dashed), the four Twitch streamers, the top 4 LLM agents of Table 1 by acts cleared and the other 16. A floor is drawn only while at least 20 human or 8 LLM runs are alive on it. Human counts are reconstructed backward from each run's final deck and relics, removing every change the record itemizes after the floor. 3.4% against 18.6%, and the gap holds in every band of hours played (14% against 45% for Act 2 among players with over 1000 hours). Clearing Act 2 rises steadily with Act 1 elites (21, 25, 32, 40 and 50% for 0 to 4 or more) and falls with rests (37, 29, 24 and 21% for 0 to 3 or more). The telemetry has no player identifier, so the strongest design compares runs within one play session: there each Act 1 elite multiplies the odds of clearing Act 2 by 1.38 (1.34–1.43) and each rest by 0.85 (0.81–0.89), over 8,745 sessions, and model-like runs clear Act 2 7.3 points less often than the same session's other runs (permutation푝=10 −4 ). Within each of the four streamers the odds ratios are 1.68 per elite and 0.64 per rest. Resting at a rest site entered with at least 70% HP, which the models do at 31% of such sites, the population at 2% and the streamers never, predicts worse outcomes across the population but not within a session, so its association is likely selection on weaker players. How much of the gap does the route explain? Shifting the streamers' odds by the effect of the models' Act 1 profile accounts for 17% of their Act 2 advantage and 19% of their Act 3 advantage under the within-session model, and 34–61% under cross-sectional models, which also absorb the effects of deck and relics and so overstate it. At their own profile the models are not worse than humans: humans with that profile clear Act 2 in about 19% of cases, the models in 25% (the top four in 47%), while the streamers clear 85%, 45 points above what their profile predicts. These are associations: weak runs skip elites and rest partly because they are weak, and the telemetry does not record which elites the map offered. 37 0.51.01.5 0.5 1.0 1.5 2.0 2.5 Elites fought ρ = +0.37, p = 0.16 0.51.01.5 0.8 1.0 1.2 1.4 1.6 1.8 2.0 Shops entered ρ = -0.25, p = 0.36 0.51.01.5 20 40 60 80 Rest sites: rested (%) ρ = -0.20, p = 0.45 0.51.01.5 Mean acts cleared 0 10 20 30 40 50 60 Fights with no card taken (%) ρ = -0.21, p = 0.43 0.51.01.5 Mean acts cleared −8 −6 −4 −2 0 HP change per hallway fight ρ = +0.62, p = 0.01 406080100 Expected from state (%) 0 20 40 60 80 100 Observed (%) Act 1 boss beaten (%): observed against expected from state Top 4 LLMsNext 8 LLMsBottom 8 LLMsAll human players (telemetry)Twitch streamers Figure 10: Act 1 choices and combat against mean acts cleared, one point per LLM agent of Table 1 with at least three runs reaching the Act 1 boss (16 models), over those runs; lines are the telemetry population (dashed) and the four Twitch streamers. 휌is Spearman's correlation across the models and푝a two-sided permutation test. HP change per hallway fight is the net HP change over a hallway-fight floor, including the relic heal after it, computed the same way for humans. Last panel: each model's rate of beating the Act 1 boss against the rate expected from its runs' states on reaching it, by a logistic model fitted on the 143,071 telemetry runs that reached it with a complete state record; the dotted line is equality. E.3 Risk in the models' own runs Elites are where the models die before the Act 1 boss: 37 of the 65 such deaths are in elite rooms, a death rate of 17.8% per elite fight against 2.7% per ordinary fight, higher for 15 of 18 models (sign test푝=0.008). The deaths are concentrated in the weak models (Qwen3.5 122B 6 of 10 elite fights, gpt-oss-120b 5 of 7), while the top four die in 0–9% of theirs, so for a model that fights badly, avoiding elites is partly the right call. Among runs that beat the Act 1 boss (49 runs, 7 models), upgrades on reaching the boss predict clearing Act 2 (푟=0.56, Holm-adjusted 푝=0.006) and rests predict against it (푟=−0.44), while the number of elites fought does not (푟=−0.12). Part of the upgrade effect is that a run with HP to spare can afford to upgrade: controlling for HP on arrival weakens it to 푟=0.27 (푝=0.11). HP on reaching the boss itself predicts clearing Act 2 (푟=0.32), which says that a deck able to clear Act 1 unhurt is strong, not that HP should be hoarded. In 72 of the 75 Act 1 clears the transition heal was exactly 75% of missing HP. E.4 Learning over the chain Figure 11 compares each memory cohort’s mean acts cleared over seeds 1–5 with seeds 6–10. The two halves are different games, so the same model’s no-memory cohort on the same seeds is shown alongside as a control for the halves’ difficulty: learning appears as a larger improvement than the control’s. Comparing runs 1–5 with runs 6–10 of each memory chain, with no-memory chains on the same seeds as the control for how hard each half is (8 models), memory changes nothing measurable: its added effect is−0.07 on the share of rest sites spent resting (푝=0.29),−0.32 Act 1 elites (푝=0.38, 5 models reaching the boss in both halves) and null on upgrades, HP at the boss, floor and acts (푝 ≥0.29). Power is low, but the direction is towards 38 Table 13: Act 1 play and what follows, over runs that beat the Act 1 boss. Humans are Mega Crit's A20 Ironclad telemetry and the four Twitch streamers of Table 1; agents are the LLM runs of Table 1. Elites, rest share (rest-site visits spent resting rather than upgrading), HP (% of maximum), relics and upgrades are on entering the Act 1 boss. The lower block splits the telemetry by profile: model-like is at most one elite, a rest share of at least 0.5 and at least 70% HP at the boss; aggressive is two or more elites and a rest share below 0.5. Human runs pass a plausibility filter first, so their counts differ slightly from Table 11. RunsElitesRest shareHPRelicsUpgradesClear Act 2 (%)Clear Act 3 (%) Telemetry population97,2202.010.22645.73.33215 Streamers7412.610.09565.73.48577 LLM agents, all751.280.63863.92.0255 LLM agents, top four34–4712 Telemetry, model-like3,08015.53.4 Telemetry, aggressive54,16338.018.6 more caution, not less. The final memory banks lean the same way: 119 of 1,374 memories advise caution and 17 advise risk (keyword-coded, 20 of 25 sampled caution codes confirmed by hand), and 18 models hold more cautious than risky memories, none the reverse. The main exception is Claude Opus 5 (“DODGING FIGHTS IS AN ERROR, BECAUSE UNSPENT SLACK EXPIRES”). Only the four strongest models state the act heal correctly; Gemini 3.8 Flash writes that it “restores player HP up to 80% of Max HP”, GLM-5.3 that it is “likely flat ~50”, and ten models never record it. Knowing the rule does not settle the decision: 32 of 209 late-act rest-site decisions (floors 13–15, 30–32 and 47–49) reason about the heal, mostly by Claude Fable 5.1 and Opus 5, and still rest in 21 (66%, against 77% for the others). Opus's own bank records the conflict: “PRE-BOSS CAMP . . . the Sleep LOSES ON THE LEDGER . . . AND WINS ANYWAY”. E.5 Deck building and boss preparation We scored 1,621 card rewards against a choice model fitted on about two million reward screens from the telemetry, as the probability that an average human makes the same pick on the same screen. Agreement barely rises with strength (0.324, 0.313 and 0.293 by tier;휌=0.16,푝=0.52); on the 92 identical screens seen by both a top and a bottom model the top model agrees more by 0.036 (푝=0.03). The models skip 10–15% of rewards where the human model expects 24–31% on the same screens, unrelated to strength. The one clear signal is area damage: on Act 1 screens offering such a card, the top four take one 46% of the time, the next eight 30% and the bottom eight 20%, against about 34% for humans on the same screens (휌=0.76,푝=0.0002; 29 points more on 28 identical screens,푝=0.002), and their reasoning cites multi-enemy fights rather than the boss. Scaling, block and draw cards show nothing. Although 92–98% of runs in every tier name the upcoming boss before reaching it, no tier adapts its deck to it: top decks facing the Slime Boss, which splits into several enemies, hold 1.94 area-damage cards against 1.50 for other bosses (푝=0.37), and the models' picks follow none of the shifts humans make by boss. Given the state in which each run reaches the Act 1 boss (HP, maximum HP, relics, deck size and upgrades), 13% of the 75-point top–bottom gap in boss wins is explained by a model fitted on humans and 24–29% by one fitted on the models; the residual, 52–57 points (푝 ≤ 0.01), is in the fight or in what the state omits. E.6 Combat From the full logs we measured each combat turn against the state the model was shown (Table 14). The measures were validated first: a card's displayed damage matches the HP and block it removed on 99.4% of 3,030 checked plays, and the shown intents less block match the damage taken on 92% of turns; intent-based measures use only the consistent turns. Measures that do not separate the tiers are as informative as those that do: unblocked damage the hand could have prevented is about 0.1 HP per turn in every tier, unspent energy 5–6%, energy spent 2.6–2.7 per turn and the share of incoming damage blocked 48–52%. The strong models do not defend better; they attack better. Paired on the same hallway encounter of the same seed (73 encounters), the top four take 6.7 less damage than the bottom tier, in 0.99 fewer turns, dealing 5.3 more damage per turn (sign-flip푝<0.001 each); on floors 1–6, where decks are still close to the starter deck, the gaps are 2.6 HP (푝=0.002) and 0.82 turns. About half of the difference in damage taken comes from the extra turns and half from damage per turn, mostly from attackers left alive. The next-turn HP forecast adds nothing in Act 1 beyond these measures (partial휌=−0.05 after controlling for hallway damage, blocking into non-attacks and missed lethal). Two examples, with the names the model saw: on seed 6, floor 8, facing a Zealot [Cultist] whose intent is a buff, Qwen3.8 27B plays three block cards (“The 20G fully covers the T2 hit . . . at zero HP cost”) and deals 4 damage, while GPT-6 Astra on the same turn deals 30; and Qwen3.5 122B, with a 9-HP Corrosive Ooze [Acid Slime] and 6 + 4 damage in hand, computes 39 Table 14: Combat play in Act 1 by tier of Table 1 (top four, next eight, bottom seven with full logs), with Spearman's휌against mean acts cleared over the 19 models and its permutation푝. Hallway fights are ordinary combats. Block into no attack: turns on which a block card is played although no enemy intends to attack. Over-block: a block-only card played beyond the incoming damage while an attack was playable. Missed lethal: turns with one enemy left and enough damage in hand to kill it. The lower block shows measures that do not separate the tiers (푝> 0.4). Top 4Next 8Bottom 7휌푝 Damage taken per hallway fight8.29.614.0 −0.87< 0.001 Turns per hallway fight3.093.584.06 −0.680.002 Damage taken per hallway turn2.652.773.43 −0.570.012 Damage dealt per energy on attacks7.56.75.8 +0.91< 0.001 Damage dealt per energy, all cards5.74.73.8 +0.76< 0.001 Damage dealt / attack damage in hand0.950.860.74 +0.650.003 Block into no attack (% of turns)61531 −0.85< 0.001 Over-block (%)0.83.06.4 −0.680.001 Missed lethal (%)00.88.7 −0.78< 0.001 Plays refused as not in hand (per 100 turns)1.12.010.8 −0.580.012 Next-turn HP forecast error (HP)0.060.442.12 −0.83< 0.001 Potions drunk in hallway fights (%)272761 −0.77< 0.001 Preventable unblocked damage (HP per turn)0.110.060.07 Energy left unspent (%)655 Energy spent per turn2.572.682.69 Share of incoming damage blocked0.480.520.50 “Trip Attack (2 energy): 4 damage . . . not enough to kill” by applying its own Fatigue [Weak] a second time, and leaves it at 3 HP. 40 0.000.250.500.751.001.251.50 Mean acts cleared: runs 1-5 (open circle) to runs 6-10 (arrowhead) Human (an author) GLM-5.3 Flash (max) GLM-5.3 (max) Grok 4.6 (xhigh) gpt-oss-120b (high) GPT-5.6 Luna (max) GPT-5.6 Terra (max) GPT-5.6 Sol (max) GPT-6 Astra (max) Nemotron 3 Nano Muse Spark 1.3 (xhigh) Gemma 4 E4B Gemini 3.8 Flash (high) DeepSeek V4.1 Flash (max) Haiku 4.5 (12k) Sonnet 5 (max) Opus 5 (max) Fable 5.1 (max) Qwen3.5 122B (xhigh) Qwen3.6 35B (xhigh) Qwen3.8 27B (xhigh) No memory (control)Consolidated (xml)Compact summaryNotes kept verbatimHuman, first 10 A20 runs Figure 11: Do models learn from their memories? Mean acts cleared per run over the first five runs of each chain (open circle) against the last five (arrowhead), by memory mechanism; a circle alone means no change. The two halves are different seeds, so the no-memory cohort on the same seeds (grey) is the control for how hard each half is: learning shows as an arrow further right than the grey one. Only cohorts with all 10 seeds are shown. For reference, the top row is one of the authors (a non-professional player) over their first ten A20 Ironclad runs, playing the unmodified game without a harness or memory bank; with five runs per half, a single win moves a half’s mean by 0.8 acts. 41 F Difficulty and where runs end Figure 12 repeats the main configuration at a range of pinned ascension levels: every level from 0 to 20 for the non-LLM baselines, and levels 0, 1, 5, 10, 15 and 20 for the models run at more than one. Each level is a separate cohort with its own memory chain. Figure 13 breaks down in which act the runs of Table 1 ended. 0 10 20 30 40 50 Pick first optionPick random optionBottledAI w/o Heart Act 1 Act 2 Act 3 Act 4 BottledAI w/ Heart 0 10 20 30 40 50 5 6 6 Qwen3.8 27B (xhigh) Qwen3.6 35B (xhigh) Qwen3.5 122B (xhigh) 8 644 Act 1 Act 2 Act 3 Act 4 Nemotron 3 Nano A0A5A10A15A20 0 10 20 30 40 50 79 GPT-5.6 Luna (max) A0A5A10A15A20 88 gpt-oss-120b (high) A0A5A10A15A20 9 GLM-5.3 Flash (max) A0A5A10A15A20 Act 1 Act 2 Act 3 Act 4 All human players Ascension level Floor reached Figure 12: Effect of difficulty on performance. Final floor reached against the pinned ascension level, for every agent in Table 1 with runs at more than one level, on the held-out seeds under the same configuration as the table. Each level is a separate cohort (its own memory chain). The solid line is the median floor at each level, the blue band the interquartile range, and the dotted lines the lowest and highest floor reached. Background shading marks the floors of each act. Levels with fewer than 10 completed seeds are annotated with their seed count. The last panel is every Ironclad run in Mega Crit’s released telemetry (October–November 2020, final content build; abandoned runs excluded), with the dotted lines at the 5th and 95th percentiles instead, since over tens of thousands of runs per level the extremes are always the first and last floors reachable. Each level there is a different population: a level is unlocked by winning at the one below it, so higher levels hold more experienced players, and only Ascension 20 includes runs that continue to the Heart. 42 020406080100 Share of runs (%) Nemotron 3 Nano Omni 6.8 Pick first option 7.1 Pick random option 7.5 Gemma 4 E4B 7.7 Qwen3.5 122B-A10B (xhigh) 9.8 Qwen3.6 35B-A3B (xhigh) 10.2 gpt-oss-120b (high) 10.4 BottledAI w/ Heart 11.3 BottledAI w/o Heart 11.3 Claude Haiku 4.5 (12k) 11.9 GPT-5.6 Luna (max) 14.2 Qwen3.8 27B (xhigh) 14.9 Grok 4.6 (xhigh) 15.4 GLM-5.3 Flash (max) 15.6 DeepSeek V4.1 Flash (max) 16.9 GPT-5.6 Terra (max) 18.2 Claude Sonnet 5 (max) 19.3 Muse Spark 1.3 (xhigh) 21.9 GLM-5.3 (max) 22.1 Gemini 3.8 Flash (high) 22.9 GPT-5.6 Sol (max) 27.8 Claude Opus 5 (max) 32.5 Claude Fable 5.1 (max) 36.2 GPT-6 Astra (max) 37.0 mean floor Ended in Act 1Ended in Act 2Ended in Act 3Cleared Act 3Killed the Heart Figure 13: Where runs end. For each agent in Table 1, the share of its runs that ended in each act, sorted by mean floor (right). Agents with a similar mean floor can differ in whether they reliably reach the next act or split between early deaths and deep runs. 43 G Calibration The forecasts.At the end of every floor the model is asked, through thepredicttool, for two probabilities: that it survives the current act, and that it wins the run. When it ends a turn it may also give, as optional arguments of end_turn, the HP it expects at the start of its next turn and at the end of the fight (Appx. B.2). Scoring.Table 15 scores these predictions against what happened, over the runs of Table 1. A floor’s act-survival prediction is scored as correct if the run went on to clear that act, and a run-win prediction if it killed the Heart. We report the mean predicted and observed rates, the Brier score (Brier, 1950), whose reliability term (Murphy, 1973) is the calibration error of Fig. 14, and the AUC, which measures whether the model gives higher predictions in the acts it goes on to survive regardless of their overall level. HP predictions are matched to the next recorded HP on the same floor, and a prediction made in the fight that killed the player is scored against 0 HP. A prediction is easier the closer it is made to the moment it resolves: the end of the fight, the end of the act (or the death that ends it early), the end of the run. Stronger models survive longer and so make more of their predictions close to an act's end, which would flatter their raw scores. We therefore bin every prediction by its distance from resolution (Fig. 15) and report scores standardized to a common mix of distances: each error is fitted as a model effect plus a distance-bin effect, and each model's fit is averaged over the distance mix of all models' predictions pooled. The AUC is computed within each distance bin and averaged with the same weights. Next-turn HP is always one turn ahead and needs no adjustment. Results. Table 15 scores the models’ predictions of their own game; how the error depends on the distance to resolution is in Fig. 15. Figure 14 relates each model's end-of-floor calibration error to the acts it clears, and Fig. 16 does the same for its within-floor HP forecasts, whose error tracks play more closely (Table 16). The act-survival forecast barely sharpens as the act nears its end, while the forecast of HP at the end of a fight sharpens steeply as the fight ends: calibration, like play, degrades with the distance to the outcome, which matters for an agent that must decide how much risk to take on a long task. Table 15: How well the models predict their own game. At the end of every floor each model is asked for its chance of surviving the current act and of winning the run; when ending a turn it may also predict its HP at the start of the next turn and at the end of the fight. Same runs as Table 1. Pred. and Obs. are the mean predicted and observed rates (a floor's act counts as survived if the run went on to clear it; a run is won at the Heart). A prediction is easier the closer it is made to the moment it resolves, and stronger models make more of theirs close to an act's end, so the Brier scores (0 is perfect; always answering 50% scores 0.25), the AUC (0.5 is chance; – where every act had the same outcome) and the fight-end HP error are adjusted for that distance: each is scored over the same mix of distances, pooled over all models (Fig. 15). HP: mean absolute error and signed bias (predicted minus actual; positive is optimistic) in hit points, with a prediction made in the fight that killed the player scored against 0. Next-turn HP is always one turn ahead, and close to arithmetic since enemy intents show the incoming damage. Best and second-bestscores among models with all 10 seeds are highlighted. Act survivalRun winHP next turnHP at fight end AgentRunsPred. %Obs. %BrierAUCPred. %Obs. %BrierMAEBiasMAEBias Qwen3.8 27B (xhigh)1051230.240.692100.051.1 −0.110.3+4.5 Qwen3.6 35B-A3B (xhigh)105500.32–3100.113.1 −1.212.3+8.2 Qwen3.5 122B-A10B (xhigh)106300.42–3300.122.8+0.617.7+17.2 Claude Fable 5.1 (max)1059760.180.801600.050.10.07.3+0.3 Claude Opus 5 (max)106268 0.150.851400.040.2 −0.28.6+0.1 Claude Sonnet 5 (max)1068450.300.583400.120.4 −0.19.5 −2.9 Claude Haiku 4.5 (12k)1042150.250.391900.053.0 −0.68.9+1.1 DeepSeek V4.1 Flash (max)1037300.250.3990 0.010.6 −0.19.9+3.0 Gemini 3.8 Flash (high)1086440.430.494900.260.10.010.9+10.6 Gemma 4 E4B107600.59–6100.394.5 −0.313.4+11.7 Muse Spark 1.3 (xhigh)1054540.230.631300.020.6 −0.28.3+1.1 Nemotron 3 Nano Omni107200.53–5500.334.0+0.113.6+10.6 GPT-6 Astra (max)1074690.160.882200.07 0.00.0 6.5+3.7 GPT-5.6 Sol (max)1071630.220.6820200.10 0.00.014.0+13.1 GPT-5.6 Terra (max)1050470.280.491200.020.4 −0.413.8+13.7 GPT-5.6 Luna (max)1064120.370.642100.050.2 −0.111.3+6.9 gpt-oss-120b (high)1065170.400.563300.131.2+0.116.1+14.8 Grok 4.6 (xhigh)1041330.240.5490 0.010.3 −0.17.0+2.1 GLM-5.3 (max)1064460.270.592500.070.50.09.0+3.9 GLM-5.3 Flash (max)1058460.260.542400.061.0 −0.48.7 −0.1 44 0.000.250.500.751.001.251.501.75 Mean acts cleared per run 0 10 20 30 40 50 60 70 Calibration error per run (percentage points; lower is better) Figure 14: Calibration against play, for each LLM agent in Table 1. Calibration error is the reliability term (Murphy, 1973) of the Brier score (Brier, 1950), in percentage points, over ten bins of stated probability, for the two forecasts made at the end of every floor (surviving the act, winning the run), scored separately and averaged. It is computed per run, not adjusted for how far ahead a prediction was made (Table 15); points are the median over a model's runs and bars the interquartile range. Lower effort levels are drawn smaller and paler, joined to the same model. The Spearman correlation is over the levels in Table 1, one per model. Table 16: Which forecasts track play. The Spearman correlation, over the 20 models of Table 1, between mean acts cleared and the error of each forecast, pooled over each model's runs: mean absolute error for the two HP forecasts, made when ending a turn, and the calibration error of Fig. 14 for the two probability forecasts, made at the end of every floor. Negative means the models that forecast better play better. Every model plays Act 1, so the Act 1 column compares all of them on forecasts about the same part of the game, while the first column also counts later acts, which only the stronger models reach and where hits are larger. Next-turn HP is close to arithmetic once enemy intents are shown, so its error measures how well a model reads the fight rather than what it knows of its own play. Almost no run is won (1.5% of run-win forecasts), so the run-win error mostly measures how much chance a model gives itself. The bottom rows correlate the scores plotted in the two figures, which combine two forecasts each and are medians over runs. ForecastMade atAll predictionsAct 1 only HP at the start of the next turnending a turn−0.84 −0.86 HP at the end of the fightending a turn−0.70 −0.82 Surviving the actend of a floor−0.84 −0.77 Winning the runend of a floor−0.45 −0.58 Both HP forecasts (Fig. 16)−0.92– Both probabilities (Fig. 14)−0.72– 45 9+5–82–40–1 Floors before the act resolves .28.21.23.20 .36.31.32.26 .44.40.43.39 .20.17.18.10 .16.16.16.14 .29.31.31.28 .27.25.23.22 .30.25.22.19 .38.47.47.47 .62.63.59.53 .25.27.18.21 .62.54.51.50 .15.14.21.13 .20.22.27.20 .28.28.28.25 .42.36.34.31 .38.38.42.40 .27.25.21.18 .24.29.30.29 .25.30.26.25 .25.27.28.25 Qwen3.8 27B (xhigh) Qwen3.6 35B (xhigh) Qwen3.5 122B (xhigh) Fable 5.1 (max) Opus 5 (max) Sonnet 5 (max) Haiku 4.5 (12k) DeepSeek V4.1 Flash (max) Gemini 3.8 Flash (high) Gemma 4 E4B Muse Spark 1.3 (xhigh) Nemotron 3 Nano GPT-6 Astra (max) GPT-5.6 Sol (max) GPT-5.6 Terra (max) GPT-5.6 Luna (max) gpt-oss-120b (high) Grok 4.6 (xhigh) GLM-5.3 (max) GLM-5.3 Flash (max) All models Act survival (Brier) 30+15–295–140–4 Floors before the run ends .05.06.05 .12.10 .12.13 .01.03.07.10 .02.03.05.04 .11.12.14.11 .03.05.06 .01.01.01 .19.24.27.28 .43.35 .02.02.03.01 .36.31 .05.05.09.09 .33.08.03.04 .03.02.02 .05.06.04 .06.13.14 .01.01.01 .08.08.07.07 .05.08.05 .08.06.09.10 Run win (Brier) 5+3–4210 Turns left in the fight 2310840 21138716 341812829 149622 1110830 12111072 1111967 228751 2611525 2813967 148754 2813974 119510 391752 0 241274 3516 168510 199744 168852 1910745 HP at fight end (MAE) Figure 15: Prediction error against distance from the outcome, for the runs of Table 1. Left: Brier score of the act-survival prediction against the number of floors before the act resolves (at its end if the run cleared it, otherwise where the run died). Middle: the same for the run-win prediction, against floors before the run ends. Right: mean absolute error, in HP, of the predicted HP at the end of the fight against the number of turns still to be played in it. Darker is worse, on a scale per panel; blank cells have fewer than 5 predictions. The bottom row pools all models, and its mix of distances is the one each model is scored over in Table 15. Fight-end predictions sharpen steeply as the fight ends; act-survival predictions barely sharpen before the act resolves. 46 0.000.250.500.751.001.251.501.75 Mean acts cleared per run 0 1 2 3 4 HP prediction error per run, relative to the all-model mean (lower is better) Figure 16: As Fig. 14, for the forecasts made within a floor: the HP a model expects at the start of its next turn and at the end of the fight, both optional when ending a turn. Each is scored by its mean absolute error in HP, divided by that forecast's mean error over the predictions of every model in Table 1 (so 1 is average), and the two are averaged for each run. The division matters: fight-end errors are about ten times next-turn ones, so without it a run that made only next-turn predictions would look excellent. A run with no HP predictions has no score. Within-floor error tracks play more closely than end-of-floor calibration (Table 16). 47 H Further analyses The analyses below support the Results and Discussion (Sections 4 and 5). Unless stated otherwise, all are over the runs of Table 1, with the tests of Appx. C.3. H.1 Time horizon Slay the Spire interleaves four kinds of decision, on four time scales, over one irreversible pool of resources: tactics within a turn (seconds), pricing HP within a fight (minutes), building a deck within an act (10–20 minutes) and setting a risk posture over a run (an hour). Playing it over such spans also needs memory management: deciding what to keep across floors, where each floor is a fresh conversation, and across runs, where the environment starts again. The time scales are the ones on which agent ability is now charted: Kwa et al. (2025) measure an agent by the length of task, in human working time, that it completes half the time, and find it doubling about every seven months on software tasks. A run here is an hour of human play, of the order of the horizons they report for frontier models, but it is one task made of decisions on every shorter scale, so the benchmark can show which scale a model fails on rather than only how long a task it survives. Measured this way (Table 17), every model succeeds on at least 87% of its turns and 77% of its fights, and falls below 50% between a fight and a run: for all but the top four at the act, for those four (53–67% of acts) only at the run. The fitted 50% horizons run from about one minute of human play (Gemma 4 E4B) to about 19 minutes (Claude Fable 5.1), on act and run times taken from Mega Crit's telemetry (players who complete Acts 1, 2 and 3 take about 9, 12 and 20 minutes over them, and a run that kills the Heart takes a median of 50 minutes); they order the models as acts cleared does, largely by construction, since the act is the only scale near 50%. What the measure does show is that the fall is a step rather than a slope. Of the 20 models, 18 win 93–100% of their ordinary Act 1 fights, but boss fights, won 0–81% of the time, decide where each model stops, and human players at this level show the same step, clearing 43% of the acts they enter. The difficulty is therefore not a limit on how long a model can keep going, as a time horizon on a smooth task distribution would suggest, but rare, high-stakes decisions inside a long task, whose preparation has to begin floors earlier. H.2 Public benchmarks and reasoning effort The 30 model and effort points of Fig. 2b are 20 models at their highest reasoning effort and 10 of them again at low effort. Acts cleared tracks the Intelligence Index about as closely within each rung as across both (푟=0.80 at the highest effort, 0.91 at low). Within a model the Index is a poorer guide. Lowering the effort costs acts for 8 of the 10 models with an Artificial Analysis entry at both levels (Table 26), but the size of the loss does not follow the size of the drop on the Index (Spearman휌=0.06): Gemini 3.8 Flash and Qwen3.8 27B each lose about 7.5 Index points and clear slightly more acts at low effort, while Claude Fable 5.1 loses 6.5 and clears 0.7 fewer acts. SpireBench ranks models much as the public suite does, but measures something the suite does not already capture. It is what the long-horizon literature would predict: if success on a long task compounds per-step reliability (Sinha et al., 2026), small differences on short evaluations should widen into large ones over a run, and the steep rise of acts cleared with the Index (Fig. 2b) is consistent with that. H.3 Agentic residuals Do models that score well on public agentic evaluations nonetheless fail at the game? Regressing mean acts cleared on each agentic evaluation Artificial Analysis reports, one point per model at its highest effort (Table 19), Grok 4.6, GLM-5.3 Flash and GPT-5.6 Luna clear fewer acts than all four predict, and Claude Fable 5.1, GPT-6 Astra and Claude Opus 5 more. The shortfall is not specific to agentic skill, however: the same models fall short of their scores on Humanity's Last Exam and GPQA Diamond by about as much (the mean residuals correlate at 푟=0.92). Only one agentic evaluation, Terminal-Bench 4.0, predicts the game better than Humanity's Last Exam (푟=0.89 against 0.81) and adds information once it is controlled for (partial푟=0.64, permutation푝=0.006, 푛= 18); 휏 -Bench Banking, Terminal-Bench 2.1 and GDPval predict it less well and add nothing. H.4 Runs ended by the harness The Orak comparison runs Orak at each model's own reasoning setting; its 85% saving in tokens per floor, 31% in cost once our cached prefix is priced at its own rate, comes largely from its rule bot playing every non-combat floor without a model call and from its shorter prompts, which is why we compare models on cost fronts (Fig. 2a) rather than on score alone. A harness can also end runs itself, and those endings are charged to the model. A harness that supports most of a game but not all of it stops a run when it meets what it cannot handle, and every 48 Table 17: Success at each decision scale of Fig. 1, for the LLM agents of Table 1 (10 runs each). Turn is the share of combat turns without a rejected action, format error or energy mismatch (a lower bound, since every error is charged to a different turn). Combat is the share of fights won; A1 hallway is the share of ordinary Act 1 fights (not elites or the boss) won, and Boss the share of boss fights won (superscript: boss fights played). Act is the share of acts cleared among acts entered, pooled over Acts 1–3. Run is the share of runs that cleared Act 3.ℎ 50 is the human time at which a logistic inlogtime, fitted with each scale weighted equally, crosses 50%, with each scale placed at a human time from Mega Crit's telemetry: 10 s for a turn, 10 s per turn of an average fight, the mean of the median times to complete Acts 1–3 (8.8, 11.9 and 20.3 min, taken from players who completed each act), and the median run that kills the Heart (50 min). Human rows come from Appx. D and have no turn or combat data. Success (%) AgentTurnCombatA1 hallwayBossActRunℎ 50 Acts Claude Fable 5.1 (max)98.69510080 20 672019 min1.60 GPT-6 Astra (max)98.3959875 20 621014 min1.50 Claude Opus 5 (max)99.39410081 16 57011 min1.30 GPT-5.6 Sol (max)98.99410073 15 531011 min1.10 Muse Spark 1.3 (xhigh)99.6919678 9 4107 min0.70 Gemini 3.8 Flash (high)99.89210050 12 3807 min0.60 GLM-5.3 (max)99.1919655 11 3806 min0.60 GPT-5.6 Terra (max)99.39010056 9 3306 min0.50 Claude Sonnet 5 (max)99.49010050 10 3306 min0.50 GLM-5.3 Flash (max)99.1889867 6 2905 min0.40 DeepSeek V4.1 Flash (max)99.48910033 9 2304 min0.30 Grok 4.6 (xhigh)95.7889850 6 2304 min0.30 Qwen3.8 27B (xhigh)96.6889829 7 1703 min0.20 gpt-oss-120b (high)87.5839633 3 902 min0.10 GPT-5.6 Luna (max)99.4879814 7 903 min0.10 Claude Haiku 4.5 (12k)96.4849825 4 902 min0.10 Qwen3.5 122B-A10B (xhigh)94.081930 1 0085 s0.00 Qwen3.6 35B-A3B (xhigh)93.984940 2 002 min0.00 Nemotron 3 Nano Omni94.67778–0077 s0.00 Gemma 4 E4B87.178890 2 0074 s0.00 Human, all A20 Ironclad (telemetry)–437–0.73 Human, 1000+ h players (telemetry)–4911–0.92 Human, Baalorlord, complete log–8970–2.98 per-run figure then records that as the model dying there. On these seeds Orak's harness ended 22 of its launches itself, by crashing on an empty model reply or looping without progress, and its own failure ended 11 of the 70 seeds; ours ended none of its launches on these arms, and 6 of the 2,432 it made across every arm. We count a run a harness ended as a finished run lost by that harness, and report these endings separately, since a benchmark that did not would score the harness's gaps as the model's mistakes. That is the practical case for a harness that plays the whole game. H.5 One conversation per floor The continuous conversation, which keeps one conversation for the whole run (Table 2), is compared on the same floors; a fresh conversation per floor, with the run's notes and history carried forward, avoids both a long context used poorly (Liu et al., 2024a) and earlier mistakes kept in view (Sinha et al., 2026). Vending-Bench finds long-running agents derailing in ways unrelated to when the context fills (Backlund & Petersson, 2025); our comparison is narrower, holding the task fixed and changing only how much of its history the model is shown. H.6 What the memory banks hold Consolidating notes into a memory bank does not raise the acts a model clears (Sec. 4). Of the memory arms, only carrying notes forward verbatim does better than no memory, gaining+0.06 acts,+1.6 floors and+5 points of Act 1 win rate (Table 27), which is suggestive rather than established. This agrees with the line of work on unverified self-correction and memory propagation (Huang et al., 2024; Xiong et al., 2026) rather than contradicting the 49 Table 18: SpireBench against Artificial Analysis's public benchmarks, over the 20 LLM agents of Table 1 with all 10 seeds at their highest reasoning effort, and 10 of them again at low effort (30 points), each matched to the Artificial Analysis entry for the reasoning setting it ran at (scores retrieved 2026-09-24). A low rung is included where it has all 10 seeds and Artificial Analysis lists that setting. Left: correlation of each benchmark with mean acts cleared, over both rungs together and within each rung (푛is the number of points with that evaluation). Right: how substitutable each benchmark is by the rest of the suite, with SpireBench added as one more row: its strongest correlation with any other single evaluation, its mean correlation with the others, and the share of its variance across points that the first principal component of the others predicts for a model left out of the fit, both rungs of it together (푅 2 LOO ; negative is worse than predicting the mean). The Intelligence Index is a composite of evaluations like these, so it is left out of the right-hand block, which uses the points with every evaluation (25). Correlation with SpireBenchSubstitutability Both rungsPearson 푟 by rung Benchmark푛Pearson 푟Spearman 휌HighestLowmax 푟mean 푟 푅 2 LOO A Intelligence Index300.820.890.800.91– GPQA Diamond280.580.800.560.740.860.700.51 Humanity’s Last Exam300.820.890.810.870.920.830.85 SciCode270.700.810.730.600.870.790.75 Long-context reasoning300.520.660.470.700.740.640.29 CritPt300.820.830.840.820.920.780.70 A-Omniscience300.830.910.820.870.860.770.70 SpireBench (acts cleared)25–0.830.710.50 reports of improvement, and the problem it exposes only shows where episodes are long, stochastic and repeated. Reading the banks shows why (Table 20). Of the memories the models carry out of their tenth run, 59% describe how the game works, 29% are advice, 9% are rules stated without condition and 2% recount a single run, and 17% are false. Most are the model's own generalizations from one run. Claude Opus 5 records that another boss “NEVER SUMMONS AGAIN” and generalizes it to “Most summon exactly ONCE”; Muse Spark decides on its second seed that an area attack “does NOT scale with Strength”, misreading a damage figure that already included its Strength, and never revisits it. Rules are the error-prone kind, 78% of them false against 9% of statements of fact, and models whose banks hold more errors of their own clear fewer acts (Spearman휌=−0.67, permutation 푝=0.002,푛=20). How emphatically a memory is phrased does not matter (휌=−0.03): Claude Fable 5.1 and Opus 5 write in capitals as often as Claude Haiku 4.5, but what they write is largely true. Nothing prunes the errors. Over the 200 runs the models delete 30 memories, and wrong memories are rewritten about as often as correct ones without being corrected. DeepSeek V4.1 Flash records correctly that the act transition restores most of the HP it has lost, overwrites it two seeds later with “the post-boss heal is small”, and then drops the subject. Without a check against outcomes, consolidation keeps whatever a single run suggested. Whether the models learn from their banks across seeds at all is in Appx. E.4. H.7 Reskin cost by model family We find no evidence that the reskin's cost follows model family, and so cannot tie it to particular training corpora: a permutation test on developer labels finds no more variation between families than within them (11 models from 7 developers;푝=0.29 for acts,푝=0.12 for floors), and within a family the cost differs sharply, with GLM-5.3 Flash losing 7.5 floors against GLM-5.3's 2.3, and GPT-5.6 Luna 4.3 against none for gpt-oss-120b. The cost appears only in models that clear 0.6–0.8 acts with the real names (Spearman휌=−0.67, permutation푝=0.029, between acts lost and acts cleared with real names), which is partly mechanical, since a model that clears no acts has none to lose. The reskin removes knowledge that only the stronger players were using. Where nonsense predicates alone collapse planning in Mystery Blocksworld (Valmeekam et al., 2023), here nonsense names cost least; a familiar name pulls a model towards what that name usually means, which helps when it is right and misleads when it belongs to something else (McCoy et al., 2024). H.8 Recognition of the game under the reskin Since a model's own text is never renamed (Sec. 3), we can measure when it recognizes what it is playing (Table 21). In 125 of the 190 themed runs of Table 1 with full logs (Gemma 4 E4B has none), from 18 of 19 models, the model names Slay the Spire or a real entity that the reskin replaced, almost always in its reasoning and usually on the first seed. This ranges from borrowing the real game's rules, as Nemotron 3 Nano Omni does in every run 50 Table 19: Which models play below, or above, what their public agentic scores predict. For each Artificial Analysis evaluation, mean acts cleared (Table 1, highest reasoning effort, all 10 seeds) is regressed on the evaluation score across models, one point per model; cells are internally studentized residuals (푧; negative means fewer acts than the evaluation predicts). The top rows give each evaluation's correlation with acts cleared, with two-sided permutation푝(200,000 permutations) and푛 models. Models are sorted by their mean residual over the agentic evaluations (left block). Artificial Analysis scores retrieved 2026-09-24. Terminal: Terminal-Bench. HLE: Humanity's Last Exam. AgenticReasoning ModelActs 휏 -BenchTerminal 2.1Terminal 4.0GDPval (normalized)Mean 푧HLEGPQA Diamond Pearson 푟0.590.650.890.670.810.56 푝0.0130.004<0.0010.002<0.0010.014 푛171918202019 DeepSeek V4.1 Flash0.30– −1.41−1.16 −1.29 −0.91– GPT-5.6 Luna0.10 −0.87 −1.41 −0.77−1.36 −1.10 −1.59−1.21 GLM-5.3 Flash0.40 −0.97 −0.75 −1.60−0.99 −1.07 −0.63−0.52 Grok 4.60.30 −1.00 −1.12 −0.45−1.24 −0.95 −1.32−0.87 Qwen3.8 27B0.20 −1.50 −1.120.29−1.02 −0.84 −0.80−0.95 GLM-5.30.60 −0.65 −0.23 −1.68−0.46 −0.75 −0.16−0.08 GPT-5.6 Terra0.50 −0.37 −0.61 −1.42−0.27 −0.67 −0.55−0.35 Qwen3.5 122B-A10B0.00 −0.34 −0.70 −0.03−0.31 −0.34 −0.75−1.17 Qwen3.6 35B-A3B0.00 −0.02 −0.63 −0.03−0.47 −0.29 −0.50−1.09 Claude Sonnet 50.50 −0.22 −0.380.72−0.31 −0.05 −0.41−0.28 Gemini 3.8 Flash0.60 −0.37 −0.340.590.03 −0.02 −0.63−0.25 Claude Haiku 4.50.100.25 −0.350.420.150.120.89−0.03 Muse Spark 1.30.70 −0.24 −0.021.33−0.150.23 −0.260.04 gpt-oss-120b0.100.050.190.420.420.270.06−0.57 Nemotron 3 Nano Omni0.00–0.56–0.360.461.071.00 Gemma 4 E4B0.00–0.75–0.360.551.180.24 GPT-5.6 Sol1.100.870.920.741.010.880.920.97 Claude Opus 51.301.451.400.711.311.221.161.48 GPT-6 Astra1.501.961.940.592.171.661.871.82 Claude Fable 5.11.601.942.111.792.071.981.872.16 (“Usually in Slay the Spire, ‘Coiled’ is a buff that gives guard when you take damage”), to identifying entities from their behaviour, as Claude Fable 5.1 does with the Act 1 boss: “This 250 HP figure actually matches Slay the Spire's Guardian at Ascension 9+ . . . I'm recalling its exact moveset”. Only Muse Spark never does. Recognition rarely reaches the memory bank (15 of 200 themed runs, 10 of them Fable's), and models recognize the game on their first seed, before any memory exists, about as often as later. Nor does it buy back what the reskin costs: across the ten models with both arms, the share of runs that recognize the game is unrelated to the cost (Spearman 휌=0.18, permutation푝=0.62), and within a model, runs that recognize it in their first six floors clear no more acts than those that do not (+0.06,푝=0.85, 170 runs). Knowing which game this is does not recover knowledge that was keyed to its names. Changing the vocabulary barely changes recognition. For the eight models run under every mapping, 56 of 80 themed runs recognize the game, against 44 of 79 with nonsense names and 45 of 73 with the permuted themed ones. Gemini 3.8 Flash at low effort goes further under nonsense names, decoding them back to the real ones in every run and correctly about nine times in ten, and it scores its best result there. When real names are swapped between entities, six of the eight models conclude they are playing a modded Slay the Spire (GLM-5.3 Flash: “The mod card-identity trap: I played†Defend thinking it gave channel [block] (base-StS instinct)”), and their appeals to the real game mostly correct the borrowed meaning rather than act on it; runs that make such appeals clear no fewer acts (+0.05, 푝= 0.71, 80 runs). H.9 Deck building does not fail by accumulation In Act 1 almost any card improves a starting deck, but by Act 3 a marginal card dilutes the deck that has to beat the boss, so the right policy for card rewards reverses during a run. The models do reverse it on average (Table 22): their skip rate rises from 0.09 in Act 1 to 0.25 and 0.42, as steeply as human players' (0.21, 0.39 and 0.53 over every Ascension 20 Ironclad run in the telemetry), and within runs as well as between them (of the runs whose rate changes, 35 rise and 8 fall; exact sign test푝=4×10 −5 ). But 41% of the model runs that reach a second act never skip a card at all, against 10% of human runs, and the deepest models are the most reluctant to skip early (3% in Act 1, against 18% for players with over 1000 hours). Neither deck size at a matched floor, within a model, nor how much a model adapts its skip rate predicts how many acts it clears (Spearman휌=0.04,푝=0.91, over the 11 models with enough decisions after Act 1). Models do not lose by filling their decks; many never revise the policy they start with, and among those that do, revising it more does not carry them further. 51 Table 20: What the models keep in their memory bank after the tenth seed of Table 1, classified by hand. Bank is the final number of memories; Created, Rewritten and Deleted count consolidation operations over the ten runs (Rewritten excludes verbatim re-saves). Composition is the share of the final bank in each category (a random 30 per model where the bank is larger): a game fact describes how the game works, a heuristic advises, an absolute rule prescribes without condition, an observation recounts one run; Other is route plans and notes about the harness. Wrong is the share stating a false game fact or a rule that fails in common situations, and H. how many of the coded wrong memories trace to harness text that was itself wrong (a rest-site key label, a state diff across the two final bosses, and debuff expiry shown as an enemy’s action); Absolute the share phrased as ALWAYS, NEVER or MUST anywhere; One run the share drawn from a single run. Models are ordered by mean acts cleared. Operations are read from the full run logs, except for Gemma 4 E4B, whose are read from the harness output log (its full logs are not available). MemoriesComposition (%) AgentActsBankCreatedRewrittenDeletedFactHeur.RuleObs.OtherWrong (%)H.Absolute (%)One run (%) Claude Fable 5.11.6050501580602701301346323 GPT-6 Astra1.50149149110087130000–353 Claude Opus 51.308283228133633001716033 GPT-5.6 Sol1.1010610676010000000–767 Muse Spark 1.30.7010310364060307037–3790 GLM-5.30.6044445706320100720–3047 Gemini 3.8 Flash0.6079793509370003–1063 Claude Sonnet 50.5010810854073270003–1057 GPT-5.6 Terra0.50292915083170000–383 GLM-5.3 Flash0.40747667287103002012063 Grok 4.60.30989881070230070–2093 DeepSeek V4.1 Flash0.3034341609307001313060 Qwen3.8 27B0.207979560632730710–2057 Claude Haiku 4.50.106980311737507060–60100 GPT-5.6 Luna0.10323217060330337–1393 gpt-oss-120b0.10797920277030023–33 Nemotron 3 Nano Omni0.006666402043300757–2710 Gemma 4 E4B0.0024247009280012–84 Qwen3.6 35B-A3B0.00132725146215230054–4638 Qwen3.5 122B-A10B0.005658522730377040–5387 H.10 Where models go wrong Reading the runs of Table 1 shows recurring failures (excerpts in Table 24). Weaker models spend reasoning on routine calls: the end-of-floor forecast asks for two probabilities, yet Claude Haiku 4.5, gpt-oss-120b and Grok 4.6 spend more than 1,000 reasoning tokens on 58–83% of theirs, against at most 9% for the strongest models (Spearman휌=−0.49 between the median spent and mean acts cleared). Grok 4.6's reasoning often settles on one action while its tool call takes another: in 20 of the 68 decisions where its reasoning ends by naming a card or an option (29%), the call differs, seven times skipping a card reward it had just said it would take, and the prose it writes beside the call agrees with the call rather than the reasoning. gpt-oss-120b does so in 2 of 31 such decisions, and none of the other 16 models in 1,352. Most models sometimes lose track of their own turn: 15 of the 20 end a turn with a different amount of energy from the one they declared (Table 23). Within a run it is the strongest models that over-write their notes: Claude Opus 5 writes 143 notes per run at a median of 192 words, and 23–42% of the notes of Gemini 3.8 Flash, GPT-6 Astra, Claude Fable 5.1, Muse Spark and GLM-5.3 are turn-by-turn accounts of a single fight, while the weakest write a few short, generic notes. What the harness leaves unstated is learnt unevenly. It gives Vulnerable's 50% but not that damage is rounded down once every modifier is applied; gpt-oss-120b and Nemotron 3 Nano Omni work the rounding out afresh in all ten of their runs, Qwen3.6 35B-A3B and Claude Haiku 4.5 in nine, and none of them ever writes it down, while eleven models keep it in their memory bank. Of the 16 models that clear Act 1, nine end with the fact that the act transition restores most lost HP in their bank; Muse Spark and Grok 4.6 never record it and weigh their low HP when choosing the boss relic, GPT-5.6 Terra credits the heal to the relic it chose, and DeepSeek V4.1 Flash records it and later overwrites it with the claim that the heal is small. End-of-run consolidation accumulates rather than curates: over the 200 runs the models create 1,404 memories (two of them left by Gemma 4 E4B's crashed first attempt), rewrite 1,080, re-save another 863 unchanged and delete 30, and 15 of the 20 never delete one; what the banks end up holding is classified in Table 20. The miscounted turns echo what is known of long executions: a slip early in a turn is carried into the rest of it, as errors in context make further errors more likely (Sinha et al., 2026). The notes and memories are the memory failure of Appx. H.6, seen from inside a run: nothing tells the model which of its observations were worth keeping, so it keeps the wrong ones (Xiong et al., 2026). 52 Table 21: Recognition of the reskinned game, from each run’s own text (reasoning, prose and tool-call arguments). Per arm: Rec. runs whose text names the real game or a real entity name the arm does not display; Game runs naming “Slay the Spire”/“StS”; Names median distinct real entity names per run. Themed also gives the first seed (floor) with recognition. The Permuted arm displays the title and real names (for the wrong entities), so there only undisplayed names and “StS” count. Memory, None: runs whose newly written memories use real names when the arm shows them (a sanity check; the real-names arm has no local logs). ThemedRandomPermutedPerm.+themedMemory AgentRec.GameNames1stRec.GameNamesRec.GameNamesRec.GameNamesNone Qwen3.8 27B10/1010/100S1 (F0)10/1010/10010/1010/1003/33/30– Qwen3.6 35B-A3B1/101/100S9 (F3)0/100/1000/100/1001/101/100– Qwen3.5 122B-A10B2/102/100S2 (F9)1/101/1000/100/1004/104/10010/10 Claude Fable 5.110/1010/1022S1 (F0)– Claude Opus 59/107/102S1 (F6)– Claude Sonnet 58/108/100S1 (F3)– Claude Haiku 4.51/101/100S1 (F0)–10/10 DeepSeek V4.1 Flash10/1010/101S1 (F6)–10/10 Gemini 3.8 Flash8/107/102S1 (F7)–2/2 Gemini 3.8 Flash (low)8/108/101S1 (F0)10/108/10462/101/1004/102/100– Muse Spark 1.30/100/100– Nemotron 3 Nano Omni10/1010/100S1 (F5)1/91/901/101/1008/108/100– GPT-6 Astra5/102/100S1 (F18)– GPT-5.6 Sol7/105/100S1 (F0)– GPT-5.6 Terra6/105/100S3 (F12)– GPT-5.6 Luna5/105/100S1 (F9)2/102/1002/102/1006/106/1009/10 gpt-oss-120b10/1010/100S1 (F1)10/1010/1006/105/10010/1010/10110/10 Grok 4.63/103/100S1 (F22)– GLM-5.310/1010/102S1 (F2)–10/10 GLM-5.3 Flash10/1010/101S1 (F0)10/1010/10610/1010/1009/109/10010/10 53 Table 22: Card-reward skip rate by act on the main-table cohorts (seeds 1–10, A20). A decision is a fight survived; it is a skip when no card joined the deck.푛is the number of decisions in the act.Δis the model’s Act 2+ skip rate minus its Act 1 rate (pooled over its decisions), shown where it has at least 10 Act 2+ decisions. Humans are Mega Crit telemetry, A20 Ironclad, where a Singing Bowl pick also counts as a skip. The first- and random-option baselines never skip by construction; Bottled AI skips every card not on its strategy's wish list. ActsAct 1Act 2Act 3 Agent(mean)skip푛skip푛skip푛Δ LLM agents, deepest first Claude Fable 5.11.600.00740.08520.1224+0.09 GPT-6 Astra1.500.05770.32600.5735+0.36 Claude Opus 51.300.01810.20510.2015+0.18 GPT-5.6 Sol1.100.04790.44430.7311+0.47 Muse Spark 1.30.700.04740.14290.003+0.08 GLM-5.30.600.03710.1625–0+0.13 Gemini 3.8 Flash0.600.14850.57210.867+0.50 Claude Sonnet 50.500.01730.00170.001 −0.01 GPT-5.6 Terra0.500.04800.0010–0 −0.04 GLM-5.3 Flash0.400.02630.2512–0+0.23 Grok 4.60.300.59611.0011–0+0.41 DeepSeek V4.1 Flash0.300.00720.006–0– Qwen3.8 27B0.200.00640.258–0– Claude Haiku 4.50.100.54500.002–0– GPT-5.6 Luna0.100.06640.003–0– gpt-oss-120b0.100.02470.001–0– Nemotron 3 Nano Omni0.000.0033–0–0– Gemma 4 E4B0.000.0034–0–0– Qwen3.6 35B-A3B0.000.2752–0–0– Qwen3.5 122B-A10B0.000.1144–0–0– All LLM runs (pooled)0.490.091,2780.253510.4296+0.20 5 deepest LLMs (pooled)–0.033850.242350.3988+0.25 Human players (telemetry, own seeds) all A20 Ironclad–0.211,320,0530.39486,0150.53191,827+0.22 1000+ hours–0.1855,9420.3723,5580.5010,773+0.23 54 Table 23: Tool-call errors per 100 model calls, for the LLM agents of Table 1. No usable call: a response in which no tool call could be executed, because every call was malformed or refused or none could be parsed; the harness asks again. Refused call: one call refused before it reached the game, as illegal on the current screen or with invalid arguments; the model is told why. Energy mismatch: a turn ended with a different amount of energy left than the model declared when ending it, a sign that it lost track of its own turn. Recovered from prose: a call the model wrote as text rather than through the tool interface, which the harness recovered and executed. Errors sums the first three, which are the ones the model is told about. Rates are totals over the cohort divided by its calls. Lowest and second lowestin each column, among agents with all 10 seeds. AgentCalls/runNo usable callRefused callEnergy mismatchRecovered from proseErrors Qwen3.8 27B (xhigh)1240.240.000.730.000.97 Qwen3.6 35B-A3B (xhigh)1040.480.380.580.381.44 Qwen3.5 122B-A10B (xhigh)770.390.391.290.132.07 Claude Fable 5.1 (max)2680.000.150.220.000.37 Claude Opus 5 (max)3010.030.100.000.000.13 Claude Sonnet 5 (max)1590.060.060.060.000.19 Claude Haiku 4.5 (12k)1010.000.001.090.001.09 DeepSeek V4.1 Flash (max)1190.080.080.000.000.17 Gemini 3.8 Flash (high)2080.050.000.000.00 0.05 Gemma 4 E4B820.851.831.100.123.78 Muse Spark 1.3 (xhigh)2280.040.000.040.000.09 Nemotron 3 Nano Omni710.280.420.994.381.69 GPT-6 Astra (max)5460.020.220.020.000.26 GPT-5.6 Sol (max)3310.060.150.000.000.21 GPT-5.6 Terra (max)2300.000.130.000.000.13 GPT-5.6 Luna (max)1680.000.060.060.000.12 gpt-oss-120b (high)1750.230.001.778.642.00 Grok 4.6 (xhigh)1711.290.000.060.001.34 GLM-5.3 (max)1600.000.000.250.000.25 GLM-5.3 Flash (max)1070.090.000.190.000.28 Table 24: Excerpts from the runs of Table 1, verbatim. Names are as the model saw them under the reskin, with the real game term in brackets. Excerpts are taken from the full run logs and cut at the ellipsis. FailureModel (seed, floor)Excerpt Deliberating over a routine forecast Nemotron 3 Nano Omni (S3, 6) “We need to estimate probabilities. The player is at camp after sleeping, HP 24/75 (32%). Actually after sleeping they healed 22 HP, so HP should be 24+22 = 46? Wait initial HP 24/75, sleeping heals 22 . . . ” Reasoning and action disagree Grok 4.6 (S9, 10) “. . . I think Blood Temper [Armaments] is the better pick here. . . I'l go with Blood Temper.” Then skips the reward, writing beside the call: “Skip Pulse and Smash as filler. . . ” Miscounted turnGLM-5.3 (S5, 13)“I miscounted energy! . . . My error: I planned Trip(2) + Slash(1) + Slash(1) = 4E but only have 3E! Oops.” Blow-by-blow noteClaude Fable 5.1 (S10, 18) “CRIMSON WARLOCK [Chosen] solo 100 HP (Act 2 R18 opener), run 10 script: T1 Strong Debuff = Jinx [Hex] . . . → T2 Debuff = Fatigue [Weak] 3 on ME + Str 3 to itself . . . ” (280 words) Learnt, not written down Nemotron 3 Nano Omni (S3, 2) “. . . player is Exposed [Vulnerable], taking 50% more damage, so effective damage = 7 * 1.5 = 10.5, rounded down? Probably 10 or 11.” Misattributed act heal GPT-5.6 Terra (S5, end) “Wanderer's Pack [Tiny House] is an excellent low-HP boss reward: . . . +5 Max HP/current HP . . . In this run it converted 16/75 to 65/80 before the next Act-2 fight . . . ” Correct memory overwritten DeepSeek V4.1 Flash (S8, end) “KEY NEW LESSON (this run): . . . the post-boss heal is small — I entered the act at 11/75” (its log that run: +44 HP, 16→60) Rule from one runClaude Haiku 4.5 (S9, end) “Arriving at any combat-potential room (R13+) with<35% HP on thin deck without full Guard [Block] infrastructure is terminal. This run: R13 reached at 27/75 HP (36%) . . . ” 55 I Ablation results in full Figure 17 summarizes each ablation model by model. The subsections below then give each ablation in turn, pooled over models and model by model: the harness, reasoning effort, memory, the reskin, and material added to the system prompt. How each ablation was run is described in Appx. C.6. DefaultSingle action Single action, by index Contin- uous Orak 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Mean acts cleared per run n=7n=7n=7n=6n=7 n.s. n.s. n.s. n.s. Harness NoneKeep notes Consoli- date None (compact) Compact summary 0.0 0.2 0.4 0.6 0.8 Mean acts cleared per run n=8n=8n=8n=8n=8 n.s. n.s. n.s. Memory Real names ThemedRandomPermutedPermuted + themed 0.0 0.2 0.4 0.6 0.8 1.0 Mean acts cleared per run n=11n=11n=8n=8n=7 n.s. n.s. n.s. n.s. Reskin Figure 17: Each ablation, model by model. Each grey line is one LLM agent: its mean over the 10 seeds in each condition, shown only where it has all 10 seeds in both that condition and the condition’s baseline, so every comparison is paired. Bars are the mean over those agents, and푛beneath each bar is how many there are. Baselines are the first bar of each panel, except that the compact summary is read against no memory bank with compact notes. Panels show mean acts cleared per run; the pooled tables are Tables 2, 27 and 29. Brackets test each condition against its baseline with a paired sign-flip permutation test over agents (*푝<0.05; n.s. not significant). Orak is shown for the four models with all 10 of its seeds scored; its full paired comparison is in Table 2. I.1 Harness The pooled comparison is Table 2 in the main text; Table 25 gives it model by model. 56 Table 25: Effect of the harness, per model (supplementary to Table 2); Orak cells score runs killed by Orak's harness as a run lost at the floor it had reached; all harness variants were run without the reskin or a cross-run memory bank. All cells are on A20, with each model at its main-table reasoning setting (grey). Acts: mean number of acts cleared per run. Floor: mean final floor. A1: Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best andsecond best are per row, among cells with all 10 seeds. DefaultSingle actionSingle action, by indexContinuousOrak AgentActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1% Qwen3.8 27B (xhigh)0.4015.7400.3013.6300.3014.8300.20 16.3200.0010.00 Qwen3.6 35B-A3B (xhigh)0.008.600.008.300.008.500.007.500.009.40 Qwen3.5 122B-A10B (xhigh)0.00 10.600.008.500.108.8100.008.900.009.10 Gemini 3.8 Flash (low)0.3016.0300.4017.0300.3017.920 0.50 19.6 400.1013.510 Nemotron 3 Nano Omni0.009.200.005.200.006.800.007.200.004.20 gpt-oss-120b (high)0.30 15.3 300.0012.900.2013.7200.009.600.1010.010 Mean (6 agents)0.17 12.6 170.1210.9100.1511.8130.1211.5100.039.43 I.2 Reasoning effort One would expect the models to perform better when they can reason more before deciding their outputs. To investigate this, we ran the models at different reasoning effort levels. We ran all models with both the lowest and highest effort levels available, and some models at in between levels too. Table 26 tabulates the full results. Controlling for the model, we find that play performance is generally higher with higher reasoning effort, but not always (e.g. GPT-5.6 Luna peaks at high effort, with reduced performance at xhigh and max). The overall distribution is graphed in Fig. 18, which does not show a clear overall relationship between number of thinking tokens and performance when taken across models. This is because more performant models are more efficient with their reasoning tokens—the best model when thinking tokens are constrained is GPT-6 Astra with low reasoning effort. 57 0.05k0.1k0.2k0.5k1k2k5k10k20k50k Thinking tokens per floor (log scale) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Mean acts cleared per run Figure 18: Performance against thinking: mean acts cleared per run against thinking tokens per floor (log scale), for the same points as Fig. 2a. 58 Table 26: Effect of reasoning effort. Each model at every effort level it was run at, A20 on the held-out seeds with the main-table configuration (themed reskin, xml notes, read-write memory, generic guidance). Columns as in Table 1, with Acts the mean number of acts cleared per run; Act 4 (a Heart kill) is not shown but counts in Acts. Only levels with all 10 seeds completed are shown, and only models with two or more such levels. Bold marks the best level per model on each column. † Self-hosted; priced at Artificial Analysis headline rates. Act win rate (%)ActsFloorThinkingCost ($) AgentEffortSeeds123(mean)meanmedian[min–max](k/floor)/floor/run Qwen3.8 27Blow10 30000.30 15.216.0[7–24]7.8 0.045 † 0.74 † medium1020000.2012.810.5[6–24]9.30.055 † 0.76 † xhigh1020000.2014.916.0[5–29]18.10.091 † 1.45 † Qwen3.6 35B-A3Blow100000.008.77.0[4–16]4.10.083 † 0.83 † medium100000.0010.09.0[5–16]3.5 0.065 † 0.71 † xhigh100000.00 10.28.0[6–16]4.20.078 † 0.87 † Qwen3.5 122B-A10Blow1010000.109.49.0[4–18]3.80.074 † 0.78 † medium100000.009.57.0[4–16]3.80.077 † 0.76 † xhigh100000.009.89.0[7–16]3.7 0.072 † 0.77 † Claude Fable 5.1low10702000.9027.527.5[7–51]1.0 0.275 8.12 max10 90 50201.60 36.239.5[7–52]17.71.66461.91 Claude Opus 5low1070000.7022.722.0[10–33]0.9 0.204 4.84 max10 904001.30 32.533.0[7–51]21.31.74360.62 Claude Sonnet 5low1010000.1012.313.5[4–24]0.8 0.074 1.03 max10 401000.50 19.316.0[7–36]23.20.50910.70 Gemini 3.8 Flashlow10 60 1000.7022.622.0[5–50]2.5 0.063 1.56 high1050 1000.60 22.918.0[16–47]13.70.1172.89 GPT-6 Astralow1070 6001.3034.340.0[7–51]0.1 0.383 13.66 max10 9050101.50 37.040.0[5–55]3.30.83433.09 GPT-5.6 Sollow10501000.6021.818.5[13–50]0.5 0.133 3.15 max10 70 20101.10 27.826.0[7–57]4.00.2728.39 GPT-5.6 Terralow1010000.1012.313.0[5–20]0.6 0.060 0.81 max10 50000.50 18.218.0[10–25]5.00.1492.91 GPT-5.6 Lunalow100000.0010.88.5[6–16]0.4 0.005 0.06 medium1020000.2011.910.5[5–21]0.80.0070.09 high10 30000.30 14.516.0[6–25]1.60.0080.14 xhigh1020000.2014.414.5[5–24]2.70.0100.16 max1010000.1014.216.0[7–24]5.00.0150.24 gpt-oss-120blow100000.007.36.5[1–16]1.5 0.042 † 0.36 † medium100000.009.28.5[2–16]4.30.047 † 0.47 † high1010000.10 10.48.0[5–21]17.00.085 † 0.91 † GLM-5.3low1020000.2014.316.0[5–24]0.8 0.046 0.72 max10 60000.60 22.122.0[5–33]13.40.1673.92 GLM-5.3 Flashlow100000.0011.110.0[6–16]0.7 0.005 0.07 medium1020000.2012.913.5[7–20]2.70.0100.14 max10 40000.40 15.616.0[5–27]14.40.0240.40 59 I.3 Memory Table 27 pools the memory arms over models, and Table 28 gives them model by model. Figure 19 shows how the bank grows over a chain; whether the models learn from it is in Appx. E.4. 1510 0 50 Memories Qwen3.8 27B (xhigh) 1510 0 10 Qwen3.6 35B (xhigh) 1510 0 25 50 Qwen3.5 122B (xhigh) 1510 0 20 Gemini 3.8 Flash (low) 1510 Run 0 100 200 Bank size (k chars) 1510 Run 0 20 1510 Run 0 20 40 1510 Run 5 10 15 1510 0 25 50 Memories Nemotron 3 Nano 1510 0 20 GPT-5.6 Luna (max) 1510 0 50 gpt-oss-120b (high) 1510 0 50 GLM-5.3 Flash (max) 1510 Run 0 10 20 Bank size (k chars) 1510 Run 0 20 40 1510 Run 0 50 100 1510 Run 0 100 200 xml: memories heldxml: addedxml: deletedConsolidated (xml)Compact summaryNotes kept verbatim Figure 19: How the memory bank grows over a 10-run chain. For each model, top: the number of xml memories held after each run (line), with the memories added (blue, up) and deleted (grey, down) in that run’s consolidation. Bottom: the size of the bank carried into the next run, in thousands of characters, for xml consolidation, the single compact summary, and notes kept verbatim. Size is measured exactly, from the system prompt of the following run, relative to the first run’s empty bank; the bank after run 10 is never read, so that curve ends at run 9. 60 Table 27: Effect of cross-run memory. Within a run the model writes either keyed xml notes or a single compact summary; between runs they are consolidated into the memory bank (xml memories or one compact summary), kept verbatim (xml notes only), or discarded (None, the baseline for its kind of notes). All with the themed reskin and generic guidance. Pooled over models: each method is compared with its baseline only on matched runs (the same model on the same seed; Pairs, over Agents models), and Acts, Floor and A1 (mean acts cleared, mean final floor and Act 1 win rate) are over those runs. Usage is compared on like-for-like floors only: the same model, seed, act and room type, and the same occurrence of that room type within the act, kept only where every method run on that model and seed completed it (Floors), so a method is neither credited with cheap early floors nor charged for expensive late ones. Calls and k tokens (input, including cached, plus output, in thousands) are means per floor; the token difference is the ratio of totals. EachΔis the method minus its baseline. All runs are A20 on the held-out seeds, each model at its main-table reasoning setting. Models: Qwen3.8 27B, Qwen3.6 35B-A3B, Qwen3.5 122B-A10B, Gemini 3.8 Flash (low), Nemotron 3 Nano Omni, GPT-5.6 Luna, gpt-oss-120b, GLM-5.3 Flash. Per-model results are in Table 28. Performance (matched runs)Usage per floor (matched floors) MethodAgentsPairsActsΔFloorΔA1%ΔFloorsCallsΔk tokensΔ XML notes Consolidate8800.19+0.0013.1+0.518 −16478.9+0.6245+36% Keep notes8800.25+0.0614.1+1.624+56478.6+0.4297+65% None (baseline)8800.19–12.5–19–6478.2–180– Compact notes Consolidate8800.19+0.0112.9+0.619+16477.9+0.3175+7% None (baseline)8800.17–12.3–18–6477.6–163– Table 28: Effect of cross-run memory, per model (supplementary to Table 27). Within a run the model writes either keyed xml notes or a single compact summary; between runs they are consolidated into the memory bank (xml memories or one compact summary), kept verbatim (xml notes only), or discarded (None). All with the themed reskin and generic guidance. All cells are A20 on the held-out seeds, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run, Floor the mean final floor, and A1 the Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best and second bestare per row, among cells with all 10 seeds. XML notesCompact notes ConsolidateKeep notesNoneConsolidateNone AgentActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1% Qwen3.8 27B (xhigh)0.2014.9200.3014.2 30 0.3013.6 300.1013.2100.2015.020 Qwen3.6 35B-A3B (xhigh)0.00 10.200.0010.100.008.600.008.400.008.00 Qwen3.5 122B-A10B (xhigh)0.009.800.109.6100.008.300.007.500.006.80 Gemini 3.8 Flash (low)0.70 22.6 600.2015.3200.6022.3600.4017.6400.4017.140 Nemotron 3 Nano Omni0.006.800.007.700.108.1100.008.100.008.10 GPT-5.6 Luna (max)0.1014.2100.3017.4200.1012.910 0.60 21.1 600.2014.520 gpt-oss-120b (high)0.1010.410 0.40 16.3 400.2012.0200.0011.800.1011.410 GLM-5.3 Flash (max)0.4015.640 0.70 22.2 700.2014.4200.4015.9400.5017.650 Mean (8 agents)0.1913.118 0.25 14.1 240.1912.5190.1913.0190.1812.318 61 Table 29: Effect of the knowledge reskin. None shows the real game names and is the baseline; Themed is our default Lich's Lair reskin; Random replaces every name with pronounceable nonsense; Permuted swaps each name for another real game name of the same kind; Perm.+themed applies the same permutation in the themed vocabulary. All with xml notes, a read-write memory bank and generic guidance. Pooled over models: each method is compared with its baseline only on matched runs (the same model on the same seed; Pairs, over Agents models), and Acts, Floor and A1 (mean acts cleared, mean final floor and Act 1 win rate) are over those runs. Usage is compared on like-for-like floors only: the same model, seed, act and room type, and the same occurrence of that room type within the act, kept only where every method run on that model and seed completed it (Floors), so a method is neither credited with cheap early floors nor charged for expensive late ones. Calls and k tokens (input, including cached, plus output, in thousands) are means per floor; the token difference is the ratio of totals. EachΔis the method minus its baseline. All runs are A20 on the held-out seeds, each model at its main-table reasoning setting. Models, by method: Themed: Qwen3.8 27B, Qwen3.6 35B-A3B, Qwen3.5 122B-A10B, Claude Haiku 4.5, DeepSeek V4.1 Flash, Gemini 3.8 Flash, Gemini 3.8 Flash (low), Nemotron 3 Nano Omni, GPT-5.6 Luna, gpt-oss-120b, GLM-5.3, GLM-5.3 Flash; Random, Permuted and Perm.+themed: Qwen3.8 27B, Qwen3.6 35B-A3B, Qwen3.5 122B-A10B, Gemini 3.8 Flash (low), Nemotron 3 Nano Omni, GPT-5.6 Luna, gpt-oss-120b, GLM-5.3 Flash. Per-model results are in Table 30. Performance (matched runs)Usage per floor (matched floors) MethodAgentsPairsActsΔFloorΔA1%ΔFloorsCallsΔk tokensΔ None (baseline)121150.37–16.1–33–11217.7–229– Themed121150.23 −0.1414.5 −1.622 −1111217.8+0.1238+4% Random8800.25 −0.0113.9 −0.121 −46328.5+0.2234+13% Permuted8800.16 −0.1012.3 −1.715 −106328.6+0.3233+13% Perm.+themed8750.15 −0.1212.4 −1.715 −115898.8+0.4223+9% I.4 Knowledge reskin Table 29 pools the five mappings over models (examples in Table 4), and Table 30 gives them model by model. 62 Table 30: Effect of the knowledge reskin, per model (supplementary to Table 29). None shows the real game names; Themed is our default Lich's Lair reskin; Random replaces every name with pronounceable nonsense; Permuted swaps each name for another real game name of the same kind; Perm.+themed applies the same permutation in the themed vocabulary. All with xml notes, a read-write memory bank and generic guidance. All cells are A20 on the held-out seeds, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run, Floor the mean final floor, and A1 the Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best andsecond bestare per row, among cells with all 10 seeds. NoneThemedRandomPermutedPerm.+themed AgentActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1%ActsFloorA1% Qwen3.8 27B (xhigh)0.2014.0200.2014.9200.30 15.9 300.2014.7200.40 5 19.8 5 40 5 Qwen3.6 35B-A3B (xhigh)0.009.700.0010.200.009.500.009.500.00 11.00 Qwen3.5 122B-A10B (xhigh)0.10 10.9100.009.800.007.600.009.200.008.00 Claude Haiku 4.5 (12k)0.1011.410 0.10 11.9 10– DeepSeek V4.1 Flash (max) 0.80 22.0 700.3016.930– Gemini 3.8 Flash (high)1.20 5 31.0 5 80 5 0.6022.950– Gemini 3.8 Flash (low)0.5020.2400.7022.660 0.80 24.1 600.4017.6300.3016.430 Nemotron 3 Nano Omni0.005.300.006.800.008.300.007.400.007.00 GPT-5.6 Luna (max)0.6018.5600.1014.210 0.60 19.6500.4015.0400.1012.910 gpt-oss-120b (high)0.0010.400.1010.4100.00 10.700.009.200.009.80 GLM-5.3 (max)0.70 24.4 600.6022.160– GLM-5.3 Flash (max)0.70 23.1 700.4015.6400.3015.3300.3016.1300.5018.050 Mean (7 agents)0.27 14.0 260.1912.8170.2413.6200.1612.0140.1311.913 I.5 Prompt material Table 31: Effect of adding material to the system prompt. Reskin is the themed reskin with nothing added and is the baseline; Bestiary adds the enemy compendium, Reddit guide the Reddit high-ascension strategy guide and Jorbs the Jorbs transcript set, each renamed by the reskin (Appx. C.6). All with xml notes, no memory bank and generic guidance. Pooled over models: each method is compared with its baseline only on matched runs (the same model on the same seed; Pairs, over Agents models), and Acts, Floor and A1 (mean acts cleared, mean final floor and Act 1 win rate) are over those runs. Usage is compared on like-for-like floors only: the same model, seed, act and room type, and the same occurrence of that room type within the act, kept only where every method run on that model and seed completed it (Floors), so a method is neither credited with cheap early floors nor charged for expensive late ones. Calls and k tokens (input, including cached, plus output, in thousands) are means per floor; the token difference is the ratio of totals. EachΔis the method minus its baseline. All runs are A20 on the 10 evaluation seeds, each model at its main-table reasoning setting. Per-model results are in Table 32. Performance (matched runs)Usage per floor (matched floors) MethodAgentsPairsActsΔFloorΔA1%ΔFloorsCallsΔk tokensΔ Reskin (baseline)8790.23–14.2–23–6318.4–182– Bestiary8710.15 −0.0413.1 −0.614 −65378.7 −0.1259+41% Reddit guide8790.18 −0.0512.4 −1.814 −96318.0 −0.4237+30% Jorbs transcripts5500.22 −0.0613.7 −2.320 −84258.5 −0.6940+379% Table 31 pools the material added to the system prompt over models (described in Appx. C.6), and Table 32 gives it model by model. 63 Table 32: What the system prompt carries, per model, on the 10 evaluation seeds (disjoint from those of Table 1). The baseline is our themed reskin; each other arm adds one thing to the reskinned prompt: the enemy compendium (Bestiary), the Reddit high-ascension strategy guide, or the Jorbs transcript set (his tutorial series and six hours on the Ironclad, 7 h in all; Appx. C.6). Guides and transcripts are renamed by the reskin. All arms take xml notes within a run, write no memory bank, and use generic guidance, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run and Floor the mean final floor. A superscript gives the seed count where fewer than 10 seeds completed, and – marks an arm with no valid runs. Best and second bestare per row, among cells with all 10 seeds. Pooled over models in Table 31. BaselineAdded to the prompt ReskinBestiaryReddit guideJorbs AgentActsFloorActsFloorActsFloorActsFloor Qwen3.8 27B (xhigh)0.3015.10.50 19.20.1013.30.3015.0 Qwen3.6 35B-A3B (xhigh)0.00 11.00.0010.60.109.60.007.3 Qwen3.5 122B-A10B (xhigh)0.009.70.007.60.008.3– Gemini 3.8 Flash (low)0.90 26.20.5021.30.8022.90.8024.8 Nemotron 3 Nano Omni0.006.30.005.80.005.7– GPT-5.6 Luna (max)0.1015.20.1015.20.30 15.60.0010.2 gpt-oss-120b (high)0.10 12.40.0011.70.0010.80.0011.3 GLM-5.3 Flash (max)0.4017.50.00 1 16.0 1 0.11 9 13.0 9 – Mean (5 agents)0.28 16.00.2215.60.2614.40.2213.7 J Cost and reasoning This appendix gives the cost frontiers in full, then what a run costs in time and money, as a guide to budgeting a new cohort. How costs are priced is described in Appx. C.4. For performance against reasoning budget, see Fig. 18. J.1 Cost Figure 2a (in the main text) names only the points on its front. Figure 20 repeats it with every point named, and Fig. 21 gives the front against cost per floor, which ranks models on efficiency rather than on what a whole run costs. 64 $0.02$0.05$0.1$0.2$0.5$1$2$5$10$20$50$100 Mean cost per run (USD, log scale) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Mean acts cleared per run Figure 20: Figure 2a with every point named, with its reasoning-effort level and, where it has fewer than 10 seeds, its seed count. 65 $0.005$0.01$0.02$0.05$0.1$0.2$0.5$1$2 Mean cost per floor (USD, log scale) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Mean acts cleared per run Figure 21: Performance against cost per floor: mean acts cleared per run against mean cost per floor (log scale), for the same points as Fig. 2a. † Self-hosted; priced at Artificial Analysis headline rates. ‡ Peak-rate list price, an upper bound. 66 J.2 Time and cost of a run Table 33 gives what a run costs in wall-clock time, money, tokens and mechanical errors, as a guide to budgeting a new cohort. A cohort's seeds are played in series against one game instance, so its wall-clock time is the sum of its runs'. Table 33: What a run costs, for the agents of Table 1: wall-clock minutes per seed (one game instance, seeds in series), dollars per run and per 10-seed cohort (priced as in Table 1), model calls and tokens per run (input including cached, plus output, in thousands); tool-call errors are in Table 23. Time depends on the provider's throughput on the day as well as on the model. Minutes/seedCost ($)Per run AgentRunsmedianmean/run/10 seedsCallsk tokens Pick first option1022– Pick random option1022– BottledAI w/o Heart1022– BottledAI w/ Heart1022– Qwen3.8 27B (xhigh)10991071.45151244,641 Qwen3.6 35B-A3B (xhigh)1030310.8791042,036 Qwen3.5 122B-A10B (xhigh)1013140.778771,601 Claude Fable 5.1 (max)1023122961.9161926824,604 Claude Opus 5 (max)1023121460.6260630144,517 Claude Sonnet 5 (max)1010715710.7010715914,122 Claude Haiku 4.5 (12k)1021311.23121013,395 DeepSeek V4.1 Flash (max)1023240.3531194,256 Gemini 3.8 Flash (high)1044622.89292088,382 Gemma 4 E4B1017610.030821,349 Muse Spark 1.3 (xhigh)1052502.93292286,972 Nemotron 3 Nano Omni101301350.545711,595 GPT-6 Astra (max)1010816533.0933154620,032 GPT-5.6 Sol (max)1055588.398433110,435 GPT-5.6 Terra (max)1035342.91292305,667 GPT-5.6 Luna (max)1021490.2421684,384 gpt-oss-120b (high)1084870.9191756,498 Grok 4.6 (xhigh)10582055.68571714,649 GLM-5.3 (max)101191213.92391607,194 GLM-5.3 Flash (max)101031180.4041074,641 67