Paper deep dive
$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution
Muyu He, Adit Jain, Anand Kumar, Vincent Tu, Soumyadeep Bakshi, Sachin Patro, Nazneen Rajani
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/2/2026, 3:34:26 AM
Summary
YC-Bench is a new long-term coherence benchmark for LLM agents, simulating a one-year startup operation. It evaluates agents on their ability to manage resources, infer adversarial client behavior, and maintain strategic planning over hundreds of turns in a partially observable environment. Results show that only a few frontier models achieve profitability, with scratchpad usage being a critical factor for success.
Entities (5)
Relation Signals (3)
YC-Bench â evaluates â AI Agent
confidence 100% · a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup
Claude Opus 4.6 â performedbeston â YC-Bench
confidence 95% · Claude Opus 4.6 achieving the highest average final funds at $1.27 M
Scratchpad â predicts â SUCCESS
confidence 90% · Scratchpad usage... is the strongest predictor of success
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce $\texttt{YC-Bench}$, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of \$200K, with Claude Opus 4.6 achieving the highest average final funds at \$1.27 M, followed by GLM-5 at \$1.21 M at 11$\times$ lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for $47\%$ of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. $\texttt{YC-Bench}$ is open-source, reproducible, and configurable.
Tags
Links
- Source: https://arxiv.org/abs/2604.01212v1
- Canonical: https://arxiv.org/abs/2604.01212v1
Trouble viewing inline? Open PDF directly â
Full Text
50,663 characters extracted from source content.
Expand or collapse full text
YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution Muyu He* Adit Jain* Anand Kumar* Vincent Tu* Soumyadeep Bakshi Sachin Patro Nazneen Rajani As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduceYC-Bench, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open-source, across 3 seeds each. Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27M, followed by GLM-5 at $1.21M with 11Ălower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for 47% of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance.YC-Benchis open-source, reproducible, and configurable at https://github.com/collinear-ai/yc-bench. Correspondence: research@collinear.ai Date: April 2, 2026 1 Introduction LLM-based agents have become increasingly capable in their ability to interact with environments and use tools. Because tasks may span 100+ tool calls, itâs crucial for the agent to know how to plan well. As such, a growing body of work has focused on planning such as PlanBench (Valmeekam et al., 2023) and BALROG (Paglieri et al., 2025). One key attribute that emerges in long-horizon planning is coherence: over the course of hundreds or even thousands of interactions/steps, the agent must stay aligned with its goal, retain crucial facts or knowledge from its past, and avoid collapse into repetitive or hallucinated behavior. Existing works such as Vending-Bench (VB) (Backlund & Petersson, 2025) and Vending-Bench 2, have aimed to test the ability of frontier models to maintain long-term coherence. These works have made meaningful progress in uncovering interesting planning failure modes and behaviors in frontier models, such as âmeltdownâ looping, hallucinations about non-existent facts, and even legal threats. They also have real-world randomness, including missed deliveries and unsatisfied customers. However, their environmental attributes have immediate tangible ramifications. For example, if the price is set incorrectly there, the agent observes a drop in sales the next day. In a real-world setting, running a business involves taking calculated risks without knowing the exact outcome immediately, and it involves taking actions that are suboptimal in the short term, in the hope of better long-horizon rewards. Further, if the current strategy is flawed and doesnât deliver maximum profits, the business must improve based on experience and adapt to a changing market landscape. * indicates equal contribution 1 arXiv:2604.01212v1 [cs.CL] 1 Apr 2026 To fill this gap, we introduceYC-Bench, a long-term coherence benchmark that evaluates an LLM agentâs ability to run a simulated startup over a one-year horizon spanning hundreds of turns. The agent operates through a CLI tool interface, deciding each turn which contracts to accept from a marketplace, which employees to assign based on their skill profiles, and how to manage cash flow against recurring monthly payroll. Our environment is adversarial: roughly a third of clients are untrustworthy, and their tasks are designed to fail. The agent must learn to identify these clients by analyzing its own history of successes and failures, then avoid them going forward. This is another key distinction with VB, where the adversaries are known before commitment (e.g, adversarial suppliers). The benchmark tests long-term coherence through a 20-turn context window that forces the agent to use a persistent scratchpad for memory: agents that fail to record which clients are adversarial will repeat costly mistakes after their conversation history is truncated. Compounding dynamics reward sustained good decisions â trust builds with repeated clients, reducing future workload, while over-staffing tasks inflates payroll costs over time. Performance is measured by final company funds at year end, a single scalar that reflects the cumulative impact of hundreds of sequential decisions around task selection, resource allocation, and risk management. Our contributions can be summarized as follows : 1.We introduceYC-Bench, a POMDP-based benchmark with deterministic but unknown transition and observation dynamics. As illustrated in Figure 1, the LLM agent must plan and manage the day-to-day operations of a startup â accepting tasks from diverse clients and domains, and assigning them to employees with varying skill sets. YC-Bench is designed to evaluate long-term planning and adaptive execution under delayed, sparse rewards. 2. In our main experimental results, we extensively test on frontier models including GPT-5.4, GPT-5.4 Mini and Nano (OpenAI, 2026), Claude Opus 4.6 (Anthropic, 2026a), Claude Sonnet 4.6 (Anthropic, 2026b), Gemini 3.1 Pro Preview, Gemini 3 Flash, Gemini 3.1 Flash Lite (Google DeepMind, 2025), Grok 4.20-beta as well as popular open-source models like Qwen3.5-397B-A17B (Qwen Team, 2026), GLM-5 (GLM-5 Team, 2026), Kimi-K2.5 (Kimi Team, 2026). Our main results are summarized in Section 4.2 with Figure 1 containing the averaged temporal runs. 3.Through our error analysis of Section 4.2 and Section 5, we observe that most model go bankrupt and there is still a significant gap in frontier models. Furthermore, we observe most models are unable to be profitable, tending to fall into adversarial client traps or under-staffing. There is also a significant gap in cost-efficiency with open-source models being more Pareto optimal than their counterparts. In summary, we introduceYC-Bench, a long-term coherence benchmark that evaluates an agentâs ability to simulate running a startup. Concretely, our benchmark tests an agentâs ability to allocate resources in a complex organization LLM Agent Observation Your Company (YC) Employees Bank Balance Prestige 200k Start + on success, - on failure 8 Sta Skills + Salary grow 4 domains Grows on success Action Role: You are the CEO of Bench Co. Goal: Maximize funds Manage employees Grow Prestige Commands: market browse, task accept, task inspect, sim resume, scratchpad write, ... Scratchpad Avoid Vanguard (inflate work). Focus Cortex (trust 4.2). Emp_4 best at research. ... YC- Bench Goal: Maximize Funds over 1 Yr Market Tasks Clients Rewards, Deadline, Client Gated by Prestige & Trust 35% Adversarial Secretly increase work Trust Grows with success Decays over time market browse id-42, $21k, research ... id-56, $14k, inference ... task accept status: planned, deadline: 9th Jan task assign assigned to: emp-4 & emp-6 sim resume task_complete funds +, prestige + monthly payroll scratchpad write Updated scratchpad Figure 1 Overview ofYC-Bench. The agent interacts with the environment through CLI commands (blue) and receives structured observations (green). The environment tracks observable state (tasks, employees, finance, prestige, client trust) and one hidden element: adversarial clients whose work inflation must be inferred from repeated task failures. 2 under uncertainty, manage risk, and adapt to adversarial dynamics over long horizons. 2 Related Work YC-Bench sits at the intersection of several active research areas: LLM agent benchmarks, long-horizon planning, simulation-based evaluation, tool use and memory architectures. LLM Agent Benchmarks. The rapid growth of agentic capabilities of LLMs has spawned a rich ecosystem of benchmarks. AgentBench (Liu et al., 2024) evaluates LLMs across eight distinct environments, including operating systems and games; and finds that poor long-term reasoning is the primary bottleneck for open-source models. SWE- bench (Jimenez et al., 2024) grounds evaluation in real GitHub issues, requiring agents to navigate codebases and generate correct patches; however, each task is an independent, bounded problem rather than a sustained sequential process. GAIA (Mialon et al., 2024) tests general assistant capabilities across 466 real-world questions requiring tool use and multi-modal reasoning. AgentBoard (Ma et al., 2024) introduces the âProgress Rateâ metric that captures incremental advancement rather than binary success, a design philosophy shared byYC-Bench, which tracks revenue trajectories and prestige progression rather than pass/fail outcomes. TheAgentCompany (Xu et al., 2024) embeds agents as digital workers in a simulated software company, finding that the best agent completes only 30% of professional tasks. Both shareYC-Benchâs emphasis on domain-grounded, consequential decision-making, but neither evaluates the sustained strategic coherence over hundreds of compounding turns that YC-Bench requires. Long-Horizon Planning and Reasoning. Effective performance onYC-Benchrequires agents to maintain strategic coherence over hundreds of turns, a capability that pushes beyond the horizons tested by most existing work. Plan- Bench (Valmeekam et al., 2023) evaluates LLMs on classical planning domains (Blocksworld, Logistics), finding that even frontier models fall well short on plan generation and verification, providing evidence that apparent planning capabilities may be retrieval rather than genuine reasoning, a distinction thatYC-Benchâs novel simulation setting, immune to pretraining contamination, is well-positioned to test. BALROG (Paglieri et al., 2025) is also closely related, evaluating LLMs across six procedurally generated game environments spanning easy to extremely challenging sequen- tial tasks. Its key finding, that current frontier models still struggle significantly with longer-horizon tasks, directly motivates YC-Benchâs focus on sustained decision-making. Most directly related to our work is Vending Bench (VB) (Backlund & Petersson, 2025), which served as a key inspiration forYC-Bench. VB tests whether LLM agents can maintain coherent behavior while operating a simulated vending machine business, and finds that all models exhibit failure loops from which they rarely recover.YC-Bench substantially extends this paradigm by introducing multi-domain task allocation, hidden employee skill rates that create an information-asymmetry puzzle, compounding financial dynamics (prestige decay, salary growth), and a multi-episode learning framework that tests whether agents can improve across restarts.YC-Benchdifferentiates itself through its deterministic business-simulation setting and its status as an open-source, extensible, and configurable benchmark. Simulation Environments and Strategic Games.YC-Benchbelongs to a growing family of simulation-based benchmarks that test AI decision-making in complex, multi-turn environments. Generative Agents (Park et al., 2023) introduced LLM-powered agents that simulate believable human behavior in a sandbox, using memory storage, reflection, and dynamic retrieval for planning; their architecture for persistent agent memory directly informsYC-Benchâs scratchpad-based memory carryover between episodes. Cicero (, FAIR) demonstrated that LLMs can handle negotiation and long-term planning under imperfect information in Diplomacy, capabilities analogous to those tested inYC-Benchâs information-asymmetric setting where agents must infer hidden employee skill rates from task outcomes. CivRealm (Qi et al., 2024) provides a Civilization-based benchmark requiring resource management and diplomacy over extended horizons, but both RL- and LLM-based agents struggle with the full gameâs complexity, motivating benchmarks like YC-Bench that isolate specific aspects of strategic decision-making in a controlled, deterministic setting. 3 YC-Bench This section describes theYC-Benchbenchmark, which we mathematically formalize as a Partially Observable Markov Decision Process (POMDP) and describe the different components and dynamics involved in the benchmark including Tasks, Employees, Clients, Memory and Simulation Clock. 3 Environment Definition. We formalizeYC-Benchas a Partially Observable Markov Decision Process (POMDP) M = (S,O,A,T , R), whereSis the state space,Ois the observation space,Ais the action space,T :SĂAâSis the deterministic transition function, andR :SĂAâRis the reward function defined as the net change in company funds. The agent operates a company over a one-year simulated horizon, starting with $200,000 in funds. At each turnt, the agent receives a partial observationo t âOand selects a sequence of actionsa t âA â , after which the environment transitions deterministically tos t+1 = T (s t ,a t ). The agent controls time progression explicitly by issuing asim resume command; between time advances, it may issue arbitrarily many actions within a single turn. The episode terminates when funds drop below zero (bankruptcy) or the horizon ends. The objective is to maximize final funds f T . We now describe the core mechanics of the environment that determine agent performance. At a high level, the agent must balance competing pressures: completing tasks to generate revenue while avoiding adversarial clients that inflate work requirements, building domain prestige to unlock higher-reward tasks, and managing a payroll that grows with every successful completion. The full agent action space is in Appendix A. Tasks and domains. The agentâs sole source of income is completing tasks accepted from a marketplace. Each task belongs to one of four industry domainsâtraining, inference, research, and data engineeringâand is issued by a client. A task comes with a predetermined reward, a deadline that activates upon acceptance, and a work quantity in that domain that employees must complete before the deadline. Accepting a task may require a minimum prestige level in its domain and a minimum trust level with its client. The agent maintains a prestige level in each domain: higher prestige unlocks higher-reward tasks and linearly scales their payout. Completing a task grants both funds and a prestige increase in its domain, creating a progression dynamic where early task selection determines which high-value tasks become accessible later. If the agent fails to complete a task by its deadline, it incurs a penalty of 35% of the advertised reward and a prestige reduction, incentivizing accurate estimation of completion time relative to employee capacity. Employees and productivity. The company has a fixed roster of staff members that cannot be hired or fired. Each employee has a per-domain productivity level, the quantity of work they complete per hour in that domain. The agent can query productivity levels directly viaemployee list. Each employee also has a tier label (junior, mid, or senior) reflecting their average productivity, but productivity distributions are spiky: a senior-tier employee may have senior-level throughput in training but junior-level throughput in research. Effective employee assignment therefore requires the agent to match employees to tasks based on their domain-specific strengths rather than relying on tier alone. Upon successful task completion, assigned employees receive a productivity boost, a percentage increase to their productivity in the taskâs domain, capped at a maximum rate. This makes employees more productive over time, rewarding consistent domain-specialized assignment. However, each completion also triggers a salary bump, a fixed raise based on the employeeâs tier. The monthly payroll therefore grows monotonically with the number of completed tasks, creating pressure to prioritize high-reward tasks that outpace rising costs. Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec $0K $200K $400K $600K $800K $1.00M $1.20M Net Worth Starting Funds ($200K) Claude Opus 4.6 GLM-5 GPT-5.4 Kimi-K2.5 Gemini 3 Flash Gemini 3.1 Flash Lite GPT-5.4 Mini Claude Sonnet 4.6 Qwen 3.5-397B Gemini 3.1 Pro GPT-5.4 Nano Grok 4.20 Greedy Bot $394K $409K $1.00M $1.21M $1.27M Figure 2 Out of the12models that we benchmark onYC-Bench,5models are profitable and only3turn a substantial profit (5Ă profit). The figure plots the funds across time averaged across three seeds for each model. Comprehensive results can be found in Appendix D. 4 010203040506070 Trust-required tasks / successful tasks Opus 4.6 GLM-5 GPT-5.4 Kimi K2.5 Gemini Flash Gemini Flash Lite GPT-5.4-Mini Sonnet 4.6 Qwen 3.5 Gemini Pro GPT-5.4-Nano Grok 4.20 Do models complete tasks that require trust? (a) Proportion of successfully completed tasks that require client trust to complete. Error bars show per-seed range across 3 seeds. Vanguard ML Equinox Labs Cortex Intelligence Prism Analytics Helix Systems Stratos Cloud Opus 4.6 GLM-5 GPT-5.4 Kimi K2.5 Gemini Flash Gemini Flash Lite GPT-5.4-Mini Sonnet 4.6 Qwen 3.5 Gemini Pro GPT-5.4-Nano Grok 4.20 0.00.02.24.62.10.8 0.00.23.62.42.43.6 0.00.03.32.72.92.9 0.20.01.71.61.42.0 0.00.43.33.62.52.0 0.00.02.01.63.02.5 0.00.01.30.11.21.1 0.00.01.51.21.90.0 0.00.01.51.00.70.8 0.50.01.51.52.21.0 0.00.01.52.02.50.7 0.30.01.61.72.51.5 ADVADV How much trust do models build with clients? 0 1 2 3 4 5 Avg final trust level (b) Final trust level for different models per client averaged across seeds (ADV=Adversarial). Figure 3 We observe that better models are able to build client trust over time by strategically selecting clients. What is surprising is smaller distilled models (Sonnet-4.6) do worse than contemporaries (Gemini-3-Flash) unlike VB. Clients. Each task is issued by one of several clients. The agent builds trust with a client by completing its tasks successfully; higher trust reduces the work required on future tasks from that client and unlocks higher-tier tasks. However, completing a task for one client slightly decays trust with all others, so the agent must choose which client relationships to invest in. Adversarial clients. A subset of clients are adversarial: after the agent accepts one of their tasks, the environment inflates the work quantity, making the deadline nearly impossible to meet. Adversarial status is hidden and never revealed directly. Crucially, adversarial clients offer competitively high rewards, so the agent cannot simply filter by price. The agent must instead infer which clients are adversarial from the pattern of repeated task failures. Observations. At the start of each turn, the agent receives a structured status summary containing the current timestamp, funds, monthly payroll, runway estimate, active task count, and a list of events since the previous turn (task completions, fund changes, salary bumps, deadline margins). All other state information (employee skill tiers, market tasks, client trust levels, per-task progress) must be actively queried through observe actions. The key hidden quantity is client reliability levels, which must be inferred from the pattern of task outcomes. Simulation Clock. The agent controls the passage of simulated time explicitly. Callingsim resumeadvances the clock to the next scheduled event, typically a task checkpoint (25%, 50%, 75%, or 100% completion), a monthly payroll deduction, or the horizon end. Between events, the agent may issue arbitrarily many actions to reassess strategy, accept new tasks, or reassign employees. Work progresses only during business hours (weekdays), and payroll is deducted on the first business day of each month. Memory. The agentâs conversation history is truncated to the most recentKturns (we useK=20in our experiments). To maintain long-term coherence, the agent may write to a persistentscratchpadthat is injected into the system prompt on every turn. The scratchpad is the agentâs sole mechanism for retaining information across context truncation, for example recording which clients are unreliable, which employees are strongest in which domains, or strategic rules derived from past performance. This design does not bias toward any particular memory strategy; instead, it tests whether agents can autonomously determine what information is worth persisting. 4 Experiments 4.1 Experimental Setup Models We evaluate 12 models spanning seven providers: GPT-5.4, GPT-5.4-Mini, and GPT-5.4-Nano (OpenAI); Claude Opus 4.6 and Claude Sonnet 4.6 (Anthropic); Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 3.1 Flash Lite 5 (Google); Qwen 3.5-397B (Alibaba); GLM-5 (Zhipu AI); Kimi-K2.5 (Moonshot AI); and Grok 4.20 (xAI). Each model is evaluated on 3 seeds using thedefaultpreset, for a total of 36 LLM runs. We use theLiteLLMframework for inference and OpenRouter as the model provider. Full configuration hyperparameters are listed in Appendix B and the agent system prompt can be found in Appendix C. Baseline We compare against a greedy baseline that, in each turn, accepts the highest-reward task available on the market, assigns all employees to it, and advances the simulation clock. This baseline does not check client history, specialize employees by domain, nor does it use the scratchpad. An LLM-based agent should be able to surpass the greedy baseline by intelligently selecting the right tasks from non-adversarial clients and the correct employees for that task over the long-term. 4.2 Main Results YC-Benchreveals large performance gaps between frontier models that score similarly on standard benchmarks. As shown in Figure 1, of the twelve models evaluated, only three (Claude Opus 4.6, GLM-5, and GPT-5.4) exceed $1,000,000 in average final funds, 2â3Ăhigher than the next-best model. Only five models turn a profit on average; the remaining seven finish below their $200,000 starting capital, and several go bankrupt in at least one seed. Figure 1 shows that the divergence emerges by FebruaryâMarch, roughly 60 days into the simulation. Top models concentrate on one or two clients early, triggering a trust snowball: each success reduces future task workloads (up to 50%), enabling more completions per month, which builds further trust. Models that spread work across many clients never reach meaningful work reduction and enter a payroll-driven decline. Only a few models can stick to improving client trust in the long term; most models choose clients indiscriminately. Figure 3a shows the ratio of completed tasks that require the model to gain non-zero trust level with the client that issues the task. A model is motivated to choose tasks that require trust if they are accessible because they come with both higher rewards and smaller workloads. Despite the clear motivation to selectively focus on a particular subset of clients, most models do not build trust consistently with clients. This is seen in Figure 3a, where we see most models maintain a minimal trust level (1-2) with most clients, barring themselves from access to tasks with higher returns. Analysis of the model scratchpads in Figure 4b shows that only 4 out of 10 models across 6 out of 30 runs explicitly maintain a "whitelist" of clients to work with. The rest of the runs distribute tasks among clients discriminately: âAvoid onlyâ shows that the model only maintains a list of clients not to work with; âSelect + Avoidâ shows that the model in addition maintains a list of clients to focus on; âNo policyâ means no list is maintained. Identifying adversarial clients in the market remains a challenge for all but a few models. Figure 4a computes the average ratio of adversarial client tasks in all tasks accepted by each model across seeds. Half of all the models accept adversarial tasks at a rate higher than the natural market share of those tasks, showing that these models either are 020406080100 Adversarial tasks / all tasks Opus 4.6 GLM-5 GPT-5.4 Kimi K2.5 Gemini Flash Gemini Flash Lite GPT-5.4-Mini Sonnet 4.6 Qwen 3.5 Gemini Pro GPT-5.4-Nano Grok 4.20 market ratio (~32%) How often do models accept adversarial tasks? (a) Average ratio of adversarial client tasks among all accepted tasks. Error bars show per-seed range. Dashed line indicates the natural market share of adversarial tasks ( 32%). Seed 1Seed 2Seed 3 Opus 4.6 GLM-5 GPT-5.4 Kimi K2.5 Gemini Flash Gemini Flash Lite GPT-5.4-Mini Sonnet 4.6 Qwen 3.5 Gemini Pro GPT-5.4-Nano Grok 4.20 Do models maintain a client selection policy? Select + Avoid Avoid only No policy (b) Client selection policy as observed in agent scratchpads for each of the 3 seeds. Figure 4 Analyzing how the models deal with adversarial clients, who have appealing rewards when accepting a task but have a lot more work than claimed (scope creep). 6 020406080100 % of failures Opus 4.6 GLM-5 GPT-5.4 Kimi K2.5 Gemini Flash Gemini Flash Lite GPT-5.4-Mini Sonnet 4.6 Qwen 3.5 Gemini Pro GPT-5.4-Nano Grok 4.20 Why do models fail on tasks? Failed to identify adversarial clients Assigned incapable employees Over-committed employees to tasks (a) Failure mode breakdown. Models miss task deadlines due to (1) failing to identify adversarial clients; (2) assigning incapable employees; or (3) over-committed employees to tasks. 0.000.250.500.751.001.251.501.75 In-game revenue / API cost 1e6 Kimi K2.5 Gemini Flash Lite Gemini Flash GLM-5 GPT-5.4-Nano Qwen 3.5 GPT-5.4 GPT-5.4-Mini Opus 4.6 Gemini Pro Grok 4.20 Sonnet 4.6 How cost-efficient are models? (b) Cost efficiency measured as revenue per dollar of API cost. Error bars show per-seed range. Figure 5 Other than identifying adversarial clients there are two other primary task failure modes, especially wrong employee assignment. We observe that Kimi-K.5 is the most cost efficient, whereas the second highest ranking model, GLM-5 is substantially (10Ă) better than Opus-4.6 on cost efficiency. indifferent to the adversarial clients or come to prefer them over good clients by misjudgment. As shown in Figure 4b, two-thirds of all runs make no mention of âblacklistingâ any adversarial client in their strategy, leading them to choose clients indiscriminately. However, as Figure 4a also reveals, the three top-performing models on average strategically accept adversarial tasks significantly less frequently than the baseline, at 1/4 the rate of the next best model. As the scratchpad reveals, the top-performing models correctly spot the increase in work quantity before and after accepting the problematic tasks and subsequently write explicit guidelines to never accept tasks from those clients. Despite occasional inconsistencies in following the guidelines, these instructions effectively prevent excessive task failures caused by these adversarial tasks. Even the top-performing models still suffer from repercussions of suboptimal employee assignment decisions. Figure 5a breaks down all failed tasks for each model across runs by the cause of failure. Although failure to identify adversarial clients is still the most common pitfall by design, 7 out of 11 models have a substantial portion of failure attributed to suboptimal employee assignment. Specifically, they either assign employees whose productivity can never theoretically complete the task in time or spread employeesâ effort across multiple tasks so that their throughput on each task cannot meet the minimal requirement. When the model assigns employees to tasks, they have perfect information about the amount of work in each task and the employeeâs productivity relevant to its domains. As a result, the failure stems from not properly estimating the speed of completion against the deadline and not taking into account the fact that working on multiple tasks slows each employee down. Models drastically differ in cost-efficiency, with medium-performing models such as Kimi-K2.5 striking a superior balance between cost and revenue over top-performing ones. Figure 5b calculates how much in-game revenue in million dollars each model generate per dollar in API costs.Kimi-K2.5achieves superior performance over all other models, having a2.5Ăperformance gap with the next most cost-effective model,Gemini-3-Flash. The three top-performing models significantly lag behind in terms of cost-effectiveness, especially the strongest model Claude-Opus-4.6, despite yielding3Ăthe return of the most cost-effective model. The ranking has practical weight for production because it shows how much actual return is to be expected from each model if we take the inference cost into account. Refer to Appendix F for summary statistics on token usage, time, and cost. 5 Error Analysis We analyze four models to understand the behavioral mechanisms behind their performance, focusing on their error modes. The brackets are (Final Revenue, Number of Bankruptcies). 7 Claude Opus 4.6 ($1.27M avg, 0/3 bankrupt). Opus actively uses the scratchpad to strategize, rewriting itâŒ34 times per run across four topics: calibrating environment mechanics from observed completions, following a one-task-at-a- time workflow, building a client blacklist, and tracking per-client success rates to optimize trust-gated task selection. These topics emerge in a consistent order: calibration dominates the firstâŒ10 turns, workflow rules solidify by Turn âŒ20, adversarial clients are flagged by TurnâŒ25, and trust-based optimization continues for the remainder of the run. It also inspects every task before acceptance (âŒ155 calls per run). Despite this, Opus is not flawless: in Seed 2 ($750K), it accepts a task from a blacklisted client in August and accumulates 9 legitimate failures in the second half as higher-prestige tasks prove harder to complete. Gemini 3 Flash ($394K avg, 0/3 bankrupt). Flash averages 2 scratchpad entries per run, rarely inspects tasks, and executes an identical 4-command cycle every turn: accept, assign all 8 employees, dispatch, resume. This lack of adaptation is costly: Flash accepts 12 adversarial tasks across the run, all of which fail, capping its earnings well below the top tier. Its 0/3 bankruptcy rate reflects sufficient throughput to absorb these losses, not the ability to avoid them. Claude Sonnet 4.6 ($103K avg, 2/3 bankrupt). Sonnet exhibits a reasoningâexecution gap: it derives correct strategies but fails to act on them. At Turn 7 in Seed 1, it writes a correct feasibility formula to the scratchpad (ârequired_qty / total_rate must be<deadline hoursâ) and a âone task at a timeâ rule. It then ignores both: Turn 8 accepts four tasks without inspection, and over the full run it averages 7.23 concurrent active tasks (max 16). Of its 17 failed tasks, 41% result from understaffing (tasks that would succeed with all 8 employees assigned), 18% from assigning employees poorly matched to the task domain, and 12% from re-accepting adversarial clients without ever recording the pattern in its scratchpad. Sonnet averages 8 scratchpad entries per run but stops updating early, leaving stale financial data ($152K) while the actual balance drops toâ$3K. Grok 4.20 ($14K avg, 2/3 bankrupt). Grok shows aware inaction: its scratchpad accurately identifies critical issues (âRunway down to 1 month,â âAvoid Equinoxâ) but these observations do not translate into changed behavior: it accepts a task from Vanguard ML (0% historical success rate) with 6 days of runway remaining. It also does not manage its active task portfolio, leaving one task accepted in March uncompleted and uncancelled for 81 days until bankruptcy. Grok averages 0.92 commands per turn, the lowest of any model evaluated (Table 3), suggesting it spends most turns on observation and deliberation rather than task execution. Overall, the four failure profiles reveal a spectrum of long-term incoherence. Flash fails entirely due to the absence of reflection. Grok fails despite accurate reflection, unable to close the loop between diagnosis and action. Sonnet fails from reflection that is structurally correct but temporally inconsistent, ie, rules written and immediately abandoned. Only Opus achieves sustained, self-correcting reflection, though even it is not immune to occasional blacklist violations. This spectrum suggests that long-horizon coherence is not a single capability but a pipeline: perceiveârecordâ retrieveâ act consistently, and current models fail at different stages of that pipeline. 6 Conclusion In this work, we presentYC-Bench, a simulation-based benchmark that comprehensively evaluates LLMsâ long-term coherence, planning, and consistent execution abilities. The benchmark introduces design components that can only be tackled by a model excelling in long-term optimization: evolving client trust relationships, disruption by adversarial clients, and compounding financial pressure from company growth. YC-Benchreveals that long-horizon coherence remains a critical, under-tested capability: only 3 of 12 frontier models grew their starting capital, and scratchpad usage was the strongest single predictor of success. Adversarial client detection, accounting for47%of bankruptcies, exposed a failure mode that benchmarks with immediate environmental feedback cannot surface. Our error analysis reveals a recurring reasoningâexecution gap as they derive correct strategies but consistently fail to act on them, suggesting that deliberation and execution are not yet unified capabilities in current frontier models. Limitations and Future Work. The current environment holds several simplifying assumptions worth relaxing in future iterations: employees cannot be hired or fired, disruptions are limited to adversarial clients rather than random exogenous events, and all quantitative signals (work quantities, productivity rates) are provided numerically rather than in natural language. Introducing these elements would create a richer, more realistic decision surface and likely surface additional failure modes in frontier models. We will releaseYC-Benchas an open-source, configurable benchmark and invite the community to stress-test future models against its compounding, adversarial dynamics. 8 ! "#$%&'()*'+,)-./0112345674689:;8< =44>?<@&$AB)5=CD "'&,&"+B'EB)@- F0#$)5=CD=5=5628 G0+H87 5I6<C78H5I=C6J<I K6C7=5H;000 L0 MNOPQRSTUVWXYNZV[TZ\]^_XS`a UVbcZ`UWVd %+&B@-,=CDeF/fg=<J3=9Khi,=CDeFjF f)k36<?lh000 m nop! q5=CD6<C78H5,=CDeFFr q5=CD=H875,=CDeF/GfH468<5- RSTUVWXsZtuh n vwxyzon | |~ ! |~Â| ÂÂÂon Â  ÂÂ||  ~  p  || ~ÂÂx |~Â|  |~ xÂÂ~ mÂÂÂÂz Âp  non! %?H3C?<"?958l&<58446J8<H8=<KÂ96C2 +<=4Â56HC5?Â364K593C50+4:=ÂC =C6J<=44>8274?Â88C5?;6J;e Â=438 5=CDC0 fF:96585?5=40$?=KÂ89C=96=4 6K8<56Â6H=56?<0h m nonÂnÂ! q5=CD=H875,=CDeFjFfH468<5- RSTUVWXsZtuh q5=CD=C6J<e8274?Â88C+B> q5=CDK6C7=5H; qC6298C32 8 n ~ ~ v ||  |~ ÂÂ~ ~ Â|   v  ~ ~ Â~ÂwÂ~|ÂÂ~   ~ ~ ~ÂÂÂ~  xÂÂÂÂv~ Â! ! Â"'&,&"+B)@@#$@ F0 MsÂMÂÂUVuÂÂÂ``Zu ]ÂZ]cUVÂu tÂbW\ÂZÂÂÂ`ÂĄ G098k3698KÂąk5ÂÂŁ5?5=4Âą9=58€;?39C <88K8K0 Â"B&)$,@ eg=<J3=9K(B-Â982632i98C8=9H;00 0 fK8CH96756Â8?<4Âi<?:=9<6<JCh m no „nŠ! q5=CD=H875,=CDeGL q5=CD=H875,=CDeF>1 f/5=CD6<C78H5H=44Ch n vw yzonÂp Â! vwÂyzon  Â! Â|§ ÂÂÂ~ ~~|Â~  Ì~ÂÂwÂÂ~ Â  ÂÂ~ ÂÂ~ÂÂÂ~ m~v x ÂÂÂÂv~ Â! ©x! ÂMÂȘ«P«ÂÂȘTVÂźZa]WÂźV`W Ì°WV`±Zb`Â\ ÂZa ÂĄ ,=CDeÂČ/6C=H56Â8Â35K8=K46<8:=C(=9 Fr=<KH3998<56CÂł3<G0&5:644 46D84ÂÂ=640$88K5?H? 2748582?98 ;6J;e98:=9K"?958l5=CDC+@+Â0 M ÌWU]VÂÂź`Zu ubW ^ÎŒ ` `dÂĄ m no ! q5=CD=H875,=CDeFF>fH468<5- NZV[TZ\]¶sh f/·;6C5?96H=4C3H8CChf ̧K=ÂC93<:= 982=6<6<Jh n ÂÂv~  Â~ vwx ÂÂ~  ! ~|Â|ÂÂ~~  zÂ~ ÂÂ|~~|ÂÂ| ÂÂv~  ÂÂ~|| ÂÂÂxÂw || | Figure 6 Representative failure moments for four models. Each panel shows the scratchpad state, agent action, and outcome. From left to right: Opus violates its own blacklist; Flash blindly accepts adversarial tasks through a rigid loop; Sonnet writes a correct rule and immediately ignores it; Grok diagnoses its bankruptcy but takes no corrective action. References Anthropic.Claudeopus4.6systemcard.https://w-cdn.anthropic.com/ 14e4fb01875d2a69f646fa5e574dea2b1c0f7b5.pdf, 2026a. Accessed: 2026-03-23. Anthropic. Claude sonnet 4.6 system card.https://anthropic.com/claude-sonnet-4-6-system-card, 2026b. Accessed: 2026-03-23. Axel Backlund and Lukas Petersson. Vending-bench: A benchmark for long-term coherence of autonomous agents. arXiv preprint arXiv:2502.15840, 2025. Meta Fundamental AI Research Diplomacy Team (FAIR)â , Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067â1074, 2022. GLM-5 Team. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Google DeepMind.Gemini 3 pro model card.https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf, 2025. Accessed: 2026-03-23. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. Kimi Team. Kimi k2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), 2024. arXiv:2308.03688. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In Advances in Neural Information Processing Systems (NeurIPS), Oral, Datasets and Benchmarks Track, 2024. arXiv:2401.13178. GrĂ©goire Mialon, ClĂ©mentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. In International Conference on Learning Representations (ICLR), 2024. arXiv:2311.12983. OpenAI. Introducing gpt-5.4. https://openai.com/index/introducing-gpt-5-4/, 2026. Accessed: 2026-03-23. 9 Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kucinski, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktaschel. BALROG: Bench- marking agentic LLM and VLM reasoning on games. In International Conference on Learning Representations (ICLR), 2025. arXiv:2411.13543. Joon Sung Park, Joseph C. OâBrien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium on User Interface Software and Technology (UIST), 2023. doi: 10.1145/3586183.3606763. arXiv:2304.03442. Siyuan Qi, Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, Nian Liu, Wei Wang, Yaodong Yang, and Song-Chun Zhu. CivRealm: A learning and reasoning odyssey in Civilization for decision-making agents. In International Conference on Learning Representations (ICLR), 2024. arXiv:2401.10568. Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, 2026. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2206.10498. Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024. 10 Appendix A Action Space The complete list of CLI commands the model can use inYC-Benchis provided in Table 1. Together they define the possible action space for the model. CategoryCommandEffect Observe company statusFunds, prestige, payroll Observe employee listNames, tiers, salaries, productivity Observe market browseAvailable tasks with client, reward, domains Observe task listAccepted tasks with status and progress Observe task inspect -task-id TPer-domain progress, deadline, assignments Observe client listClient trust levels and tiers Observe client historyPer-client success/failure counts Observe finance ledgerFull transaction history Task task accept -task-id TAccept from market; starts deadline Task task assign -task-id T -employees EAssign employees to task Task task dispatch -task-id TBegin work on assigned task Task task cancel -task-id T -reason RAbandon task; prestige penalty Sim sim resumeAdvance clock to next event Memory scratchpad write -content COverwrite persistent notes Memory scratchpad append -content CAppend to persistent notes Table 1 Agent action space. All actions are CLI commands with arguments constructed as free-form text. Multiple actions may be issued per turn. B Default Configuration for YC-Bench The default configuration parameter for each variable in the environment is provided in Table 2. We selected this configuration because it is adequately difficult without being unfair or unnecessarily difficult for frontier models. C System Prompt The system prompt for each evaluation run is provided as follows. The model can access the system prompt in the chat history at all times during each turn. You are the CEO of a startup in a business simulation. Maximize funds and prestige while avoiding bankruptcy. All actions use`yc-bench` CLI commands via`run_command`. All return JSON. ## Core Workflow (repeat every turn) ** You must always have active tasks running. Every turn, follow this loop: ** 1.`yc-bench market browse` - pick a task 2.`yc-bench task accept --task-id Task-42` - accept it 3.`yc-bench task assign --task-id Task-42 --employees Emp_1,Emp_4,Emp_7` - assign employees (check`employee list` for skill rates) 4.`yc-bench task dispatch --task-id Task-42` - start work 5.`yc-bench sim resume` - advance to next event (requires active tasks) 11 CategoryParameterValueDescription SimulationHorizon1 yrEvaluation period SimulationAuto-advance5 turnsIdle turns before forced advance SimulationBusiness hours9â18Weekdays only WorkforceEmployees8Fixed roster WorkforceInitial funds$200KStarting capital WorkforceSalary bump1%/taskAll assigned employees MarketMarket tasks200Initial pool MarketBrowse limit50Visible per query MarketClients6Total count PrestigeRange[1, 10]Per domain PrestigeDecay0/dayNo passive loss PrestigeReward scale0.30Reward multiplier PrestigeReq. prestigeTri(1,5,1)Task requirement dist. DeadlinesQty/day150For deadline calc DeadlinesMin days7Minimum deadline DeadlinesFail penalty35%Of advertised reward DeadlinesCancel penalty 1.5Ă ÎŽPrestige lost TrustMax5.0Per client ceiling TrustBuild rate5.0âŒ5 tasks to significant TrustWork reduction50%At max trust TrustGated fraction30%Tasks requiring trust TrustFocus pressure0.3Cross-client decay AdversarialFraction35%Adversarial clients AdversarialScope creepâ„ 3.0ĂWork inflation floor Salary TiersJunior (50%)$2â4K/moRate 1â4 units/hr Salary TiersMid (35%)$6â8K/moRate 4â7 units/hr Salary TiersSenior (15%)$10â15K/moRate 7â10 units/hr DistributionsTask rewardTri($2K,$12K,$5K)Per task DistributionsDomain count1Domains per task DistributionsWork qtyTri(400,1500,800)Units per domain MemoryContext window20 turnsBefore truncation MemoryScratchpadPersistentIn system prompt Table 2 Default configuration hyperparameters for YC-Bench. 12 Run multiple tasks concurrently when possible. Accept -> assign -> dispatch a second task before calling sim resume. ** Use`yc-bench scratchpad write` ** to save strategy notes - your conversation history is truncated after 20 turns, but scratchpad persists in the system prompt. Write reusable rules, not one-off observations. ## Commands ### Observe -`yc-bench company status` - funds, prestige, payroll -`yc-bench employee list` - employees with skill rates per domain -`yc-bench market browse [--domain X] [--reward-min-cents N] [--limit N]` - available tasks -`yc-bench task list [--status X]` - your tasks -`yc-bench task inspect --task-id Task-42` - task details -`yc-bench client list` - clients with trust levels -`yc-bench client history` - per-client success/failure rates -`yc-bench finance ledger` - financial history ### Act -`yc-bench task accept --task-id Task-42` - accept from market -`yc-bench task assign --task-id Task-42 --employees Emp_1,Emp_4,Emp_7` - assign employees (comma-separated) -`yc-bench task dispatch --task-id Task-42` - start work (must assign first) -`yc-bench task cancel --task-id Task-42 --reason "text"` - cancel (prestige penalty) -`yc-bench sim resume` - advance time -`yc-bench scratchpad write --content "text"` - save notes -`yc-bench scratchpad append --content "text"` - append notes ## Key Mechanics - ** Salary bumps ** : completed tasks raise salary for every assigned employee. More employees assigned = higher payroll growth. - ** Throughput split ** : employees on multiple active tasks split their rate (rate/N). Two tasks run at 50% each. - ** Deadlines ** : success before deadline = reward + prestige. Failure = prestige penalty, no reward. - ** Trust ** : completing tasks for a client builds trust -> less work per task, access to gated tasks. Working for one client erodes trust with others. - ** Not all clients are reliable. ** Check`client history` for failure patterns. - ** Payroll ** : deducted monthly. Funds < 0 = bankruptcy. - Prestige grows per domain. Higher prestige unlocks better-paying tasks. Listing 1 Agent system prompt. D Main Results & Tool-Usage Statistics Figure 7 shows the average revenue of each model across all three seeds, with the horizontal line showing the starting fund. Figure 8 shows the revenue progression of all models per seed. The relative performance difference across models is consistent across seeds. Figure 9 and Table 3 further break down specific behavioral patterns by models. 13 Claude Opus 4.6 GLM-5 GPT-5.4 Kimi-K2.5 Gemini 3 Flash Gemini 3.1 Flash Lite GPT-5.4 Mini Claude Sonnet 4.6 Qwen 3.5-397B Gemini 3.1 Pro GPT-5.4 Nano Grok 4.20 Greedy Bot $0K $200K $400K $600K $800K $1.00M $1.20M Net Worth $1.27M $1.21M $1.00M $401K $394K $192K $137K $103K $68K $61K $34K $14K $0 Starting Funds ($200K) Figure 7 Average final funds across three seeds for all 12 models and the greedy baseline, sorted by performance. Only three models (Claude Opus 4.6, GLM-5, GPT-5.4) consistently surpass the starting capital. Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Month $0K $200K $400K $600K $800K $1.00M $1.20M $1.40M $1.60M $1.80M $2.00M $2.20M Net Worth Seed 1 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Month Seed 2 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Month Seed 3 Claude Opus 4.6 GLM-5 GPT-5.4 Kimi-K2.5 Gemini 3 Flash Gemini 3.1 Flash Lite GPT-5.4 Mini Claude Sonnet 4.6 Qwen 3.5 Gemini 3.1 Pro GPT-5.4 Nano Grok 4.20 Greedy Bot Figure 8 Monthly funds trajectory across three seeds. 14 ModelSP/100TInspect/AcceptAvg Conc.Cmd/Turn Claude Opus 4.65.61.102.01.41 GLM-52.70.261.52.16 GPT-5.410.61.432.52.63 Kimi-K2.50.60.001.31.18 Gemini 3 Flash0.20.111.20.24 Gemini 3.1 Flash Lite0.90.021.22.16 GPT-5.4 Mini0.20.221.00.04 Claude Sonnet 4.64.60.647.22.01 Qwen 3.5-397B0.70.012.51.00 Gemini 3.1 Pro0.00.001.11.06 GPT-5.4 Nano0.00.081.20.08 Grok 4.20 Beta0.40.651.50.92 Table 3 Behavioral statistics per model, averaged across 3 seeds and sorted by average final funds. The top three models use the scratchpad and task inspection significantly more than others. Sonnetâs average concurrency of 7.2 is a clear outlier, consistent with its high understaffing failure rate. GPT-5.4 Mini and Nano execute fewer than 0.1 commands per turn despite running thousands of turns, indicating most turns produce reasoning without action. SP/100T : scratchpad writes per 100 turns. Inspect/Accept: ratio of task inspecttotask acceptcalls. Avg Conc.: mean simultaneously active tasks. Cmd/Turn: commands executed per turn. task accept task assign task dispatch task inspect market browse sim resume task list client list employee list scratchpad write 0 20 40 60 80 100 120 140 160 Average Tool Calls per Run Claude Opus 4.6 GLM-5 GPT-5.4 Grok 4.20 GPT-5.4 Nano Gemini 3.1 Pro Figure 9 Average number of tool calls per run for the top three and bottom three performing models, grouped by command type. Top models execute 3-4x more actions overall, with the largest gaps in task inspection and scratchpad usage. E Ablation: Context Window Size We vary the number of retained conversation turnsK â 20, 30, 40, 50across four models and three seeds each (Figure 10). Three of four models peak atK = 30, with GPT-5.4 Mini improving by 123% and Gemini 3 Flash by 27%, before degrading sharply at higher values. GPT-5.4 performs best at the defaultK = 20: atK = 30, it issues fewer commands per turn and never learns to bundle accept-assign-dispatch into a single action, reducing task throughput by over 60%. All four models degrade beyond their optimalK, suggesting that larger context windows reduce per-turn action efficiency faster than they improve decision quality. F Summary of Token Usage, Cost, and Time Table 4 summarizes the API cost, number of total tokens, and runtime in minutes of each model. 15 20304050 Context Window Size (K turns) $0K $200K $400K $600K $800K $1.00M Net Worth Starting Funds ($200K) GPT-5.4 Gemini 3 Flash GPT-5.4 Mini GPT-5.4 Nano Figure 10 Average final funds vs. K (number of retained turns), across three seeds. ModelBankruptCost ($)TokensRuntime (min) Claude Opus 4.60/386.07± 41.1716.7M70± 36 GLM-50/37.62± 1.068.3M40± 7 GPT-5.40/323.08± 7.079.5M19± 5 Kimi-K2.51/31.79± 2.743.8M21± 30 Gemini 3 Flash0/31.83± 0.323.8M6± 1 Gemini 3.1 Flash Lite1/30.59± 0.222.6M6± 2 GPT-5.4 Mini2/31.34± 1.831.8M3± 4 Claude Sonnet 4.62/318.06± 18.875.8M23± 24 Qwen 3.5-397B2/31.16± 0.792.8M13± 8 Gemini 3.1 Pro2/37.36± 8.182.8M7± 6 GPT-5.4 Nano2/30.41± 0.202.0M3± 1 Grok 4.202/35.03± 2.402.9M4± 2 Table 4 Computational cost per run, averaged across three seeds (Ό± Ï). Models are sorted by average final funds. 16