Paper deep dive
Code Owns the Simulation, Jev Owns the Evaluation
Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 1:55:22 AM
Summary
The paper investigates the capabilities of the judgment model Jev, finding a sharp boundary between 'evaluation' (judging options based on input) and 'simulation' (predicting outcomes not in the input). Jev excels at evaluation (99% on Cognitive Reflection Tests) but fails at simulation tasks like one-shot games and ALFWorld when it must predict opponent actions or future states in a single call. However, when code performs the simulation (e.g., via lookahead or physics engines) and supplies the result, Jev becomes an expert controller. The core contribution is the 'Code Owns the Simulation, Jev Owns the Evaluation' paradigm.
Entities (19)
Relation Signals (20)
Yaodong Yang → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Preprint Code Owns the Simulation, Jev Owns the Evaluation Yaodong Yang 1
Yi Ma → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Yaodong Yang 1 Hongyao Tang 2 Yi Ma 3
Tianpei Yang → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Jinpeng Li 5 Tianpei Yang 6
Jinpeng Li → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Weixun Wang 4 Jinpeng Li 5
Weixun Wang → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Xingyu Fan 1 Weixun Wang 4
Xingyu Fan → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Yi Ma 3 Xingyu Fan 1
Hongyao Tang → authored → Code Owns the Simulation, Jev Owns the Evaluation
confidence 95% · Preprint Code Owns the Simulation, Jev Owns the Evaluation Yaodong Yang 1 Hongyao Tang 2
Jev → failsat → Simulation
confidence 95% · However, it fails when the right option depends on simulation (i.e., predicting something not in the input)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Judgment models such as \jev{} return, in a single call and without reasoning text, a probability for each described option. This makes them attractive as an agent's action-selection layer, but it is unclear which decisions they can be trusted with. We test \jev{} on reflection tests, one-shot matrix games, the text game ALFWorld and robot control, and find a sharp boundary. \jev{} succeeds when the right option can be judged from what the input describes, which we call \emph{evaluation}. Specifically, it solves 99\% of the counterintuitive Cognitive Reflection Test questions. However, it fails when the right option depends on \emph{simulation} (i.e., predicting something not in the input), such as the opponent's action or the subgoal that must come first. In games, \jev{} plays suboptimally as if its rational opponent acted at random, because the opponent's action is not given. In ALFWorld, \jev{} favors commands that mention an object or place named in the task description. For example, given the task ``put a clean knife in the drawer'', \jev{} carries an unwashed knife straight to the drawer instead of first washing it at the sink. Surprisingly, many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, \jev{} usually answers correctly, and it responds well given the opponent's action. It fails when one call must both perform the simulation and evaluate based on it. This suggests letting code make the prediction or simulation. When code supplies it, such as a lookahead in ALFWorld and physics simulation in robot control, \jev{} becomes an expert controller through its general evaluation ability.
Tags
Links
- Source: https://arxiv.org/abs/2610.01834v1
- Canonical: https://arxiv.org/abs/2610.01834v1
Trouble viewing inline? Open PDF directly →
Full Text
76,495 characters extracted from source content.
Expand or collapse full text
Preprint Code Owns the Simulation, Jev Owns the Evaluation Yaodong Yang 1 Hongyao Tang 2 Yi Ma 3 Xingyu Fan 1 Weixun Wang 4 Jinpeng Li 5 Tianpei Yang 6 1 The Chinese University of Hong Kong 2 Tianjin University 3 Shanxi University 4 Independent Researcher 5 CAIR, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences 6 Nanjing University yydapple@gmail.com, tanghongyao@tju.edu.cn, mayi@sxu.edu.cn, xyfan@link.cuhk.edu.hk, wxwang@tju.edu.cn, jinpeng.li@cair-cas.org.hk, tianpei.yang@nju.edu.cn Abstract Judgment models such as Jev return, in a single call and without reasoning text, a probability for each described option. This makes them attractive as an agent’s action-selection layer, but it is unclear which decisions they can be trusted with. We test Jev on reflection tests, one-shot matrix games, the text game ALFWorld and robot control, and find a sharp boundary. Jev succeeds when the right option can be judged from what the input describes, which we call evaluation. Specifically, it solves 99% of the counterintuitive Cognitive Reflection Test questions. However, it fails when the right option depends on simulation (i.e., predicting something not in the input), such as the opponent’s action or the subgoal that must come first. In games, Jev plays suboptimally as if its rational opponent acted at random, because the opponent’s action is not given. In ALFWorld, Jev favors commands that mention an object or place named in the task description. For example, given the task “put a clean knife in the drawer”, Jev carries an unwashed knife straight to the drawer instead of first washing it at the sink. Surprisingly, many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, Jev usually answers correctly, and it responds well given the opponent’s action. It fails when one call must both perform the simulation and evaluate based on it. This suggests letting code make the prediction or simulation. When code supplies it, such as a lookahead in ALFWorld and physics simulation in robot control, Jev becomes an expert controller through its general evaluation ability. 1 Introduction Many decisions of an agent are choices within a short list of options: the next command in a text game, the next move in a game or the next skill of a robot. Judgment models such as Jev, built as a general “System One”, target exactly this kind of choice (TypeSafe, 2026). In a single call and without reasoning text, Jev returns a probability for each described option. It needs no task-specific training and no output parsing. This makes Jev attractive as an agent’s action-selection layer, called at every step as a fast policy over listed actions. But it is unclear which kind of decisions such a model can be trusted with. Benchmark scores do not provide the answer. For example, Jev alone solves 31% of ALFWorld games, but this number says how often it fails, not at which decisions. Two common views of such models do not provide the answer either. The first view comes from the “System One” label in the dual-process theory of Kahneman (2011) and Evans & Stanovich (2013), which predicts failure wherever an intuitive option is wrong. Yet Jev solves 99% of Cognitive Reflection Test questions, which are built around an intuitive but wrong answer (Frederick, 2005; Hagendorff et al., 2023). The second view holds that small models lack reasoning. Yet when asked to predict the opponent’s action in a matrix game, Jev is usually right. We test Jev on reflection tests, one-shot matrix games, the text game ALFWorld (Shridhar et al., 2021) and robot control, and find a sharp boundary. Specifically, we find that Jev succeeds when the right option can be judged from what the input describes, which we call evaluation. For example, in a CRT question, a bat and a ball cost $1.10 in total, and the bat costs $1.00 more. By checking the listed prices against the question, Jev selects $0.05 for the ball and avoids the intuitive $0.10. 1 arXiv:2610.01834v1 [cs.AI] 1 Oct 2026 Preprint (a) Jev alone Task: put a clean butterknife in drawer. Just done: take butterknife 2 (not washed). 42 candidates. OptionJev probability go to drawer 1 lure: names the destination 0.60 go to sinkbasin 1 correct: wash it first 0.17 examine butterknife 2 0.09 (b) code owns the simulation Same decision. Code runs each command in a forked copy of the game and adds its outcome to the option. OptionJev probability go to drawer 1 → “The drawer 1 is closed.” Then: open drawer 1. [already seen] 0.07 go to sinkbasin 1 → “You see a lettuce 3, . . . ” Then: clean butterknife 2 with sinkbasin 1. [already seen] 0.83 examine butterknife 2 → “There’s nothing special about butterknife 2.” <0.01 No prediction needed Prediction needed, surface shortcut wrong Code supplies prediction 0.0 0.5 1.0 Accuracy 1.00 0.33 0.98 One-shot games (3×3 matrix games) No prediction needed Prediction needed, surface shortcut wrong Code supplies prediction 0.0 0.5 1.0 Accuracy 0.93 0.36 0.79 ALFWorld decisions (seen split, 116 games) Jev alone + obs. (v1) + cmds (v2) Word- overlap 0.0 0.5 1.0 Fraction solved 0.31 0.59 0.87 0.13 ALFWorld games (unseen split, 134 games) Figure 1: Jev judges what its input describes. It fails when a prediction must be made first, and recovers when code supplies it. Top: one ALFWorld decision in a kitchen scene. (a) Jev sees only the commands and picks the drawer, which the task mentions. (b) Code runs every command in a copy of the game and adds the result to the option, which reveals why the sink matters. Bottom: accuracy in one-shot games (left) and at ALFWorld steps (middle), and ALFWorld games solved out of 134 as the lookahead grows (right). Seen and unseen are two evaluation splits defined by ALFWorld. The middle panel tests single decisions: at each step of the 116 seen games that ALFWorld’s hand-coded expert solves, Jev chooses the next command given the expert’s history so far. The dashed line in the left panel marks chance (1/3). The hatched bar replaces Jev with a simple rule that picks the option sharing the most words with the task, given the same lookahead. However, Jev fails when the right option depends on simulation (i.e., predicting something not in the input), such as the opponent’s action or the subgoal that must come first. In RL terms, Jev behaves like a policy without a world model. It scores actions by the current input and does not look ahead. In games, Jev plays suboptimally as if its rational opponent acted at random, because the opponent’s action is not given. In ALFWorld, we observe that Jev favors commands that mention an object or place named in the task description. Its accuracy is 99% when such a command is correct, but 36% when such commands are wrong. For example, given the task “put a clean butterknife in drawer”, Jev carries an unwashed butterknife straight to the drawer instead of first washing it at the sink (Figure 1a). Surprisingly, we find that many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, Jev usually answers correctly (90%), and it plays the best response 98% of the time given the opponent’s action. We therefore trace the failure to a single call that must both perform the simulation and evaluate based on it. This leads us to let code make the prediction or simulation. When code supplies it, Jev becomes an expert controller through its general evaluation ability. In one-shot games, stating the opponent’s action raises accuracy from 33% to 98% when the action with the highest average payoff is not the best action against a rational opponent. Even when the opponent’s action is not supplied, Jev can predict it in one call. Given this prediction in a second call, Jev plays the best action in 94% of cases. Similarly, with a lookahead in a copy of ALFWorld, solved games rise from 31% to 87% (Figure 1, bottom right). Finally, in robot control, Jev never picks up a box when it chooses raw torques. In contrast, with skills for which code simulates the physical dynamics, it matches a hand-written 2 Preprint controller on the native task and further generalizes to new task variants (22 of 40 episodes against 3). Our contributions are as follows. • Diagnosis. Across reflection tests, one-shot games, ALFWorld and robot control, we find a sharp boundary. Jev succeeds at evaluation but follows surface shortcuts when the right option depends on simulation (Sections 3–6). • Cause. We show that many of these failures are not due to a lack of knowledge. Jev fails when one call must both perform the simulation and evaluate based on it. Separating simulation and evaluation recovers from these failures (Sections 4.3 and 5.2). • Remedy. We propose that code owns the simulation and Jev the evaluation. With this divi- sion, Jev becomes an expert controller in games, ALFWorld and robot control (Sections 4.3, 5.3 and 6). 2 Evaluation, Simulation, and How We Label Decisions This section defines evaluation and simulation and explains how we label each decision. Setting. Jev (version jev-1.13.0) takes a JSON state and a list of candidate options, each with a text description (TypeSafe, 2026). It returns the chosen option, a probability for every option and a confidence, but no reasoning text. We use it zero-shot, with one fixed instruction per condition (Appendix A). A decision is a triple (x,A,a ⋆ ), where x is the state, A the set of described candidate options and a ⋆ ⊆ A the correct ones. Definition 1 (Evaluation and simulation). A decision needs only evaluation if a ⋆ can be judged from what x and the candidate options describe. On the other hand, a decision depends on simulation if finding a ⋆ requires a prediction z with information not in the input. The prediction can be shared by all candidate options, like the opponent’s action or the next subgoal. It can also belong to one candidate option, like the observation that follows it. If code supplies z by adding it to x, the decision needs only evaluation. Code can obtain z by running a simulator, such as the game rules, a copy of the environment or a physics engine, or by asking the model in a separate call. Whether a decision depends on simulation is a property of the task as presented. Whether a model can perform that simulation is a property of the model (Example 1). Example 1 (one decision, presented two ways). When the agent holds an unwashed butterknife, going to the sink is right because the knife can be cleaned there. If an option only names a command, such as “go to sinkbasin 1”, the fact that the knife can be cleaned there must be simulated by code or predicted by the model (Figure 1a). If an option also shows what the command makes possible, such as “Then: clean butterknife 2 with sinkbasin 1”, the same decision needs only evaluation (Figure 1b). Knowing that a decision depends on simulation, however, does not separate real success from a lucky guess. A model can get such a decision right when a surface shortcut happens to point to the correct candidate option. Consider the task “put a clean butterknife in drawer”. Once the butterknife is clean, going to the drawer is correct, and a model that simply follows the words of the task gets it right. We therefore fix such a surface shortcut for each domain. Definition 2 (Surface shortcut and lure). For each domain, the surface shortcut h is a simple rule, fixed in advance, that picks the candidate options that look most relevant from the input alone, without predicting anything. A decision that depends on simulation is aligned if h picks a correct candidate option and misleading otherwise. In a misleading decision, the candidate options that h picks are the lure. In games, h picks the action with the highest average payoff, as if the opponent acted at random. This is the level-1 rule of behavioral game theory (Stahl & Wilson, 1995; Costa-Gomes et al., 2001). In ALFWorld, h picks the commands that mention an object or place named in the task description. Following Hagendorff et al. (2023), we classify each answer to a misleading decision as correct, lure or other. If most errors are caused by lures, the model is being drawn to the surface shortcut. If errors are spread over the other wrong options, they are more likely random noise. 3 Preprint Predictions. Our claim concerns what Jev does, not how it works internally. If Jev evaluates but does not simulate, five things should hold. (i) Decisions that need only evaluation are answered well, even against an intuitive but wrong option. (i) Aligned decisions that depend on simulation are answered well, because the surface shortcut picks a correct candidate option. (i) Misleading decisions that depend on simulation fail, and the errors go to the lure. (iv) Stating the missing prediction z in the input, such as the opponent’s action, restores accuracy. (v) If Jev can predict z in a separate call, writing its answer back also restores accuracy. Prediction (v) tells whether Jev cannot predict z at all, or can predict it correctly when asked separately but fails when it must also make the choice in the same call. By contrast, if intuitive lures were enough to mislead Jev, it would fail the CRT, whose questions are built around intuitive but wrong answers. If Jev simply lacked reasoning, it would also fail when asked to predict the opponent’s action on its own. Protocol. The order of the candidate options shifts Jev’s probabilities, so we average each decision over several orders. Whether a decision needs only evaluation or depends on simulation, and whether it is aligned or misleading, are computed from the environment, never from Jev’s choices. 3 Jev Owns the Evaluation: Cognitive Reflection Table 1: Jev avoids the intuitive answer on the CRT. Fractions over the 150 questions, each in four option orders. In the control versions, the intuitive answer is correct. Published results are open-ended answers, shown for reference only. The 95% interval of Jev’s correct rate on lure questions is [0.97, 1.00]. Four optionsPublished, open-ended answers (Hagendorff et al., 2023) JevHumans (n=455) GPT-3 davinci-003 ChatGPT-3.5 ChatGPT-4 Lure questions, correct0.990.380.050.590.96 Lure questions, intuitive0.010.550.800.150.00 Control versions, correct0.96– Setup. We first test Jev on decisions that need only evaluation. Does an intuitive but wrong option make Jev fail? The Cognitive Reflection Test (CRT) puts prediction (i) to a strong test, because its questions are built around intuitive but wrong answers, so that the first answer that comes to mind is wrong. We use the 150 new questions of Hagendorff et al. (2023). Each question becomes four candidate options, namely the correct answer, the intuitive answer and two other wrong answers, shown in four orders. Results. Jev picks the correct answer on 99% of questions and the intuitive answer on 1% (Table 1). For reference, people answering the same questions in open form give the intuitive answer 55% of the time. Each question also has a control version without the trap, in which the intuitive answer is correct. Jev is right on 96% of these, so it does not simply avoid intuitive-looking options. Because the answers are listed as options, Jev only needs to check each one against the conditions in the question (Example 2). Such checking needs only evaluation, so prediction (i) holds: an intuitive lure alone does not make Jev fail. Next, we further validate this boundary on one-shot matrix games, where the right action can depend on a prediction of the opponent’s action. Example 2 (bat and ball). A bat and a ball cost $1.10 in total, and the bat costs $1.00 more than the ball. The intuitive answer for the ball is $0.10. Checking the listed options against the question settles it. A ball of $0.05 gives a bat of $1.05 and a total of $1.10, as the question states, so Jev picks $0.05. 4 Jev Does Not Own the Simulation: One-Shot Games We now test Jev on decisions that depend on simulation. Can Jev predict what the opponent will do and act on it? One-shot games let us control exactly what must be simulated. The payoff table is fully visible, but the value of each action depends on the opponent’s action, which the table does not describe. We vary how many steps of reasoning about the opponent the answer needs and whether the surface shortcut happens to pick the correct action. 4 Preprint (a) A misleading game, D=1 Row LMR A 24, 571, 8312, 94 B 54, 212, 983, 34 C 86, 5685, 062, 99 Avg. PlainTold 35.70.080.00 46.30.13 0.96 77.7 0.79 0.04 eq. lure Cells give Row’s payoff, then Column’s. R pays Column the most in every row, so a rational Column plays R (shaded). Against R, B pays Row the most. The surface shortcut picks C, which has the highest average payoff. Plain and Told give Jev’s probabilities, averaged over three option orders. (b) Accuracy over all 210 games D0D1D2D3 0.0 0.5 1.0 Accuracy Aligned Chance (1/3) Plain (table only) Reason (rule out first) Told (Column's action given) D1D2D3 Misleading Figure 2: One-shot games: failures follow the surface shortcut, not depth. (a) An example game. (b) Accuracy is how often Jev chooses the equilibrium action (the right answer). Error bars are 95% bootstrap intervals over games. In aligned games the surface shortcut picks the equilibrium, and in misleading games it does not. Plain, reason and told are the three conditions of Section 4.1. Reason does not help, while told restores accuracy at every depth. 4.1 Failures follow the surface shortcut, not depth Setup. We sample random 3×3 matrix games with distinct payoffs in 0–99 that iterated elimination of strictly dominated strategies solves to a unique outcome. Jev plays Row against a Column described as rational and informed. The depthD is the number of elimination rounds Row needs, minus one, so D=0 means Row has a dominant action, one that pays more than every other action whatever Column plays. At D=k, Row must reason k steps about Column, such as “Column will never play this action, so I should play that one”. A game is aligned if Row’s best action against a rational Column, the equilibrium action, is also the action with the highest average payoff, and misleading otherwise. We sample 30 games for each value of depth and alignment, resulting in 210 games in total. Each game is shown with Row’s three actions in three different orders. We compare three conditions. In the plain condition, Jev sees only the payoff table. In the reason condition, the instruction also tells Jev to first rule out the actions that Column would never play. In the told condition, code adds Column’s equilibrium action to the state, as in “Column will play Y.”, which tests prediction (iv). Results. Figure 2 shows that alignment, not depth, determines accuracy. With a dominant action, which is best whatever Column plays and so needs only evaluation, Jev is perfect. In aligned games it stays at 77–93% for every depth. In misleading games it falls to chance (28–37%) and picks the lure in 53–71% of games. A regression confirms that depth has no detectable effect once alignment is taken into account (Appendix C). An instruction to reason leaves misleading games at chance, whereas stating Column’s action lifts them to 97–99%. This confirms predictions (i)–(iv). Contribution 1: diagnosis. Jev succeeds at decisions that need only evaluation. When the right option depends on simulation, it follows the surface shortcut, and its errors go to the lure. 4.2 What Jev computes Which rule produces these choices? A one-parameter softmax over each action’s average payoff, a soft version of the surface shortcut, fits Jev’s probabilities best (Appendix C). Adding whether an action is the equilibrium barely improves the fit. Jev thus plays as if its rational opponent acted at random, a quantal level-1 player in the terms of behavioral game theory (McKelvey & Palfrey, 1995; Stahl & Wilson, 1995). Level 1 involves no reasoning about the opponent, so it is not the same as depth D=1. 4.3 Knowing is not using Does Jev fail because it cannot predict Column’s action, or because it cannot use that prediction while choosing? Following Hagendorff et al. (2023), we ask for the prediction in a separate call on the same games. Jev correctly predicts Column’s equilibrium action 90% of the time, mostly by 5 Preprint picking the column that pays Column the most on average. Yet Jev’s own choice does not use this prediction. In the plain condition, Jev chooses correctly in 32% of the calls with a right prediction and in 35% of the calls with a wrong prediction. A choice that used the prediction would be right in almost all calls with a right prediction, as in the told condition (98%). Jev thus fails when one call must both perform the simulation and evaluate based on it. Separating simulation and evaluation recovers the decision. Code first asks Jev to predict Column’s action and then writes the answer into the state, as in the told condition. Accuracy in misleading games rises from 33% to 94%, as prediction (v) expects. The gain follows the written content. When Jev’s own prediction is wrong, code writes a wrong action into the state, and Jev still plays the best response to that written action in 24 of 26 calls. Contribution 2: cause. Jev can often make the prediction and make correct decisions based on the prediction, but it fails when one call must do both. Games isolate the failure, but they are small and artificial. In contrast, a household agent faces long episodes and dozens of commands, and the missing prediction may be a step that the task never states, such as washing a butterknife before putting it away. The next section asks whether the same failure appears in more realistic settings. 5 Code Owns the Lookahead: Text Games ALFWorld (Shridhar et al., 2021) tests the same boundary in a sequential task. Its household tasks, such as “put a clean butterknife in drawer”, are text games in which each step is a choice among admissible commands. Playing on its own, Jev solves only 31% of the 134 unseen games. We ask where it fails, what it knows and whether letting code perform the simulation fixes it. 5.1 Where Jev fails Setup. To test single decisions, we replay the successful trajectories from ALFWorld’s hand-coded expert. At each step Jev sees the task, the observation, the full history and the admissible commands. A choice is correct if it lies in an acceptable set derived from the environment. For example, when the expert goes to sinkbasin 1 to wash the knife, going to sinkbasin 2 also counts as correct. We label each step with one of four types that we define in advance and compute automatically from the task and the history (Appendix D). E steps directly complete part of the task, such as taking the butterknife, cleaning it or putting it in the drawer. Whether such a command is right can be read from the command itself, so these steps need only evaluation. S-known steps complete no part of the task themselves but make a later step possible, such as going to the sink while holding an unwashed butterknife. They depend on simulation, although all information needed for them is in the history. S-search steps explore for an object not yet seen, so the needed information is missing from both the history and the current state. Therefore, they also depend on simulation. Other steps, mostly looking around or closing a receptacle, neither complete part of the task nor make a later step possible. We exclude them (22% of expert steps). We test on both of ALFWorld’s evaluation splits, called seen and unseen. The 140 seen games take place in rooms that also appear in ALFWorld’s training data, and the 134 unseen games take place in new rooms. We use Jev zero-shot on both. The surface shortcut splits S steps into aligned and misleading (Section 2). We defined the ALFWorld surface shortcut according to the unseen games, such as picking “go to drawer 1” for the task “put a clean butterknife in drawer”, and then tested it on the seen games (Appendix D). Results. Figure 3a follows one seen game. Jev gets the E steps right, such as taking and cleaning the butterknife. At the misleading step it goes to the drawer, which the task mentions, instead of the sink. After the butterknife is washed, the same command becomes correct, and Jev chooses it. Figure 3b shows the same pattern over all games. Here we focus on E and S-known steps. S-search steps mix simulation with exploring for an object not yet seen, and many commands are acceptable there, so even random choice scores high (Appendix D). On the seen games, Jev chooses an acceptable command at 93% of E steps that take, put or clean, and at 99% of aligned S-known steps. At misleading ones it drops to 36%, and it picks the lure 56% of the time, five times the rate of uniform choice. The unseen games give similar results (96%, 95%, 43% and 51%). This confirms 6 Preprint (a) One trajectory: “put a clean butterknife in drawer” Step Step typeExpert commandJev’s top choice 1 Etake butterknife 2same (0.98) 2 S-known, misleading go to sinkbasin 1go to drawer 1 (0.60) 3 Eclean butterknife 2 with sinkbasin 1 same (0.96) 4 S-known, alignedgo to drawer 1same (0.69) 5 Emove butterknife 2 to drawer 1same (1.00) Consecutive decisions of the expert’s trajectory (opening the drawer omitted). go to drawer 1 is the lure before the butterknife is washed and the right command after. (b) Accuracy over all games E S-known aligned S-known misleading 0.0 0.5 1.0 Accuracy Uniform randomSeenUnseen Figure 3: ALFWorld: Jev is right when the surface shortcut is right. (a) Jev’s top choice at five decisions of one game, with probabilities averaged over two option orders. (b) Accuracy is how often Jev chooses an acceptable command. Error bars are 95% bootstrap intervals over games, and dashes mark uniform random choice. The aligned/misleading split was defined on the unseen games and tested on the seen games. Table 2: Jev can predict in a separate call and use a given prediction, but not both in one call. z is the missing prediction. Own z is Jev’s answer from the separate call. Rows are the 90 misleading games and the 64 misleading ALFWorld steps of the seen games from task types that have both an ordered and a goal form. PredictDecide DecisionsMissing z alone no z given correct z given own z given Games, misleadingColumn’s action 0.900.330.980.94 ALFWorld, ordered form next subgoal1.000.680.860.86 ALFWorld, goal formnext subgoal0.330.060.920.33 predictions (i)–(i). Why does Jev go straight to the drawer instead of first washing the butterknife at the sink? 5.2 What Jev knows Does Jev lack the order of subgoals, or fail to use it? ALFWorld writes each task in one of two forms, the ordered form and the goal form (Example 3). Asked separately for the next subgoal, Jev answers correctly 100% of the time under the ordered form but only 33% under the goal form. Under the ordered form, Jev thus knows the next subgoal, yet its decisions made in a single call are right only 68% of the time. Writing its own answer into the input raises them to 86%, as prediction (v) expects (Table 2). Under the goal form, Jev’s own prediction is wrong 67% of the time, and its wrong answers always skip to putting the object in place. The order of subgoals must therefore come from code, in one of two ways. Code can add the correct next subgoal to the input, which raises decisions to 92%. Code can also rewrite the task into the ordered form, which raises them from 6% to 73% over both seen and unseen games (Appendix F). Example 3 (two ways to write one task). The ordered form lists the steps, as in “clean some butterknife and put it in drawer”. The goal form only describes the final goal, as in “put a clean butterknife in drawer”. The goal form states only one action, putting. Washing is implied only by the word “clean”. Asked for the next subgoal, Jev answers “clean” under the ordered form, but under the goal form it usually answers “put”. 5.3 Code owns the simulation Lookahead. We now use code to perform the simulation. Before each decision, code runs every admissible command in a copy of the game. Each option then shows the command, what the game would print next and the commands that become possible only afterwards, such as “then newly possible: clean butterknife 2 with sinkbasin 1” (“Then:” in Figure 1b). This writes the missing prediction into each option. At misleading S-known steps, accuracy rises from 36% to 79% (Figure 1, bottom middle), and the rate of choosing the lure drops from 56% to 21%, as prediction (iv) expects. Whole games. The same lookahead turns Jev into a strong controller for whole games (Figure 1, bottom right, and Table 6 in Appendix D). Jev alone solves 42 of 134 unseen games. Showing what 7 Preprint Instruction: “Place box0 on top of box1, wherever box1 currently is, then move the hand away from the boxes. The target marker is irrelevant for this task.” Jev choosing among skills that code simulatesHand-written target box1 box0 move_above_box0 descend_to_box0 close_gripper place_on_box1 retreat place_at_target Step 1: 0.96Step 2: 0.94Step 3: 0.79Step 4: 1.00Step 5: 0.71Step 4 (Jev: 0.00) Figure 4: Stacker: Jev follows a new instruction by choosing among simulated skills. Each frame shows the scene after one decision, with the chosen skill and Jev’s probability for it. Code executes and simulates every skill, and Jev never outputs a torque. On the same scene, the hand-written controller carries box0 to the red target marker at its fourth decision (right). the game would print next raises this to 79, and adding the newly possible commands raises it to 116 (87%). A word-overlap rule with the same lookahead information solves only 18. The gain therefore comes from Jev’s evaluation of the predicted outcomes, not from the lookahead information alone. 6 Code Owns the Physics: Robot Control Does the same division hold in robot control? Here the missing prediction is the physics. Whether a torque is good depends on how the arm will move next, and the state describes only where the arm is now. We therefore expect Jev to fail when it chooses raw torques. In contrast, we expect it to succeed when code performs the physics simulation and Jev only chooses among described outcomes. We test both cases on the same task, Stacker from the DeepMind Control Suite (Tassa et al., 2018), where a planar arm with a two-finger gripper must bring a box to a target marker, a red square that shows where the box should go, either on the floor or one box above it. Without physics simulation. We first use the native task, “stack boxes so that one box sits at the target marker”. Every 0.02 s, Jev sets the torques of the arm’s five actuators, choosing for each among five described torque levels. Jev’s input describes the arm, gripper, boxes and joint angles. The input contains no outcome that code predicts by simulation, such as whether a move can be reached. In 20 episodes, Jev with raw torque options never picks up a box and solves none. Random torques also solve none (Table 3). With physics simulation. We then give Jev a small library of manipulation skills, tuned on the native task (Appendix E). The skills move above a box, descend, push a crowded box aside, close or open the gripper, lift, place the held box, retreat and hold still. Code implements them with inverse kinematics (Tunyasuvunakool et al., 2020) and joint PD control (Lynch & Park, 2017). As a result, every skill corresponds to a physics simulation implemented by code. In other words, when Jev reads the skill options, it knows the outcome of that simulation. On the same 20 episodes of the native task, Jev now solves 9, as prediction (iv) expects, and the hand-written controller solves 8. Generalization to new instructions. To test generalization, we extend the native task with five new instructions, such as “put box0 on box1” or “move the farther box to the target”, each run on 8 seeds with different starting conditions. We change the instruction text, while the skills are shared. Jev solves 22 of the 40 episodes, against 3 for the hand-written controller and none for random choice. We find that Jev’s evaluation ability, with skills developed for one task, provides promising generalization to unseen task variants. Figure 4 shows one such episode. Although the settings with and without physics simulation differ in the action level, their contrast shows that code provides the simulation, so that Jev only needs to evaluate. Contribution 3: remedy. Code should own the simulation, and Jev the evaluation. With predictions or simulated skills supplied by code, Jev becomes an expert controller in games, ALFWorld and robot control. 8 Preprint Table 3: Stacker without and with physics simulation performed by code. Success counts over episodes not already solved at reset. Random chooses uniformly among the same actions. The hand-written controller is the state machine for the native task, run unchanged. Raw torques, no simulationSkills simulated by code ControllerNative taskNative task5 new instructions Jev0/20 (0%)9/20 (45%)22/40 (55%) Hand-written controller–8/20 (40%)3/40 (8%) Random choice0/20 (0%)1/20 (5%)0/40 (0%) 7 Related Work Language models as policies over candidate actions. A common way to use a language model as an agent is to let code list valid actions and let the model score them, as in SayCan (Ahn et al., 2022) and admissible-action mapping (Huang et al., 2022). Text-game agents choose among TextWorld and ALFWorld commands (Côté et al., 2018; Shridhar et al., 2021), with or without reasoning traces (Yao et al., 2023b; Shinn et al., 2023). GLAM and TWOSOME train such scorers with RL (Carta et al., 2023; Tan et al., 2024), and TALES collects text-adventure benchmarks (Cui et al., 2025). These works report how often an agent succeeds. We instead ask which decisions such a scorer gets right. The answer depends on whether a decision needs only evaluation or depends on simulation, which we determine from the environment, not from the model. Who simulates and who evaluates. Tree of Thoughts, RAP and LATS let one model both propose future states and score them (Yao et al., 2023a; Hao et al., 2023; Zhou et al., 2024). PlanBench shows that language models plan poorly on their own (Valmeekam et al., 2023), and LLM-Modulo pairs a generating model with external verifiers (Kambhampati et al., 2024). We use the opposite split. Code predicts the consequences of each option, and a judgment model without reasoning text evaluates them. In the terms of Daw et al. (2005), Jev alone resembles model-free control, which acts on values learned from past experience instead of predicting consequences. Code adds the model-based part by predicting the consequences for it. Cognitive and strategic probes. The CRT is built so that the intuitive answer is wrong (Frederick, 2005). Hagendorff et al. (2023) found that earlier GPT-3 models fall for it while ChatGPT-4 does not, and we borrow their error scoring and direct knowledge probes. Other work applies cognitive tasks to language models (Binz & Schulz, 2023; Coda-Forno et al., 2024), and HANS splits items by whether a known heuristic answers them (McCoy et al., 2019), which our aligned/misleading split mirrors. Behavioral game theory describes players by their depth of reasoning about others (Nagel, 1995; Stahl & Wilson, 1995; Camerer et al., 2004; Costa-Gomes et al., 2001), and human play departs from equilibrium in systematic ways (Goeree & Holt, 2001; Arad & Rubinstein, 2012). We vary depth and surface alignment separately and add a condition in which code states the opponent’s action. 8 Discussion The boundary lies inside a single call. Across the four settings, Jev succeeds at evaluation when the input contains sufficient information. It fails when the right option depends on simulation to provide information missing from the input. When it fails, it picks the lure. In misleading matrix games, 90% of its errors go to the action with the highest average payoff, against 50% expected under random errors. At misleading ALFWorld steps, 89% go to a command that mentions the task, against 12% expected (Figure 6 in the appendix). In robot control, it never picks up a box when it must choose raw torques. Table 2 shows which part fails. Asked in a separate call, Jev predicts Column’s action correctly in 90% of misleading games. When code writes a correct prediction into the input, Jev chooses correctly in 98% of them. It fails when one call must both perform the simulation and evaluate based on it, where it is right in only 33%. This also explains why an instruction to reason does not help, since the instruction adds no prediction. Two limits remain. First, some facts are relations between options, such as one action paying more than another whatever the opponent does. Such a fact cannot be read from any single option. Jev mostly misses it, even when asked in a 9 Preprint separate call, so code must compute it (Appendix F). Second, Jev follows the order in which the task mentions the actions, even when a word such as “after” says otherwise (Example 4). Example 4 (a stated order that Jev does not follow). “Put some butterknife in drawer after cleaning it” mentions putting first, although cleaning must come first. With this wording, Jev decides correctly in 24% of cases, against 69% when the task mentions cleaning first (Appendix F). When can code make the prediction or simulation? In RL terms, code supplies the world model that Jev lacks. With a simulator, such as the game rules, a copy of the game or a physics engine, code can compute the prediction directly. Jev then only evaluates. Without a simulator, code can still separate simulation and evaluation. It asks Jev for the prediction in one call and writes the answer into the next. This works when Jev can make the prediction from the input. When Jev lacks the ability to simulate a given environment, as with the unwashed knife, the prediction must come from elsewhere, for example a learned world model (Ha & Schmidhuber, 2018; Hafner et al., 2020) or a larger reasoning model. Limitations. We study one model and one API version. We have not tested other models that answer without reasoning text, nor how the boundary moves when reasoning is allowed. We therefore cannot yet say whether the effect is specific to Jev. The ALFWorld lookahead and the Stacker simulation use an exact copy of the environment, which a real agent may not have. Our ALFWorld lookahead looks one command ahead and lists the commands it makes possible. When a consequence appears only after several commands, code must search deeper, and the cost grows quickly with the number of commands. Our analysis of which predictions Jev can make from the input is partly post hoc. 9 Conclusion We began with a contrast. Jev solves the counterintuitive CRT questions, yet it carries an unwashed butterknife to the drawer instead of the sinkbasin. The two lures differ in where the answer lies. A CRT answer can be checked against the question, which is evaluation. The sinkbasin is right only because the knife must be washed first, which the input does not describe and Jev must simulate. Jev evaluates well, and in games it can often predict the opponent’s action when asked separately. It fails when one call must both perform the simulation and evaluate based on it. Letting code make the prediction or simulation closes this gap. Code can state the opponent’s action, write back Jev’s own prediction, show the outcome of each command, or simulate each skill of a robot arm. With these simulations, Jev becomes an expert controller through its general evaluation ability, and in robot control it follows new instructions that a hand-written controller cannot. Code should own the simulation, and Jev the evaluation. Reproducibility Statement Appendix A gives the instruction of every condition and the model version, and Appendix F gives the probe questions. The other appendices describe the CRT items, the game generator, the behavioral models, the ALFWorld labeler and lookahead, the DMC skills and the knowledge probes. Every model call is stored in a raw run file. One script recomputes every number, figure and table of the CRT, game and ALFWorld studies from these files. The DMC numbers come from the run files of the DMC evaluation scripts, and the frames of Figure 4 come from a script that replays the logged decisions in the simulator. We will release the code, the raw runs and these scripts. Jev is accessed through a public API, so its outputs can change across versions. Repeating a single call changes its probabilities by 0.04 on average. References Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691, 2022. 10 Preprint Ayala Arad and Ariel Rubinstein. The 11–20 Money Request Game: A Level-k Reasoning Study. American Economic Review, 102(7):3561–3573, 2012. ISSN 0002-8282. doi: 10.1257/aer.102. 7.3561. Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6), 2023. ISSN 1091-6490. doi: 10.1073/pnas.2218523120. C. F. Camerer, T.-H. Ho, and J.-K. Chong. A Cognitive Hierarchy Model of Games. The Quarterly Journal of Economics, 119(3):861–898, 2004. ISSN 1531-4650. doi: 10.1162/ 0033553041502225. Thomas Carta, Clément Romac, Thomas Wolf, S. Lamprier, Olivier Sigaud, and P. Oudeyer. Ground- ing Large Language Models in Interactive Environments with Online Reinforcement Learning. In International Conference on Machine Learning (ICML), 2023. Julian Coda-Forno, Marcel Binz, Jane X. Wang, and Eric Schulz. CogBench: a large language model walks into a psychology lab. In International Conference on Machine Learning (ICML), 2024. Miguel Costa-Gomes, Vincent P. Crawford, and Bruno Broseta. Cognition and Behavior in Normal- Form Games: An Experimental Study. Econometrica, 69(5):1193–1235, 2001. ISSN 1468-0262. doi: 10.1111/1468-0262.00239. Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, and Marc-Alexandre Côté. TALES: Text Adventure Learning Environment Suite. arXiv preprint arXiv:2504.14128, 2025. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, B. Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew J. Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. TextWorld: A Learning Environment for Text-based Games. In Computer Games Workshop at IJCAI (CGW), 2018. Nathaniel D Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12):1704–1711, 2005. ISSN 1546-1726. doi: 10.1038/n1560. Jonathan St. B. T. Evans and Keith E. Stanovich. Dual-Process Theories of Higher Cognition: Advancing the Debate. Perspectives on Psychological Science, 8(3):223–241, 2013. ISSN 1745- 6924. doi: 10.1177/1745691612460685. Shane Frederick. Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19 (4):25–42, 2005. ISSN 0895-3309. doi: 10.1257/089533005775196732. Jacob K Goeree and Charles A Holt. Ten Little Treasures of Game Theory and Ten Intuitive Contradictions. American Economic Review, 91(5):1402–1422, 2001. ISSN 0002-8282. doi: 10.1257/aer.91.5.1402. David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nature Computational Science, 3(10):833–838, 2023. ISSN 2662-8457. doi: 10.1038/s43588-023-00527-x. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, D. Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In International Conference on Machine Learning (ICML), 2022. 11 Preprint Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011. Subbarao Kambhampati, Karthik Valmeekam, L. Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil B Murthy. Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks. In International Conference on Machine Learning (ICML), 2024. Kevin M. Lynch and Frank C. Park. Modern Robotics: Mechanics, Planning, and Control. Cambridge University Press, 2017. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. Richard D. McKelvey and Thomas R. Palfrey. Quantal Response Equilibria for Normal Form Games. Games and Economic Behavior, 10(1):6–38, 1995. doi: 10.1006/game.1995.1023. Rosemarie Nagel. Unraveling in Guessing Games: An Experimental Study. American Economic Review, 85(5):1313–1326, 1995. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations (ICLR), 2021. Dale O. Stahl and Paul W. Wilson. On Players’ Models of Other Players: Theory and Experimental Evidence. Games and Economic Behavior, 10(1):218–254, 1995. ISSN 0899-8256. doi: 10. 1006/game.1995.1031. Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2024. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, Andrew Lefrancq, T. Lillicrap, and Martin A. Riedmiller. DeepMind Control Suite. arXiv preprint arXiv:1801.00690, 2018. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and Tasks for Continuous Control. Software Impacts, 6:100022, 2020. TypeSafe. TypeSafe System One models: Jev (jev-1.13.0). https://typesafe.ai, 2026. Accessed September 2026. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, S. Sreedharan, and Subbarao Kambhampati. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, T. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023a. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR), 2023b. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models. In International Conference on Machine Learning (ICML), 2024. 12 Preprint A The Jev Interface and Prompts Every call is one request to the TypeSafe System One API with the model jev-1.13.0 (TypeSafe, 2026). A request carries a JSON state and one choice question. The raw-torque condition of Stacker sends one question per actuator instead. A question consists of an instruction and a set of options, each with a key and a text description. In the CRT and ALFWorld the keys are neutral, such as o0 and c0. In games they are the action labels A, B, C or X, Y, Z. In Stacker they are the names of the skills or torque levels, such as place_at_target. The response gives the chosen key, a probability for each option and a confidence score. Each box below lists the state, the options and the instruction for one domain. A row named after a condition applies only to that condition. CRT. State question: the question text. OptionsFour candidate answers with keys o0 to o3, for example “$0.05”. InstructionAnswer the question in question. Games. State game: “one-shot simultaneous-move game, played once”. you: “Row player: choose A, B or C”. opponent: “Column player: chooses X, Y or Z at the same time as you, without seeing your choice. Column is rational, knows all payoffs, wants to maximize their own payoff, and knows that you are rational too.” payoffs_row_column: the payoffs of both players in each cell, for example "A,X": [82, 94]. OptionsRow’s actions with keys A, B and C, for example “Play A: you get 82 if Column plays X, 36 if Y, 20 if Z”. InstructionYou are the Row player. Choose the action that maximizes your own payoff against this opponent. ReasonAppends to the instruction: First work out which actions Column would never play given Column’s own payoffs, then which of your actions is best against what remains, and repeat until one action is left. ToldAdds to the state: opponent_prediction: “Column will play X.” ALFWorld. State task: the task text. current_observation: the game’s last output. history: the earlier commands, each with the output it produced. step: the step number. OptionsThe admissible commands with keys c0, c1, . . . , for example “go to sinkbasin 1”. InstructionYou control an agent in a household text game. Choose the next command that makes the most progress toward completing task. Use history to remember what you have already seen and done, avoid revisiting places you already checked, and pick up / use objects as the task requires. Lookahead v1Each option becomes “command -> output”. Appends to the instruction: Each option shows a command and, after ‘->’, what the game would print right after it (computed by running the command in a copy of the game), marked when you have already seen exactly that output. Lookahead v2Each option also ends with “then newly possible:” and the new commands. Appends to the instruction: Each option shows a command and, after ‘->’, what the game would print right after it and which commands would newly become possible (computed by running the command in a copy of the game), marked when you have already seen exactly that output. In v1 and v2, the mark is the tag “[already seen]” at the end of the option. It appears when the predicted output is identical to an output printed earlier in the same game. 13 Preprint Stacker with skills. State task: the task text. Target: target_xz_m, target_height, box_under_target. Hand: hand_xz_m, hand_to_target_m, gripper, held_box, held_box_at_target. Progress: task_complete, valid_primitives_now, last_primitive, last_outcome. boxes: for each box xz_m, distance_to_target_m, at_target, under_target, upright, gap_to_nearest_box_m, crowded, graspable_from_above, resting_on_arm, hand_above_it, between_fingers, held. Code computes all of these from the simulator. OptionsThe skills valid in the current gripper state, keyed by name: move_above_box, descend_to_box, nudge_box, close_gripper, open_gripper, lift, place_at_target, place_under_target, retreat and hold_still. Each has a what field, which says what the skill does, and a not_for field, which says when not to use it. For example, place_at_target has what “Carry the held box to the target position and release it there” and not_for “when no box is held, or the target is elevated and nothing supports it yet”. InstructionChoose the next primitive for the robot arm so the stacking task is completed and rewarded as early as possible. Use target_height, box_under_target, boxes, held_box and gripper. New tasks task holds the new instruction (Appendix E). Adds to the state: box_start_xz_m, box_has_visited_target and, while a box is held, held_box_relations (at_target, at_start_position, above_box). Adds the skills place_at_start and place_on_box. Replaces the instruction: Choose the next primitive for the robot arm so that the instruction in task is completed as early as possible. Use boxes, held_box, gripper, box_start_xz_m and box_has_visited_target. Stacker with raw torques. State rules: A planar arm in the x-z plane with a root, shoulder, elbow and wrist joint and a two-finger gripper. Commands are normalized torques, not position targets. Gravity pulls the arm down, so a zero command lets it sag, and holding a pose needs a steady torque. Half torque held for 0.1 s moves the hand by 2 to 8 cm. A box is picked up by placing it between the open fingers and closing them. Boxes are 4.4 cm cubes. The other fields are those of Stacker with skills, without valid_primitives_now, last_primitive and last_outcome, plus joints: the angle and velocity of each joint and the gripper opening. OptionsOne question for each actuator (root, shoulder, elbow, wrist, grasp). Five torque levels: strong_positive, gentle_positive, none, gentle_negative, strong_negative. For example, strong_negative for the elbow reads “Maximum torque that lowers the elbow angle, so the hand moves right (+x) and slightly up”. InstructionChoose the torque command for the actuator actuator for the next 0.02 s so that the arm completes the instruction in task as early as possible. Use joints, hand_xz_m, gripper and boxes. Repeating an identical request changes the returned probabilities by 0.04 on average (maximum 0.12). It changes the chosen option in 12% of cases. Averaging over option orders also reduces this noise. B CRT Questions We use the 150 questions of Hagendorff et al. (2023) (OSF w5vhp), 50 of each type. Type 1 follows the bat and ball, as in “A pear and a fridge together cost $140. The pear costs $100 more than the fridge. How much does the fridge cost?”. Type 2 follows the machines that make widgets, and type 3 the doubling lily pads. Each question has a control version from the same materials, in which the intuitive answer is correct. In type 1 the control states the price of one item. In type 2 the number of workers stays the same. In type 3 the doubling clause is removed. Each question has four options: the correct answer, the intuitive answer and two other wrong answers. In type 1 the other two are half the correct price and the price of the other item. In type 2 they are two of the product, difference or sum of the correct and intuitive times, or twice or three times the 14 Preprint correct time. In type 3, where t is the total time given in the story, the correct answer is t− 1, the intuitive answer is t/2, and the other two are two of t− 2, t/4, t + 1 and 2t. Our first run reached 88% on the control versions. The errors came from two ambiguities. First, a type-2 control asks “How long does it take 7 people to tailor 4 jackets, if it takes 7 people 7 hours to tailor 7 jackets?”. The intended answer is 4 hours, but 7 hours is also right if each person makes a whole jacket. Second, the sentence we appended to type-3 controls to state a constant rate did not fit every story. A falling branch, for example, does not “grow”. The reported run fixes both. It appends “They share the work evenly, so any job can be split among all of them.” to both versions of type-2 questions. This sentence does not change the answer of the lure version. It appends “Assume it progresses at a constant rate, so equal amounts are added in equal periods of time.” to type-3 controls. Controls then rose to 96%, with type 2 going from 78% to 92% and type 3 from 85% to 96%. Most remaining control errors are type-2 questions such as asking how long 8 painters take to paint 1 painting when 8 painters paint 8 paintings in 8 hours. The intended answer is 1 hour, but splitting one painting among eight painters conflicts with common sense. The revision does not affect the main result. In both runs, Jev answers 99% of the lure questions correctly. C Games: Generation and Behavioral Models Table 4: Jev plays as if the opponent acted at random (the logit level-1 rule). One-parameter models P(a) ∝ exp(λv(a)) are fitted to Jev’s probabilities over 630 calls per condition. The columns give the KL divergence from Jev’s distribution (lower is better), the fraction of Jev’s choices the model predicts, and the fitted λ per 100 payoff units. Bold marks the best fit per condition. plaintold Decision ruleKL choice λKL choice λ Level-1: average own payoff0.068 0.77 5.5 0.771 0.576.2 Maxmax: best own payoff0.137 0.66 4.2 0.919 0.503.4 Maxmin: worst own payoff0.172 0.66 3.0 0.889 0.553.3 Level-2: best reply to a level-1 opponent 0.192 0.65 1.9 0.485 0.897.5 Equilibrium action0.203 0.65 0.8 0.076 0.994.2 Payoff given the stated action0.194 0.65 2.0 0.103 0.99 26.9 Uniform0.286– 1.028– Generation. Games are 3× 3. Each player’s nine payoffs are distinct numbers drawn at random from 0–99, so no ties occur. Iterated elimination proceeds in rounds. In each round, and for both players simultaneously, it removes every action that is strictly dominated by another remaining pure action. Games are kept if a single outcome survives. The depth is the round, counted from zero, after which Row has one action left. Games are sampled until there are 30 of each of seven game types, namely depth 0 and depths 1 to 3 each aligned or misleading. Effect of depth. We fit a logistic regression to the plain calls with D ≥ 1: logitP(correct) = β 0 + β 1 m + β 2 a + β 3 ma + β 4 1[D=2] + β 5 1[D=3], where m is the margin and a is 1 in aligned games and 0 in misleading ones. The margin is the gap between the highest and the second-highest average payoff among Row’s actions, where each average is taken over Column’s three actions. Because m and a are in the model, β 4 and β 5 measure the effect of depth at the same margin and alignment. The margin lowers accuracy in misleading games (β 1 =−9.8 per 100 payoff units) and raises it in aligned ones (β 1 +β 3 = 17.4), as the surface shortcut predicts. Depth has no detectable effect (β 4 = 0.22 [−0.52, 1.03], β 5 = −0.38 [−1.22, 0.45]). Brackets give 95% bootstrap intervals over games. Behavioral models. We fit one-parameter softmax models P(a)∝ exp(λv(a)) to Jev’s probabili- ties, one for each decision rule v (Table 4). In plain, the level-1 rule fits best. It values each action by its average payoff against a random opponent. It has a KL divergence of 0.068 and predicts 77% of Jev’s choices. Adding an indicator for the equilibrium to this rule barely improves the fit. Its fitted weight equals that of only 6 payoff units of average payoff, while the two best actions typically differ by more than 10 points. It also fits best under the reason instruction (KL 0.049), with a lower λ (3.9 against 5.5). In told, the equilibrium rule fits best (KL 0.076) and predicts 99% of choices. 15 Preprint D ALFWorld Details Expert and filtering. We run ALFWorld’s hand-coded expert, which searches receptacles in the order of ALFWorld’s table of where each object type is usually found. We record every step until the game ends or 60 steps pass. The expert solves 122 of 134 unseen and 116 of 140 seen games. We keep only solved games, because the labels assume sensible expert steps. State tracking. The labeler reads only the task and the history. From them it tracks the current location, the held object, which receptacles have been looked into, where each object was last seen, which objects have been cleaned, heated or cooled, and which have been placed. The task type fixes the subgoals. For example, in a clean task the agent must find the object, bring it to the sink and then bring it to the destination. Step types. The type depends on the expert’s command and the tracked state. A step is E if the command changes a goal fact: taking the object, cleaning, heating or cooling it, putting it in the destination, or using the desk lamp while holding the object. A step is S-known if the command changes no goal fact but goes to or opens a receptacle that a later goal step needs, such as the sink while holding an unwashed object, or returns to where the object was seen. A step is S-search if the needed object has not been seen yet and the command goes to or opens a receptacle to look for it. All other steps, such as looking around, closing a receptacle or discarding a wrong object, are labeled other and excluded. Surface shortcut. A step is aligned if an acceptable command names the task’s object or destination. It is misleading if no acceptable command does but another command does. Such commands are lures. Otherwise the step has no surface shortcut. We fixed this rule on the unseen games before testing it on the seen games. Acceptable sets. For E steps, a command is acceptable if it has the same verb and object class as the expert’s command. For a put command, the destination class must also match. This rule marks reasonable commands as wrong at E steps that heat, cool or use a lamp, such as opening a closed microwave before heating. We therefore report only E steps that take, put or clean, which make up 82% of E steps. For S-known steps, going to a receptacle of the needed class is acceptable. So is going to where the needed object was seen, or opening the needed receptacle. For S-search steps, going to a receptacle not yet checked is acceptable. So is opening the current closed receptacle. Table 5: ALFWorld decisions on both splits. The decisions are replayed from the expert’s successful trajectories. The table gives the fraction of acceptable choices, with 95% bootstrap intervals over games, and the rate under uniform random choice. The surface shortcut was defined on the unseen split and tested on the seen split. Unseen (shortcut defined)Seen (shortcut tested) Decision typeJevrandomJevrandom E (take, put or clean)0.96 [0.94, 0.98] 0.04 0.93 [0.90, 0.96] 0.04 S-known, aligned0.95 [0.89, 1.00] 0.08 0.99 [0.97, 1.00] 0.07 S-known, misleading0.43 [0.33, 0.53] 0.04 0.36 [0.26, 0.48] 0.04 S-search, aligned0.79 [0.72, 0.85] 0.66 0.79 [0.74, 0.85] 0.74 S-search, misleading0.34 [0.24, 0.43] 0.55 0.50 [0.36, 0.64] 0.55 S-search, no surface shortcut 0.40 [0.33, 0.49] 0.51 0.59 [0.40, 0.90] 0.48 On search steps with no surface shortcut, Jev is below random choice on the unseen split (0.40 versus 0.51) and above it on the seen split (0.59 versus 0.48). We draw no conclusion from them. Lookahead. ALFWorld keeps its game state inside a C library, so the game cannot be copied from Python. At each decision, the process holding the game forks once per admissible command. Each child runs its command, sends back the game’s output, the new admissible commands and the win flag, and exits, so the parent’s game is left untouched. A random policy with a fixed seed produces identical trajectories with and without lookahead, so the lookahead does not change the game. The predicted outputs also match the real ones at every step. Per decision, the forked lookahead took about 3 s and the Jev call about 0.4 s. Table 6 gives the closed-loop results. Lookahead v1 solves 41 games that Jev alone fails and fails 4 that it solves (exact McNemar test, p = 9× 10 −9 ). Lookahead v2 gains 38 games over v1 and loses 1 (p = 1.5× 10 −10 ). The word-overlap rule uses no model. It 16 Preprint Table 6: ALFWorld games solved in closed loop. 134 unseen games, at most 50 steps. The columns from Pick to Two give the fraction solved for each task type in the first run, and Run 2 repeats the Jev conditions. Stuck counts failed games whose last ten commands contain at most three distinct commands. Tokens are Jev’s input tokens. The game copy shows what each command would reveal, so these numbers are not comparable to agents without a simulator. ConditionSolved 95% CIRun 2 Pick Look Clean Heat Cool Two Stuck Tokens Jev alone42/134 [0.24, 0.40] 39/134 0.58 0.56 0.19 0.09 0.05 0.53 54/92 10.6M Jev + lookahead v1 (next observation) 79/134 [0.51, 0.67] 84/134 0.96 0.72 0.42 0.39 0.19 1.00 45/55 9.2M Jev + lookahead v2 (+ new commands) 116/134 [0.81, 0.92] 117/134 0.96 0.94 0.77 0.78 0.81 1.00 14/18 5.9M Word-overlap rule + lookahead v218/134 [0.08, 0.19]–0.17 0.44 0.13 0.04 0.05 0.00 1/116– Word-overlap rule + lookahead v17/134 [0.01, 0.09]–0.12 0.22 0.00 0.00 0.00 0.00 1/127– Random1/134 [0.00, 0.02]–0.00 0.06 0.00 0.00 0.00 0.00 0/133– scores each option, the command together with its predicted output, by the words it shares with the task and picks the best option whose output was not already seen. E DMC Details Stacker skills. Each skill computes target joint angles and drives the arm there with a PD (proportional-derivative) controller at 100 Hz, with gains k p = (21, 14, 7, 3.5) and k d = (2.1, 1.4, 0.7, 0.35). The target angles come from inverse kinematics (IK), which finds joint an- gles that put the hand at a given position. IK is run from several random starting points and can return several solutions. Code simulates each solution in a copy of the physics and keeps the one that moves the other box least and needs the least joint motion. The place skills run as one unit: carry the box, release it and wait for it to settle. An episode succeeds if the task’s success check holds at its end. For the native task, the check requires a box at the target and the hand at least 10 cm away. Once the check holds, code offers only hold still, so a finished task is not undone. Episodes last 10 s on the native task and 20 s on the new instructions. Instructions. The native task reads “Stack boxes so that one box sits at the target marker. Reward accrues only while a box is within about one box width of the target AND the hand is at least 10 cm away from the target.” For each new instruction, we redraw the initial scene, up to a fixed number of times, until the task can be done, for example until each needed box rests on the floor with room around it and can be reached from above. This is similar to how dm_control redraws scenes in which objects overlap. We take seeds in order from 0 and skip scenes where the instruction already holds at the start or no workable scene is found, until 8 scenes remain. The five new instructions read as follows. • “Place box0 on top of box1, wherever box1 currently is, then move the hand away from the boxes. The target marker is irrelevant for this task.” • “Place box1 on top of box0, wherever box0 currently is, then move the hand away from the boxes. The target marker is irrelevant for this task.” • “Bring box0 to the target marker on the floor, then carry it back to where it started (see box_start_xz_m), release it there and move the hand away.” • “Bring box1 to the target marker (on the floor at the target x) and move the hand away. Do not move box0 at all.” • “Take whichever box started farther from the target marker and bring it to the target (on the floor at the target x), then move the hand away. Leave the other box alone.” Stacker without simulation. Every 0.02 s, Jev answers one question per actuator, choosing among five torque levels. Each level is described by the direction in which it moves the hand. We measured these directions once, by applying half torque for 0.1 s from the arm’s start pose. We run seeds 0 to 21. In seeds 7 and 9 a box already sits at the target when the episode starts, so we exclude them, which leaves 20 episodes. In these, over about 440 decisions each, Jev never holds a box. On average, the closest its hand comes to a box is 25 cm, against 16 cm with random torques. The skill runs in Table 3 use the same seeds. 17 Preprint F Knowledge Probes Game questions. The probes use the 180 games with D ≥ 1, each in three orders of Row’s actions, with the state of the plain condition. The prediction question asks “Which action will Column play?”. Its options describe Column’s payoffs for each of its actions. An elimination question asks which of one player’s remaining actions is strictly dominated, that is, pays less than another remaining action of the same player against every remaining action of the other player. Its options are the remaining actions and “none”. In the elimination write-back, code asks Column and Row in turn and writes each answer into the state as a note, such as “Column will never play Z.”. It stops when Row has one action left or both players answer “none”. The prediction write-back writes Jev’s own prediction into the state as “Column will play Y.”, exactly as in the told condition. Game results. When Column had a dominated action, Jev named it in 19% of steps and wrongly answered “none” in 81%. When Column had none, it answered correctly in 98%. For Row, the same two accuracies were 58% and 85%. Overall, single elimination steps are correct in 57% of calls. With correct elimination notes, decisions in misleading games reach 94%, but with Jev’s own elimination steps written back they reach only 43%. In the prediction probe, Jev picks the column with the highest average payoff to Column in 88% of calls. In the 17 misleading games where that column is not the equilibrium, it names Column’s action correctly only 55% of the time. The 244 calls with a correct prediction and the 26 with a wrong one give the decision accuracies of 32% and 35% in Section 4.3. ALFWorld questions. The subgoal question reads “Which of these must happen next to complete task? Use history to see what has already been done.”. Its six options are to find and pick up the object, clean it, heat it, cool it, put it in the destination, and turn on the desk lamp while holding it. The labeler of Appendix D gives the correct answer. The told condition writes the correct subgoal into the state as a field next_subgoal, and the write-back writes Jev’s own answer there. Both then ask the decision question of Section 5.1 unchanged. ALFWorld results. At the 149 aligned S-known steps, where the next subgoal is to put the object, predicting the subgoal, deciding with the correct subgoal given and deciding with Jev’s own subgoal written back all reach 100%. At misleading steps, Jev names the subgoal correctly in 63% of calls, and every error names putting the object. By task type, it is right in 77% of clean, 57% of heat and 50% of cool steps. With the correct subgoal given, clean and heat steps reach 100% and cool steps 70%. On cool steps Jev then often goes to the sink or to the destination. Table 2 uses the 64 misleading seen-split steps at which the object must still be cleaned, heated or cooled, 28 in the ordered form and 36 in the goal form, each in two option orders. The other 6 misleading steps belong to lamp and two-object tasks, which have neither form. The game row of Table 2 uses the 90 misleading games in three orders. “put a clean T in D” “put some T in D after cleaning it” “clean some T and put it in D” 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy 0.28 0.42 0.48 0.57 1.00 1.00 Predicts the next subgoal (separate call) Task originally in goal form (69 steps)Task originally in ordered form (70 steps) “put a clean T in D” “put some T in D after cleaning it” “clean some T and put it in D” 0.06 0.11 0.17 0.31 0.73 0.66 Chooses the next command Figure 5: Jev knows what comes first only when the task mentions it first. Misleading ALFWorld steps of both splits under three task templates. Code swaps the template and changes nothing else. Left: Jev predicts the next subgoal in a separate call. Right: Jev chooses a command. Task wording. ALFWorld writes each clean, heat or cool task in the ordered form or the goal form (Example 3), using fixed templates such as “heat some T and put it in D.” and “put a hot T in D.”, 18 Preprint where T is the object and D the destination. Code swaps only the template. This moves decisions from 6% to 73% when the goal form becomes the ordered form, and from 66% to 11% the other way, and the subgoal answers flip with them (Figure 5). A third template, “put some T in D after cleaning it”, says with “after” that cleaning comes first but mentions putting first. With it, decisions reach only 24% (Example 4). G Error Outcomes Figure 6 scores every decision that has a lure as correct, as the lure, or as another error (Hagendorff et al., 2023). If Jev’s errors were spread evenly over the wrong options, the lure would receive one in three errors in the CRT, one in two in games and about one in eight in ALFWorld. Instead, almost all errors go to the lure. In the CRT, the few errors (1%) all pick the intuitive answer. In misleading games, 90% of errors pick the action with the highest average payoff. At misleading S-known ALFWorld steps, 89% pick a command that mentions the task. When code supplies the missing prediction, most decisions move from the lure to the correct option. Stating Column’s action leaves 2% errors in games, and the lookahead cuts the lure rate in ALFWorld from 56% to 21%. Jev alone 0.99 CRT, lure questions correct: Jev alonecorrect: + reason instructioncorrect: + code, partialcorrect: + codelureother error + code states Column's action + code writes back Jev's own prediction + code writes back Jev's own elimination steps + “reason” instruction Jev alone 0.98 0.94 0.43 0.52 0.37 0.55 0.07 0.33 0.61 0.07 One-shot games, misleading, depth 1–3 0.00.20.4 0.6 0.81.0 Fraction of decisions + lookahead by code + code states the subgoal + code writes back Jev's own subgoal Jev alone 0.790.21 0.90 0.600.36 0.36 0.56 0.07 ALFWorld steps, misleading (S-known) Figure 6: When Jev is wrong, it picks the lure. The decisions with a lure in each domain, scored as correct, lure or other error (Hagendorff et al., 2023). The panels cover the 150 CRT questions, the 90 misleading games with D ≥ 1 and the 70 misleading S-known decisions of the ALFWorld seen split. The lure is the intuitive answer in the CRT and the action with the highest average payoff in games. In ALFWorld it is any wrong command that mentions an object or place named in the task description. The color of the correct segment marks the condition, as in the other figures. Light aqua marks partial help from code. Here code writes back Jev’s own answer to a separate question. Rates are averaged over items and option orders. H Compute and Cost All Jev calls use the public API, which charges only for input tokens. Table 7 lists the input tokens of each experiment. In total, the reported runs used about 94M input tokens. At the price TypeSafe listed in September 2026, $0.042 per million input tokens, they cost about $4. The two closed-loop ALFWorld runs account for about $2.2 of this, and the Stacker runs for about $0.9. The environments ran on the CPUs of a single server. 19 Preprint Table 7: Input tokens of each experiment. ExperimentInput tokens CRT0.73M Games1.18M Game probes4.1M ALFWorld subgoal probe1.3M ALFWorld template swaps1.1M + 0.7M ALFWorld decisions: unseen / seen / seen with lookahead4.8M / 4.5M / 3.4M ALFWorld whole games, two runs each: alone / v1 / v22× 10.6M / 2× 9.2M / 2× 5.9M Stacker with skills: native task / new instructions0.33M / 0.98M Stacker with raw torques19.6M Total≈ 94M I Use of Large Language Models The authors conceived the research idea, designed the experiments and set the argument of the paper. They decided which experiments to run and report, and they shaped the logic and wording of every section through iterative review. An AI coding assistant, Claude Code (Anthropic), implemented most of the experiment and analysis code and ran the experiments under the authors’ direction. It also drafted text, which the authors revised sentence by sentence. Another assistant, OpenAI’s Codex, reviewed drafts for logic and consistency. Every reference was checked against its published source. The authors checked every number against the raw runs and take full responsibility for the content. Jev, the model we study, is part of the method and is described in the main text, not here. 20