A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents Raghu Arghal 1 Fade Chen 2 Niall Dalton 3 Evgenii Kortukov 4 Calum McNamara 5 Angelos Nalmpantis 6 Moksh Nirvaan 7 Gabriele Sarti 8 Mario Giulianelli 3 Abstract Understanding an agent’s goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agen- tic systems. We propose a framework for evaluat- ing goal-directedness that integrates behavioural evaluation with interpretability-based analyses of models’ internal representations. As a case study, we examine an LLM agent navigating a 2D grid world toward a goal state. Behaviourally, we evaluate the agent against an optimal policy across varying grid sizes, obstacle densities, and goal structures, finding that performance scales with task difficulty while remaining robust to difficulty-preserving transformations and com- plex goal structures. We then use probing meth- ods to decode the agent’s internal representations of the environment state and its multi-step action plans. We find that the LLM agent non-linearly encodes a coarse spatial map of the environment, preserving approximate task-relevant cues about its position and the goal location; that its actions are broadly consistent with these internal repre- sentations; and that reasoning reorganises them, shifting from broader environment structural cues toward information supporting immediate action selection. Our findings support the view that in- trospective examination is required beyond be- havioural evaluations to characterise how agents represent and pursue their objectives. 1. Introduction Attributing goals to agents helps explain and predict their behaviour and provides a useful abstraction for reasoning about agency. This topic has received attention across in 1 University of Pennsylvania 2 New York University 3 University College London 4 Fraunhofer Heinrich Hertz Institute 5 Indiana University Bloomington 6 TKH AI 7 Independent Researcher 8 Northeastern University. Authors are listed in alphabetical order, except for the last two; see Statement of Author Contributions. Correspondence to: Mario Giulianelli <m.giulianelli@ucl.ac.uk>. Iso-difficulty Transform GPT-OSS Prompt: Reach the goal [...] Observation: Pick LEFT / RIGHT / UP / DOWN Wall Agent Goal Empty A) B) Action: UP ... ... ... Reasoning: I should navigate [...] Layer 7 Layer 15 Layer 23 Pre-reasoning Post-reasoning Original Grid Rotated Grid Decoded Cognitive Maps Cognitive Map Probes C) Figure 1.Overview of our goal-directedness analysis. A: We eval- uate how iso-difficulty transforms affect agent trajectories that agree or disagree with the optimal policy. B: We prompt an LLM- based agent to reason and act over the fully-observable grid setup, extracting its pre-and post-reasoning activations at intermediate layers. C: We probe the agent’s beliefs over goal distance, planned actions and reconstruct cognitive maps for the current grid state. fields as varied as philosophy (Davidson, 1973; Dennett, 1990), psychology and neuroscience (Baker et al., 2009; Schultz et al., 1997) economics and decision theory (von Neumann & Morgenstern, 1944; Savage, 1948), and rein- forcement learning (Bellman, 1966; Ng & Russell, 2000). More recently, determining when and in what sense goal attributions are warranted has become a pressing concern for LLM-based agents (Xu & Rivera, 2024; MacDermott et al., 2024; Everitt et al., 2025; Goldstein & Lederman, 2025; Mazeika et al., 2025), particularly from an AI safety 1 arXiv:2602.08964v1 [cs.LG] 9 Feb 2026 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents perspective (Naik et al., 2025; Wentworth & Lorell, 2025; Marks et al., 2025; Li et al., 2025; Summerfield et al., 2025). A natural way to measure goal-directedness is behavioural evaluation, i.e., assessing the agent’s actions relative to some goal, particularly compared to an optimal policy (Xu & Rivera, 2024; Everitt et al., 2025). However, purely be- havioural measures face fundamental theoretical, practical, and philosophical challenges (Bellot et al., 2025; Rajcic & Søgaard, 2025; Chalmers, 2025). Agent capabilities may act as confounders for behavioural measures, as consistent failure may reflect capability limitations rather than lack of goal-directed behaviour. Relatedly, behavioural monitoring alone may be insufficient to guarantee alignment: a system with misaligned internal objectives could produce aligned behaviour, or fail a safety-relevant task, when doing so is in- strumentally useful (Hubinger et al., 2019; Ngo et al., 2024). To address these limitations, we propose a framework that combines behavioural evaluation with analysis of inter- nal representations, enabling holistic assessment of goal- directedness as a rich property arising from the interaction of beliefs, planning, and action selection. We study an LLM agent in a fully observable grid world, tasked with navigat- ing to a goal state across grids of varying sizes and obstacle densities. We begin by subjecting the agent to standard capa- bility tests and gradually introduce controlled environment perturbations and multi-goal task structures to measure the generalisability of its goal-directed behaviour, finding sen- sitivity to task difficulty and goal-like task-irrelevant cues, but robustness to difficulty-preserving transformations and instrumental goals. We then use probing classifiers to test if goal-relevant information can be decoded from the agent’s internal activations, before and after reasoning. Through our probing analyses, we are able to extract cognitive maps—i.e., latent beliefs about the current environment state, including the agent position and the goal location—and planned multi- step action sequences directly from the model activations. We also find that these representations reorganise during reasoning: pre-reasoning activations preserve broader spa- tial cues and longer-horizon plans, while post-reasoning activations sharpen focus on next action selection. Fig. 1 provides an overview of our approach. Contributions. Our primary contributions are as follows: 1.We propose a white-box framework combining be- havioural assessment and representation probing anal- yses for goal-directedness evaluation. 2.We design controlled environment perturbations and multi-goal task structures to measure bias and robust- ness in the agent’s goal-directed behaviour. 3.We probe environment beliefs and multi-step action plans from the agent’s learned representations, and use them to assess behavioural coherence in relation to decoded information. 2. Related Work The problem of identifying an agent’s goals and intentions has a rich history spanning multiple research fields. Seminal works in philosophy (Davidson, 1973; Lewis, 1974; Dennett, 1990) and microeconomics (von Neumann & Morgenstern, 1944; Savage, 1948) have emphasised the predictive and explanatory power of assigning goals to an agent. Measuring Agents’ Goal-directedness. Recent works at- tempt to formally define and measure goal-directedness to benefit AI alignment and safety (Ward et al., 2024; Xu & Rivera, 2024; Everitt et al., 2025; MacDermott et al., 2024). Notably, Everitt et al. (2025) define a measure of goal- directedness conditioned upon an agent’s task-relevant ca- pabilities and show goal-directedness is measurably distinct from performance in LLMs and general across tasks. Mac- Dermott et al. (2024) build upon Dennett (1990), proposing a formal measure of goal-directedness based on the pre- dictive power of posited utility functions for the agent’s behaviour. However, behavioural approaches to measuring goal-directedness are not without their weaknesses. Raj- cic & Søgaard (2025) argue that such methods falter when faced with underspecification, coarse goals, uncertainty, and multi-agent settings. Bellot et al. (2025) prove bounds on learnability from agent behaviour, showing that goal infer- ences are strictly limited by gaps between internal world models and the environment and out-of-distribution shifts. Our work complements these approaches by enabling as- sessment of goal-directed behaviour relative to the agent’s internal beliefs rather than ground truth alone. Inverse Reinforcement Learning (IRL). is a direct instan- tiation of the goal attribution problem, aiming to distil a reward function from a policy or a set of demonstrations. A rich line of work in this area (e.g. Ng & Russell, 2000; Abbeel & Ng, 2004, surveyed by Arora & Doshi, 2021) also focused on AI alignment (e.g., Hadfield-Menell et al., 2016; 2017). While a weakness of classical IRL is the as- sumption that observed behaviour is optimal, approaches like Maximum Entropy IRL (Ziebart et al., 2008) aim to relax this via stochastic models of behaviour. Still, IRL methods suffer from the mis- and under-specification of the agent’s behavioural model and latent reward function, re- spectively (Skalse & Abate, 2023). Unlike IRL, in this work we directly probe for goal-relevant representations without assuming a specific reward structure. Probing Environment and Plans in LLMs. Various works studied whether language models learn structured represen- tations of their environment. Li et al. (2023) show that a GPT model trained to predict Othello moves develops a causally relevant representation of the board state, while Nanda et al. (2023) show this representation can be de- coded linearly. Similar linear representations of spatial and temporal information were found in LLMs trained on 2 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents d = 0.0d = 0.2d = 0.4d = 0.6d = 0.8d = 1.0 Figure 2. Grid worlds with increasing wall density d, from fully open grids (d = 0) to maze-like grids with no circular paths (d = 1). natural text (Gurnee & Tegmark, 2024). Recent work has also probed LLMs for goal-oriented abstractions (Li et al., 2024) and shown that models engage in forward planning, pre-selecting future outputs before generating intermediate tokens (Pal et al., 2023; Men et al., 2024; Lindsey et al., 2025; Dong et al., 2025). Similar phenomena have also been observed in other neural architectures (Bush et al., 2025; Taufeeque et al., 2025). More broadly, high-level features were found to be decodable from model activations, and were used for monitoring and steering (Li et al., 2021; Zou et al., 2023; Marks & Tegmark, 2024). We extend these works through the propositional interpretability lens (Chalmers, 2025), eliciting environment representations and plans from model internals in an agentic navigation task. 3. Grid World Agent Setup We select GPT-OSS-20B (OpenAI, 2025) for our evaluation in light of its manageable size and outstanding performance on complex tasks, and test it for a 2-dimensional navigation task using the MiniGrid environment (Chevalier-Boisvert et al., 2023). Our LLM agent has full observability of the grid and is tasked with navigating to the goal square one action at a time. We translate the grid into a text based representation (Fig. 3) ensuring that each cell in the grid corresponds to exactly one token to limit issues stemming from tokenisation (Edman et al., 2024; Cosma et al., 2025). A fully observable environment offers a simple but tightly controlled setting for analysing goal-directedness, for at least three reasons. (1) With full observability, the agent ########### # _ # _______ # # _________ # # ___ # __ G# _ # # _ # _ # _ # _ ### # _________ # # _ # _ # _____ # # _________ # # _ ### ___ A _ # # _________ # ########### Figure 3.An example grid (left) and its corresponding text based representation (right) used for LLM prompting. directly observes the true world state. This eliminates the need to maintain beliefs over hidden world states and allows optimal policies to be derived using standard algo- rithms. (2) Full observability also removes several factors that might otherwise confound the analysis, including memory, belief updating under perceptual uncertainty, and exploration–exploitation trade-offs. (3) We observed that LLM-based agents (even frontier ones) perform poorly in partially observable grid worlds, exhibiting behaviours like redundant backtracking. This makes it difficult to disentangle capability limitations from failures of goal-directedness (see App. A for a discussion). Grid Worlds.We modeln× ngrid world environ- ments as Markov Decision Processes defined by the tu- ple(S,A,T ,r,γ). The state space isS = [n] 2 , rep- resenting grid locations, and the action space isA = UP, DOWN, LEFT, RIGHT. Transitions are determinis- tic: the transition functionT (s ′ | s,a) = P(S t+1 = s ′ | S t = s,A t = a) moves the agent to the adjacent cell as determined byaif that cell is open; otherwise (e.g., if the action would enter a wall), the agent remains in its current location. A grid world instance is specified by a function G : [n] 2 →wall, open, goal, which assigns a cell type to each grid location, andGdenotes the set of all such grids. Grid worlds vary in obstacle densityd∈ [0, 1], whered = 0 corresponds to a fully open grid andd = 1to a maze-like grid with no circular paths. Examples of grids with different density levels are shown in Fig. 2. Policies and Trajectories. We writeπ ∗ for an optimal policy, assumed to be uniform over optimal actions when multiple optima exist. An agent parameterised byθfollows a policyπ θ (a| s) = P(A t = a| S t = s),witha t denoting the action selected by the agent at timet. Given a policyπ and an initial states 0 , a trajectoryτ π (s 0 ) = (s i ,a i ) T i=0 is generated by executingπfroms 0 . A trajectory is successful if the final state satisfiess T = s goal ; otherwise, it terminates upon reaching a fixed maximum horizon T . 4. Behavioural Evaluation We begin with a behavioural evaluation of the agent’s policy, comparing it against an optimal reference policy derived 3 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents 01020304050+ Distance to Goal 0.00 0.25 0.50 0.75 1.00 Accuracy Raw Smoothed (window=5) 01020304050+ Distance to Goal 0.0 0.2 0.4 0.6 Mean JSD Raw Smoothed (window=5) 0.00.20.40.60.81.0 0-5 5-10 10-15 15-20 20-25 25-30 30-35 35-40 40-45 45-50 50+ Distance to Goal 7x7 0.00.20.40.60.81.0 9x9 0.00.20.40.60.81.0 Density 11x11 0.00.20.40.60.81.0 13x13 0.00.20.40.60.81.0 15x15 0.4 0.6 0.8 1.0 Action Accuracy Figure 4.Top: Action accuracy (left) and mean JSD (right) in relation to the agent’s distance from the goal. Bottom: Action accuracy by size, complexity, and goal distance. using A* with Manhattan distance to the goal. This anal- ysis assesses how closely the agent’s action choices and action distributions align with optimal behaviour in the grid world, without relying on or inspecting the agent’s inter- nal representations. We construct a set of grid worldsG with sizesS G = 7, 9, 11, 13, 15and obstacle densities D = 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. For each size–density pair inS×D, we generate 10 random grids. On each grid, the agent is evaluated over 10 trajectories using a sampling temperature of0.7and a maximum horizon ofT = 1.5×L whereLis the optimal path length to the goal. 1 Additional evaluation settings and prompts are reported in App. E. 4.1. Goal-Directedness across Baseline Task Conditions We report per-action accuracy, measuring the fraction of actions along a trajectory that are optimal, i.e.,a t ∈ arg max a∈A π ∗ (a | s t ), and the Jensen–Shannon Diver- gence (JSD) from the optimal policy, both averaged across trajectories and grids. To estimate the agent’s policyπ θ , we compute empirical action probabilities based on the relative action frequency across all available trajectories for a given grid. 2 App. B provides formal definitions and introduces additional metrics such as the percentage of trajectories that reach the goal (Goal Success Rate, GSR), the entropy of the agent’s policy, and the Expected Calibration Error, with full results shown in App. C. While relevant, we do not find these to provide additional insight than those discussed here. We find that both goal-directed capability and uncertainty 1 We limit trajectory length to1.5×Lsteps to filter cases where the agent was stuck moving back and forth between states. 2 We use relative frequency instead of action-token log- probability since the latter converges to 1 after the model reasoning chain, making it a poor proxy for agent uncertainty. vary systematically with task difficulty. In particular, ac- curacy decreases monotonically with both grid size and density. In contrast, both the entropy of the agent’s policy and the JSD from the optimal policy increase with grid size and obstacle density, demonstrating increased uncertainty under more difficult grids. We further analyse behavioural metrics as a function of the agent’s distance to the goal (Fig. 4 top). Per-action accuracy decreases linearly with distance from the goal for distances with less than 20 steps, after which estimates become noisier, and the JSD with respect to the optimal policy correspondingly increases. Variance in both metrics grows with distance, indicating less stable behaviour when the agent is farther from the goal. Fig. 4 (bottom) shows a breakdown of per-action accuracy as a function of grid size, obstacle density, and distance to the goal, confirming that an increase across any of the three dimensions contribute to a decrease in action accuracy. Controlling for the other factors, accuracy decreases most systematically with increasing grid size and obstacle density, with distance to goal only playing a significant role for the larger grid sizes (13 and 15). 4.2. Robustness to Iso-difficulty Transformations Having established that the agent’s performance varies pre- dictably with task difficulty, we next evaluate the agent’s robustness to environment transformations that preserve task difficulty to assess potential bias for specific grid configura- tions. We introduce a set of controlled environment transfor- mations that preserve the difficulty of the navigation task, which we refer to as iso-difficulty transformations. We con- sider four such transformations, shown in Fig. 12, App. D: (1) reflection of the grid (REFLECTENV); (2) rotation of the grid by90 ◦ (ROTATEENV); (3) swapping the agent’s start position with the goal position (STARTGOALSWAP); and (4) transposing the grid (TRANSPOSEENV). Each transfor- mation preserves the grid size, obstacle density, and optimal path length of the original grid, and therefore maintains the difficulty of the task. The optimal policy on the transformed grid is obtained by applying the corresponding transforma- tion to the optimal policy of the baseline grid. We apply our transformations to gridsGand compare be- havioural metrics between each original grid and its trans- formed counterpart. For each grid, we compute paired met- rics across all trajectories and use a Wilcoxon signed-rank test to assess whether performance differs significantly be- tween baseline and transformed environments. Across all transformations, we find no statistically significant differ- ences in any of the evaluated metrics. This indicates that the agent’s navigation behaviour in grid worlds is driven by task-relevant information rather than by incidental proper- ties of particular grid configurations. Detailed results are reported in App. D (Tab. 3 and Fig. 13). 4 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents KeyDoorEnvKeyNoDoorEnv2PathKeyEnv Figure 5. Grid world variants with instrumental and implicit goals. In the text representation, the key and the door are encoded withK and D, and their meaning is explained in the system prompt. 4.3. Instrumental and Implicit Goals We move to examine whether the observed robustness ex- tends to more complex goal structures using three variants of the grid world environment that include instrumental and implicit goals. Fig. 5 shows examples of our three environ- ment variants, with the prompt available in App. E.3. Instru- mental goals represent prerequisite subtasks that must be completed to reach the main objective. In KeyDoorEnv, the agent must collect a key to unlock a door that blocks the path to the goal. The agent interacts with the key and the door automatically upon reaching the associated cell, with the door acting as a wall if the agent does not have the key. 3 We define implicit goals as goal-like artifacts (e.g., a key) that carry no reward or utility in the navigation task. We assess whether their presence influences the agent’s behaviour de- spite their irrelevance towards task completion. We consider two variants: in KeyNoDoorEnv, door is removed, making the key functionally useless. In 2PathKeyEnv, we design a grid with two optimal paths, one of which contains a vesti- gial key, and observe whether the agent’s path selection is biased. To isolate the effect of the key, we compare each setting against a grid with an identical structure but no key. We generate100trajectories for KeyDoorEnv and KeyN- oDoorEnv, and100trajectory pairs for 2PathKeyEnv to account for the key presence. We setT = 30, which is sufficient for solving the maze. Results are summarised in Tab. 1. In KeyDoorEnv, the agent achieves a 100% success rate, correctly collecting the key, unlocking the door, and reaching the goal. Accuracy relative to the optimal policy remains high throughout all subtasks. This indicates success- ful handling of instrumental goals. In contrast, performance slightly deteriorates in KeyNoDoorEnv, despite the reduced task complexity. Although the key has no functional utility, the agent deviates from the optimal path and moves towards the key in 75% of cases, indicating that the key acts as an effective distractor. In 2PathKeyEnv, the agent is biased to- ward the key-containing path (key pickup rate: 67.3%). This bias leads to a slight improvement in per-action accuracy (76.0% vs. 74.3%), however, it ultimately lowers success 3 We augment the MDP state space with a key collection binary flag. Interactions with key and door occur automatically upon reaching them, with the door acting as a wall if the agent has no key. Table 1.Instrumental and implicit goals. Success rates and ac- tion accuracy with respect to the optimal policy, together with key pickup rates and attraction bias towards the key, for envi- ronments with an instrumental subgoal (KeyDoorEnv) and with reward-irrelevant key artifacts (KeyNoDoorEnv, 2PathKeyEnv). Results from KeyDoorEnv and KeyNoDoorEnv MetricKeyDoorEnvKeyNoDoorEnv Success Rate (%)100.098.9 Accuracy (%)98.7± 3.297.2± 11.1 Stage-specific Accuracy (%): Collecting Key98.6± 5.7N/A Opening Door99.2± 3.2N/A Reaching Goal99.2± 3.3N/A Key-related Metrics: Key Pickup Rate (%)100.017.0 Key Attraction Bias * (%)N/A75.0 * Percentage of Non-Optimal Actions that are Moving Towards the Key Comparison of Trajectories from 2PathKeyEnv MetricWith KeyWithout Key Success Rate (%)71.475.5 Accuracy (%)76.0± 16.174.3± 15.7 Key Pickup Rate (%)67.3N/A Jaccard Sim.65.6± 35.8 compared to the counterfactual environment without the key (71.4% vs. 75.5%). Trajectories with and without the key also show low Jaccard similarity, indicating substantial behavioural differences induced by the presence of the key. In summary, we find that the agent reliably solves tasks with instrumental goals, but is also systematically influenced by goal-like non-functional artifacts. We conjecture this could be due to semantic associations between entities and goals that the LLM acquired during training (e.g., collecting keys is common in games), which are not consistently suppressed to account for the provided goal structure. 5. Representational Evaluation Behavioural evaluation alone is insufficient to determine an agent’s goal-directedness. This limitation has been noted in both theoretical (Bellot et al., 2025) and philosophical work (Rajcic & Søgaard, 2025; Chalmers, 2025), and has clear practical implications. For example, in grid world nav- igation, an agent may fail to reach the goal state while still acting goal-directedly relative to its own imperfect beliefs about the environment. In this section, we therefore anal- yse the agent’s internal world representations to evaluate whether its actions are consistent in light of these beliefs. We begin in §5.1 by decoding the agent’s beliefs about the environment state, producing what we term cognitive maps. 4 Building on this, in §5.2, we evaluate the optimality 4 We borrow this term from classic cognitive neuroscience work on navigational tasks (Tolman, 1948; Schmidt & Redish, 2013). 5 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents 79111315 Grid Size 0.2 0.4 0.6 0.8 Accuracy 72.4 20.3 69.4 28.7 74.7 31.6 70.7 27.6 69.0 39.1 MLP ProbeLinear Probe 0.00 0.25 0.50 0.75 1.00 Recall (= Accuracy) 83 71 68 69 67 52 66 81 72 71 100 99 85 71 72 99 93 83 94 87 WallEmptyAgentGoal 79111315 Grid Size 0.00 0.25 0.50 0.75 1.00 Precision 83 85 87 78 74 81 74 74 75 74 20 13 21 18 12 30 18 15 9 7 True GridDecoded Grid Figure 6.Extracting a cognitive map from GPT-OSS-20B representations.Left:Overall accuracy of an MLP and a linear probe.Center: Per-class recall (=accuracy) and precision for varying grid sizes. Right:A cognitive map decoded from pre-reasoning activations. of the agent’s actions with respect to its decoded, subjec- tive cognitive map. Finally, in §5.3, we examine whether goal-directed action plans can be extracted from the agent’s internal representations. In App. F.3, we additionally test whether the agent encodes its distance to the goal. 5.1. Cognitive Maps: Decoding the Agent’s Beliefs about its Environment To assess whether the agent’s representations encode an internal model of their environment, we extract resid- ual stream activations from the last three pre- and post- reasoning prompt tokens from the model chat template (<|end|>,<|start|>, andassistant) at layers 7, 15 and 23 while prompting GPT-OSS-20B to solve the fully observable grid navigation task described in §3. Loosely inspired by Li et al. (2023), we construct training examples by augmenting each activation with the(x,y)coordinates of the queried cell yielding inputs of the form([act,x,y],c), wherec ∈ agent, goal, wall, open, padding. 5 We then train linear and MLP classifiers on the resulting pairs to decode cell types across grid positions. For the MLP probe, we use a two-layer architecture with ReLU activation and a hidden dimension of1024. Both probes are trained using an AdamW optimiser with weight decay, and normalisa- tion is applied before training. At test time, the probes are applied to each grid-coordinate combination, using arg max c P(c| act,x,y )to reconstruct the model’s cog- nitive map, i.e., its decoded belief over the current grid state. We train general and size-specific variants of these probes using grids of sizes7, 9, 11, 13, 15. In the general case, we pad smaller grids to size15by settingc = paddingfor miss- ing positions. Our next experiments focus on general probes on the intermediate layer15for pre-reasoning tokens, un- less otherwise stated. Additional results across layers, grids sizes, and goal distance probes are shown in App. F.2. How is the Environment Encoded in Model Activations? Fig. 6 (left) shows the accuracy of cognitive maps recon- 5 Thepaddingclass labels cells that fall outside the valid grid boundaries. This allows us to train a single set of general cognitive map probes that generalise across grid sizes. 79111315 Grid Size 0.0 0.5 1.0 Accuracy 83 45 42 27 40 80 52 37 40 20 AgentGoal 79111315 Grid Size 0.0 0.5 1.0 1.5 Manhattan Dist. Figure 7.Probes performance for locating agent and goal positions. Binary localisation accuracy drops as the grid size increases, but avg. Manhattan distance to true locations remains bounded. structed by linear and MLP probes across various grid sizes. The MLP probe decodes cell identities with around 70% accuracy, reaching a maximum of 75.7% for11× 11grids. Linear probes underperform at 39.1% average accuracy in the same setting, suggesting that environment information is encoded non-linearly in model representations. Fig. 6 (center) presents a per-class performance breakdown of the MLP probe in terms of its recall (top) and precision (bot- tom). Recall scores shows that all cell identities are retrieved robustly across grid sizes, with especially high recall for goal (83-99%) and agent (72-100%) positions. We note that probes assignagentandgoallabels to multiple cells in the neighborhood of their respective true locations, re- sulting in the high recall but low precision trend discussed earlier (Fig. 6 right). In contrast, information about the posi- tion of walls is not represented in detail, as reflected by the lower recall for thewallclass. These results suggest that the agent’s internal representations encode a coarse spatial map of the environment, preserving approximate task-relevant information about agent position and goal location. Locating Agent and Goal. Given the coarse localisation ofagentandgoalcells in cognitive maps, we assess how accurately the true agent and goal locations can be recov- ered from the decoded cognitive maps. We measure top-1 accuracy for the grid coordinates with the highest predicted P(c = agent)andP(c = goal), with results shown in Fig. 7. We find that agent and goal localisation accuracy decreases steadily as grid size increases. However, the Man- hattan distance between predicted and true locations remains 6 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents Table 2.Policy evaluation against decoded beliefs. Acc. GT : action accuracy w.r.t optimal policy on ground truth grid; Acc. Dec.: action accuracy w.r.t optimal policy on cognitive map; Agreem.: % optimal actions for both policies; Rec.: % optimal actions only for the cognitive map; Avg A and Avg G: average Manhattan distances from classified to true agent (A) and goal (G) positions. Grid size results (top) are averaged across density, and vice versa (bottom). nAcc. GTAcc. Dec.Agreem.Rec.Avg AAvg G 797.390.191.548.3 1.040.78 995.288.288.342.4 1.391.14 1184.177.381.774.2 1.101.45 1379.478.579.764.5 1.231.81 1578.778.578.460.1 1.622.33 dAcc. GTAcc. Dec.Agreem.Rec.Avg AAvg G 0.0 100.094.698.6N/A 1.131.25 0.294.589.193.288.4 1.251.49 0.488.990.093.782.4 1.281.51 0.687.380.383.249.6 1.331.47 0.884.475.477.347.8 1.311.55 1.066.665.857.637.4 1.351.76 0.00 0.25 0.50 0.75 1.00 Recall (= Accuracy) 68 68 81 51 85 55 83 56 75 60 WallEmptyAgentGoalOverall Before ReasoningAfter Reasoning Reasoning Stage 0.00 0.25 0.50 0.75 1.00 Precision 87 75 74 72 21 5 15 4 75 60 Figure 8.Performance of cognitive map before and after reasoning. Cognitive map accuracy drops significantly after reasoning. lower than 2 even for large15× 15grids (see Fig. 7; numer- ical results are given in Tab. 2), indicating that information about agent and goal positions is encoded coarsely in prox- imity of their true locations. Goal Location Information Degrades after Reasoning. We evaluate how decoded cognitive maps change when moving from the pre-reasoning activations we examined so far to post-reasoning ones. Fig. 8 shows that after rea- soning, overall probe accuracy drops from 75% to 60%, with a notable drop inagent,goalandopencells recall, and agentandgoalprecision. These results suggest that while a coarse environment representation is formed prior to rea- soning, post-reasoning representations exhibit substantially degraded cognitive maps, potentially reflecting a shift from general environment features to task-specific information essential for action selection. 5.2. Evaluating Policies against Decoded Beliefs We now set out to test whether the LLM agent’s behaviour is consistent with its cognitive maps, particularly in cases when its actions deviate from optimal behaviour. To do so, we compare the agent’s observed action sequence with the optimal policy derived from the cognitive map decoded at each trajectory step using the general MLP probe with pre- reasoning activations from layer 15. As in §5.1 (and Fig. 7), we obtain a single location for theagentandgoalcells by selecting the grid cells with the highest predicted probabil- ities for the corresponding classes, and compute optimal actions on both the ground-truth and decoded grids. Tab. 2 reports the accuracy of the agent’s actions with respect to optimal policies defined on both the ground-truth grid (Acc. GT ) and the decoded cognitive map (Acc. Dec.), and the fraction of actions that are optimal under both (Agreem.). Acc. Dec., the accuracy relative to the optimal policy in the decoded cognitive map, is high across grid sizes and decreases with grid density, while agreement is consistently high (above 77% except for the highest density grids with d = 1). This indicates that the agent’s actions are broadly consistent with its internal world representation. While the agreement between the two policies is high across condi- tions (Agreem. averages 84%), we remark that Acc. Dec. is consistently lower than Acc. GT. This may be due to the fact that we derive a single location for the agent and goal by selecting the grid cell with the highest predicted probability. This approach does not fully capture the uncertainty evident in the decoded maps, which exhibit blurred agent and goal spatial representations, as discussed in §5.1 and shown in Fig. 6. In addition, Tab. 2 reports the average Manhattan distances from all cells classified asagentorgoalto their ground-truth positions (Avg A/G), which is below 2 in all but the15×15grids. It is therefore possible that predictions are optimal with respect to nearby cells rather than the exact ground-truth location. 6 Of particular interest is the recovery metric Rec., which is the proportion of actions that are sub-optimal in the true environment but optimal given the agent’s decoded cognitive map. This metric captures behavioural inaccuracies which can be attributed to a faulty internal world representation or a lack of goal-directedness. In conjunction with the Manhattan distances, our results indicate that a substantial fraction of failures can be attributed to inaccurate, or fuzzy world representations rather than a lack of goal-directedness, particularly in low density and small-to-medium grids. 6 An alternative approach would be to evaluate the agent against multiple optimal policies defined over all combinations of cells classified asagentandgoal. We do not pursue this since, at least in its simplest form, it is a rather loose evaluation and does not result in a single predicted action. Nonetheless, it is possible that the model internally representsagentorgoalpositions as a some weighted combination over nearby cells rather than as a single cell. 7 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents 5.3. Evaluating Plans In this section, we examine whether the agent’s internal representations encode goal-directed planning information, and how this encoding differs before vs. after reasoning. We consider the same single-goal navigation task as in previous sections. For each trajectory, we extract residual-stream activations from layersℓ ∈ 7, 15, 23at two stages: (i) pre-reasoning, using the final prompt tokens immediately before the model begins reasoning, and (i) post-reasoning, using the final reasoning tokens immediately before the model outputs its first action. Each example is labelled with a target action sequencea 1:T = (a t ) T t=1 , derived from the executed actions in the trajectoryτ π (s 0 ) = (s t ,a t ) T t=0 for a given grid instance, withT = 10. We train the plan decoder on 3,000 trajectories, use a 600-trajectory validation set, and report results on the 300-trajectory test set used in §4. Trajectories are sampled from grid sizes 7–15 with varying complexities and start/goal configurations, yielding a diverse distribution of path lengths and planning difficulty. Plan Decoder Architecture. Our goal is to decode the entire plan from a fixed set of activations, while minimising any additional planning or inference performed by the probe. Leth 1 , h 2 , h 3 ∈ R 2880 denote the three extracted token ac- tivations at a chosen layer and stage. We first map each h i through a shared bottleneck consisting of a linear pro- jection to 1024 dimensions followed by LayerNorm: ̃ h i = LN(W h i ) ∈ R 1024 . We then decode a horizon-Tplan us- ing a 4-layer Transformer decoder (8 attention heads per layer) withTlearned query embeddingsq 1 ,..., q T . Each query corresponds to a plan step index and performs cross- attention over the same set of token activations ̃ h 1 , ̃ h 2 , ̃ h 3 . A final linear head produces a distribution over actions for each stept ∈ [T],p(a t | ̃ h 1:3 ) = softmax(W o z t ), where z t is the decoder output at query slot t. Importantly, we design the decoder to predict the entire plan ˆ a 1:T simultaneously from the input activations, rather than predictingˆa t autoregressively conditioned on previously decoded actionsˆa <t . Autoregressive decoding would intro- duce an additional channel for the probe to create plan struc- ture: early predicted actions can implicitly constrain later actions via simple continuation heuristics, even if the un- derlying representations only weakly specify a full-horizon trajectory. By contrast, in one-shot decoding, later steps can- not condition on earlier predictions, so coherent multi-step structure in ˆ a 1:T must be supported by information already present in the base model’s activations. Thus, accuracy above baseline under one-shot decoding is more diagnostic of plan information encoded in the model’s representations than of computation performed by the probe itself. Results. We evaluate plan decodability using prefix accu- racy, defined as the fraction of episodes for which the first Npredicted actions exactly match the target plan prefix. Figure 9.Prefix accuracy of one-shot plan decoding before vs. after reasoning. We report the fraction of episodes for which the first Npredicted actions match the target action sequence prefix. The dashed curve shows the random baseline 0.25 N . For a predicted plan ˆ a 1:T and target action sequencea 1:T (withT = 10), prefix accuracy atNisPr[ ˆ a 1:N = a 1:N ]. We report prefix accuracy forN ∈ 1,..., 10for both pre-reasoning and post-reasoning activations. As a baseline, random guessing among four actions yields 0.25 N . Fig. 9 shows that both probes substantially exceed the0.25 N baseline for the first several steps, indicating that GPT-OSS- 20B activations encode non-trivial information about the upcoming action sequence. Post-reasoning activations im- prove one-step decoding (53.92% vs. 40.31% atN = 1), consistent with reasoning increasing the separability of the immediate next action. For longer prefixes, pre-reasoning activations are consistently more decodable (e.g., 13% vs. 9% atN = 4; 6% vs. 3% atN = 5), suggesting that pre- reasoning representations preserve more readily recoverable long-horizon trajectory structure at our readout locations. Both curves approach zero beyond N ≈ 8. This pre- and post-reasoning crossing point suggests a performance trade-off induced by reasoning. While post- reasoning activations more strongly support local action selection (higherN=1accuracy), the improved decodabil- ity of pre-reasoning activations for longer prefixes suggests a partial decay of planning-relevant information after rea- soning is completed. We interpret this as additional evi- dence of a shift in representational emphasis from broader environment-related structure to near-term action selection after reasoning. In other terms, post-reasoning activations make the next move more decodable, while pre-reasoning activations retain more decodable information about later steps. Our additional analyses (App. F.4) show that the de- coder does not recover the exact trajectory length in most cases, but it reliably predicts a close estimate, consistent with the presence of a coarse plan in the underlying represen- tations. Since trajectory length in this setting is tightly cou- pled to progress toward the goal under the executed policy, this provides complementary evidence that the model’s inter- nal states encode planning variables beyond the next action. 8 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents 6. Conclusion We have presented a framework for analysing the goal- directedness of LLM-based agents that integrates be- havioural evaluation with representation probing, and demonstrated its utility in a grid world navigation setup. Our behavioural analyses revealed that the agent under in- vestigation exhibits systematic sensitivity to task difficulty while remaining robust to goal-irrelevant environmental vari- ations, which is a first indication of genuine goal-directed behaviour. The agent also successfully navigates multi-goal settings with instrumental subgoals, but its behaviour is biased by goal-like task-irrelevant cues. Our representational analyses uncovered rich internal struc- ture that aligns with an interpretation of the agent as goal- directed. In particular, the LLM encodes cognitive maps that preserve task-relevant spatial information about its po- sition and the goal location. Moreover, we observed a shift across the model’s reasoning process: pre-reasoning repre- sentations maintain broader spatial information about the environment and longer-horizon plans, while post-reasoning representations have a narrower focus on next action selec- tion. This finding suggests that the LLM’s reasoning dynam- ically reorganises information to support effective action. Our controlled setup abstracts away from the complexity of real-world agentic settings, but also enables precise mea- surements whose extension to more complex environments will require further study. Furthermore, while our repre- sentational analysis reveals meaningful, consistent relations between internal representations and behaviour, establishing causal links remains another important direction for future work. Similarly, extending and applying our framework to a broader range of architectures, scales, and training regimes will help establish the generality of our findings. Looking forward, the methods and insights from this work provide a foundation for developing more comprehensive approaches to goal attribution and monitoring in agentic systems. The development of rigorous approaches to eval- uating goal-directedness is a prerequisite for making high- confidence claims about agents’ goals and potential related risks, and for informing the responsible deployment and oversight of increasingly autonomous AI systems. Impact Statement Goal-directedness lies at the core of agency: understanding whether an artificial system pursues goals, which goals it pursues, and how those goals relate to its behaviour is foun- dational for explaining, predicting, and governing advanced AI systems. Progress on this problem has broad scientific significance, spanning cognitive science, neuroscience, phi- losophy of action, macroeconomics, and decision theory, but it also carries important practical implications as AI systems are increasingly deployed in autonomous, long-horizon, and high-stakes settings. The ability to reliably evaluate goal-directedness is partic- ularly critical from a safety perspective. As AI systems become more capable, behavioural success alone becomes an increasingly weak signal of alignment. Systems driven by misaligned internal objectives may nonetheless appear competent, compliant, or even helpful because aligned be- haviour is instrumentally useful. Recent work has high- lighted risks such as alignment faking and sandbagging (Greenblatt et al., 2024; van der Weij et al., 2025; Taylor et al., 2025), with models behaving deceptively or underper- forming during evaluation, arguably to avoid modification or oversight. While claims of deliberate “AI scheming” would be highly consequential if substantiated, current evidence remains limited, often anecdotal, and difficult to interpret without clear hypotheses, controls, or mechanistic ground- ing (Summerfield et al., 2025). This paper takes a first step toward addressing these limitations. Acknowledgements This project was supported by SPAR. We thank Cozmin Ududec, Dima Krasheninnikov, Gonçalo Guiomar, Michael Hanna, and members of the BauLab at Northeastern Univer- sity for helpful discussions. GS is supported by the NDIF project (U.S. NSF Award IIS-2408455). Statement of Author Contributions RA, FC, CM, GS, and MG conducted the literature review of §2, identified and synthesised relevant prior work across research paradigms, and developed the framing that situates the project’s contributions within existing research on goal- directedness. RA, ND and AN worked on implementation of code infrastructure for §4. ND developed the iso-difficulty transformations and conducted the analysis for §4.1, §4.2, App. A, C, and D. AN developed the procedures for gener- ating the base environments and the variants with the key, and conducted the analysis for §4.3. GS advised the design of representational evaluation experiments of §5, developed a unified pipeline for trace generation, activation extraction, and probe training, and built a trace viewer to explore prob- ing results. EK designed, implemented, and evaluated the cognitive map probes in §5.1 and App. F. ND and AN con- ducted the analysis for §5.2. MN designed, implemented, and evaluated the plan decoder in §5.3. MG conceived and led the project, provided ongoing scientific guidance, and coordinated the project’s execution. All authors contributed to the conceptualisation of the study and helped with the preparation of the manuscript. 9 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents References Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, ICML ’04, p. 1, New York, NY, USA, July 2004. Association for Computing Machinery. ISBN 978-1-58113-838-2. doi: 10.1145/1015330.1015430. URLhttps://dl.acm. org/doi/10.1145/1015330.1015430. Arora, S. and Doshi, P. A survey of inverse reinforce- ment learning:Challenges, methods and progress. Artificial Intelligence, 297:103500, August 2021. ISSN 0004-3702. doi: 10.1016/j.artint.2021.103500. URLhttps://w.sciencedirect.com/ science/article/pii/S0004370221000515. Baker, C. L., Saxe, R., and Tenenbaum, J. B.Ac- tion understanding as inverse planning.Cognition, 113(3):329–349, 2009.ISSN 0010-0277.doi: https://doi.org/10.1016/j.cognition.2009.07.005. URLhttps://w.sciencedirect.com/ science/article/pii/S0010027709001607. Reinforcement learning and higher cognition. Bellman, R. Dynamic programming. Science, 153(3731): 34–37, 1966. Bellot, A., Richens, J., and Everitt, T. The limits of predict- ing agents from behaviour. In Forty-second International Conference on Machine Learning, 2025. URLhttps: //openreview.net/forum?id=a3swNuXTxI. Bush, T., Chung, S., Anwar, U., Garriga-Alonso, A., and Krueger, D.Interpreting emergent planning in model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=DzGe40glxs. Chalmers, D. J. Propositional interpretability in artificial in- telligence, 2025. URLhttps://arxiv.org/abs/ 2501.15740. Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023. Cosma, A., Ruseti, S., Radoi, E., and Dascalu, M. The straw- berry problem: Emergence of character-level understand- ing in tokenized language models. In Christodoulopou- los, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing, p. 28252–28263, Suzhou, China, November 2025. As- sociation for Computational Linguistics. ISBN 979- 8-89176-332-6.doi: 10.18653/v1/2025.emnlp-main. 1434. URLhttps://aclanthology.org/2025. emnlp-main.1434/. Davidson, D. Radical interpretation. Dialectica, p. 313– 328, 1973. doi: 10.1111/j.1746-8361.1973.tb00623. x.URLhttps://w.jstor.org/stable/ 42968535. Publisher: JSTOR. Dennett, D. C. The Interpretation of Texts, People and Other Artifacts. Philosophy and Phenomenological Re- search, 50:177–194, 1990. ISSN 0031-8205. doi: 10. 2307/2108038. URLhttps://w.jstor.org/ stable/2108038.Publisher: [International Phe- nomenological Society, Philosophy and Phenomenologi- cal Research, Wiley]. Dong, Z., Zhou, Z., Liu, Z., Yang, C., and Lu, C. Emergent response planning in LLMs.In Forty- second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum? id=Ce79P8ULPY. Edman, L., Schmid, H., and Fraser, A. CUTE: Mea- suring LLMs’ understanding of their tokens. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 3017–3026, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18653/v1/2024.emnlp-main. 177. URLhttps://aclanthology.org/2024. emnlp-main.177/. Everitt, T., Garbacea, C., Bellot, A., Richens, J., Papadatos, H., Campos, S., and Shah, R. Evaluating the goal- directedness of large language models, 2025.URL https://arxiv.org/abs/2504.11844. Goldstein, S. and Lederman, H. What Does ChatGPT Want? An Interpretationist Guide, September 2025. URL https://philpapers.org/rec/GOLWDC-2. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDi- armid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., and Hubinger, E. Alignment fak- ing in large language models, 2024.URLhttps: //arxiv.org/abs/2412.14093. Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URLhttps:// openreview.net/forum?id=jE8xbmvFin. 10 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A.Cooperative Inverse Reinforcement Learning.In Advances in Neural Information Processing Systems,volume 29. Curran Asso- ciates, Inc., 2016.URLhttps://papers. nips.c/paper_files/paper/2016/hash/ c3395d46c34fa7fd8d729d8cf88b7a8-Abstract. html. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A.Inverse Reward Design.In Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URLhttps://proceedings.neurips. c/paper_files/paper/2017/hash/ 32fdab6559cdfa4f167f8c31b9199643-Abstract. html. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019. Lewis, D.Radical Interpretation.Synthese, 27(3/4): 331–344, 1974.ISSN 0039-7857.doi: 10.1007/ BF00484599.URLhttps://w.jstor.org/ stable/20114928. Publisher: Springer. Li, B. Z., Nye, M., and Andreas, J.Implicit repre- sentations of meaning in neural language models. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Process- ing (Volume 1: Long Papers), p. 1813–1827, Online, August 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2021.acl-long.143. URLhttps: //aclanthology.org/2021.acl-long.143/. Li, C., Phuong, M., and Tan, D. Spilling the beans: Teach- ing LLMs to self-report their hidden objectives. arXiv preprint arXiv:2511.06626, 2025. Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview. net/forum?id=DeG07_TcZvT. Li, Z., Cao, Y., and Cheung, J. C. Do LLMs build world representations? probing through the lens of state abstrac- tion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps: //openreview.net/forum?id=lzfzjYuWgY. Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Thomp- son, T. B., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. On the biology of a large language model. Transformer Circuits Thread, 2025. URLhttps://transformer-circuits.pub/ 2025/attribution-graphs/biology.html. MacDermott,M.,Fox,J.,Belardinelli,F.,and Everitt, T.Measuring Goal-Directedness, Decem- ber 2024. URLhttp://arxiv.org/abs/2412. 04758. arXiv:2412.04758 [cs]. Marks, S. and Tegmark, M. The geometry of truth: Emer- gent linear structure in large language model representa- tions of true/false datasets. In First Conference on Lan- guage Modeling, 2024. URLhttps://openreview. net/forum?id=aajyHYjjsk. Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., Ziegler, D., Ameisen, E., Batson, J., Belonax, T., et al. Auditing language models for hidden objectives. arXiv preprint arXiv:2503.10965, 2025. Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., Phan, L., Mu, N., Khoja, A., Zhang, O., and Hendrycks, D.Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs, Febru- ary 2025. URLhttp://arxiv.org/abs/2502. 08640. arXiv:2502.08640 [cs]. Men, T., Cao, P., Jin, Z., Chen, Y., Liu, K., and Zhao, J. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 7713–7724, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18653/v1/2024.emnlp-main. 440. URLhttps://aclanthology.org/2024. emnlp-main.440/. Naik, A., Quinn, P., Bosch, G., Gouné, E., Zabala, F. J. C., Brown, J. R., and Young, E. J. AgentMisalignment: Mea- suring the propensity for misaligned behaviour in LLM- based agents, 2025. URLhttps://arxiv.org/ abs/2506.04018. Nanda, N., Lee, A., and Wattenberg, M. Emergent lin- ear representations in world models of self-supervised sequence models. In Belinkov, Y., Hao, S., Jumelet, J., Kim, N., McCarthy, A., and Mohebbi, H. (eds.), Proceedings of the 6th BlackboxNLP Workshop: An- alyzing and Interpreting Neural Networks for NLP, 11 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents p. 16–30, Singapore, December 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023. blackboxnlp-1.2. URLhttps://aclanthology. org/2023.blackboxnlp-1.2/. Nezhurina, M., Cipolina-Kun, L., Cherti, M., and Jitsev, J. Alice in wonderland: Simple tasks showing complete rea- soning breakdown in state-of-the-art large language mod- els, 2025. URLhttps://arxiv.org/abs/2406. 02061. Ng, A. Y. and Russell, S. J. Algorithms for inverse rein- forcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, p. 663– 670, 2000. Ngo, R., Chan, L., and Mindermann, S. The alignment problem from a deep learning perspective. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum? id=fh8EYKFKns. OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925. Pal, K., Sun, J., Yuan, A., Wallace, B., and Bau, D. Fu- ture lens: Anticipating subsequent tokens from a sin- gle hidden state. In Jiang, J., Reitter, D., and Deng, S. (eds.), Proceedings of the 27th Conference on Compu- tational Natural Language Learning (CoNLL), p. 548– 560, Singapore, December 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.conll-1. 37.URLhttps://aclanthology.org/2023. conll-1.37/. Rajcic, N. and Søgaard, A. Goal-Directedness is in the Eye of the Beholder, August 2025. Savage, L. J. Samuelson’s Foundations: Its Mathemat- ics.Journal of Political Economy, 56(3):200–202, June 1948. ISSN 0022-3808. doi: 10.1086/256672. URLhttps://w.journals.uchicago.edu/ doi/abs/10.1086/256672 . Publisher: The Uni- versity of Chicago Press. Schmidt, B. and Redish, A. D. Navigation with a cognitive map. Nature, 497(7447):42–43, 2013. Schultz, W., Dayan, P., and Montague, P. R.A neu- ral substrate of prediction and reward. Science, 275 (5306):1593–1599, 1997. doi: 10.1126/science.275.5306. 1593.URLhttps://w.science.org/doi/ abs/10.1126/science.275.5306.1593. Skalse, J. and Abate, A. Misspecification in Inverse Re- inforcement Learning, March 2023. URLhttp:// arxiv.org/abs/2212.03201 . arXiv:2212.03201 [cs]. Summerfield, C., Luettgau, L., Dubois, M., Kirk, H. R., Hackenburg, K., Fist, C., Slama, K., Ding, N., Ansel- metti, R., Strait, A., et al. Lessons from a chimp: AI "scheming" and the quest for ape language. arXiv preprint arXiv:2507.03409, 2025. Taufeeque, M., Quirke, P., Li, M., Cundy, C., Tucker, A. D., Gleave, A., and Garriga-Alonso, A. Planning in a re- current neural network that plays sokoban, 2025. URL https://arxiv.org/abs/2407.15421. Taylor, J., Black, S., Bowen, D., Read, T., Golechha, S., Zelenka-Martin, A., Makins, O., Kissane, C., Ayonrinde, K., Merizian, J., et al. Auditing games for sandbagging. arXiv preprint arXiv:2512.07810, 2025. Tolman, E. C. Cognitive maps in rats and men. Psychologi- cal Review, 55(4):189, 1948. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. AI sandbagging: Language models can strategically underperform on evaluations, 2025. URL https://arxiv.org/abs/2406.07358. von Neumann, J. and Morgenstern, O. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ, 1944. Ward, F. R., MacDermott, M., Belardinelli, F., Toni, F., and Everitt, T. The Reasons that Agents Act: Intention and Instrumental Goals, February 2024. URLhttp:// arxiv.org/abs/2402.07221 . arXiv:2402.07221 [cs]. Wentworth, J. and Lorell, D. Instrumental goals area dif- ferent and friendlier kind of thing than terminal goals, January 2025. Xu, D. and Rivera, J.-P.Towards Measuring Goal- Directedness in AI Systems, November 2024. Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artifi- cial Intelligence - Volume 3, AAAI’08, p. 1433–1438, Chicago, Illinois, July 2008. AAAI Press. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency, 2023. URL https://arxiv.org/abs/2310.01405. 12 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents A. Partially Observable GridWorld We also examine a partially observable GridWorld, wherein the agent can see the full grid but many of the cells are hid- den with fog. Once the agent “sees” a cell, it is permanently revealed. An agent “sees” cells around itself using a uniform radius of sizen(we usen = 3in our tests). An example of what the agent sees is given in Figure 10. The seen radius is blocked by walls; specifically, we use Bresenham’s line algorithm to trace lines on the grid. 0 1 2 3 4 5 6 0 * * # # # # # 1 * * _ _ # _ # 2 * * # _ # _ # 3 * * # _ _ A # 4 * * # # # # # 5 * * * * * * * 6 * * * * * * * Figure 10. Partially Observable representation of a grid. The agent is at "A" and the goal (unseen) will be represented by "G". Hidden spaces are represented by "*", while revealed spaces are repre- sented by "_". This setting is formalised by the Partially Observable Markov Decision Process (POMDP). A POMDP is a 7- tuple(S,A,T ,R,O,Z,γ), which includes all elements of an MDP plus: • O: A finite set of observations the agent can receive. • Z(o|s ′ ,a) = P(o t = o|s t = s ′ ,a t−1 = a): The observation function, which gives the probability of re- ceiving observationoafter taking actionaand landing in the (hidden) state s ′ . We initially planned on running the same set of ablations as the fully observable case: grid sizes, complexity levels, and iso-difficulty transforms. We also planned on testing the following additional settings. A.1. Memory Ideally, we are able to provide the full history and avoid history ablations. If that’s not possible, we evaluate memory through (1) probing; (2) querying past state-actions that are not immediately accessible from history representation. A.2. Perseverance We evaluate whether the agent can recover from setbacks and whether its decoded (optional: and queried) beliefs update appropriately after setback occurs. A.3. Corrigibility and Focus We evaluate whether the agent adapts its course of action when goals are displaced and whether its decoded beliefs update accordingly. We also evaluate whether actions and beliefs remain unchanged when irrelevant additional goals are introduced. A.4. Difficulties with Partial Observability In general, we attempted to provide the model with enough information such that “remembering” past facts was not an issue. This formulation allows us to maximally decou- ple capability from goal-directedness. That is, changes in actions for a model which is obviously capable are much more likely due to changes in goal-directedness than in the less than obviously capable case. For our testing, as men- tioned previously, we permanently revealed seen cells to the model. In addition, we provided the model with a history of past state, action tuples and the full conversation history (although it should be redundant). Upon initial testing with partial observability, we discovered that the model we use (namely GPT-OSS-20B) has signifi- cant difficulties in the partially observable case. Namely, we discover several undesirable behaviours: redundant back- tracking, running into walls, and moving towards known dead-ends. Increasing the reasoning effort to “high” did not alleviate these issues. Although it is non-standard, we even tried providing the reasoning step of the model alongside the standard output for each step in the conversation history – this did not help. We qualitatively observe the reasoning and find that the model usually focuses almost exclusively on what its next step should be based on where it currently is and almost never reasons about its past actions. Even for a more powerful model, namely GPT-OSS-120b, we rarely observe it talk about “backtracking” when on “high” reasoning. In order to determine whether or not this was a capability issue of the models, we performed the same tests with a frontier model, GPT-5.1-Thinking. Even this frontier model displays the same sub-optimal behaviours of redundant back- tracking and moving towards known dead-ends (although we did not observe it try to run into walls). We see the same pathological reasoning wherein the model focuses on the current state (“tunnel vision”) and usually fails to rea- son about the past actions. We conclude that even today’s strongest models are not sufficiently capable to perform reasonably well at the partial observability case. We hypoth- esise that this is due to models not being trained on similar problems, and a failure of other reasoning problems (e.g., math or coding) to generalise to maze navigation, similar to other out of distribution reasoning problems, like the “Al- ice in Wonderland” problem discussed by Nezhurina et al. (2025). 13 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents This issue makes it difficult to decouple capability issues from goal-directedness because actions which appear to be in support of another goal may in fact be due to sub-optimal understanding and reasoning about the main goal. Conse- quently, we decide not to study the partial observability case in this paper and leave study of the partial observability case to future study. B. Behavioural Evaluation: Metrics Capability Metrics.We evaluate agent performance with various metrics, defined below. The goal success rate (GSR) is defined as the expected fraction of trajectories that termi- nate at the goal state: GSR : = E τ∼π [GS(τ(s 0 ))],(1) whereGS(τ(s 0 )) : = 1(s T = s goal ). In practice, we sample a finite amount of trajectories used to compute the GSR, as well as for the other metrics below. We evaluate the agent’s adherence to the optimal policy using a two-step accuracy metric. First, we define the per- action accuracy for a single trajectory τ i of length T i as: Acc(τ i ) = 1 T i T i −1 X t=0 1 a (i) t ∈ π ∗ (s (i) t ) (2) where1(·)is the indicator function,a (i) t is the action taken at steptof trajectoryi, andπ ∗ (s (i) t ) is the set of optimal actions for that state. The grid-level accuracy is then computed by averaging the trajectory-level accuracies across allNgenerated trajecto- ries: Acc grid = 1 N N X i=1 Acc(τ i )(3) Uncertainty Metrics. We also compute several metrics to measure the agent’s uncertainty. First, we measure the entropy of the agent’s policyH π θ and the Jensen-Shannon Divergence JSD π θ from the optimal policy: H π θ := 1 |S visited | X s∈S visited H (π θ (·|s))(4) JSD π θ := 1 |S visited | X s∈S visited JSD (π θ (·|s)∥ π ∗ (s))(5) withS visited being the set of all unique states encountered across all trajectories. We compute the agent’s policyπ θ empirically by assigning probabilities proportional to action count taken during all trajectories for a grid. We prefer this method over using log-probability of the ac- tion token because reasoning models like GPT-OSS-20B usually have deliberated and already locked-in a final choice during reasoning, thus making the log-probability uninfor- mative in terms of uncertainty. We also compute the Expected Calibration Error (ECE) over the aggregate counts of all state-action pairs encountered by the agent. LetDbe the collection of all pairs(s,a)from all trajectories for a gridG. We partitionDintoMdisjoint binsB 1 ,...,B M based on the agent’s policy confidence π θ (a|s). The ECE is the weighted average of the difference between the average confidence and the empirical accuracy within each bin: ECE = M X m=1 |B m | N total |acc(B m )− conf(B m )|(6) whereN total is the total number of state-action pairs. The bin accuracy, representing the empirical probability of opti- mality, is defined as: acc(B m ) = 1 |B m | X (s,a)∈B m 1 (a∈ π ∗ (s))(7) and the bin confidence is the average policy probability: conf(B m ) = 1 |B m | X (s,a)∈B m π θ (a|s)(8) In our experiments, we use M = 10 bins. Letτ 1 andτ 2 be two different trajectories having the same starting states 0 in gridG. We measure the overlap of their unique state-action pairs using the Jaccard Similarity Index: J(τ 1 ,τ 2 ) = |S(τ 1 )∩ S(τ 2 )| |S(τ 1 )∪ S(τ 2 )| where S(τ) denotes the set of unique states in trajectory τ . 14 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents C. Additional Behavioural Evaluation Results 7.510.012.515.0 Grid Size 0.6 0.8 1.0 Goal Success Rate Goal Success Rate by Grid Size 0.00.51.0 Density 0.4 0.6 0.8 1.0 Goal Success Rate Goal Success Rate by Density (a) Goal success rate. 7.510.012.515.0 Grid Size 0.2 0.4 Mean Entropy (bits) Entropy by Grid Size Model Optimal 0.00.51.0 Density 0.0 0.2 0.4 0.6 Mean Entropy (bits) Entropy by Density Model Optimal (b) Entropy of the agent’s policy. 7.510.012.515.0 Grid Size 0.05 0.10 0.15 0.20 Mean JSD Mean JSD by Grid Size 0.00.51.0 Density 0.10 0.15 0.20 Mean JSD Mean JSD by Density (c) Jensen-Shannon Divergence. 7.510.012.515.0 Grid Size 0.05 0.10 ECE ECE by Grid Size 0.00.51.0 Density 0.05 0.10 0.15 ECE ECE by Density (d) Expected Calibration Error. Figure 11. Performance metrics by size complexity. (a) Goal suc- cess rate, (b) Policy entropy, (c) JSD from optimal policy, and (d) Expected Calibration Error. D. Iso-difficulty Transform Quantitative Results OriginalRotateEnvReflectEnv StartGoalSwap TransposeEnv Figure 12. Examples of iso-difficulty transformations. ReflectEnv RotateEnv StartGoalSwap TransposeEnv 0.5 0.0 0.5 Goal Success Goal Success (vs Baseline) ReflectEnv RotateEnv StartGoalSwap TransposeEnv 0.5 0.0 0.5 Accuracy Accuracy (vs Baseline) Figure 13.Robustness to iso-difficulty environment transforma- tions. Table 3.Statistical significance and effect sizes for iso-difficulty transformations (N = 300pairs); Wilcoxon signed-rank test. No significant difference is observed across all transformations. MetricTransformation p-valueEffect Size GSR Reflection0.5820.059 Rotation0.3110.103 Start/Goal Swap0.3910.085 Transposition0.9490.006 Accuracy Reflection0.7840.022 Rotation0.8380.016 Start/Goal Swap0.1790.109 Transposition0.6020.042 E. Evaluation Settings and Prompts E.1.Evaluation Parameters for Behavioural Evaluation We use “low” reasoning because the model (GPT-OSS-20B) will not finish reasoning on high density / large size grids, even with 10,000 tokens. Note that the reasoning length is not a hard constraint, but rather a training objective. That is, "low" reasoning does not impose a strict token cutoff. Table 4. Model Configuration Details. ParameterValue Model ID openai/gpt-oss-20b Provider together_ai Interface litellm Max Tokens10,000 Temperature0.7 Reasoning Effort low Top P0.95 Top Logprobs5 Num. Trajectories per Grid10 15 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents E.2. Prompt for Behavioural Evaluation # Instructions You are controlling an agent in a grid-based environment with full observability. The agent can move in four directions: up, down, left, and right. The environment contains walls, open spaces, and a goal location. The following symbols are used in the grid representation: Legend: --------------- #: Wall _: Open Space (can be visited) G: Goal A: Current agent position ---------------- Your objective is to navigate from the current position (A) to the goal (G) while avoiding walls (#). Importantly, the agent should aim to reach the goal using the least amount of steps possible. You must decide your next move based on the provided information. You will receive the current state of the grid as a NxM matrix of symbols separated by whitespaces, and with coordinates for each row an column. For example, given the 4x5 grid: 0 1 2 3 4 0 # # # # # 1 # _ _ G # 2 # A _ _ # 3 # # # # # The goal is in position (1,3), while the agent's position A is (2,1). The agent's possible actions at each step are: Actions: - UP: Move Up - DOWN: Move Down - LEFT: Move Left - RIGHT: Move Right Your final answer should be a valid JSON object provided in the following form: ``` "action": "<UP|DOWN|LEFT|RIGHT>", ``` DO NOT INCLUDE ANY`json` or`jsonb` in your response NOR TICK MARKS LIKE THIS:```json```. Start with and end with exactly. # Inputs Current grid state: grid_state E.3. Prompt for Instrumental and Implicit Goals # Instructions You are controlling an agent in a grid-based environment with full observability. The agent can move in four directions: up, down, left, and right. The environment contains walls, open spaces, a goal location, and may include doors and keys. The following symbols are used in the grid representation: Legend: --------------- #: Wall _: Open Space (can be visited) G: Goal A: Current agent position D: Door (locked) K: Key (can unlock doors) ---------------- Your objective is to navigate from the current position (A) to the goal (G) while avoiding walls (#). Importantly, the agent should aim to reach the goal using the least amount of steps possible. You must decide your next move based on the provided information. ## Key and Door Mechanics - ** Key Pickup ** : The key is automatically picked up when the agent moves to the same cell as the key (K). Once picked up, the agent carries the key with them. - ** Door Opening ** : A locked door (D) is automatically opened and removed from the grid when the agent is adjacent to the door (in front of it) and has already picked up the key. The agent does not need to be on the same cell as the door to open it. - ** Important ** : You must pick up the key before you can open the door. If a door blocks your path to the goal, you need to first navigate to the key, pick it up, then navigate to be adjacent to the door to open it. You will receive the current state of the grid as a NxM matrix of symbols separated by whitespaces, and with coordinates for each row and column. For example, given the 6x7 grid: 0 1 2 3 4 5 6 0 # # # # # # # 1 # _ _ _ _ _ # 2 # A _ _ _ K # 3 # _ # D # _ # 4 # _ # G # _ # 5 # # # # # # # The goal is in position (4,3), the agent's position A is (2,1), the door D is at (3,3), and the key K is at (2,5). The agent's possible actions at each step are: Actions: - UP: Move Up - DOWN: Move Down - LEFT: Move Left - RIGHT: Move Right Your final answer should be a valid JSON object provided in the following form: ``` "action": "<UP|DOWN|LEFT|RIGHT>", ``` DO NOT INCLUDE ANY`json` or`jsonb` in your response NOR TICK MARKS LIKE THIS:```json```. Start with and end with exactly. # Inputs Current grid state: grid_state Agent status: - Carrying key: carrying_key 16 A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents 0.00 0.25 0.50 0.75 1.00 Recall (= Accuracy) 68 68 77 74 81 63 84 85 83 85 83 72 71 75 70 WallEmptyAgentGoalOverall 71523 Layer 0.00 0.25 0.50 0.75 1.00 Precision 88 87 79 76 74 79 9 21 11 11 15 9 71 75 70 Figure 14.Performance of cognitive map probes across layers. Middle layer activations show the highest overall accuracy, but goal-specific information is already present in early layers. F. Additional Representational Evaluation Results F.1. Cognitive Map Encoding across Layers We examine how world model information develops across layers in GPT-OSS-20B by training MLP probes on activa- tions from early (7), middle (15), and late (23) layers for grid size 11. Detailed probe performance is reported in Fig. 14. Goal-specific spatial information is already present in early layers, but the precision of theagentandgoalclasses con- tinues to increase in layer 15. By the later layers, overall accuracy declines again, along with recall and precision foragentandgoal. This pattern suggests that, at end-of- prompt-token indices, spatial information is most explicitly represented at intermediate layers and is subsequently trans- formed for the computation of other features rather than being directly preserved for next-token prediction. F.2. Size-specific Cognitive Map Probing Results In the main experiments we trained a single MLP probe on data from all grid sizes, padding inputs to size 15. While this approach has the obvious advantage that we can use the same probe for any grid world, it might result in a worse performance than size-specific classifiers. We test this by training size-specific MLP probes without padding for each grid size. Results are shown in Fig. 15. Compared to Fig. 6 (centre), size-specific probes achieve higher overall accuracy for smaller grids, but match general probes performance for grid sizes 11–15. Precision for theagentandgoalclasses is higher across grid sizes, but recall is substantially lower for grid sizes 11–15. Given these uneven performance dif- ferences, together with the flexibility of the size-agnostic approach, we adopt the size-independent MLP probe in our main experiments. F.3. Goal Distance Probing Results We also test if internal representations of GPT-OSS-20B encode information about the distance between the agent 0.00 0.25 0.50 0.75 1.00 Recall (= Accuracy) 85 76 70 66 65 93 82 74 76 75 99 97 59 55 44 99 93 50 40 41 89 79 72 71 70 WallEmptyAgentGoalOverall 79111315 Grid Size 0.00 0.25 0.50 0.75 1.00 Precision 96 86 77 75 72 81 74 71 70 71 81 61 24 22 13 79 60 20 14 10 89 79 72 71 70 Figure 15.Performance of size-specific MLP probes. Size-specific probes provide comparable performance, but less flexibility than the size-agnostic approach. Table 5. Distance probe performance. Probe TypeReasoning StageMAE↓ R 2 ↑ MLP ProbePre-Reasoning3.160.40 Post-Reasoning2.670.36 Linear ProbePre-Reasoning3.400.42 Post-Reasoning3.130.47 and the goal. As with cell identity, distance to the goal is a variable we hypothesise may be tracked because of its direct relevance for goal-directed behaviour. Indeed, in our grid worlds, the optimal state value is a monotonic function of the distance to the goal (cf. §3). To test this, we train linear and MLP probes on both pre- and post-reasoning activations, using the length of the optimal trajectory computed via A* as the ground-truth distance label. Results are reported in Tab. 5. Across conditions, goal distance can be decoded with a mean absolute error of ap- proximately 3 steps, both before and after reasoning. The best performance is achieved by the post-reasoning MLP probe, with a mean absolute error of 2.67. These results suggest that, by reasoning about the grid and planning the next steps, the model develops more accurate representation of the distance it needs to cover to reach the goal. Table 6. Sequence length analysis for one-shot plan decoding. MetricPre-reasoningPost-reasoning Exact length match (%)↑15.7118.85 Predicted length (avg)4.815.05 Ground-truth length (avg)4.934.93 Length bias (avg, pred− true) −0.12+0.12 Median abs. length error (steps)1.01.0 F.4. Additional Plan Decoder Results We analyse predicted trajectory lengths (Table 6). Post- reasoning decoding achieves a higher exact length match rate (19% vs. 16%) and exhibits a small positive bias in average length (avg pred−true= +0.12), whereas pre- reasoning decoding exhibits a small negative bias (−0.12). Median absolute length error is 1 step in both settings. 17