Paper deep dive
A 3D Characterization Framework for Intelligent Sequential Decision Making
Sadig Gojayev, Carolina Fortuna
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/10/2026, 2:16:54 AM
Summary
The paper introduces a 3D characterization framework for evaluating AI methods in sequential decision making, focusing on MDP projection, human prior, and skill/cost trade-offs. Using the Tower of Hanoi puzzle as a benchmark, the authors compare graph-based (Neurosolver), reinforcement learning (FBRL), and LLM-based (AutoToS, DA-ToS) approaches. The study finds that LLM-based methods incur significantly higher memory and runtime costs due to weakly constrained action spaces that shift complexity to inference-time verification.
Entities (12)
Relation Signals (10)
3D Characterization Framework â evaluates â AutoToS
confidence 95% · Using this framework, we analyze how representative... large language model (LLM)-based approaches differ... instantiated respectively by... AutoToS
3D Characterization Framework â evaluates â Neurosolver
confidence 95% · Using this framework, we analyze how representative graph-based... approaches differ... instantiated respectively by Neurosolver
3D Characterization Framework â evaluates â FBRL
confidence 95% · Using this framework, we analyze how representative... reinforcement learning... approaches differ... instantiated respectively by... FBRL
3D Characterization Framework â evaluates â DA-ToS
confidence 95% · including a double-agent extension of thought-of-search (DA-ToS)
LLM-based methods â hashighercostthan â Neurosolver
confidence 95% · LLM-based methods... leading to substantially higher memory and runtime costs than Neurosolver and FBRL.
LLM-based methods â hashighercostthan â FBRL
confidence 95% · LLM-based methods... leading to substantially higher memory and runtime costs than Neurosolver and FBRL.
3D Characterization Framework â uses â Tower of Hanoi
confidence 95% · Using this framework, we analyze how representative graph-based, reinforcement learning, and large language model (LLM)-based approaches differ... The analysis relies on the Tower of Hanoi puzzle
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Puzzles are widely used to evaluate the reasoning capabilities of artificial intelligence (AI) systems for sequential decision making, yet approaches originating from different paradigms are rarely compared under unified conditions. To address this gap, we introduce a three-dimensional characterization framework that enables the analysts of AI methods by 1) projecting them to the Markov decision process (MDP) sequential decision making formalism, 2) degree of autonomy through human prior ranking of their designs and, 3) skill and computational cost. Using this framework, we analyze how representative graph-based, reinforcement learning, and large language model (LLM)-based approaches differ in their design choices and performance characteristics, instantiated respectively by Neurosolver, forward-backward reinforcement learning (FBRL), and automated thought-of-search (AutoToS), including a double-agent extension of thought-of-search (DA-ToS). The analysis relies on the Tower of Hanoi puzzle that provides a controlled benchmark with well-defined rules and scalable complexity, enabling consistent comparison across increasing problem sizes. The 3D characterization reveals that LLM-based methods, due to their weakly constrained action-space design, shift complexity from architecture to inference-time verification, leading to substantially higher memory and runtime costs than Neurosolver and FBRL.
Tags
Links
- Source: https://arxiv.org/abs/2610.11696v1
- Canonical: https://arxiv.org/abs/2610.11696v1
Trouble viewing inline? Open PDF directly â
Full Text
62,645 characters extracted from source content.
Expand or collapse full text
AI Artificial Intelligence MDP Markov decision process SOA Service-oriented Architecture ToH Towers of Hanoi RL Reinforcement Learning FBRL Forward-backward reinforcement learning NCM Neural Cognitive Modelling LNN Logical Neural Networks LLM Large Language Model PDDL Planning Domain Definition Language FCN Fully-connected Network DDQN Double Deep Q networks LIF Leaky Integrate-and-Fire VSA Vector Symbolic Architecture ML Machine Learning MBRL Relational Model-based Reinforcement Learning ILP Inductive Logic Programming ToS Thought-of-Search DA-ToS Double-Agent Thought-of-Search AutoToS Automated Thought-of-Search STRIPS Stanford Research Institute Problem Solver A 3D Characterization Framework for Intelligent Sequential Decision Making Sadig Gojayev, Carolina Fortuna Affiliation: Department of Communication Systems, JoĆŸef Stefan Institute, SI-1000 Ljubljana, Slovenia sadig.gojayev, carolina.fortuna@ijs.si Affiliation: Abstract Puzzles are widely used to evaluate the reasoning capabilities of artificial intelligence (AI) systems for sequential decision making, yet approaches originating from different paradigms are rarely compared under unified conditions. To address this gap, we introduce a three-dimensional characterization framework that enables the analysts of AI methods by 1) projecting them to the Markov decision process (MDP) sequential decision making formalism, 2) degree of autonomy through human prior ranking of their designs and, 3) skill and computational cost. Using this framework, we analyze how representative graph-based, reinforcement learning, and large language model (LLM)-based approaches differ in their design choices and performance characteristics, instantiated respectively by Neurosolver, forwardâbackward reinforcement learning (FBRL), and automated thought-of-search (AutoToS), including a double-agent extension of thought-of-search (DA-ToS). The analysis relies on the Tower of Hanoi puzzle that provides a controlled benchmark with well-defined rules and scalable complexity, enabling consistent comparison across increasing problem sizes. The 3D characterization reveals that LLM-based methods, due to their weakly constrained action-space design, shift complexity from architecture to inference-time verification, leading to substantially higher memory and runtime costs than Neurosolver and FBRL. I Introduction The rise of agentic artificial intelligence (AI) [1] is expected to significantly impact the economy and society as a whole. While agents relying on AI techniques have been around for several decades [2], the new generations of such agents are designed around the recent breakthrough in auto-regressive models, namely large language models (LLMs) [3]. These models, based on transformer architectures [4], essentially learn a probabilistic language model that, conditioned by an input generates an output. These models exhibit broad coverage over human-generated text because they are trained on massive web-scale corpora. Whether the training process enables them to go beyond memorization towards eliciting abstraction and generalization that would subsequently enable reasoning is still under investigation [5]. To improve reasoning, various training and prompting approaches such as reinforcement learning [6] and chain of thought [7] were proposed, however their reasoning capabilities are mostly unclear [8]. Reasoning capabilities of models are often evaluated on puzzles [9, 10, 8]. Some works focus more on logic puzzles [9] while others focus on text-based puzzles, classified into numeric, sequence, and physics categories [10]. toh (toh) is one recently used puzzle to investigate reasoning differences between LLMs and LRMs [8]. toh is also widely used in education to develop algorithmic thinking through inductive and recursive approaches [11, 12]. It also has wide applicability beyond education, being used in logistics and robotics, where weighted variants have been proposed to determine optimal move sequences with minimal cost [13, 14, 15]. 112233n1n_1n2n_2 (a) Initial state: n1,n2n_1,n_2 on peg 11 112233n1n_1n2n_2 (b) Action: Move n2n_2 peg 22 112233n1n_1n2n_2 (c) Goal: n1,n2n_1,n_2 on peg 33 Fig. 1: toh for n=2n=2 disks and p=3p=3 pegs. toh involves discs n and pegs p, where all the discs are initially placed on the first peg, and the goal is to move all the discs to the last peg. The key limitation is that only a smaller disc can be placed on top of a larger disc, as illustrated in Figure 1. To solve this puzzle, at each step, a human or AI player needs to understand the current state and take an action that will transition the puzzle to the next state until it is solved. This sequential decision making is referred to as a mdp (mdp) and some AI approaches, especially the reinforcement learning ones, explicitly leverage the mdp formalism. Over time, several AI techniques, other than LLMs and LRMs, have been used to solve the ToH. For instance, [16] employed reinforcement learning, [17] approached it with logical neural networks, [18] proposed graph-based solvers, while [19] investigated neural cognitive modeling. Currently, it remains unclear what the design trade-offs are when selecting one class of AI models over another for scalable sequential decision making problems such as toh. State-of-the-art studies mostly focus on decision-quality comparisons between LLMs and RL [20], LLMs and AI planners [21]. Beyond performance metrics, very recent work [22] provides theoretical formulations that cast LLM- and AI planners within mdp for optimization. However, while these efforts highlight differences across paradigms, they largely lack an explicit linkage between formulation choices, decision-making skill, computational cost, and the level of human-imposed biases. To fully characterize an AI agent, one must consider three key aspects: (1) how the agent represents the world, (2) the degree of external guidance or supervision it receives, and (3) the computational cost required. In this paper, we propose a three-dimensional characterization framework that provides a structured and systematic basis for comparing different AI methods. The framework captures mdp formulation projecting to understand each methodâs approach to sequential decision making, human prior to provide clearer insight into required guidance, and skill and computational cost to illustrate resource requirements. To illustrate the applicability of the proposed framework we employ it for comparing selected AI techniques. Our key contributions are as follows: âą We introduce a 3D characterization framework for sequential decision making methods that jointly analyzes (i) technique projection to the mdp-formalism, (i) human prior across architecture/training/inference, and (i) skillâcost trade-offs. âą We propose an operational human-prior rubric with explicit levels (High/Medium/Low) for representation/mechanism, knowledge/learning, and guidance/decision/validation, enabling qualitative ranking across paradigms. âą We instantiate the framework on toh by comparing a graph-based solver (Neurosolver) [18], a RL method fbrl (fbrl) [16], and LLM-based planning via autotos (autotos) [23], including a new datos (datos) variant that modularizes code generation and repair. âą We show that weakly constrained LLM action spaces shift complexity from model design to inference-time verification loops, resulting in substantially higher runtime/memory than graph-based/RL approaches under the same task scaling. This paper is organized as follows. Section I summarizes related work, Section I introduces the three dimensional characterization framework while Section IV elaborates on the methods. Sections V, VI and VII analyze the selected AI methods from the perspective of the proposed framework. Finally, Section VIII concludes the paper. I Related Work Comparisons of sequential decision making methods across different paradigms and backgrounds usually focus on outcome-based metrics, such as success rate or optimality. The UNO Arena [21] benchmark evaluates agents from reinforcement learning, heuristic, and LLM-based approaches by emphasizing decision-level quality, using metrics such as hit rate and decision rank rather than only win rate. For puzzle-solving tasks, PUZZLES [9] provides a benchmark of logical puzzles for neural reasoning in reinforcement learning, evaluating various RL algorithms primarily through success rate. PUZZLEWORLD [24], in contrast, evaluates open- and closed-source LLMs by focusing on stepwise accuracy, where models are tasked with producing step-by-step reasoning toward a final solution. Recent work by Wang et al. [20] compares human decision-making with LLM and reinforcement learning agents, evaluating coordination behavior and path cost under identical environmental conditions. Overall, these comparisons adopt a largely single computational perspective, focusing primarily on result-based measures such as inference quality, even across different paradigms. They emphasize external performance metrics while lacking explicit analysis and direct linkage to representation-level action spaces, architecture-level transition mechanisms, and human guidance, leaving internal decision-making processes treated as black boxes. Beyond purely computational evaluations, theoretical efforts such as FRAP [22] decompose AI planning and reinforcement learning algorithms into explicit dimensions (e.g., root states, trials, bootstrapping, backups) and analyze them through the lens of mdp optimization. Similarly, Burden et al. [25] classify AI evaluation approaches into paradigms such as benchmarking and exploratory evaluation. While these frameworks provide valuable formulations for comparing methods across paradigms, they do not offer insight into decision-making skill and computational cost in relation to mdp formulation. As a result, it remains unclear how action-space design, human guidance, and architectural constraints affect inference quality and computational cost. There is a lack of unified frameworks across different paradigms, particularly between reinforcement learning and large language modelâbased approaches. Most comparisons focus on outcome metrics, rather than the formulation of sequential decision making (e.g., mdps) and the role of human insight in shaping model behavior. I Proposed 3D Characterization Framework In this section, we introduce a three-dimensional framework for comparing AI methods used in sequential decision making. The framework focuses on three aspects: how each method projects to the elements of a mdp, what is the level of human prior introduced throughout the AI model development, and what skill vs computational trade-offs it involves. I-A Projecting through Markov Decision Process The first dimension, projecting to mdp, clarifies how each method handles the core components of sequential decision makingâsuch as state representation, transitions, rewards, and goal identification. Understanding this projection shows how a method reasons about progress and how it identifies correct or optimal next steps. Sequential decision making is typically formulated as a mdp [26], which is defined as âł=âšS,A,T,R,Gâ©M= S,A,T,R,G , where states (S) denote environment configurations, actions (A) specify possible moves, the transition function (T) gives the probability of reaching sâČs after taking action a in state s, the reward function (R) assigns values to transitions, and goal states (G) indicate task completion. State space representation refers to how the mdp state space S is specified in the sense of mdp. The levels of human prior are defined as follows: High Humans explicitly design the state representation, fully specifying its structure and semantics, as in symbolic planners based on PDDL [27]. Medium Humans prescribe a high-level representational structure or template, while allowing the system to learn internal parameters or details, as in logical neural networks or partially structured encodings [17]. Low No explicit representational structure is imposed by humans; the model learns state representations end-to-end from interaction or data, as in fully end-to-end reinforcement learning systems trained via self-play [28]. Mechanism refers to the algorithmic structure responsible for reasoning and inference. The levels of human prior are defined as follows: High Humans design the inference procedure, such as logical rules or search heuristics. Medium Humans select the general type of mechanism to employ, while the algorithm handles the details; for example, in Value Iteration Networks [29], humans prescribe a value-iterationâbased planning structure, while the network learns the reward and transition representations end-to-end. Low Humans provide only high-level instructions, such as requesting an LLM to âgenerate code for running a DDQN,â without specifying internal structure. TABLE I: Levels of Human Prior in Architecture Design. Knowledge Input concerns how training data, constraints, or reward structures are introduced, directly shaping the transition function T and reward function R in mdp formulation. High Humans explicitly specify the training data, define relevant constraints, and design or shape reward structures, fully guiding how the model learns from examples, as in reinforcement learning with human-structured environments and hand-crafted training scenarios (e.g., Q-learning with expert-designed tasks). Medium Humans provide curated or loosely structured data without fully specified objectives, allowing the system to organize patterns and infer useful structure while still retaining partial human guidance, such as transformers trained on human-curated datasets. Low The model autonomously extracts training signals from large-scale interaction or self-generated experience, with minimal human specification beyond initial setup, as in self-supervised pretraining of large language models on unlabeled data [30]. Learning reflects how the model updates its parameters in response to the training signal and the extent of human involvement in this process. High Humans explicitly shape the optimization process through curated objectives, staged training procedures, or tightly controlled learning phases, as in reinforcement learning from human feedback (RLHF) for large language models. Medium Humans specify general optimization settingsâsuch as learning rates, network architectures, or update rulesâwhile the system autonomously manages the internal learning dynamics, as in Double Deep Q-Networks (DDQN) [31]. Low Learning proceeds end-to-end with minimal human involvement beyond initialization, with the system autonomously generating training signals, as in self-play reinforcement learning [28]. TABLE I: Levels of Human Prior at Training Stage. Guidance captures the extent to which humans specify task details or constrain model behavior at runtime. High Humans provide detailed task instructions, including explicit constraints, edge cases, and step-level guidance, often through fully specified prompts or rule descriptions, as in Least-to-Most prompting techniques [32]. Medium Humans specify high-level task constraints or goals without defining the complete execution structure, such as providing domain rules (e.g., âa smaller disk cannot be placed beneath a larger oneâ in toh) while leaving reasoning to the model. Low Humans provide only minimal task informationâsuch as specifying the start and goal states in graph-based solversâwhile the system autonomously determines the reasoning process and action sequence. Decision describes how actions or outputs are selected by the system during inference and the extent of human involvement in this process. High Humans explicitly drive or supervise the decision-making process, often through interactive or step-wise instructions. The model depends on human-provided reasoning steps, clarifications, or corrective feedback to determine each action, as in interactive planning with large language models (e.g., DEPS) [33]. Medium Humans specify the decision framework or evaluation criteriaâsuch as accuracy thresholds, selection rules, or preferred heuristicsâwhile the system autonomously executes decisions within these constraints, as in safe reinforcement learning approach in constrained mdp [34]. Low The system selects actions entirely autonomously based on its learned internal dynamics or probability distributions, without any human-defined decision logic at inference time, as in end-to-end transformer-based language models. Validation describes how the correctness of model outputs is assessed and the extent of human involvement in this process. High Humans manually inspect model outputs, identify errors, and provide corrective feedback after each execution. Medium Humans specify validation criteria or automated test suites, while the system autonomously executes these checks and uses the results to guide refinement or improving tests, as in TestGen-LLM framework [35]. Low Validation and correction are performed entirely by the system itself, without human oversight, as in self-correcting large language models [36]. TABLE I: Levels of Human Prior at Inference Stage. I-B Human Prior in AI for Sequential Decision Making The second dimension, human prior, reflects how much human know-how is embedded in a method during its development: architecture design, training, and inference stages. This helps assess the degree of intrinsic intelligence that characterizes a method. We propose a categorization of human prior in AI techniques for sequential decision making focusing on three major aspects: architecture design, model training and model inference as follows. Architecture Design The architecture design involves the development of the structure of the model prior to learning or execution. We identify two components in the design: Representation and Mechanism as defined in Table I. Training Stage The training stage describes how a model acquires skill and how humans shape this process. It consists of two substages: Knowledge Input and Learning as defined in Table I. Inference Stage Inference describes how the model behaves once training is complete. It includes how humans guide the model, how decisions are produced, and how outputs are checked. It consists of three substages: Guidance, Decision and Validation as defined in Table I. Fig. 2: datos Architecture Overview I-C Performance and Computational Cost The third dimension, focuses on assessing decision making skill versus computational costs required to provide that level of performance. By performance we refer to AI specific metrics measuring model performance at a single step, such as mean squared error distance or F1 or reward and regret characterizing the entire decision sequence. By computation cost we refer to metrics such as memory footprint, floating point operations, time and energy. This axis goes beyond measuring single domain specific performance metric towards a more holistic cost benefit analysis. This proposal is in line with prior work [20] that has emphasized the importance of understanding computational behavior across planning, reinforcement learning, and LLM-based reasoning, and broader overviews [22] of sequence models in sequential decision making that discuss how model architecture and training dynamics impact performance and cost. In this paper, we focus on time and memory consumption as the primary indicators of computational cost. In addition to measured time and memory, we also provide theoretical memory estimates for each selected method. For reproducibility purposes, we publish the code11 1 GitHub link to be inserted after double blind review completes.. IV Methods This section presents the selection of diverse AI techniques to be analyzed according to the three-dimensional framework introduced in Section I. We consider LLM-based agents, a graph-based learning method, and a reinforcement learningâbased approach. IV-A LLM-based Agents IV-A1 tos (tos) autotos is a framework that automates search-based planning by using a large language model (LLM) to generate two core components: a successor function (succ), which enumerates valid next states, and a goal predicate (isgoal), which determines whether a state satisfies the task objective. In toh domain, succ encodes legal disk moves, while isgoal identifies the configuration in which all disks are stacked on the target peg. The generated functions are embedded in an iterative refinement loop, where a Breadth-First Search (BFS) is used to test correctness and optimality. Detected errors, such as missing transitions or incorrect goal checks, are automatically converted into feedback prompts for the LLM, which regenerates the functions until soundness and completeness are achieved. IV-A2 datos We extend autotos by introducing a dual-agent architecture that separates code generation from error interpretation. As illustrated in Figure 2, in datos Coding agent generates implementations of succ and isgoal, while a Repair agent analyzes compilation errors and test failures to produce targeted feedback. The system operates in a closed-loop workflow: generated code is validated against predefined test cases, failures are interpreted by the Repair agent, and corrective guidance is fed back to the Coding agent for refinement. This process repeats until all tests pass. IV-A3 Estimated Memory The total memory used by the model during inference mainly comes from two parts â the model weights and the keyâvalue (KV) cache that stores hidden states for each token in the context. It can be expressed as MtotalâMweights+MKVM_totalâ M_weights+M_KV where Mweights=PĂbM_weights=PĂ b depends on the number of model parameters P and the precision b (bytes per parameter) and MKVâ2ĂLĂTĂdmodelĂbM_KVâ 2Ă LĂ TĂ d_modelĂ b is proportional with the number of L, hidden size dmodeld_model, and sequence length T (tokens in the conversation). The factor of 2 accounts for storing both keys and values. In practice, MKVM_KV increases linearly with the token count, while MweightsM_weights remains constant, so the total memory grows slowly with longer interactions. IV-A4 Experimental Setting for LLM-based Agents We use standard (non-reasoning) LLMs instead of explicitly trained reasoning models, including low-parameter variants, to avoid introducing additional inductive biases from specialized reasoning supervision. In our approach, reasoning is externalized through search and verification, ensuring correctness independently of the modelâs internal deliberation. This allows performance differences to be attributed to the planning framework rather than hidden reasoning priors, making standard LLMs a more neutral baseline. In our implementation, we used DeepSeek Coder (deepseek-coder-v2-lite-instruct, 16B), Yi Coder (yi-coder-9b-chat, 9B), and Qwen Coder (qwen2.5-coder-32b-instruct, 32B) as Coding agents, while Gemma (8.5B) and Mistral (7.2B) served as Repair agents. By employing coder fine-tuned LLMs for the Coding agent and general-purpose LLMs for the Repair agent, we seek to improve the effectiveness of smaller-parameter coding models while reducing bias through cross-model validation. Memory estimation is obtained by combining the model weight with an analytically computed keyâvalue (KV) cache cost, using MKVM_KV from Section IV-A3. The weight values are taken directly from the quantized GGUF models used in our experimentsâDeepSeek (9.65 GB), Yi (4.96 GB), Qwen (11.47 GB), Gemma (4.67 GB), and Mistral (3.83 GB). The KV-cache is estimated assuming fp16 precision (b=2b=2 bytes), with architectural parameters L=40,dmodel=5120L=40,d_model=5120 for DeepSeek, L=32,dmodel=4096L=32,d_model=4096 for Yi and Mistral, L=64,dmodel=7168L=64,d_model=7168 for Qwen, and L=28,dmodel=3072L=28,d_model=3072 for Gemma, yielding a conservative upper-bound estimate where all tokens are assumed to remain in the context window. TABLE IV: Comparison of selected ToH models in terms of their State, Action, Transition, Reward, and Goal Representations. Model State (S) Action (A) Transition (T) Reward (R) Goal (G) datos âš prompt, code c, generated feedback FâĄ(c,e)F(c,e) (token sequences) Invoke coding or repair LLM once Probabilistic token generation: P(yt+1:t+mâŁx1:t)P(y_t+1:t+m x_1:t) 1 = working code, 0 = non-working code Verified succ and isgoal functions autotos [23] âš prompt, function, verifier feedbackâ© (token prefixes) Single LLM generation step Probabilistic token generation: P(yt+1:t+mâŁx1:t)P(y_t+1:t+m x_1:t) 1 = working code, 0 = non-working code Verified succ and isgoal functions Neurosolver [18] sâ1,2,3nsâ\1,2,3\^n (symbolic diskâpeg assignments) Legal disk moves Deterministic: sâČ=TâĄ(s,a)s =T(s,a), guided by goal-conditioned value signal Goal reward; penalties for illegal or redundant moves All disks stacked on target peg (optimal 2nâ12^n-1 steps) FBRL [16] Bit string (disk positions) Valid disk moves (bit changes) Deterministic: QâĄ(st,At)ââĄ[R+ÎłâmaxâĄQâĄ(st+1,A)]Q(s_t,A_t) [R+Îł Q(s_t+1,A)] +1 valid, -1 invalid, +10 for goal Bit string for disks on Peg C (binary pattern) IV-B Neurosolver Neurosolver [18] is a graph-based planner that models puzzle state as a node with two divisions. States are encoded as strings, for example in toh [123,000,000] for three disks stacked on the first peg. The upper division propagates goal-directed activity backward from the target state, tracing feasible transitions in reverse. The lower division encodes the current configuration and propagates signals forward to execute valid moves. Training builds the state space and transitions, the search phase spreads activation backward from the goal, and the solving phase reconstructs a valid path forward from the start state. IV-B1 Estimated Memory The memory requirements of Neurosolver can be understood from the way the puzzleâs state space grows as discussed in this section. Because Neurosolver stores both states and transitions (in forward and reverse directions), the memory footprint grows in proportion to both of these quantities. If we let α be the average storage cost per state and ÎČ the average storage cost per edge, the overall memory can be approximated as Mnâeâuârâoâ(n)âαâ 3n+ÎČâ nâ 3n=(α+ÎČân)â 3nM_neuro(n)âα· 3^n\;+\;ÎČ· n· 3^n=(α+ÎČ n)\,3^n. This formula shows that memory usage increases exponentially with the number of disks (3n3^n), with an additional linear factor in n due to the growing number of possible moves. IV-B2 Experimental Setting for Neurosolver Neurosolver was implemented such that each disk-number configuration was trained using 100 iterations of random exploration. To ensure robustness, each configuration was trained and solved in ten independent runs, with results averaged across runs reported for metrics such as step count and execution time. To estimate memory usage, MnâeâuârâoM_neuro, each state is assigned a constant storage cost of α=64α=64 bytes, corresponding to a compact fixed-size encoding (e.g., 4Ă4Ă44Ă 4Ă 4 bytes per peg). Each transition is assigned ÎČ=16ÎČ=16 bytes, representing a stored reference (sourceâtarget identifiers). Since both forward and reverse transitions are stored, this cost is doubled using a duplication factor Îș=2Îș=2. We further assume an average out-degree of d=3d=3 valid moves per state. These assumptions are combined to estimate the total memory footprint, which is reported in megabytes. IV-C Forward-Backward Reinforcement Learning fbrl [16] is a reinforcement learning method that combines forward and backward exploration during training. Built on the Double Deep Q-Network (DDQN) framework with a replay buffer, FBRL exploits both real and imagined experiences to accelerate learning in sparse-reward environments. In the forward step, the agent updates Q-values using real trajectories collected from the start state, while in the backward step it generates imagined trajectories by predicting reverse transitions from known goal states. Replay buffer is prioritized where transitions with higher temporal-difference errors are sampled more frequently [37]. IV-C1 Estimated Memory The total memory consumption of FBRL can be decomposed into three principal components: the Q-network, the backward (reverse) model, and the replay buffer. The Q-network learns a mapping QâĄ(s,a,Ξ)Q(s,a;Ξ), and its memory cost is determined by the number of parameters in that network. If the Q-network has LQL_Q hidden layers each of width hQh_Q, with input dimension 3ân3n and output dimension |A| A , then its memory cost can be approximated as MQ=αQâ(3ânâ hQ+(LQâ1)âhQ2+hQâ |A|)M_Q= _Q (3n· h_Q+(L_Q-1)\,h_Q^2+h_Q· A ), where αQ _Q is the average cost per parameter (in bytes or memory units). Similarly, the backward modelâwhich predicts reverse transitions or goal-conditioned dynamicsâhas an analogous architecture. Let it have LBL_B hidden layers of width hBh_B. Then its memory cost is MB=αBâ(3ânâ hB+(LBâ1)âhB2+hBâ |A|)M_B= _B (3n· h_B+(L_B-1)\,h_B^2+h_B· A ) where αB _B is its parameter storage cost. The replay buffer stores past transitions (s,a,r,sâČ)(s,a,r,s ) and thus scales proportionally with the number of stored entries. Denoting the buffer capacity by N and the average memory per stored tuple by Îł, we write MR=ÎłâNM_R=Îł N. Adding up the three components, the total memory consumed by FBRL is MFâBâRâLâ(n)â(αQ+αB)â(3ânâh+(Lâ1)âh2+hân)+ÎłâNM_FBRL(n)â( _Q+ _B) (3n\,h+(L-1)\,h^2+h\,n )+Îł\,N. This expression makes clear that while the network components scale roughly linearly with n (given fixed architecture), the replay buffer term ÎłâNÎł N is independent of n and may dominate memory usage for sufficiently large N. IV-C2 Experimental Setting for FBRL We implemented FBRL for toh using a string-based state representation. Each disk is encoded according to its peg location, expressed as a binary tuple. For example, the state [(100), (100), (100)] denotes the initial configuration of a three-disk problem, where all disks are placed on the first peg. In the experimental setup, the replay buffer was set to a capacity of 10,000 transitions, and the DDQN learning rate was fixed at 5Ă10â45Ă 10^-4. To estimate the memory consumption of FBRL, we use MFâBâRâLM_FBRL introduced in this section, which decomposes total memory into the Q-Network, Backward Model, and Replay Buffer components. The estimation was based on the architecture parameters: number of disks n=4n=4, input dimension din=3ân=12d_in=3n=12, action space |A|=3ân=12|A|=3n=12, hidden widths hQ=64h_Q=64 and hB=100h_B=100, and two linear layers per network (LQ=LB=2L_Q=L_B=2). The replay buffer capacity was set to N=10,000N=10,000 transitions, with all values stored in 32-bit floating-point format (4 bytes each). V Analysis from the mdp Perspective Along the first dimention of the proposed framework in Section I, in this section, we project the methods from Section IV to the elements of the mdp. The columns represent State, Action, Transition, Reward, and Goal, while each row corresponds to a method, as shown in Table IV. datos Fig. 3: LLM-based Agents Projection to MDP. In datos, mdp is formulated over sequential language-model interactions between a Coding agent and a Repair agent as illustrated in Figure 3. Let t denote the iteration step. The initial state is defined as s0=âštask promptâ©,s_0= prompt , where the task prompt specifies the problem instance (Boxes 1 and 2 from the Annex). At step t, the state is defined as st=âštask prompt,ct,FâĄ(ct,et)â©,s_t= prompt,\,c_t,\,F(c_t,e_t) , where ctc_t denotes the current candidate code implementation, ete_t represents the corresponding evaluation signal (e.g., unit-test or verifier output), and FâĄ(ct,et)F(c_t,e_t) denotes the feedback generated by the Repair agent based on the code and its evaluation. The action space A consists of invoking a language model once. An action corresponds either to generating updated code (coding agent) or generating feedback FâĄ(ct,et)F(c_t,e_t) (repair agent). The transition function is induced by autoregressive generation. Let the internal token representation of sts_t be denoted by (x1,âŠ,xn)(x_1,âŠ,x_n), where n is the current token length. The transition probability is defined as: T(st,at)=P(yn+1:n+mâŁx1:n)=âi=n+1n+mP(yiâŁx1:n,y<i),T(s_t,a_t)=P(y_n+1:n+m x_1:n)= _i=n+1^n+mP(y_i x_1:n,y_<i), (1) where yn+1:n+my_n+1:n+m corresponds to the newly generated tokens. The next state is obtained by concatenation, st+1=st|(yn+1,âŠ,yn+m).s_t+1=s_t\,\|\,(y_n+1,âŠ,y_n+m). The reward function is binary, RâĄ(st)â0,1R(s_t)â\0,1\, assigning 11 only when the generated planning functions satisfy soundness and completeness constraints, and 00 otherwise. Iteration proceeds until RâĄ(st)=1R(s_t)=1, at which point the goal state is reached. The goal state G is defined as the state in which both succ and isgoal are verified to be sound and complete, ensuring that the toh problem instance can be solved correctly with 100% validity. autotos In autotos, the mdp follows the same formulation as in datos (see Figure 3), but without a separate Repair agent providing feedback. Let t denote the iteration step. The initial state is s0=âštask promptâ©.s_0= prompt . At step t, the state is defined as st=âštask prompt,ct,etâ©,s_t= prompt,\,c_t,\,e_t , where ctc_t denotes the current candidate implementation (e.g., succ, isgoal), and ete_t denotes the verification feedback obtained directly from predefined unit tests and BFS soundness checks. Unlike datos, no secondary model generates feedback; the verifier output ete_t is directly appended to the state. The action space consists of a single language-model invocation conditioned on sts_t. The transition dynamics follow the same autoregressive formulation as in Eq. 1, and the next state is obtained via token concatenation, st+1=st|(yn+1,âŠ,yn+m).s_t+1=s_t\,\|\,(y_n+1,âŠ,y_n+m). The reward function remains binary, RâĄ(st)â0,1R(s_t)â\0,1\, with termination when RâĄ(st)=1R(s_t)=1. The goal state G is reached when both succ and isgoal are verified to be sound and complete, ensuring correct solution of the toh problem. Neurosolver In Neurosolver, the problem is modeled as a discrete mdp. The state space is defined as sn=1,2,3ns_n=\1,2,3\^n, where a state s=(s1,âŠ,sn)s=(s_1,âŠ,s_n) encodes the peg position of each disk, and the initial state is s0=123,000,000s_0=\123,000,000\, corresponding to all disks stacked on the source peg. The action space consists of all legal disk moves, and the transition function is deterministic, T:snĂAâsn,sâČ=TâĄ(s,a)T:s_nĂ Aâ s_n,s =T(s,a), with transitions defined only for legality-preserving actions. Action selection is guided by a goal-conditioned value signal, such that ÏâĄ(aâŁs,g)âexpâĄ(Vgâ(sâČ))Ï(a s,g) \! (V_g(s ) ), biasing exploration toward states closer to the goal. The reward function assigns maximal reward when the goal is reached, penalizes illegal or redundant moves, and otherwise reflects progress toward the target state. The goal state G is defined as the configuration where all disks are stacked on the destination peg, yielding the optimal solution length 2nâ12^n-1. TABLE V: Human Prior ranking across Architecture, Training, and Inference Stages. Method Model Architecture Training Inference Representation Mechanism Knowledge Learning Guidance Decision Validation datos DeepSeek + Mistral High High DeepSeek + Gemma Medium High Medium Medium Medium Low Medium autotos [23] DeepSeek High High Qwen Medium Medium Yi Medium High Medium Medium High Low Medium Neurosolver [18] Graph-based model Low High Low Low Medium Low N/A FBRL [16] Reinforcement Learning Low High High Medium Low Low N/A FBRL In FBRL, the state space s is encoded as a bit string of length 3ân3n, where each group of three bits specifies the peg location of a disk. For example, with two disks, the initial state is represented as [1,0,0,1,0,0][1,0,0,1,0,0] (both disks on Peg A), and the goal state as [0,0,1,0,0,1][0,0,1,0,0,1] (both disks on Peg C). The action space A consists of legal disk moves, expressed as updates to the relevant 3-bit segment of the bit string while ensuring only valid transitions occur. The transition function T is deterministic; approximated with a DDQN, where transitions are sampled from a replay buffer and updated using the Bellman equation QâĄ(st,At)ââĄ[R+ÎłâmaxâĄQâĄ(st+1,A)]Q(s_t,A_t) [R+Îł Q(s_t+1,A) ]. The reward function (R) assigns +1+1 for valid moves, â1-1 for illegal moves (e.g., placing a larger disk on top of a smaller one), and a terminal reward of +10+10 when the goal is reached. The goal state G is the bit-string configuration where all disks are successfully moved to Peg C. Summary: The projecting of methods to the mdp formalism enables in depth analysis and understanding of their details. From the analysis it is possible to notice that the state space for the ToS/LLM-based methods consists of a sequence of tokens and itâs significantly larger than in the case of Neurosolver and FBRL. The state, transition and action structure of the ToS/LLM rely on probabilistic distributions (input, output tokens and the LLM) while in the case of RL and graph based they rely on graphs encoding states in nodes and transitions on edges that dictate the action. The reward in ToS is binary realized through the verifier loop, while the other two rely on maximizing continuous reward. VI Analysis from the Human Prior Perspective In this section, we analyze the degree of human prior embedded in each method, as defined in Section I-B. Table V reports a stage-wise comparison across Architecture, Training, and Inference, highlighting how human knowledge and design choices shape each system. Architecture In autotos and datos, representation is implicit and function-based (Table I): the model generates successor and goal-checking functions without an explicitly defined symbolic state encoding. This corresponds to a Medium level of human prior, since the task formulation and verification criteria are human-defined, while the internal state representation is learned implicitly by the language model. In contrast, Neurosolver and FBRL use low-level encodings that capture only disk positions on pegs, without higher-level semantics, resulting in a Low level of human prior. For all methods, the reasoning mechanism, as defined in Table I, carries a High level of human prior. autotos and datos rely on Transformer-based language models, whose architectures are fully human-designed. FBRL is built on the Double Deep Q-Network framework, which explicitly specifies the learning and decision-making mechanism. Neurosolver employs a graph-based planning mechanism with manually defined propagation rules. In all cases, the core reasoning process is explicitly designed by humans rather than discovered autonomously. Training For LLM-based methods (autotos and datos), the level of human prior in knowledge input as defined in Table I, depends on the underlying language model. DeepSeek, Yi, and Mistral are trained primarily on large-scale, human-curated datasets, which directly embed substantial human knowledge into the models, resulting in a High level of human prior [38, 39, 40]. Qwen and Gemma, in contrast, incorporate a significant amount of synthetic data and automated reinforcement learning signals during training, reducing direct human involvement in knowledge specification and leading to a Medium level of human prior [41, 42]. FBRL exhibits a High level of human prior, as humans explicitly define the reward structure, penalties, goal conditions, and environment constraints that directly shape the learning signal. In contrast, in Neurosolver, humans do not provide training data, rewards, or explicit transition constraints; the model explores valid moves autonomously based on basic puzzle rules, corresponding to a Low level of human prior. In autotos and datos, the learning as defined in Table I, process depends on the underlying LLM. DeepSeek is trained primarily through large-scale supervised learning on human-curated code data, resulting in a High level of human prior. Qwen incorporates automated reinforcement learning with synthetic data generation, reducing direct human involvement and leading to a Medium level of human prior [41]. Yi relies on continued pretraining from an existing base model using curated data [39], which also corresponds to a Medium level. Among repair agents, Mistral is trained via supervised instruction tuning with human-curated datasets, yielding High human prior, whereas Gemma explicitly employs reinforcement learning from human feedback [42] (RLHF), introducing partial automation and resulting in a Medium level. In contrast, Neurosolver and FBRL do not use human feedbackâbased learning. Neurosolver learns autonomously through exploration over valid state transitions, and FBRL updates its policy via predefined reinforcement learning rules. Since no human feedback is involved during learning, both correspond to a Low level of human prior. Inference Guidance, as defined in Table I, differs across methods in how task constraints are provided at inference time. autotos exhibits a High level of guidance, as human-provided test cases and error types are directly used to steer the refinement loop during inference. In datos, this guidance is partially automated: although human-defined test cases remain, error interpretation and correction are handled by a secondary language model, resulting in a Medium level of guidance. Neurosolver also exhibits Medium guidance, since human-defined legality rules constrain admissible moves during execution without interactive input. In contrast, FBRL operates with Low guidance, executing a learned policy without task-specific input or constraints at inference time. For all evaluated methods, the level of human prior in decision-making as defined in Table I, is Low. Action selection and planning are performed by the systems themselves during inference, without direct human involvement across autotos, datos, Neurosolver, and FBRL. Validation, as defined in Table I, differs substantially across methods. In autotos and datos, humans explicitly define test cases that are used to verify solution correctness, resulting in a Medium level of human prior in validation. In contrast, Neurosolver and FBRL do not employ an explicit validation stage: correctness is determined implicitly by goal satisfaction within the environment, making validation not applicable for these methods. Summary: The human prior ranking of the methods makes a significant view on understanding autonomous level. In general, it can be clearly seen that ToS/LLM-based approaches need more human insight compare to FBRL/Neurosolver. Due to constrain-based architectural design, FBRL and Neurosolver donât need verification after final decision, this shows clear superiority compare to ToS/LLM-based methods. While all four paradigm include high human prior in their architectural design, FBRL and Neurosolver donât need verification after final decision, due to stricter constraining at architectural stage. TABLE VI: Comprehensive Performance and Memory Comparison across datos, autotos, FBRL, and Neurosolver. All values are reported as mean ± MADâ. Model Setup Disk Att. / Steps Tokens Time (s) Total E.Mem (MB) Total O.Mem (MB) datos DeepSeek + Gemma n 6.8±1.56.8± 1.5 17998.2±3552.517998.2± 3552.5 209.21±47.84209.21± 47.84 1.82Ă1041.82Ă 10^4 1.55Ă104±01.55Ă 10^4± 0 DeepSeek + Mistral n 5.0±1.55.0± 1.5 6718.4±1571.06718.4± 1571.0 32.69±10.3732.69± 10.37 1.55Ă1041.55Ă 10^4 1.43Ă104±01.43Ă 10^4± 0 autotos [23] DeepSeek n 4.2±1.04.2± 1.0 5257.5±975.05257.5± 975.0 23.61±8.8023.61± 8.80 1.40Ă1041.40Ă 10^4 1.16Ă104±01.16Ă 10^4± 0 Yi n 2.0±0.02.0± 0.0 1162.3±64.51162.3± 64.5 26.37±5.0626.37± 5.06 5.47Ă1035.47Ă 10^3 6.16Ă103±06.16Ă 10^3± 0 Qwen n 2.0±0.02.0± 0.0 831.4±3.5831.4± 3.5 33.56±1.1633.56± 1.16 1.27Ă1041.27Ă 10^4 1.23Ă104±01.23Ă 10^4± 0 Neurosolver [18] Graph-based 2 3 â 12.06±0.0412.06± 0.04 1.37Ă10â31.37Ă 10^-3 5.41Ă10â3±9.77Ă10â75.41Ă 10^-3± 9.77Ă 10^-7 3 7 â 19.75±0.1819.75± 0.18 4.12Ă10â34.12Ă 10^-3 5.62Ă10â3±8.30Ă10â65.62Ă 10^-3± 8.30Ă 10^-6 4 15 â 28.13±0.1728.13± 0.17 1.24Ă10â21.24Ă 10^-2 1.40Ă10â2±01.40Ă 10^-2± 0 5 31 â 38.00±0.3538.00± 0.35 3.71Ă10â23.71Ă 10^-2 2.51Ă10â2±02.51Ă 10^-2± 0 6 63 â 49.76±0.4549.76± 0.45 1.11Ă10â11.11Ă 10^-1 1.05Ă10â1±01.05Ă 10^-1± 0 7 127 â 60.40±0.1560.40± 0.15 3.34Ă10â13.34Ă 10^-1 3.30Ă10â1±5.47Ă10â53.30Ă 10^-1± 5.47Ă 10^--5 FBRL [16] Reinforcement Learning 2 â â 2.42±0.072.42± 0.07 6.65Ă10â16.65Ă 10^-1 (0.018 + 0.037 + 0.61) 3.29Ă10â1±6.25Ă10â33.29Ă 10^-1± 6.25Ă 10^-3 (0.004 + 0.005 + 0.32) 3 â â 7.68±0.277.68± 0.27 9.03Ă10â19.03Ă 10^-1 (0.020 + 0.043 + 0.84) 7.03Ă10â1±07.03Ă 10^-1± 0 (0.005 + 0.008 + 0.69) 4 â â 31.20±2.0231.20± 2.02 1.14Ă1001.14Ă 10^0 (0.022 + 0.048 + 1.07) 9.37Ă10â1±09.37Ă 10^-1± 0 (0.007 + 0.010 + 0.92) â MAD denotes the Median Absolute Deviation, defined as MAD=medianâĄ(|xiâmedianâĄ(x)|)MAD=median\! ( x_i-median(x) ), computed over repeated runs. VII Analysis from the Skill and Computational Cost Perspective In this section, we analyze the skill and computational cost of each method, as defined in Section I-C by focusing on the cost to solve the puzzle correctly. We consider the methods and experimental settings outlined in Section IV. In the analysis on toh problem depicted in Figure 1 we assume three pegs and an number of disks nâ2,3,âŠâ7nâ2,3,...7. Table VI presents a comprehensive comparison of experimental results across datos, autotos, FBRL, and Neurosolver, with all values reported as mean ± Median Absolute Deviation. The table reports the model configuration, including the LLM combinations used for LLM-based agents, the number of disks, the number of attempts or steps, token usage for LLM-based methods, total execution time (including both training and solving), and both estimated and observed memory consumption. VII-A LLM-based Agents For LLM-based approaches, the reported results correspond to the combined generation of the Successor and Goal functions. As a result, the number of attempts (LLM call) reflects successful generation for both components, with one step required for the Goal function and one for the Successor function in the optimal case. datos For datos, it is evident that the double-agent design incurs higher computational cost in terms of both token usage and execution time. Although the DeepSeek+Gemma and DeepSeek+Mistral configurations exhibit comparable memory consumption, the Mistral-based setup is markedly more efficient, requiring fewer steps, fewer tokens, and shorter execution time, while also exhibiting lower variance across runs. Specifically, the DeepSeek+Gemma configuration consumes over 17k tokens and approximately 209 seconds, whereas DeepSeek+Mistral completes the task with roughly 6.5k tokens in about 32 seconds. The estimated memory consumption exceeds the observed usage, as the estimation assumes simultaneous high-precision execution of both agents. In practice, quantized deployments such as DeepSeek+Gemma (17.73 GB â 15.13 GB) and DeepSeek+Mistral (15.12 GB â 14.00 GB) use compressed weights and lighter keyâvalue caches, resulting in a lower actual memory footprint. Auto-ToS For autotos, the results indicate a substantially lower computational overhead compared to datos, as the approach relies on a single LLM without an explicit repair agent. Across all configurations, autotos typically converges in one to three attempts, with lower token usage and shorter execution times. In particular, Qwen and Yi achieve the most efficient performance, requiring fewer than 1k tokens on average, completing execution in approximately 25â35 seconds, and consistently producing the optimal step size, whereas DeepSeek exhibits higher token usage and additional refinement cycles in the Successor function, leading to increased attempts and longer execution times. In terms of memory consumption, autotos follows the expected trend based on model size, with DeepSeek exhibiting the largest footprint (â 11.3 GB observed), followed by Qwen (â 12.0 GB) and Yi (â 6.0 GB). As with datos, estimated memory exceeds observed usage due to conservative precision and cache assumptions, while quantized execution reduces the actual footprint during inference. VII-B Neurosolver Across all configurations, Neurosolver consistently achieves the optimal number of steps (2nâ12^n-1), while runtime and memory consumption increase with puzzle size. Runtime grows gradually from 12.06 seconds for two disks to 60.40 seconds for seven disks, remaining below one minute even for the largest instance. In contrast, memory consumption increases more sharply, from 0.005 MB to 0.321 MB, reflecting the exponential growth of the state space. While observed memory usage remains within the same order of magnitude as theoretical estimates, deviations are expected due to random exploration and higher-level state representations in the implementation. Overall, this behavior aligns with the theoretical scaling law from Section IV-B1, highlighting the trade-off between guaranteed optimality and rapidly growing memory requirements. VII-C Forward-backward Reinforcement Learning For FBRL, computational cost increases rapidly with the number of disks, primarily due to its dual exploration mechanism that combines forward trajectories from the initial state with backward rollouts from the goal. Runtime grows from approximately 2.4 seconds for two disks to over 31 seconds for four disks, reflecting the increased training effort required as the search space expands. In terms of memory consumption, the breakdown shown in the table highlights contributions from the Q-Network, Backward Model, and Replay Buffer. While the Q-Network and Backward Model remain lightweight and scale linearly with the input and output dimensions of 3ân3n, their memory usage is negligible compared to the replay buffer. In contrast, experience storage dominates the overall footprint, increasing from about 0.32 MB for two disks to approximately 0.92 MB for four disks. Both observed and estimated results indicate that FBRLâs memory behavior is driven primarily by replay buffer growth rather than network complexity, resulting in a predictable but increasingly costly memory profile. Summary: The skill to solve the ToH puzzle and computational comparison illustrates quantitative insight about pros and cons in terms of memory and time. From the table, it is clear that ToS/LLM-based approaches need large amount of memory resource compared to Neurosolver and FBRL. On the other hand the ToS/LLM-based ones are not limited by about disk numbers. Focusing on FBRL vs Neurosolver, it seems that Neurosolver has superior scalability performance since time and memory does not increase as much as for FBRL. VIII Conclusion In this paper we addressed the challenge that existing evaluations of sequential decision making methods largely rely on isolated prosâcons analyses, which fail to reveal how design choices redistribute complexity across system components. To address this gap, we proposed a three-dimensional characterization that jointly captures mdp formulation, human prior, and skillâcost trade-offs. Using the proposed framework, three representative graph-based, reinforcement learning, and large language model (LLM)-based approaches and the Tower of Hanoi puzzle we found that in LLM-based methods, where the action space is weakly constrained by design, : 1) correctness is enforced through increased human insight at inference time, primarily via verification and correction loops and 2) complexity is shifted to computation, resulting in significantly higher memory and runtime costs compared to graph-based and reinforcement learning models. Our findings enable structured cross-paradigm comparison linking theoretical formulation with empirical performance. The framework provides a principled basis for analyzing such trade-offs across AI techniques even beyond the toh. References [1] S. Murugesan (2025) The rise of agentic ai: implications, concerns, and the path forward. IEEE Intelligent Systems 40 (2), p. 8â14. Cited by: §I. [2] C. Castelfranchi (1998) Modelling social action for ai agents. Artificial intelligence 103 (1-2), p. 157â182. Cited by: §I. [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877â1901. Cited by: §I. [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ć. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §I. [5] X. Wang, A. Antoniades, Y. Elazar, A. Amayuelas, A. Albalak, K. Zhang, and W. Y. Wang (2025) Generalization v.s. memorization: tracing language modelsâ capabilities back to pretraining data. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §I. [6] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §I. [7] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824â24837. Cited by: §I. [8] P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar (2025) The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941. Cited by: §I, §I. [9] B. Estermann, L. Lanzendörfer, Y. Niedermayr, and R. Wattenhofer (2024) Puzzles: a benchmark for neural algorithmic reasoning. Advances in Neural Information Processing Systems 37, p. 127059â127098. Cited by: §I, §I. [10] J. Hawkins (2025) Enigme: generative text puzzles for evaluating reasoning in language models. arXiv preprint arXiv:2505.04914. Cited by: §I. [11] P. Merrotsy (2015) The tower of hanoi and inductive logic. Australian Senior Mathematics Journal 29 (1), p. 16â24. Cited by: §I. [12] O. D. Anderson (1992) Induction, recursion, and the towers of hanoi. International Journal of Mathematical Education in Science and Technology 23 (3), p. 339â343. Cited by: §I. [13] J. A. H. Aguilar, A. R. P. Mayo, S. Y. Ortuño, A. Ochoa-Zezzatti, and J. C. P. Gallegos (2016) The application of hanoi towers game in logistics management. In Handbook of Research on Military, Aeronautical, and Maritime Logistics and Operations, p. 422â433. Cited by: §I. [14] E. Mehiri and H. Belbachir (2024) The weighted tower of hanoi. Discrete Mathematics, Algorithms and Applications 16 (05), p. 2350051. Cited by: §I. [15] G. Havur, K. Haspalamutgil, C. Palaz, E. Erdem, and V. Patoglu (2013) A case study on the tower of hanoi challenge: representation, reasoning and execution. In 2013 IEEE International Conference on Robotics and Automation, p. 4552â4559. Cited by: §I. [16] A. D. Edwards, L. Downs, and J. C. Davidson (2018) Forward-backward reinforcement learning. arXiv preprint arXiv:1803.10227. Cited by: 3rd item, §I, §IV-C, TABLE IV, TABLE V, TABLE VI. [17] D. J. Agravante, D. Kimura, and M. Tatsubori (2023) Learning neuro-symbolic world models with logical neural networks. In PRL Workshop Series Bridging the Gap Between AI Planning and Reinforcement Learning, Cited by: §I, TABLE I. [18] A. Bieszczad and S. Kuchar (2015) Neurosolver learning to solve towers of hanoi puzzles. In 2015 7th International Joint Conference on Computational Intelligence (IJCCI), Vol. 3, p. 28â38. Cited by: 3rd item, §I, §IV-B, TABLE IV, TABLE V, TABLE VI. [19] T. Stewart and C. Eliasmith (2011) Neural cognitive modelling: a biologically constrained spiking neuron model of the tower of hanoi task. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 33. Cited by: §I. [20] L. Wang, Z. Jiang, C. Hu, J. Zhao, Z. Zhu, X. Chen, Z. Wang, T. Liu, G. He, Y. Yin, et al. (2025) Comparing ai and human decision-making mechanisms in daily collaborative experiments. iScience 28 (6). Cited by: §I, §I, §I-C. [21] Z. Qin, H. Wang, D. Liu, Z. Song, C. Fan, Z. Lv, J. Wu, Z. Lei, Z. Tu, D. Chu, et al. (2024) UNO arena for evaluating sequential decision-making capability of large language models. arXiv preprint arXiv:2406.16382. Cited by: §I, §I. [22] T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker (2022) A unifying framework for reinforcement learning and planning. Frontiers in Artificial Intelligence 5, p. 908353. Cited by: §I, §I, §I-C. [23] M. Katz, H. Kokel, K. Srinivas, and S. Sohrabi Araghi (2024) Thought of search: planning with language models through the lens of efficiency. Advances in Neural Information Processing Systems 37, p. 138491â138568. Cited by: 3rd item, TABLE IV, TABLE V, TABLE VI. [24] H. Li, B. Jiang, A. Naehu, R. Song, J. Zhang, M. Tjandrasuwita, C. Ekbote, S. Chen, A. Balachandran, W. Dai, et al. (2025) PuzzleWorld: a benchmark for multimodal, open-ended reasoning in puzzlehunts. arXiv preprint arXiv:2506.06211. Cited by: §I. [25] J. Burden, M. TeĆĄiÄ, L. Pacchiardi, and J. HernĂĄndez-Orallo (2025) Paradigms of ai evaluation: mapping goals, methodologies and culture. arXiv preprint arXiv:2502.15620. Cited by: §I. [26] D. Z. Chen, P. Verma, S. Srivastava, M. Katz, and S. ThiĂ©baux (2024) AI planning: a primer and survey (preliminary report). arXiv preprint arXiv:2412.05528. Cited by: §I-A. [27] T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-PĂ©rez, and L. P. Kaelbling (2022) PDDL planning with pretrained large language models. In NeurIPS 2022 foundation models for decision making workshop, Cited by: TABLE I. [28] Y. Bai and C. Jin (2020) Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning, p. 551â560. Cited by: TABLE I, TABLE I. [29] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel (2016) Value iteration networks. Advances in neural information processing systems 29. Cited by: TABLE I. [30] R. R. Hegde and S. Sharma (2024) Self supervised llm customizer (sslc): customizing llms on unlabeled data to enhance contextual question answering. In Proceedings of the 4th International Conference on AI-ML Systems, p. 1â11. Cited by: TABLE I. [31] H. Van Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30. Cited by: TABLE I. [32] D. Zhou, N. SchĂ€rli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, et al. (2022) Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. Cited by: TABLE I. [33] Z. Wang, S. Cai, G. Chen, A. Liu, X. S. Ma, and Y. Liang (2023) Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. Advances in Neural Information Processing Systems 36, p. 34153â34189. Cited by: TABLE I. [34] A. Wachi and Y. Sui (2020) Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning, p. 9797â9806. Cited by: TABLE I. [35] N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang (2024) Automated unit test improvement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, p. 185â196. Cited by: TABLE I. [36] S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi (2022) Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053. Cited by: TABLE I. [37] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §IV-C. [38] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Li, et al. (2024) DeepSeek-coder: when the large language model meets programmingâthe rise of code intelligence. arXiv preprint arXiv:2401.14196. Cited by: §VI. [39] A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen, et al. (2024) Yi: open foundation models by 01. ai. arXiv preprint arXiv:2403.04652. Cited by: §VI, §VI. [40] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed (2023) Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §VI. [41] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. (2024) Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §VI, §VI. [42] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. RiviĂšre, M. S. Kale, J. Love, et al. (2024) Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: §VI, §VI.