Paper deep dive
HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents
Hongbo Jin, Rongpeng Zhu, Jiayu Ding, Guibo Luo, Ge Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 3:36:17 AM
Summary
The paper introduces HiMAC, a hierarchical agentic reinforcement learning framework for Large Language Model (LLM) agents designed to address limitations in long-horizon tasks. HiMAC decomposes decision-making into macro-level planning (blueprint generation) and micro-level execution (goal-conditioned action). It employs a critic-free hierarchical policy optimization paradigm extending Group Relative Policy Optimization (GRPO) and an iterative co-evolution training strategy to stabilize non-stationary bi-level optimization. Experiments on ALFWorld, WebShop, and Sokoban demonstrate state-of-the-art performance and improved sample efficiency compared to flat policy baselines.
Entities (17)
Relation Signals (18)
HiMAC → evaluatedon → Sokoban
confidence 95% · Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms
HiMAC → evaluatedon → WebShop
confidence 95% · Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms
HiMAC → evaluatedon → ALFWorld
confidence 95% · Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms
HiMAC → uses → Macro-Policy
confidence 95% · HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution
HiMAC → uses → Micro-Policy
confidence 95% · HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution
HiMAC → uses → Micro-Policy
confidence 95% · Conditioned on this blueprint, the Micro-Policy operates as a focused executor
HiMAC → outperforms → GiGPO
confidence 92% · HiMAC achieves an overall success rate of 89.9%, surpassing the strongest multi-turn RL baseline, GiGPO, by 3.8%.
Micro-Policy → executes → Blueprint
confidence 90% · Conditioned on this blueprint, the Micro-Policy operates as a focused executor, generating atomic actions for each sub-goal
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model (LLM) agents have recently demonstrated strong capabilities in interactive decision-making, yet they remain fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to inefficient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adaptation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms strong prompting and reinforcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and visually grounded environments. Our results show that introducing structured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence.
Tags
Links
- Source: https://arxiv.org/abs/2603.00977v2
- Canonical: https://arxiv.org/abs/2603.00977v2
Trouble viewing inline? Open PDF directly →
Full Text
51,303 characters extracted from source content.
Expand or collapse full text
HiMAC: Hierarchical Macro–Micro Learning for Long-Horizon LLM Agents Hongbo Jin † , Rongpeng Zhu † , Jiayu Ding, Guibo Luo, and Ge Li ∗ School of Electronic and Computer Engineering Peking University Abstract. Large language model (LLM) agents have recently demon- strated strong capabilities in interactive decision-making, yet they re- main fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to ineffi- cient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a struc- tured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage es- timation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adap- tation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demon- strate that HiMAC consistently outperforms strong prompting and rein- forcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and vi- sually grounded environments. Our results show that introducing struc- tured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence. Keywords: LLM Agent· Hierarchical RL· Long-Horizon Planning 1 Introduction The rapid evolution of Large Language Models (LLMs) has sparked a paradigm shift in agentic AI, transitioning from passive answering systems to autonomous agents capable of active environment interaction [39, 40, 53]. By grounding lin- guistic reasoning in sequential decision-making, LLM-based agents have demon- strated impressive proficiency in short-horizon tasks [11, 26]. However, current †: Equal key contributions. ∗: Corresponding author. arXiv:2603.00977v2 [cs.AI] 5 May 2026 2F. Author et al. LLM agents exhibit three coupled failure modes in long-horizon settings: ex- ponential exploration complexity, delayed credit assignment, and semantic drift across extended reasoning trajectories. The prevailing landscape of agentic control is dominated by "flat" policy architectures [12, 14, 20]. In these paradigms, a single autoregressive model is tasked with generating both high-level thoughts and low-level actions in a dense, token-by-token manner. While straightforward, this monolithic formulation suf- fers fundamentally from the curse of dimensionality in exploration [10,25,57,59]. Under this setting, the agent must navigate a vast combinatorial search space us- ing myopic next-token prediction. In such flat trajectories, errors do not merely occur but also propagate exponentially. A minor syntactic deviation in an early step often cascades into irreversible failure states, causing the agent to lose track of the global goal. This limitation indicates that relying solely on the inherent reasoning capabilities of generic LLMs is insufficient; structural inductive biases are required to decouple global planning from local control [7,13,34,49]. Fig. 1: A typical case comparsion in WebShop environment and overall performance. (a) Flat-policy architectures easily get lost in irrelevant observations (Context Drift). (b) HiMAC addresses this exploration inefficiency through a bi-level decoupled archi- tecture (Planner and Executor sharing the same parameters π θ ), mapping structured blueprints to executable atomic actions. (c) HiMAC yields consistent state-of-the-art performance against advanced RL reasoning baselines across multiple environments. This calls for a fundamental shift in how LLM agents are structured. We pro- pose HiMAC (Hierarchical Macro-Micro Agentic Control), a framework that explicitly organizes the agent as a two-level cooperation. The Macro-Policy op- erates as a strategic planner: given a task instruction, it searches over a latent se- mantic space to produce a blueprint—a structured sequence of natural language sub-goals that decompose the long-horizon objective into tractable milestones. Abbreviated paper title3 Conditioned on this blueprint, the Micro-Policy operates as a focused executor, generating atomic actions for each sub-goal in sequence. As illustrated in Fig. 1, this separation fundamentally changes the nature of the exploration problem: rather than navigating an exponentially large joint action-reasoning space in a single flat trajectory. Realizing this hierarchical vision, however, introduces a formidable optimiza- tion challenge. Training the two levels jointly constitutes a non-stationary bi-level problem: the Macro-Policy must learn to propose sub-goals for an executor whose capabilities are simultaneously evolving, while the Micro-Policy must learn to ex- ecute plans whose quality is shifting as the planner improves. Standard RL ap- proaches attempt to resolve such instability via auxiliary Value Networks (Crit- ics) [36,37,43,48], but learning accurate value functions in the high-dimensional, sparse semantic space of language is notoriously sample-inefficient and prone to divergence. Without stable optimization, hierarchical decomposition offers no practical benefit over flat policies. Driven by these challenges, HiMAC introduces two tightly coupled techni- cal innovations. (i) Critic-Free Hierarchical Policy Optimization: We ex- tend Group Relative Policy Optimization (GRPO) [44] to a bi-level structure by constructing level-specific comparison groups—sampled blueprints are evaluated against peer blueprints, and execution trajectories are evaluated against peer trajectories conditioned on the same blueprint. This hierarchical grouping yields low-variance advantage estimates tailored to each level, enabling precise credit assignment without any parametric value network. (i) Iterative Co-Evolution Training: To resolve non-stationarity, we decouple the two optimization objec- tives into alternating phases. In the Macro-Exploration Phase, the planner is updated while the executor runs in inference mode, providing a deterministic re- ward signal that drives the planner toward practically achievable blueprints. In the Micro-Adaptation Phase, a high-confidence blueprint is fixed as a constant condition, and the executor is updated in isolation. This alternation converts the unstable bi-level problem into a sequence of stationary single-level updates, naturally inducing a curriculum where the planner progressively proposes more complex strategies as executor proficiency grows. We evaluate HiMAC across three benchmarks specifically chosen to stress- test different facets of hierarchical agentic control: ALFWorld [46] (multi-step embodied reasoning in household environments), WebShop [58] (long-horizon web navigation with noisy, high-dimensional observations), and Sokoban [42] (visually-grounded spatial planning requiring precise sequential reasoning). Against strong prompting techniques [45, 60] and recent critic-free RL methods [15, 44] including GRPO and GiGPO, HiMAC achieves state-of-the-art results on all three—most notably a 16% gain over the strongest RL baseline on WebShop (83.4% vs. 67.4%), a benchmark where flat policies are most severely penalized by context drift. Beyond raw performance, HiMAC reaches comparable success thresholds with substantially fewer training iterations, and qualitative analysis reveals that the Macro-Policy spontaneously develops self-verification behaviors absent from any flat baseline. Together, these results support a broader conclu- 4F. Author et al. sion: structured hierarchy provides complementary and substantial gains beyond those achievable through model scale alone. In summary, our main contributions are as follows: – We propose HiMAC, a hierarchical framework that decomposes long-horizon agentic tasks into Macro-level blueprint generation and Micro-level goal- conditioned execution, fundamentally reducing the exploration complexity and error propagation of flat policy architectures. – We introduce a Critic-Free Hierarchical Policy Optimization objective and an Iterative Co-Evolution training strategy, extending group-based RL to a bi-level structure for precise level-specific credit assignment, while alternating Macro-Exploration and Micro-Adaptation phases to stabilize the non-stationary bi-level optimization. – Experiments on various challenging benchmarks demonstrate state-of-the- art performance across text-based and visually-grounded environments, with up to 16% gain over the strongest RL baseline and superior sample efficiency. 2 Related Work LLM Agents. Large language models (LLMs) [1, 64] and, increasingly, Vision- Language Models (VLMs) [28,51] have rapidly evolved into versatile autonomous agents capable of operating in open-ended text and visually-grounded environ- ments [23, 52, 62, 66]. Early prompting methods, such as ReAct [60] and Re- flexion [45], proved effective for short-horizon tasks by interleaving reasoning with actions. However, these "flat" autoregressive approaches struggle severely in long-horizon scenarios. In these settings, myopic next-token prediction of- ten causes minor deviations to cascade into irreversible failures. Because these errors fundamentally stem from entangling high-level reasoning with low-level execution in a single token stream, relying solely on frozen foundation models is insufficient. To robustly decouple global semantic planning from local control, agents require explicitly hierarchical architectures combined with corresponding policy optimization strategy [8,17,29,33]. Agentic RL. Applying RL to LLM-based agents has emerged as a promising direction for sequential decision-making [4, 9, 21, 22, 38, 50, 55, 61]. Traditional actor-critic methods such as PPO [43] have been adopted for embodied and web navigation agents [46,58], but learning accurate value functions over high- dimensional language states with sparse rewards is notoriously sample-inefficient and prone to divergence. This has motivated a shift toward critic-free, group- based objectives such as RLOO [2] and GRPO [44], which estimate relative advantages within sampled groups and yield more stable convergence. GiGPO [15] and RAGEN [54] further extend this paradigm to multi-turn agentic settings. However, all of these methods optimize a monolithic policy over the joint space of reasoning and action tokens, conflating high-level semantic intent with low-level execution in a single flat trajectory. As a result, they inherit the exponential exploration complexity of long-horizon tasks and remain susceptible to error Abbreviated paper title5 propagation across extended interaction sequences—limitations that hierarchical decomposition is specifically designed to address. Hierarchical Reinforcement Learning. Classical HRL [3,13,27,32,48,65] addresses long-horizon tasks via temporal abstraction. Recent robotics approaches further mitigate hierarchical non-stationarity via preference optimization [47] or alter- nating training [31], yet they remain grounded in compact, predefined state spaces distinct from the open-ended reasoning required by LLMs. In the realm of LLM agents, works largely rely on prompt engineering without parameter op- timization (e.g., HiAgent [18]), or depend heavily on offline datasets and fixed primitives (e.g., LARAP [63], GLIDER [16]). Similarly, while advanced RL meth- ods target long-horizon tasks, they often optimize flat policies (e.g., AgentGym- RL [56]) or rely on offline preference learning (e.g., HPL [19]). CoDA [30] in- troduces a context-decoupled framework for multi-hop QA via joint optimiza- tion. In contrast, HiMAC bridges existing gaps in embodied tasks by decoupling decision-making into a Macro-Planner and Micro-Executor, directly optimizing natural-language parameters through a stable, critic-free hierarchical objective driven by iterative co-evolution. 3 Methodology 3.1 Problem Formulation We formulate the long-horizon agentic task as a Goal-Conditioned POMDP [24,41], where an agent receives partial observations o t ∈O and executes atomic actions a t ∈ A to maximize the expected return given a natural language in- struction x. Standard approaches model the policy π(a t |o ≤t ,x) as a flat autoregressive process. However, for long-horizon tasks, the probability of optimal trajectory generation P(τ|x) vanishes exponentially with horizon length T. To address this, we introduce a Structured Blueprint z as a discrete semantic variable, represent- ing a complete high-level plan decomposed from x. We formulate the trajectory generation as a Hierarchical Process where the joint distribution of the blueprint z and atomic actions a is factorized as: P(a, z|x) = π macro (z|x) |z Planning · T Y t=1 π micro (a t |o ≤t , z φ(t) ) | z Execution (1) where z∈Z denotes a sequence of natural language sub-goalsg 1 ,...,g K , and φ(t) maps the timestep to the active sub-goal index. This formulation decouples the problem into two optimization objectives: (i) Macro-Optimization: Finding the optimal blueprint z ∗ = arg max z E[V (z,x)] via search in the semantic space Z. (i) Micro-Optimization: Learning a robust instruction-following policy π micro that minimizes the execution divergence given a fixed blueprint z ∗ . 6F. Author et al. 3.2 Macro-Micro Architecture Fig. 2: Overall pipeline of HiMAC. Phase A (left) optimizes the macro-policy for blueprint generation using GRPO while freezing the micro-policy. Phase B (right) op- timizes the micro-policy for atomic action execution conditioned on a fixed blueprint z ∗ . Both phases update the same underlying LLM parameters θ. The core premise of HiMAC is decoupling long-horizon reasoning from local execution. First, the Macro-Policy searches over an open-ended natural language space to generate a structured blueprint z = (z 1 ,...,z K ). Each z k represents a free-form semantic milestone, decomposing the global objective into tractable, task-agnostic sub-problems. Subsequently, the Micro-Policy π micro operates as a specialized executor. Conditioned on a selected blueprint z ∗ , it sequentially executes the plan. At each timestep t, atomic actions are generated based on the currently active sub- goal z k : a t ∼ π micro (a t |o ≤t ,z k ). Sub-goal transitions (k ← k+1) are autonomously triggered when the micro- policy generates a special <sub_done> termination token. This rigid condition- ing acts as a temporal attention mask, explicitly narrowing the agent’s context window to the relevant task segment to prevent semantic drift over long hori- zons. Finally, to handle unexpected environmental stochasticity, we implement a lightweight budget-based fallback: if the executor fails to output <sub_done> within a step limit T limit or encounters specific failure keywords, the episode is halted and penalized. 3.3 Hierarchical Policy Optimization To train the unified policy without auxiliary value networks, we extend GRPO to our hierarchical setting by constructing level-specific comparison groups. The key insight is that both levels share the same optimization structure: given a group of N sampled outputs with returns R 1 ,...,R N , the group-relative advantage for the i-th output is computed as: Abbreviated paper title7 ˆ A (i) = R i − μ group σ group + ε (2) where μ group and σ group are the mean and standard deviation of returns within the group. The corresponding GRPO-style objective is: J (θ) =E " 1 N N X i=1 min ρ i ˆ A (i) , clip(ρ i , 1−ε, 1+ε) ˆ A (i) − βD KL (π θ ∥π ref ) # (3) where ρ i = π θ (·)/π θ old (·) is the importance sampling ratio and β controls the KL penalty against the reference model. The two levels instantiate this template differently, as detailed below. Macro-Objective (Phase A). The planner samples a group of G candidate blueprints z (1) ,..., z (G) ∼ π θ old (·| x). Each blueprint is evaluated by rolling out the cur- rent micro-policy to obtain a trajectory return R(z (i) ), which serves as the group signal in Eq. (2). Substituting into Eq. (3) with ρ macro i = π θ (z (i) | x) π θ old (z (i) | x) and restricting the gradient to blueprint tokens yields the macro-objective J macro (θ). This setup forces the advantage signal to reflect the quality of the plan itself, isolated from the variance introduced by execution stochasticity. Micro-Objective (Phase B). The executor samples a group of M trajectories τ (1) ,...,τ (M) ∼ π θ old (·| x, z ∗ ) conditioned on a fixed high-confidence blueprint z ∗ . Because all trajectories share the same semantic intent, their return differ- ences are attributable solely to execution quality, forming a valid cohort for group-relative comparison. Substituting into Eq. (3) with ρ micro j = π θ (τ (j) | x, z ∗ ) π θ old (τ (j) | x, z ∗ ) and restricting the gradient to atomic action tokens yields the micro-objective J micro (θ). 3.4 Iterative Co-Evolution Training Simultaneously optimizing the planner and executor introduces a fundamen- tal non-stationarity: the planner chases a shifting execution capability while the executor adapts to a drifting sub-goal distribution. To stabilize this bi-level op- timization, we introduce an Iterative Co-Evolution strategy (Algorithm 1) that decouples the process into two alternating phases, ensuring each level is updated against a stationary counterpart. 8F. Author et al. Algorithm 1 HiMAC: Iterative Co-Evolution Training Require: Dataset D, initial policy π θ , reference model π ref , group sizes G, M, itera- tions T 1: for iteration t = 1,...,T do 2: Sample instruction x∼D 3: // Phase A: Macro-Exploration 4: Sample blueprint group z (i) G i=1 ∼ π θ old (·| x) 5: for i = 1,...,G do 6:Evaluate R(z (i) ) via greedy micro-policy rollout (no grad) 7: end for 8: Compute ˆ A (i) macro via Eq. (2) 9: Update θ ← θ− η∇ θ J macro (θ) (blueprint tokens only) 10: // Phase B: Micro-Adaptation 11: Selectz ∗ = arg max i R(z (i) ) 12: Sample trajectory group τ (j) M j=1 ∼ π θ old (·| x,z ∗ ) 13: Compute ˆ A (j) micro via Eq. (2) 14: Update θ ← θ− η∇ θ J micro (θ) (action tokens only) 15: end for Ensure: Trained hierarchical policy π θ Phase A: Macro-Exploration. We optimize J macro (θ) while treating the micro- policy as a fixed component of the environment. For each sampled blueprint z (i) ∼ π θ (·| x), the executor evaluates it via greedy decoding in inference mode (no gradient tracking), providing a deterministic return signal R(z (i) ) that re- flects plan feasibility without execution variance. Phase B: Micro-Adaptation. We optimizeJ micro (θ) conditioned on a high-confidence blueprint: z ∗ = arg max i R(z (i) )(4) selected from the group sampled in Phase A. Fixing z ∗ as a constant condition (via stop-gradient) ensures that return differences across the sampled trajectories τ (j) M j=1 are attributable solely to execution quality, providing a clean learn- ing signal for low-level control. Using the highest-return blueprint (rather than a randomly sampled one) further guarantees the executor trains on coherent, currently achievable plans, avoiding wasted rollouts (ablated in Sec. 4.3). 4 Experiments In this section, we empirically validate the effectiveness of HiMAC on complex, long-horizon agentic tasks. Our evaluation focuses on three central hypotheses: (i) the hierarchical decoupling of planning and execution yields superior per- formance compared to flat policy architectures; (i) the Iterative Co-Evolution strategy effectively stabilizes the non-stationary dynamics inherent in bi-level optimization; (i) our method achieves these gains with high sample efficiency. Abbreviated paper title9 4.1 Experimental Setup Benchmarks and Baselines. To ensure a rigorous evaluation aligned with state- of-the-art standards, we conduct experiments on various challenging bench- marks. ALFWorld [46] evaluates embodied decision-making across six distinct task types requiring multi-step reasoning in a simulated household environment. WebShop [58] tests the agent’s ability to navigate a noise-heavy e-commerce web- site to locate and purchase products matching specific attributes. Sokoban [42] introduces a visually-grounded spatial planning challenge, requiring the agent to execute precise sequential reasoning to navigate and push boxes to designated targets. We compare HiMAC against a comprehensive suite of baselines, includ- ing prompt-based methods (ReAct [60], Reflexion [45]) and recent reinforcement learning advancements (PPO [43], RLOO [2], GRPO [44] and GiGPO [15]. Implementation Details. For text-based tasks (ALFWorld and WebShop), we employ Qwen2.5-Instruct [35] (1.5B and 7B) as the backbone for both the Macro- Policy and Micro-Policy. For the visually-grounded Sokoban benchmark, we uti- lize Vision-Language Models (VLMs) including the Qwen2.5-VL [6] (3B and 7B) and Qwen3-VL [5] (8B) series. Across all tasks, we maintain a consistent response length of 512 and a rollout group size of N = 8. For ALFWorld, the maximum prompt length is 2048 with 128 parallel environments; for WebShop, the maximum prompt length is 4096 and episodes are limited to 15 steps; for Sokoban, the maximum prompt length is 1024 with a similar episode limit of 15 steps. The KL-divergence coefficient β is fixed at 0.01 across all runs. 4.2 Main Results ALFWorld. Table 1 reports the performance of all methods on ALFWorld and WebShop. On ALFWorld with the 1.5B backbone, HiMAC achieves an overall success rate of 89.9%, surpassing the strongest multi-turn RL baseline, GiGPO, by 3.8%. The gains are particularly pronounced on structurally complex task types: Pick2 (multi-target pick, 84.3 vs. 76.4%) and Clean (98.2 vs. 91.8%), which require coherent multi-stage planning. This demonstrates that the Struc- tured Blueprint effectively partitions the long-horizon task into tractable sub- goals, preventing the context drift that causes flat policies to fail. Notably, de- spite having only 1.5B parameters, our HiMAC-trained model achieves an 89.9% success rate, surpassing the prompt-only performance of the much larger closed- source model Claude Sonnet-4.5 (82.5%) and demonstrating the efficiency of our hierarchical structure. Scaling to the 7B backbone, HiMAC achieves an overall success rate of 92.1%, consistently outperforming all baselines across all six task categories. The im- provement over the best flat RL baseline (GiGPO, 90.8%) reflects that hierarchi- cal decomposition provides complementary benefits beyond those of group-based credit assignment alone. 10F. Author et al. Table 1: Performance on ALFWorld and WebShop. Results are averaged over 3 ran- dom seeds. PE refers to prompt engineering. GiGPO w/ std denotes using F norm = std, while GiGPO w/o std uses F norm = 1. Type Method ALFWorldWebShop PickLook Clean HeatCool Pick2AllScore Succ. Closed-Source Model PE GPT-4o75.360.831.256.721.649.848.031.823.7 PE Gemini-2.5-Pro92.863.362.169.026.658.760.342.535.9 PE GPT-5------65.4-33.7 PE Claude Sonnet 4.5------82.5-38.6 Qwen2.5-1.5B-Instruct PE Qwen2.55.95.53.39.74.20.04.123.15.2 PE ReAct17.420.515.76.27.72.012.8 40.111.3 PE Reflexion35.322.221.713.619.43.721.855.821.9 RL PPO64.8 ±3.5 40.5 ±6.9 57.1 ±4.9 60.6 ±6.6 46.4 ±4.0 47.4 ±1.9 54.4 ±3.1 73.8 ±3.0 51.5 ±2.9 RL RLOO88.3 ±3.0 52.8 ±8.6 71.0 ±5.9 62.8 ±8.7 66.4 ±5.5 56.9 ±4.7 69.7 ±2.5 73.9 ±5.6 52.1 ±6.7 RL GRPO85.3 ±1.5 53.7 ±8.0 84.5 ±6.8 78.2 ±7.9 59.7 ±5.0 53.5 ±5.6 72.8 ±3.6 75.8 ±3.5 56.8 ±3.8 RL GiGPO w/ std 94.4 ±5.9 67.5 ±4.6 94.8 ±3.8 94.4 ±7.8 79.8 ±4.7 76.4 ±5.4 86.7 ±1.7 83.1 ±1.6 65.0 ±3.2 RL GiGPO w/o std 96.0 ±4.7 76.5 ±1.9 91.8 ±5.5 91.3 ±6.1 71.7 ±8.4 79.5 ±7.7 86.1 ±4.7 83.5 ±1.8 67.4 ±4.5 RLHiMAC98.6 ±1.5 79.9 ±4.7 98.2 ±1.8 90.5 ±3.1 80.1 ±3.6 84.3 ±5.6 89.9 ±3.4 92.2 ±2.9 83.4 ±1.3 Qwen2.5-7B-Instruct PE Qwen2.533.421.619.36.92.83.214.826.47.8 PE ReAct48.535.434.313.218.217.631.246.219.5 PE Reflexion62.041.644.930.936.323.842.758.128.8 RL PPO92.3 ±4.0 64.0 ±8.4 92.5 ±2.4 89.5 ±7.0 80.3 ±2.0 68.8 ±8.3 80.4 ±2.7 81.4 ±3.1 68.7 ±5.1 RL RLOO87.6 ±4.3 78.2 ±8.3 87.3 ±5.8 81.3 ±7.6 71.9 ±5.2 48.9 ±8.4 75.5 ±4.6 80.3 ±3.2 65.7 ±4.0 RL GRPO90.8 ±5.1 66.1 ±6.7 89.3 ±5.4 74.7 ±6.9 72.5 ±5.4 64.7 ±7.3 77.6 ±5.2 79.3 ±2.8 66.1 ±3.7 RL GiGPO w/ std 97.7 ±1.6 82.7 ±7.9 97.8 ±1.6 83.7 ±7.2 89.3 ±8.2 79.2 ±6.6 90.8 ±1.3 84.4 ±2.9 72.8 ±3.2 RL GiGPO w/o std 91.8 ±5.4 88.6 ±6.3 95.9 ±3.2 90.2 ±2.6 86.5 ±5.5 85.2 ±7.5 90.2 ±2.3 86.2 ±2.6 75.2 ±3.8 RLHiMAC99.1 ±0.9 88.1 ±5.9 98.4 ±4.5 91.9 ±2.3 92.7 ±4.4 89.5 ±6.1 92.1 ±3.5 93.8 ±2.4 84.1 ±2.9 WebShop. The WebShop benchmark presents a substantially more challeng- ing setting due to its large action space and noisy observation signals. HiMAC demonstrates exceptional gains here: with the 1.5B backbone, it achieves a suc- cess rate of 83.4% and a score of 92.2%, surpassing GiGPO by a remarkable 16.0% in success rate (67.4%). We attribute this large margin to the Struc- tured Blueprint’s ability to decompose the web navigation task into semantic milestones, preventing the executor from losing track of the overall purchas- ing intent across long interaction sequences. With the 7B backbone, HiMAC achieves a success rate of 84.1% and a score of 93.8%, again outperforming all RL baselines. Sokoban. As shown in Table 2, on the Sokoban benchmark using the Qwen2.5- VL-7B backbone, HiMAC achieves a success rate of 87.5% and a score of 6.70, outperforming GiGPO (82.8%, 5.27) by 4.7 points in success and 1.43 points in score. This result demonstrates that the hierarchical planning paradigm general- izes beyond text-based agentic tasks to visually-grounded puzzle environments, where the Macro-Policy effectively learns to decompose box-pushing sequences into structured subgoal chains. Abbreviated paper title11 Prompt-Based vs. RL-Based Methods. Across all benchmarks, prompt-based methods (ReAct, Reflexion) lag significantly behind RL-based approaches, con- firming that pre-trained LLMs require parameter-level adaptation to master long-horizon sequential tasks. Among RL methods, the overall performance com- parison suggests that finer-grained credit assignment and hierarchical structure are both independently beneficial, and their combination in HiMAC yields the strongest performance. 4.3 Ablation Studies Table 2: Performance on the Sokoban benchmark. ’Succ.’ and ’Score’ refers to success rate and test score respectively. Method Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen3-VL-8B Succ. Score Succ. Score Succ. Score Prompt 14.1-0.8825.0-0.2621.9 -0.95 GRPO76.24.5983.95.3982.95.24 GiGPO 81.14.8682.85.2778.14.67 HiMAC 83.8 5.31 87.5 6.70 90.6 7.32 To isolate the contribution of each core design decision in HiMAC, we conduct a systematic ablation study on ALFWorld and WebShop us- ing the Qwen2.5-7B-Instruct backbone. We evaluate four variants, each removing or re- placing a single component of the full system. Results are re- ported in Table 3. w/o Hierarchy. In this variant, we remove the entire hierarchical structure—both the structured blueprint generation and the level-specific group construction—and instead apply a single GRPO objective over the complete, flat trajectory. Under this flat formulation, blueprint-level credit and execution-level credit are con- flated into a single advantage estimate, making it impossible to attribute out- come variance to planning decisions versus execution decisions independently. Note that this variant also implicitly removes the Iterative Co-Evolution strat- egy, since the alternating phase structure presupposes the existence of a two-level hierarchy. The performance drop of 14.5% on ALFWorld and 18.0% on WebShop confirms that the two-level group construction provides strictly more precise credit assignment than a flat rollout group. Without this separation, the Macro- Policy receives a noisy learning signal contaminated by execution stochasticity, slowing convergence and degrading plan quality. Table 3: Ablation study on ALFWorld and Webshop (Qwen2.5-7B-Instruct). VariantALFWorld (%) WebShop Score WebShop Succ.(%) HiMAC (Full)92.193.884.1 w/o hierarchy (Flat GRPO)77.679.366.1 w/o Iterative Co-Evolution85.386.774.8 w/o <sub_done>88.290.179.8 Random Blueprint 89.791.681.5 12F. Author et al. w/o Iterative Co-Evolution. Here we abandon the Iterative Co-Evolution strat- egy and instead update both the Macro-Policy and Micro-Policy simultane- ously within each training step. As discussed in Section 3.4, this introduces non-stationary dynamics: the planner optimizes sub-goals for an executor whose capabilities are shifting within the same gradient step, and vice versa. The re- sulting 6.8% degradation on ALFWorld validates our alternating optimization motivation—by freezing one level while updating the other, HiMAC converts the unstable bi-level problem into a sequence of stationary single-level problems. The larger relative drop on WebShop (9.3%) suggests that non-stationarity is particularly harmful in environments with longer horizons and noisier reward signals, where gradient interference accumulates more severely. w/o <sub_done> token. Rather than allowing the Micro-Policy to autonomously signal sub-goal completion via the <sub_done> termination token, this variant switches to a fixed step budget: each sub-goal is allocated a pre-determined number of steps T_fixed = T_limit, after which the system advances to the next sub-goal regardless of execution state. This eliminates the model’s ability to self-regulate pacing based on task difficulty. The resulting 3.9%/4.3% drop on ALFWorld/Webshop reveals that adaptive termination is a critical capabil- ity: some sub-goals (e.g., locating an object in a cluttered environment) require variable step counts that a fixed budget cannot accommodate. When the bud- get is too short, the executor is interrupted mid-task; when too long, it wastes steps after completion and risks drifting into erroneous actions. The learnable <sub_done> mechanism enables the Micro-Policy to develop a reliable sense of task completion, contributing meaningfully to the overall system’s robustness. Table 4: Comprehensive sample efficiency analysis across benchmarks (7B backbone). We report both the convergence speed (approximate iterations required to reach a target success threshold) and the final converged success rate. Benchmark (Target) Iters to Target ↓Final Succ. (%) ↑ HiMAC GiGPO GRPO HiMAC GiGPO GRPO ALFWorld (75%) ∼110 ∼115 ∼150 92.190.2 77.6 WebShop (65%) ∼220 ∼230 ∼380 84.172.8 65.7 Sokoban (80%)∼180 ∼195 ∼210 87.582.8 83.9 Random Blueprint. In the full HiMAC system, Phase B conditions the Micro- Policy on a high-confidence blueprint z ∗ = argmax i R(z (i) ), i.e., the blueprint from the sampled group that achieved the highest trajectory return. This design ensures that the executor is trained on plans that are both coherent and practi- cally achievable under the current planner’s capability. In this variant, we replace this selection with a randomly sampled blueprint from the same group, regard- less of its return. The performance drops by 2.4% on ALFWorld and 2.6% on WebShop, demonstrating that the quality of the conditioning blueprint during Abbreviated paper title13 Micro-training has a direct and significant impact on execution learning. When the executor is trained on low-quality or incoherent blueprints, it acquires execu- tion patterns conditioned on unreliable semantic intents, introducing noise into the Micro-Policy’s gradient signal and slowing convergence. The high-confidence selection thus serves as an implicit quality filter, ensuring that the executor’s "bootcamp" is grounded in feasible plans and avoiding the waste of rollout bud- get on unexecutable decompositions. 4.4 Analysis Sample Efficiency. Table 4 compares the sample efficiency of HiMAC against existing methods by measuring performance achieved under limited training iter- ations. HiMAC consistently reaches strong performance levels with fewer training samples, demonstrating faster convergence than flat-policy baselines. This improvement mainly stems from the hierarchical design. By decompos- ing decision-making into blueprint generation and execution, HiMAC reduces the effective exploration space and enables more stable credit assignment through hierarchical group-relative advantages. In addition, the Iterative Co-Evolution strategy introduces a natural learning curriculum, allowing the agent to progres- sively learn executable plans instead of wasting samples on infeasible trajectories. These results indicate that HiMAC not only improves final performance but also utilizes interaction samples more efficiently, which is particularly important for long-horizon agent training. Fig. 3: Evolution of HiMAC’s planning and emergent be- haviors. For the task "put a candle in toilet": (Left) Early training shows blind exploration due to lacking spatial priors. (Right) Post-convergence, the planner efficiently navigates to the target and develops emergent self-verification. Qualitative Analysis of Learned Blueprints. To gain interpretable insight into HiMAC’s planning behavior, Fig- ure 3 visualizes rep- resentative blueprints generated by the Macro- Policy at early (itera- tion 19) and late (it- eration 239) training stages on the ALF- World task "put a candle in the toi- let". Early in train- ing, blueprints reflect a lack of spatial com- mon sense and rely on naive, open-loop ex- ploration (e.g., waste- fully searching a closed 14F. Author et al. cabinet for a can- dle). After conver- gence, the generated blueprints not only exhibit accurate spatial priors by directly locating the tar- get on the countertop, but also demonstrate a striking emergent self-verification mechanism. Specifically, the mature planner autonomously appends an obser- vation step (e.g., "Inventory or look to confirm") to validate task success. This critical transition—from blind execution to closed-loop, structurally complex planning—clearly visualizes the emergent co-evolution of planner complexity and executor proficiency. This qualitative evidence strongly supports our co- evolutionary curriculum hypothesis. Scalability with Model Size. Comparing the 1.5B and 7B variants of HiMAC reveals a consistent 2–3 percentage point improvement in overall ALFWorld success rate (89.9%→ 92.1%) and improvement in WebShop (83.4%→ 84.1%), suggesting that HiMAC’s hierarchical framework scales gracefully with backbone capacity and that the architectural gains are largely orthogonal to model scale. This property is practically significant, as it implies that HiMAC can deliver strong performance even in resource-constrained deployment scenarios. Emergent Curriculum. By alternating these two phases every iteration, HiMAC fosters an emergent curriculum without any explicit difficulty scheduling. Early in training, only simple blueprints yield positive returns under the weak executor, so the group-relative advantage in Eq. (2) naturally pushes the planner toward short-horizon decompositions. As executor proficiency improves, more complex blueprints begin to yield higher returns, incentivizing the planner to progres- sively explore richer strategies. This co-evolutionary dynamic enables the stable emergence of hierarchical cognitive structures across the full training horizon. 5 Conclusion In this paper, we presented HiMAC, a novel hierarchical reinforcement learning framework designed to enable Large Language Models to robustly handle long- horizon agentic tasks. By formulating the reasoning process as a bi-level opti- mization problem, HiMAC explicitly decouples strategic planning from tactical execution, effectively mitigating the exponential error accumulation inherent in traditional flat policies. To address the optimization challenges of such decoupled architectures, we introduced two key technical innovations: a Critic-Free Hierar- chical Group-Based Optimization objective and an Iterative Co-Evolution train- ing strategy. These mechanisms allow the agent to learn precise credit assignment without unstable value networks and ensure the synchronized improvement of both the planner and the executor. Extensive experiments on complex bench- marks, including ALFWorld, WebShop and Sokoban, demonstrate that HiMAC achieves state-of-the-art success rates with superior sample efficiency compared to strong baselines. Our findings suggest that imposing structural inductive bi- ases—specifically the Macro-Micro separation—is crucial for scaling long-horizon Abbreviated paper title15 agents towards real-world complexity. Future work will explore applying HiMAC to more open-ended environments and investigating the transferability of learned blueprints across different domains. 16F. Author et al. References 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2. Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S.: Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In: Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). p. 12248–12267 (2024) 3. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al.: Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022) 4. Bai, H., Zhou, Y., Pan, J., Cemri, M., Suhr, A., Levine, S., Kumar, A.: Digirl: Training in-the-wild device-control agents with autonomous reinforcement learn- ing. Advances in Neural Information Processing Systems 37, 12461–12495 (2024) 5. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 6. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025), https://arxiv.org/abs/2502.13923 7. Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learn- ing. Discrete event dynamic systems 13(4), 341–379 (2003) 8. Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.: Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018) 9. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv preprint arXiv:1606.01540 (2016) 10. Chang, M., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., He, J.: Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems 37, 74325–74362 (2024) 11. Chen, D., Zhang, Q., Zhu, Y.: Efficient sequential decision making with large lan- guage models. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. p. 9157–9170 (2024) 12. Chen, Y., Pesaranghader, A., Sadhu, T., Yi, D.H.: Can we rely on llm agents to draft long-horizon plans? let’s take travelplanner as an example. arXiv preprint arXiv:2408.06318 (2024) 13. Dayan, P., Hinton, G.E.: Feudal reinforcement learning. Advances in neural infor- mation processing systems 5 (1992) 14. Erdogan, L.E., Lee, N., Kim, S., Moon, S., Furuta, H., Anumanchipalli, G., Keutzer, K., Gholami, A.: Plan-and-act: Improving planning of agents for long- horizon tasks. arXiv preprint arXiv:2503.09572 (2025) 15. Feng, L., Xue, Z., Liu, T., An, B.: Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978 (2025) 16. Gao, H., Sun, Z., Min, E., Cai, H., Wang, S., Yin, D., Chen, X.: Solving the granularity mismatch: Hierarchical preference learning for long-horizon llm agents. arXiv preprint arXiv:2510.03253 (2025) Abbreviated paper title17 17. Goyal, A., Bengio, Y.: Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A 478(2266), 20210068 (2022) 18. Hu, M., Chen, T., Chen, Q., Mu, Y., Shao, W., Luo, P.: Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large lan- guage model. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). p. 32779–32798 (2025) 19. Hu, Z., Liu, W., Qu, X., Yue, X., Chen, C., Wang, Z., Cheng, Y.: Divide and conquer: Grounding llms as efficient decision-making agents via offline hierarchical reinforcement learning. arXiv preprint arXiv:2505.19761 (2025) 20. Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., Chen, E.: Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716 (2024) 21. Jin, H., Lin, K., Zhang, W., Jin, Y., Li, G.: Videocurl: Video curriculum re- inforcement learning with orthogonal difficulty decomposition. arXiv preprint arXiv:2601.00887 (2025) 22. Jin, H., Wang, Q., Zhang, W., Liu, Y., Cheng, S.: Videomem: Enhancing ultra- long video understanding via adaptive memory management. arXiv preprint arXiv:2512.04540 (2025) 23. Jin, H., Xie, S., Ding, J., Lin, K., Li, G.: Tir-flow: Active video search and reasoning with frozen vlms. arXiv preprint arXiv:2601.06176 (2026) 24. Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial intelligence 101(1-2), 99–134 (1998) 25. Kulkarni, T.D., Narasimhan, K., Saeedi, A., Tenenbaum, J.: Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems 29 (2016) 26. Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.A., Akyürek, E., Anandkumar, A., et al.: Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems 35, 31199– 31212 (2022) 27. Li, Z., Chang, Y., Yu, G., Le, X.: Hiplan: Hierarchical planning for llm-based agents with adaptive global-local guidance. arXiv preprint arXiv:2508.19076 (2025) 28. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 29. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys 55(9), 1–35 (2023) 30. Liu, X., Feng, J., Zhuang, Z., Zhao, J., Que, M., Li, J., Wang, D., Tong, H., Chen, Y., Li, P.: Coda: A context-decoupled hierarchical agent with reinforcement learning. arXiv preprint arXiv:2512.12716 (2025) 31. Lu, W., Wei, Y., Xu, J., Jia, W., Li, L., Xiong, R., Wang, Y.: Reinforcement learning for adaptive planner parameter tuning: A perspective on hierarchical ar- chitecture. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). p. 3883–3889. IEEE (2025) 32. Nachum, O., Gu, S.S., Lee, H., Levine, S.: Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems 31 (2018) 33. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing sys- tems 35, 27730–27744 (2022) 34. Parr, R., Russell, S.: Reinforcement learning with hierarchies of machines. Ad- vances in neural information processing systems 10 (1997) 18F. Author et al. 35. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., Qiu, Z.: Qwen2.5 technical report (2025), https://arxiv.org/abs/2412.15115 36. Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems 36, 53728–53741 (2023) 37. Raileanu, R., Fergus, R.: Decoupling value and policy for generalization in rein- forcement learning. In: International Conference on Machine Learning. p. 8787– 8798. PMLR (2021) 38. Rawles, C., Li, A., Rodriguez, D., Riva, O., Lillicrap, T.: Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems 36, 59708–59728 (2023) 39. Sang, J., Xiao, J., Han, J., Chen, J., Chen, X., Wei, S., Sun, Y., Wang, Y.: Beyond pipelines: A survey of the paradigm shift toward model-native agentic ai. arXiv preprint arXiv:2510.16720 (2025) 40. Sapkota, R., Roumeliotis, K.I., Karkee, M.: Ai agents vs. agentic ai: A conceptual taxonomy, applications and challenges. Information Fusion p. 103599 (2025) 41. Schaul, T., Horgan, D., Gregor, K., Silver, D.: Universal value function approxi- mators. In: International conference on machine learning. p. 1312–1320. PMLR (2015) 42. Schrader, M.P.B.: gym-sokoban. https://github.com/mpSchrader/gym-sokoban (2018) 43. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017) 44. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024) 45. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in neural information processing systems 36, 8634–8652 (2023) 46. Shridhar, M., Yuan, X., Côté, M.A., Bisk, Y., Trischler, A., Hausknecht, M.: Alf- world: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768 (2020) 47. Singh, U., Chakraborty, S., Suttle, W.A., Sadler, B.M., Namboodiri, V.P., Bedi, A.S.: Dipper: Direct preference optimization to accelerate primitive-enabled hier- archical reinforcement learning. arXiv preprint arXiv:2406.10892 (2024) 48. Sutton, R.S., Barto, A.G., et al.: Reinforcement learning: An introduction, vol. 1. MIT press Cambridge (1998) 49. Sutton, R.S., Precup, D., Singh, S.: Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112(1-2), 181–211 (1999) 50. Tan, W., Zhang, W., Liu, S., Zheng, L., Wang, X., An, B.: True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learn- ing. arXiv preprint arXiv:2401.14151 (2024) 51. Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) Abbreviated paper title19 52. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., Anand- kumar, A.: Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023) 53. Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al.: A survey on large language model based autonomous agents. Frontiers of Computer Science 18(6), 186345 (2024) 54. Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M.N., Liu, L., et al.: Ragen: Understanding self-evolution in llm agents via multi- turn reinforcement learning. arXiv preprint arXiv:2504.20073 (2025) 55. Wen, M., Wan, Z., Wang, J., Zhang, W., Wen, Y.: Reinforcing llm agents via policy optimization with action decomposition. Advances in Neural Information Processing Systems 37, 103774–103805 (2024) 56. Xi, Z., Huang, J., Liao, C., Huang, B., Guo, H., Liu, J., Zheng, R., Ye, J., Zhang, J., Chen, W., et al.: Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXiv:2509.08755 (2025) 57. Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., Su, Y.: Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622 (2024) 58. Yao, S., Chen, H., Yang, J., Narasimhan, K.: Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems 35, 20744–20757 (2022) 59. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems 36, 11809–11822 (2023) 60. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: React: Synergizing reasoning and acting in language models. In: The eleventh international conference on learning representations (2022) 61. Zhai, S., Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al.: Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems 37, 110935–110971 (2024) 62. Zhang, K., Li, J., Li, G., Shi, X., Jin, Z.: Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). p. 13643–13658 (2024) 63. Zhang, N., Zhao, Y., Yang, M., Dai, S.: Llms augmented hierarchical reinforcement learning with action primitives for long-horizon manipulation tasks. Scientific Re- ports 15(1), 36779 (2025) 64. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 1(2), 1–124 (2023) 65. Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al.: Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144 (2023) 66. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. p. 2165–2183. PMLR (2023)