Paper deep dive
Can RL Improve Generalization of LLM Agents? An Empirical Study
Zhiheng Xi, Xin Guo, Jiaqi Liu, Jiazheng Zhang, Yutao Fan, Zhihao Zhang, Shichun Liu, Mingxu Chai, Xiaowei Shi, Yitao Zhai, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 6:30:18 AM
Summary
This paper presents a systematic empirical study on the generalization capabilities of Reinforcement Fine-Tuning (RFT) for LLM agents. The authors evaluate RFT across three axes: intra-environment task difficulty, cross-environment transfer, and sequential multi-environment training. Findings indicate that while RFT excels at intra-environment generalization and benefits from curriculum learning, it faces challenges with cross-environment transfer due to shifts in semantic priors and action/observation interfaces. The study concludes that sequential and mixture training strategies can mitigate these issues and improve overall agentic robustness.
Entities (8)
Relation Signals (3)
RFT → evaluatedon → WebShop
confidence 100% · We select five representative agent environments, including WebShop
Sequential Training → enhances → Transferability
confidence 95% · sequential RFT enables effective transfer to new environments while maintaining performance on upstream ones
Reinforcement Fine-Tuning → improves → Generalization
confidence 90% · RFT generalizes well across task difficulty within an environment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement fine-tuning (RFT) has shown promise for training LLM agents to perform multi-turn decision-making based on environment feedback. However, most existing evaluations remain largely in-domain: training and testing are conducted in the same environment or even on the same tasks. In real-world deployment, agents may operate in unseen environments with different background knowledge, observation spaces, and action interfaces. To characterize the generalization profile of RFT under such shifts, we conduct a systematic study along three axes: (1) within-environment generalization across task difficulty, (2) cross-environment transfer to unseen environments, and (3) sequential multi-environment training to quantify transfer and forgetting. Our results show that RFT generalizes well across task difficulty within an environment, but exhibits weaker transfer to unseen environments, which correlates with shifts in both semantic priors and observation/action interfaces. In contrast, sequential training yields promising downstream gains with minimal upstream forgetting, and mixture training across environments improves the overall balance. We further provide detailed analyses and deeper insights, and hope our work helps the community develop and deploy generalizable LLM agents.
Tags
Links
- Source: https://arxiv.org/abs/2603.12011v1
- Canonical: https://arxiv.org/abs/2603.12011v1
Trouble viewing inline? Open PDF directly →
Full Text
78,689 characters extracted from source content.
Expand or collapse full text
FudanNLPLab 2026-3-13 Can RL Improve Generalization of LLM Agents? An Empirical Study Zhiheng Xi 1∗† , Xin Guo 1∗ , Jiaqi Liu 1 , Jiazheng Zhang 1 , Yutao Fan 3 , Zhihao Zhang 1 , Shichun Liu 1 , Mingxu Chai 1 , Xiaowei Shi 2 , Yitao Zhai 2 , Xunliang Cai 2 , Tao Gui 1† , Qi Zhang 1† , Xuanjing Huang 1† 1 Fudan University 2 Meituan 3 Shanghai Artificial Intelligence Laboratory zhxi22@m.fudan.edu.cn, tgui,qz,xjhuang@fudan.edu.cn Reinforcement fine-tuning (RFT) has shown promise for training LLM agents to perform multi- turn decision-making based on environment feedback. However, most existing evaluations remain largely in-domain—training and testing are conducted in the same environment or even on the same tasks. In real-worlddeployment, agents may operate in unseen environments with differ- ent background knowledge, observation spaces, and action interfaces. To characterize the gen- eralization profile of RFT under such shifts, we conduct a systematic study along three axes: (1) within-environment generalization across task difficulty, (2) cross-environment transfer to unseen environments, and (3) sequential multi-environment training to quantify transfer and forgetting. Our results show that RFT generalizes well across task difficulty within an environment, but ex- hibits weaker transfer to unseen environments, which correlates with shifts in both semantic priors and observation/action interfaces. In contrast, sequential training yields promising downstream gains with minimal upstream forgetting, and mixture training across environments improves the overall balance. We further provide detailed analyses and deeper insights, and hope our work helps the community develop and deploy generalizable LLM agents. 1. Introduction Reinforcement fine-tuning (RFT) has emerged as a promising post-training paradigm for improving large language model (LLM) agents on complex interactive tasks such as web navigation and software engineering (Deng et al., 2023; He et al., 2024; Jimenez et al., 2023; Merrill et al., 2026; Zan et al., 2025; Zhou et al., 2023). In RFT, an agent is trained to make a sequence of intelligent decisions that maximize task-specific objectives based on environment feedback (Mai et al., 2025; Xi et al., 2025c). By optimizing for long-horizon outcomes, RFT is often associated with stronger agentic behaviors such as instruction following (Bai et al., 2022b; Ouyang et al., 2022a), planning (Shinn et al., 2023; Yao et al., 2022b), reasoning (Lightman et al., 2023; Uesato et al., 2022), and tool use (Li et al., 2023, 2025b). Despite rapid progress, most empirical evidence for RFT focuses on in-domain evaluation, where training and testing are conducted within the same environment—and often even on closely overlap- ping tasks (Liu et al., 2023; Zhou et al., 2023). In real-world deployment, however, agents frequently encounter unseen environments that differ not only in task instances, but also in background knowl- edge, observation spaces, and action interfaces (Ruan et al., 2023; Xi et al., 2025a). Such shifts can fundamentally reshape the interaction dynamics faced by agents, including which observations are informative, what actions are feasible, and how failures can be corrected. This raises a practical and * Equal contribution. † Corresponding authors. arXiv:2603.12011v1 [cs.AI] 12 Mar 2026 Can RL Improve Generalization of LLM Agents? An Empirical Study important research question: do the improvements brought by RFT generalize beyond the training distribution? Axis 1: Intra-Environment Generalization Axis 2: Inter-Environment Generalization Axis 3: Multi-Environment Training Train:Easy Task Test: Hard Task Train: Seen Env Test: Unseen Env Env2Env4 Sequential Reinforcement Fine-tuningMix Reinforcement Fine-tuning Env1 AlfWorld WebShop Env3 표 푎 표 푎 Figure 1 | An overview of three axes we study. Table 1| Characteristics of each environment we study. The columns “DEN.”, “VAL.”, “ACT.”, “KNWL.” and “STR.” indicate whether the rewards are dense, action validation is strict, valid action lists are provided per step, world knowledge is required, and observations are structured, respectively. The environments vary along these dimensions. Environment Types DEN. VAL. ACT. KNWL. STR. WebShopWeb✔✗✔ SearchQASearch✗✔✗✔ TextCraftGame✗✔✗ Alf WorldHousehold✗✔✗ BabyAIEmbodied✔✗✔✗ To bridge this gap, we conduct a sys- tematic study to explore how RFT im- pacts the generalization and transferabil- ity of LLM agents across three axes (Fig- ure 1). First, we investigate task-level generalization under the same environ- ment by reviewing varying difficulty per- formance when RFT on a specific subset of tasks (Section 4); Second, we examine environment-level generalization to assess whether RFT maintains efficacy in un- seen environments, where agents must navigate shifts in background knowledge alongside changes in observation and ac- tion spaces (Section 5). Third, we analyze sequential training across environments to characterize the dynamics of trans- fer and forgetting, comparing this ap- proach against joint training on environ- mental mixtures (Section 6). Together, these three axes provide a comprehensive, systematic framework for understanding RFT-driven generalization and transfer in LLM agents. Our analysis covers five agentic en- vironments that vary in background and core properties, as shown in Table 1 and Table 4. We highlight several key findings: (1) Intra-environment gener- alization, within a consistent environ- ment, RFT-trained agents exhibit signifi- cant generalization, and easy-to-hard cur- riculum learning further boosts perfor- mance gains. (2) Inter-environment sen- sitivity, while RFT enhances agentic capa- bilities, its generality to unseen environ- ments exhibits fluctuations. Our analysis reveals that this across-environment generalization is sen- sitive to the required prior knowledge, observation spaces, and action spaces. (3) Multi-environment training. Employing multi-environment training strategies substantially enhances generalization: se- quential RFT enables effective transfer to new environments while maintaining performance on up- stream ones, and mix RFT across multiple environments achieves consistently strong results overall. We further provide insights that shed light on the mechanisms underlying these behaviors. Overall, we hope these findings inform the development of more generalizable LLM agents and their practical deployment in real-world settings. 2 Can RL Improve Generalization of LLM Agents? An Empirical Study 2. Related Work 2.1. RL for LLM Agents Reinforcement learning (RL) has been widely adopted in LLM training, with paradigms such as reinforcement learning from human feedback (RLHF) (Bai et al., 2022a; Christiano et al., 2017; Ouyang et al., 2022b) and reinforcement learning from verifiable rewards (RLVR) (DeepSeek-AI, 2025; Rafailov et al., 2023; Schulman et al., 2017) demonstrating strong effectiveness in improving instruction following, reasoning accuracy, and behavioral alignment (Guo et al., 2025; Mu et al., 2024; Ren et al., 2025). Building upon these advances, recent work has extended reinforcement learning to LLM agents to enhance multi-step decision making (Wang et al., 2025b; Zhai et al., 2024, 2025), long-horizon planning (Song et al., 2024a; Wang et al., 2025a; Xi et al., 2025c), and interaction with external environments (Chen et al., 2025b; Feng et al., 2024; Tan et al., 2024). Across diverse agent settings, RL has been shown to strengthen agents’ abilities to decompose com- plex tasks, coordinate reasoning with action , and adapt policies through environment feedback (Tian et al., 2024), enabling more effective information gathering (Jin et al., 2025; Ramrakhya et al., 2025; Zhang et al., 2025b), iterative self-correction (Kumar et al., 2025; Ma et al., 2025; Zeng et al., 2025b), and tool utilization (Feng et al., 2025; Li et al., 2025a; Zeng et al., 2025a). Despite these successes, most RL-based agent methods are evaluated primarily under in-domain settings with similar task distributions and interfaces, leaving open the question of how well such learned decision-making policies generalize across tasks or transfer to unseen environments. 2.2. Generalization and Forgetting by Post-training Recent work has examined how post-training strategies affect the generalization and forgetting be- haviors of LLMs (Tu et al., 2025; Zhang et al., 2025c; Zhao et al., 2024). Supervised fine-tuning (SFT) is effective for improving in-distribution performance but often leads to over-specialization and degradation of general capabilities due to representation and distributional drift (Kotha et al., 2024; Kumar et al., 2022; Luo et al., 2023; Wang et al., 2024). In contrast, RL, particularly RLVR, tends to better preserve pre-trained representations by optimizing trajectory-level objectives rather than directly fitting target distributions, enabling more robust transfer across tasks and domains (Chen et al., 2025a; Cheng et al., 2025; Chu et al., 2025; Huan et al., 2025). However, RL can also induce negative interference and winner-take-all dynamics, reducing behavioral coverage and leading to reasoning boundary shrinkage (Hu et al., 2025; Nguyen et al., 2025; Sun et al., 2025). Notably, most existing studies focus on static and single-turn LLM tasks, whereas we extend the study of post-training generalization and forgetting to multi-turn LLM agents and systematically evaluate these effects across different environments with distinct observation and action spaces. 3. Preliminaries 3.1. Preliminaries Task Formulation. We follow the ReAct interaction paradigm (Yao et al., 2022b) to formulate a multi-turn decision-making task. The task can be represented by the tuple (U,S,A,O,T,R). Here, U denotes the instruction space, S is the state space, A is the action space, and O represents the observation space. The functionT :S×A →S denotes the deterministic state transition function, andR :U×S → ℝ is the reward function. Given a task instruction 푢 ∈ U, the agent operates based on a policy 휋 휃 (an LLM parameterized by 휃). In each interaction step 푡, the state 푠 푡 consists all previous dialogues and their resulting 3 Can RL Improve Generalization of LLM Agents? An Empirical Study observations, i.e., 푠 푡 = (푎 0 , 표 0 , . . . , 푎 푡−1 , 표 푡−1 ). The agent generates an action 푎 푡 ∼ 휋 휃 (· | 푢, 푠 푡 ). The action 푎 푡 includes an internal reasoning trace and a interaction to the environment. The agent then receives an observation 표 푘 ∈ O from the environment. This interaction loop continues until the tesk is completed or the maximum number of turns is reached, resulting in a complete trajectory: 휏=푢,(푎 0 , 표 0 ),(푎 1 , 표 1 ), . . . ,(푎 푇 , 표 푇 )(1) The environment provides a terminal rewardR(휏) ∈ [0, 1] upon task completion. Policy Gradient. In RL, our objective is to optimize the policy parameters 휃 to maximize the expected cumulative reward over all possible trajectories for the given task (Sutton and Barto, 1998): max 휃 퐽(휃)= 피 휏∼휋 휃 [R(휏)](2) To optimize the objective function 퐽(휃), we utilize policy gradient methods (Sutton et al., 1999). Unlike value-based methods that approximate a value function, policy gradient methods directly search the policy parameter space. The core idea is to perform gradient ascent to update 휃, increasing the probability of trajectories that yield high rewards. The vanilla policy gradient is formulated as: ∇ 휃 퐽(휃)= 피 휏∼휋 휃 " R(휏) 푇 ∑︁ 푡=0 ∇ 휃 log휋 휃 (푎 푡 |푠 푡 ) # (3) Based on this gradient estimation, the parameters are updated via 휃 푛푒푤 = 휃 표푙푑 + 훼∇ 휃 퐽(휃), where 훼 is the learning rate. However, standard policy gradient methods often suffer from high variance, leading to unstable training dynamics. To address this challenge, we adopt the widely-used GRPO algorithm (DeepSeek-AI, 2025), detailed in Appendix C. 3.2. Experimental Settings Environment Setup. We select five representative agent environments, including WebShop (Yao et al., 2022a), SearchQA (Dunn et al., 2017), TextCraft (Sanghi et al., 2022), Alf World (Shridhar et al., 2020), and BabyAI (Chevalier-Boisvert et al., 2018), with characteristics of each environ- ment summarized in Table 1. Among them, WebShop is an interactive web shopping environment; SearchQA is a Q&A environment augmented with context from a search engine; ALFWorld requires agents to explore rooms and execute tasks in a household setting; BabyAI is an interactive grid world simulator; and TextCraft is a game environment for crafting items in Minecraft. The task data U is sourced from AgentGym (Xi et al., 2025b), with detailed statistics and action spaces for each environment provided in Appendix B. Training Setup. We perform RFT training using the AgentGym-RL framework 1 (Xi et al., 2025c) with Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct (Team, 2024). Following the ReAct (Yao et al., 2022b) paradigm, each action receives real-time environment feedback. For all experiments, we sample 8 trajectories per task and set the maximum response length to 8192 tokens. Following Xi et al. (2025b), we set the maximum interaction turns 퐾 for each environment as follows: 10 for WebShop, 5 for SearchQA, 30 for Alf World, 10 for BabyAI, and 15 for TextCraft. 1 https://github.com/woooodyy/AgentGym-RL 4 Can RL Improve Generalization of LLM Agents? An Empirical Study Table 2 | Results of generalization across task difficulties within the same environment. Models WebShopSearchQATextCraftAlf WorldBabyAI easy hardall easy hardall easy hardall easy hardall easy hardall Qwen2.5-3B-Instruct base model21.7 10.615.3 63.7 6.523.7 23.7 2.314.5 26.8 8.013.2 71.5 48.061.6 train w/U easy 90.3 75.381.6 82.7 16.936.6 97.6 49.476.9 93.2 82.085.1 93.4 79.087.3 train w/U hard 86.1 84.585.2 72.5 19.435.3 95.0 42.272.3 96.1 92.993.8 92.3 77.386.0 train w/U92.8 84.3 87.9 87.6 22.141.8 93.9 47.473.9 97.0 89.891.8 93.2 77.586.6 U easy +U hard 93.6 85.288.7 82.1 19.838.5 93.9 52.976.3 97.3 93.494.4 94.5 82.289.3 U hard +U easy 90.2 82.485.7 82.0 18.737.7 97.8 40.173.0 95.5 92.393.6 94.5 80.488.6 Qwen2.5-7B-Instruct base model44.1 17.428.6 79.6 10.431.2 46.9 16.033.6 40.2 21.426.6 80.1 47.967.0 train w/U easy 88.1 77.582.0 85.8 21.140.5 98.2 47.476.4 95.2 89.791.2 95.2 76.187.2 train w/U hard 87.9 81.484.2 93.3 27.046.9 99.3 57.381.3 97.5 94.695.4 92.8 77.986.5 train w/U92.9 81.9 86.5 91.8 26.646.1 95.2 52.380.9 97.7 97.292.0 95.6 74.588.8 U easy +U hard 90.6 77.583.0 89.8 25.544.8 98.7 64.083.8 96.1 92.693.6 95.5 82.990.2 U hard +U easy 89.1 79.183.3 87.6 21.542.3 99.8 57.081.4 97.7 88.591.1 92.5 78.286.5 Evaluation Setup. As summarized in Table 1, WebShop and BabyAI provide dense rewards, while SearchQA, Alf World, and TextCraft use binary rewards. We follow DeepSeek-AI (2025) and adopt exact matching, counting an action as correct only if it exactly matches the reference. To reduce randomness, we report avg@8 as the main metric, and also measure the average interaction turns ( ̄ 푘) and generated tokens ( ̄ 푙) for efficiency. For fair comparison, all agents are evaluated with the maximum number of interaction turns set to 퐾= 20 during testing. 4. Does RFT Yield Generalization Across Task Difficulties within the Same En- vironment? Reinforcement fine-tuning trains LLM agents through interaction with the environment, allowing them to learn its dynamics and adapt over time (Cui et al., 2025; Song et al., 2024b). Yet even with a fixed action and observation space, tasks within the same environment can differ substantially in exploration depth and information accessibility. In this section, we investigate whether RFT policies learned on a subset of tasks transfer to other tasks of differing difficulty within the same environment. Setting. Following practice in previous work (Bengio et al., 2009; Mukherjee et al., 2023), we categorize the tasks U into easy and hard difficulty levels (denoted as U easy and U hard ) based on the avg@8 results of the Qwen2.5-7B-Instruct model, while ensuring a balanced distribution of data between the two difficulty levels. The same categorization is applied to the test set. Detailed data statistics for each environment are provided in Appendix B. RFT demonstrates strong tranferability across varying difficulty levels within the same envi- ronment. As shown in Table 2, RFT demonstrates strong robustness when trained on data of varying difficulty levels within the same environment. Notably, with 7B model on WebShop, training onU easy improves performance on the hard testset by 60.1 points, suggesting that RFT encourages agents to adapt to the environment, thereby enabling performance transfer across tasks with different diffi- culty levels and varying steps (Chen et al., 2025a; Huan et al., 2025). Overall, we also find that training on U hard yields more gains on test set, e.g., for 3B model in Alf World, training on U hard outperforms training onU easy by 8.7 points. We attribute this trend to the richer failure signals and 5 Can RL Improve Generalization of LLM Agents? An Empirical Study longer-horizon exploration induced by harder tasks, which push the RFT optimizer to seek better policies (Wang et al., 2023; Xu et al., 2023). Curriculum learning can further enhance performance. Additionally, we investigate the impact of the training sequence using data of varying difficulty levels. Results in Table 2 indicate that training on U (i.e., mixture of U easy and U hard ) generally does not yield optimal performance. In most cases, training on U easy first, followed by U hard , achieves the best results. Compared with using a single difficulty level, this easy-to-hard curriculum enables further performance improvement. For example, on BabyAI, training onU easy +U hard outperforms training onU easy and onU hard by 2.0 and 3.3 points, respectively. This result validates the rationale and effectiveness of curriculum learning (Qi et al., 2024; Zhang et al., 2025a). Base U easy U hard U 0 500 1000 Avg Generated Tokens Base U easy U hard U 0 5 10 15 Avg Turn Figure 2 | Average generated tokens and interac- tive turn of all environments for Qwen2.5-3B-Instruct model trained with varying-difficulty tasks. RFT training encourages agents to per- form efficient exploration. Figure 2 re- ports the average number of interaction turns and generated tokens across differ- ent environments, with a more detailed breakdown provided in Figure 8 of Ap- pendix E. We find that RFT enables agents to interact and explore more efficiently within the same environment. For exam- ple, when training the 3B model onU easy in BabyAI, the average interaction turns decrease from 10.76 to 4.19, and the av- erage trajectory length is reduced from 624.58 to 160.60 tokens. These results indicate that RFT not only improves success rates but also encourages more concise and goal-directed exploration, sub- stantially enhancing interaction efficiency (Nakano et al., 2021; Song et al., 2024b). A case can be found in Appendix G. 5. Does RFT Yield Generalization Across Different Environments? In real-world scenarios, agents may encounter previously unseen environments and tasks, which is essential for practical deployment. We therefore investigate whether RFT improves performance in unseen environments with different background knowledge, observation space and action spaces. Con- cretely, we perform RFT training within a single environment and then evaluate the trained agent on other environments to measure cross-environment generalization. Setting. We employ the base model (i.e., Qwen2.5-3B-Instruct or Qwen2.5-7B-Instruct) as the base- line and evaluate its avg@8 metric across varying environments. To make these comparisons ex- plicit, we define three metrics:ΔHeld-In,ΔHeld-Out, andΔOverall.ΔHeld-In measures the average improvement over the baseline when the training and test environments coincide.ΔHeld-Out mea- sures the average improvement when the agent is evaluated on environments different from those seen during training. Finally,ΔOverall reports the average improvement over the baseline across all evaluation settings. Performance differs significantly between held-in and held-out environments. Table 3 reveals that agents exhibit generalization across different environments, yet a significant performance gap exists between held-in and held-out settings. Specifically, substantial gains are achieved under held- 6 Can RL Improve Generalization of LLM Agents? An Empirical Study Table 3 | Results of generalization across different environments. Darker red indicates greater per- formance improvement, while darker blue signifies more pronounced performance decline. The best result is in bold, while the second-best is marked with underline. ModelsWebShop SearchQA TextCraft Alf World BabyAI ΔHeld-In ΔHeld-Out ΔOverall Qwen2.5-3B-Instruct base model15.3023.6614.5013.1961.83 − train w/ WebShop87.8625.8418.5023.0670.91 +72.56+6.29 +19.54 train w/ SearchQA22.9741.7822.7512.1365.72 +18.13 +4.69 + 4.56 train w/ TextCraft14.4622.1673.8811.0063.73 +59.38 −0.66 +11.35 train w/ Alf World21.8623.5024.8891.8164.68 +78.62 +4.91+19.65 train w/ BabyAI28.3622.6316.754.5086.55 +24.72 +1.40 + 6.06 Qwen2.5-7B-Instruct base model28.5931.1933.6326.5667.00 − train w/ WebShop86.5033.2840.7524.1379.21 +57.91+4.75 +15.38 train w/ SearchQA47.0746.1235.2516.7580.33 +14.93 +5.91+ 7.71 train w/ TextCraft38.3032.1980.8831.5077.95 +47.25 +6.65 +14.77 train w/ Alf World 34.3129.5936.1392.0072.91 +65.44 +3.13 +15.59 train w/ BabyAI10.2529.4139.2528.1388.79 +21.79 −3.23 + 1.77 in conditions. In Alf World, performance improves by 78.62 and 65.44 points for the 3B and 7B models, respectively. In contrast, in held-out conditions, generalization remains possible in most cases, but with more modest gains. On average, the 3B and 7B models yield improvements of 3.32 and 3.44 points on unseen environments, respectively. In unseen environments, positive transfer is observed in most cases. Compared to the base- line, most trained agents can demonstrate transferability to unseen environments, even when these environments demand different background knowledge and operate under different action spaces. Notably, agents trained on WebShop, Alf World, and SearchQA consistently exhibit positiveΔheld- out. When evaluated on WebShop, agents trained on SearchQA yield performance gains of 7.67 and 18.48 points for the 3B and 7B models, respectively. This can be attributed to the similarity between WebShop and SearchQA as search-based environments. Specifically, the agent trained on SearchQA learns to formulate more flexible search queries, as well as efficient information extraction from com- plex results. An illustrative case is presented in Figure 10 in Appendix G and discussed in Section 7. In unseen environments, negative effects may sometimes occur. However, agents trained on TextCraft and BabyAI may struggle to generalize to other environments, e.g., after training on BabyAI, the 7B models show average negative improvements of −3.23 on held-out tasks, and even drop sharply from 28.59 to 10.25 on WebShop. Through careful analysis, we find that because the BabyAI environment provides available actions at each step, the agent gradually becomes depen- dent on this information during training, leading to a decline in long-horizon reasoning capability. When faced with other environments, it fails to accurately use the valid action, resulting in a sharp performance drop. A case of interaction between the agent and BabyAI environment is provided in Appendix G. Cross-environment generalization performance varies significantly across target environ- ments. Moreover, our observations reveal that cross-environment generalization performance is highly dependent on the target environment. On the one hand, agents trained on nearly all source 7 Can RL Improve Generalization of LLM Agents? An Empirical Study 0 50 100 train w/ WStrain w/ TC Eval on WS Eval on TC train w/ WStrain w/ AW Eval on WS Eval on AW train w/ WStrain w/ BA Eval on WS Eval on BA train w/ WStrain w/ SQ Eval on WS Eval on SQ 100200300400 0 50 100 train w/ TCtrain w/ WS Eval on TC Eval on WS 100200300400 train w/ TCtrain w/ AW Eval on TC Eval on AW 100200300400 train w/ TCtrain w/ BA Eval on TC Eval on BA 100200300400 train w/ TCtrain w/ SQ Eval on TC Eval on SQ Training Steps Performance (%) Figure 3 | Training dynamics of forgetting and transfer in sequential two-stage cross-environment training with Qwen2.5-7B-Instruct, where blue and red denote the upstream environment and the downstream environment, respectively. environments generalize effectively to TextCraft and BabyAI, as these domains rely less on specific background knowledge, thereby allowing acquired skills to transfer more readily. On the other hand, effective transfer to Alf World and SearchQA proves to be significantly more challenging. We attribute this to the strict action validation and sparse feedback inherent in these two environments. For in- stance, Alf World responds uniformly with “Nothing happens.” to all invalid actions, offering no instructional guidance for improvement. 6. How Does Sequential RFT Across Environments Affect Transfer and Forget- ting? Beyond zero-shot transfer to unseen environments without additional training, we further investigate the effect of sequential training across multiple environments on agentic behaviors, specifically the resulting learning dynamics on memorizing downstream tasks or forgetting on upstream ones. In our experimental setup, a model initially converged in one environment is further trained on a second environment, after which we assess both its retention of performance on the original environment and its adaptation to the new one. Finally, we extend this analysis to sequential training across five environments, comparing this strategy with joint training across a mixture of environments. Setting. We conduct 20 two-stage experiments by sequentially pairing each of the five environments as the upstream and the downstream, as well as several five-stage training experiments. Using agents trained on single-environment as baselines, we evaluate anti-forgetting by upstream performance retention and transferability by downstream performance gains. Furthermore, we compare the five- stage sequential approach with joint training, in which the agent is trained on all available data from every environment in a randomly mixed manner. Sequential training demonstrates consistent anti-forgetting and transferability. Figure 3 il- lustrates the performance dynamics of 8 two-stage sequential training scenarios, with others and the final results summarized in Figure 7 and Table 6 in Appendix D. Overall, sequential training matches or exceeds single-task performance on the downstream environment while largely preserv- ing upstream performance. For instance, when the WebShop-pre-trained agent is further trained on TextCraft, it boosts performance on TextCraft from the single-task baseline of 80.88 to 82.50. Mean- while, performance on WebShop experiences only a minor fluctuation, shifting from 86.5 to 86.32. 8 Can RL Improve Generalization of LLM Agents? An Empirical Study SQ+TC SQ+AW SQ+BA 49.31 39.42 44.36 WebShop (WS) WS+TC WS+AW WS+BA 36.81 34.22 33.97 SearchQA (SQ) WS+SQ WS+AW WS+BA 44.63 43.75 49.25 TextCraft (TC) WS+SQ WS+TC WS+BA 17.81 26.63 25.50 AlfWorld (AW) WS+SQ WS+TC WS+AW 78.88 81.68 76.34 BabyAI (BA) TC+SQ TC+AW TC+BA 39.17 42.04 23.00 TC+WS TC+AW TC+BA 34.59 31.56 33.37 SQ+WS SQ+AW SQ+BA 34.50 41.00 25.00 SQ+WS SQ+TC SQ+BA 18.44 26.06 14.06 SQ+WS SQ+TC SQ+AW 78.59 78.03 81.29 AW+SQ AW+TC AW+BA 53.04 50.31 25.68 AW+WS AW+TC AW+BA 31.50 30.81 28.31 AW+WS AW+SQ AW+BA 35.88 43.50 39.25 TC+WS TC+SQ TC+BA 18.31 19.69 22.81 TC+WS TC+SQ TC+AW 78.44 78.97 62.19 0204060 BA+SQ BA+TC BA+AW 33.71 17.95 13.60 203040 BA+WS BA+TC BA+AW 30.19 33.37 29.88 20304050 BA+WS BA+SQ BA+AW 36.38 40.63 41.25 10203040 BA+WS BA+SQ BA+TC 33.38 24.06 29.31 406080100 AW+WS AW+SQ AW+TC 49.65 77.86 77.01 Figure 4 | Generalization of sequential two-stage cross-environment training with Qwen2.5-7B- Instruct. Across five environments (WS, SQ, TC, AW, BA), the figure presents the generalization performance on the three unseen environments following sequential training on two environments. Each subplot corresponds to a fixed first environment, with dashed line indicating the baseline per- formance. These findings suggest that sequential training endows agents with stable capabilities for transfer and resistance to forgetting. Generalization in multi-environment training is highly correlated with single-environment training. Figure 4 demonstrates the generalization performance of multi-environment training, re- vealing a strong alignment with the generalization patterns in single-environment training. First, environments that yield poor generalization in single-environment settings continue to be detrimen- tal. For instance, the agent trained on BabyAI exhibits severe negative effect on WebShop. Even when BabyAI is trained sequentially after other generalizable environments (i.e., TC+BA, AW+BA), it drastically degrades their WebShop performance. Notably, the score of TC+BA dropped sharply from 38.30 (score of TC in Table 3) to 23.0. Moreover, from the perspective of target environments, the results also show consistency with single-environment generalization. For target environments easy to generalization in single- environment scenarios, such as TextCraft and BabyAI, sequentially trained agents also tend to per- form well. Specifically, agents pre-trained on WebShop (i.e., WS+SQ, WS+AW, WS+BA) achieve significant gains on TextCraft, increasing by 11.00, 10.12, and 15.62 points, respectively. However, for environments challenging for generalization, like Alf World, sequential training yields limited benefits. For instance, all three agents utilizing TextCraft as the upstream environment (TC+WS, TC+SQ, TC+BA) suffer performance degradation on Alf World. Training order significantly affects generalization performance in held-out environments. As shown in Figure 4, the training order exerts a substantial influence on generalization performance. For instance, when evaluated on TextCraft and Alf World, BA+SQ outperforms SQ+BA by 15.63 and 10.00 points, respectively. This represents a substantial margin, particularly for held-out scenarios. We attribute this phenomenon to the inherent task difficulty within each environment. As noted in Section 5, BabyAI provides detailed feedback, whereas SearchQA imposes strict constrains with 9 Can RL Improve Generalization of LLM Agents? An Empirical Study Base+BA+AW+WS+TC+SQ 60 80 100 Performance (%) Eval on BabyAI Base+BA+AW+WS+TC+SQ 0 50 100 Eval on AlfWorld Base+BA+AW+WS+TC+SQ 0 50 100 Eval on WebShop Base+BA+AW+WS+TC+SQ 0 50 100 Eval on TextCraft Base+BA+AW+WS+TC+SQ 20 30 40 50 Eval on SearchQA (a) Sequential training of BabyAI, Alf World, WebShop, TextCraft, and SearchQA. Base+SQ+WS+BA+AW+TC 20 30 40 50 Performance (%) Eval on SearchQA Base+SQ+WS+BA+AW+TC 0 50 100 Eval on WebShop Base+SQ+WS+BA+AW+TC 60 80 100 Eval on BabyAI Base+SQ+WS+BA+AW+TC 0 50 100 Eval on AlfWorld Base+SQ+WS+BA+AW+TC 0 50 100 Eval on TextCraft (b) Sequential training of SearchQA, WebShop, BabyAI, Alf World, and TextCraft. Figure 5 | Training dynamics of sequential training across five environments. We present the re- sults for representative sequence combinations, monitoring how performance on each environment changes as the agent is trained on different environments sequentially. The dashed lines denote the performance achieved by joint training on a mixture of data from all five environments. limited feedback. Consequently, the BA+SQ order naturally creates an “easy-to-hard” curriculum, which in turn facilitates better generalization performance. Sequential training achieves performance comparable to joint training. Furthermore, we con- duct sequential training over five environments in different orders. Figure 5 illustrates the per- formance dynamics on each environment across the five training stages for two representative se- quences; for comparison, joint-training results are indicated by dashed lines. The results show that sequential training achieves performance comparable to joint training, even after training on five distinct tasks. Overall, the final performance is insensitive to the training order, which we at- tribute to RFT’s ability to preserve previously acquired capabilities, consistent with findings in prior work. For environments such as Alf World and SearchQA—to which other tasks struggle to general- ize—relatively pronounced forgetting may occur over the course of long-term sequential training. 7. Further Analysis and Discussion Failure Mode Analysis. We conduct a fine-grained error analysis on both intra/inter-environment evaluations, summarizing 8 common failure modes including instruction misinterpretation, action execution failure, and logical deficits. Using GPT-5-mini, we systematically categorize the error tra- jectories across all scenarios. More detail about each failure pattern can be found in Appendix F. The results presented in Figure 6 reveal that while failure modes differ by environment, errors related to “Confirmation Bias” are prevalent (> 10%) across all scenarios. This suggests that after training, agents tend to exhibit overconfidence, neglecting further verification and lacking the capac- ity for self-reflection based on environmental feedback. Moreover, in SearchQA, errors categorized as “Guessing or Fabrication” are widespread among both held-in (23.8%) and held-out (21.2%) set- tings. This highlights a critical deficiency in tool utilization, which fundamentally constrains the agent’s generalization potential. When the training and test environments differ, agents exhibit different failure modes. For exam- ple, in WebShop we observe an increase in the proportion of “State or Memory Inconsistency” errors, rising from 4.3% in held-in to 21.9% in held-out settings. This suggests that, when confronted with large volumes of information, the agent’s decision-making becomes less coherent, and it struggles to 10 Can RL Improve Generalization of LLM Agents? An Empirical Study 26.1% 26.1% 17.4% 13.0% 8.7% 4.3% 4.3% Train & Test on WebShop 26.1% 23.8% 23.0% 17.5% 6.3% 0.1% 0.7% 2.4% Train & Test on SearchQA 19.1% 19.1% 19.1% 16.2% 16.2% 10.3% Train & Test on TextCraft 24.4% 22.0% 17.1% 12.2% 12.2% 2.4% 4.9% 4.9% Train & Test on ALFWorld 33.1% 21.5% 12.4% 9.9% 7.4% 5.8% 5.0% 5.0% Train & Test on BabyAI 25.6% 21.9% 21.7% 13.9% 7.9% 2.5% 2.6% 3.7% Train on other Environments, Test on WebShop 33.8% 21.2% 21.1% 16.0% 0.3% 1.8% 2.4% 3.4% Train on other Environments, Test on SearchQA 20.5% 18.7% 15.8% 15.2% 13.8% 13.6% 0.6% 1.8% Train on other Environments, Test on TextCraft 25.2% 22.4% 12.9% 11.7% 10.7% 9.6% 5.1% 2.3% Train on other Environments, Test on ALFWorld 24.5% 20.4% 15.3% 14.8% 14.4% 5.1% 0.9% 4.6% Train on other Environments, Test on BabyAI Instruction Misinterpretation Confirmation Bias Guessing or Fabrication State or Memory Inconsistency Action Execution Failure Logic or Numerical Deficit Constraint Prioritization Failure Termination Protocol Failure Figure 6| Comparison of failure mode distributions between in-environment and out-of-environment evaluation. generalize its ability to extract salient information. A full environment-by-environment breakdown is provided in Figure 9. Case Study. We perform a comprehensive analysis of environment generalization and present spe- cific case studies in Appendix G. Notably, Figure 10 illustrates the mechanism behind the successful transfer from SearchQA to WebShop. Specifically, the base agent tends to blindly input the entire instruction into the search bar, and struggles to accurately extract key information from the volumi- nous HTML content, resulting in the retrieval and selection of irrelevant items. In contrast, the agent trained on SearchQA learns to effectively search for key details and extract information, successfully selecting the correct item. Furthermore, Figure 11 offers qualitative insight into why SearchQA generalization remains difficult, using an Alf World-trained agent as a case study. Although both agents fail initially, the SearchQA-trained agent iteratively refines its queries to become more specific, improving retrieval and ultimately recovering. By contrast, this query-refinement behavior does not reliably transfer: the Alf World-trained agent falls into a degenerate loop, repeatedly issuing near-duplicate searches, returning the same responses, and ultimately failing. 8. Conclusion In this paper, we present a systematic study of how RFT affects the transfer and generalization of LLM agents for multi-turn decision-making. Through large-scale experiments along three comple- mentary axes, we characterize when RFT generalizes within and across environments and identify generalization patterns. We further complement our quantitative results with failure mode analysis and qualitative case study to pinpoint where agents break down and what behaviors fail to transfer. Together, our findings offer practical guidance for training and evaluating agents under distribu- tion shift, and we hope they inform the development of agents that generalize reliably in real-world deployments. 11 Can RL Improve Generalization of LLM Agents? An Empirical Study References Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmless- ness from AI feedback. CoRR, abs/2212.08073, 2022a. doi: 10.48550/ARXIV.2212.08073. URL https://doi.org/10.48550/arXiv.2212.08073. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009. Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen. Retaining by doing: The role of on-policy data in mitigating forgetting. CoRR, abs/2510.18874, 2025a. doi: 10.48550/ARXIV. 2510.18874. URL https://doi.org/10.48550/arXiv.2510.18874. Kevin Chen, Marco F. Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive LLM agents. CoRR, abs/2502.01600, 2025b. doi: 10.48550/ARXIV.2502.01600. URL https: //doi.org/10.48550/arXiv.2502.01600. Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al. Revisiting reinforcement learning for llm reasoning from a cross-domain perspective. arXiv preprint arXiv:2506.14965, 2025. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272, 2018. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299–4307, 2017. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for rea- soning language models. arXiv preprint arXiv:2505.22617, 2025. 12 Can RL Improve Generalization of LLM Agents? An Empirical Study DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948, 2025. doi: 10.48550/ARXIV.2501.12948. URL https://doi.org/10. 48550/arXiv.2501.12948. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. CoRR, abs/2504.11536, 2025. doi: 10.48550/ARXIV.2504.11536. URL https://doi.org/10. 48550/arXiv.2504.11536. Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. AGILE: A novel reinforcement learning framework of LLM agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. Xu Guo, Tianyi Liang, Tong Jian, Xiaogui Yang, Ling-I Wu, Chenhui Li, Zhihui Lu, Qipeng Guo, and Kai Chen. Ifdecorator: Wrapping instruction following reinforcement learning with verifiable rewards. arXiv preprint arXiv:2508.04632, 2025. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024. Chuxuan Hu, Yuxuan Zhu, Antony Kellermann, Caleb Biddulph, Suppakit Waiwitlikhit, Jason Benn, and Daniel Kang. Breaking barriers: Do reinforcement post training gains transfer to unseen domains? CoRR, abs/2506.19733, 2025. doi: 10.48550/ARXIV.2506.19733. URL https:// doi.org/10.48550/arXiv.2506.19733. Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025. Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general LLM capabil- ities? understanding transferability of LLM reasoning. CoRR, abs/2507.00432, 2025. doi: 10.48550/ARXIV.2507.00432. URL https://doi.org/10.48550/arXiv.2507.00432. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training llms to reason and leverage search engines with reinforcement learning. CoRR, abs/2503.09516, 2025. doi: 10.48550/ARXIV.2503.09516. URL https://doi.org/10. 48550/arXiv.2503.09516. 13 Can RL Improve Generalization of LLM Agents? An Empirical Study Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=VrHiF2hsrm. Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022. URL https://openreview.net/forum?id=UYneFzXSJWh. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. In The Thirteenth Interna- tional Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025. URL https://openreview.net/forum?id=CjwERcAU7w. Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool-augmented reward modeling. arXiv preprint arXiv:2310.01045, 2023. Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, and Zhicheng Dou. Deepagent: A general reasoning agent with scalable toolsets. CoRR, abs/2510.21618, 2025a. doi: 10.48550/ARXIV.2510.21618. URL https://doi. org/10.48550/arXiv.2510.21618. Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, et al. Deepagent: A general reasoning agent with scalable toolsets. arXiv preprint arXiv:2510.21618, 2025b. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catas- trophic forgetting in large language models during continual fine-tuning. CoRR, abs/2308.08747, 2023. doi: 10.48550/ARXIV.2308.08747. URL https://doi.org/10.48550/arXiv.2308. 08747. Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2 r: Teaching llms to self-verify and self-correct via reinforcement learning. CoRR, abs/2502.12853, 2025. doi: 10.48550/ARXIV.2502.12853. URL https://doi.org/ 10.48550/arXiv.2502.12853. Xinji Mai, Haotian Xu, Zhong-Zhi Li, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773, 2025. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. 14 Can RL Improve Generalization of LLM Agents? An Empirical Study Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tom- czak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, and Khoa D. Doan. The reasoning boundary paradox: How reinforcement learning constrains language models. CoRR, abs/2510.02230, 2025. doi: 10.48550/ARXIV.2510.02230. URL https://doi.org/10. 48550/arXiv.2510.02230. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022a. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022b. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, and Roozbeh Mottaghi. Grounding multimodal llms to embodied agents that ask for help with reinforcement learning. arXiv preprint arXiv:2504.00907, 2025. Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, and Chong Ruan. Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. CoRR, abs/2504.21801, 2025. doi: 10.48550/ARXIV.2504.21801. URL https://doi.org/10.48550/arXiv.2504.21801. 15 Can RL Improve Generalization of LLM Agents? An Empirical Study Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. Aditya Sanghi, Rao Fu, Vivian Liu, Karl D. D. Willis, Hooman Shayani, Amir Hosein Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Textcraft: Zero-shot generation of high-fidelity and diverse shapes from text. CoRR, abs/2211.01427, 2022. doi: 10.48550/ARXIV.2211.01427. URL https: //doi.org/10.48550/arXiv.2211.01427. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707. 06347. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization for LLM agents. CoRR, abs/2403.02502, 2024a. doi: 10.48550/ ARXIV.2403.02502. URL https://doi.org/10.48550/arXiv.2403.02502. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502, 2024b. Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Omega: Can llms reason outside the box in math? evaluating exploratory, compositional, and transformative generalization. arXiv preprint arXiv:2506.18880, 2025. Richard S. Sutton and Andrew G. Barto. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6. URL http: //w.incompleteideas.net/book/first/the-book.html. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186, 2024. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm. github.io/blog/qwen2.5/. Haozhe Tian, Homayoun Hamedmoghadam, Robert Shorten, and Pietro Ferraro. Reinforcement learning with adaptive regularization for safe control of critical systems. Advances in Neural Infor- mation Processing Systems, 37:2528–2557, 2024. Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, and Xunliang Cai. A survey on LLM mid-training. CoRR, abs/2510.23081, 2025. doi: 10.48550/ARXIV.2510.23081. URL https://doi.org/10.48550/arXiv.2510.23081. 16 Can RL Improve Generalization of LLM Agents? An Empirical Study Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. SPA-RL: reinforcing LLM agents via stepwise progress attribution. CoRR, abs/2505.20732, 2025a. doi: 10.48550/ARXIV. 2505.20732. URL https://doi.org/10.48550/arXiv.2505.20732. Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline re- inforcement learning for llm multi-step reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8881–8893, 2025b. Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models’ capabilities back to pretraining data. arXiv preprint arXiv:2407.14985, 2024. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025a. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Ding- wen Yang, Chenyang Liao, Wei He, et al. Agentgym: Evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27914–27961, 2025b. Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and Yu- Gang Jiang. Agentgym-rl: Training LLM agents for long-horizon decision making through multi- turn reinforcement learning. CoRR, abs/2509.08755, 2025c. doi: 10.48550/ARXIV.2509.08755. URL https://doi.org/10.48550/arXiv.2509.08755. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022a. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international confer- ence on learning representations, 2022b. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605, 2025. 17 Can RL Improve Generalization of LLM Agents? An Empirical Study Yirong Zeng, Xiao Ding, Yutai Hou, Yuxian Wang, Li Du, Juyi Dai, Qiuyang Ding, Duyu Tang, Dandan Tu, Weiwen Liu, Bing Qin, and Ting Liu. Tool zero: Training tool-augmented llms via pure RL from scratch. CoRR, abs/2511.01934, 2025a. doi: 10.48550/ARXIV.2511.01934. URL https: //doi.org/10.48550/arXiv.2511.01934. Yongcheng Zeng, Xinyu Cui, Xuanfa Jin, Guoqing Liu, Zexu Sun, Dong Li, Ning Yang, Jianye Hao, Haifeng Zhang, and Jun Wang. Evolving llms’ self-refinement capability via iterative preference optimization. arXiv preprint arXiv:2502.05605, 2025b. Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via rein- forcement learning. Advances in neural information processing systems, 37:110935–110971, 2024. Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang, Bo Ding, and Huaimin Wang. En- hancing decision-making for LLM agents via step-level q-value models. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intel- ligence, February 25 - March 4, 2025, Philadelphia, PA, USA, pages 27161–27169. AAAI Press, 2025. doi: 10.1609/AAAI.V39I25.34924. URL https://doi.org/10.1609/aaai.v39i25.34924. Enci Zhang, Xingang Yan, Wei Lin, Tianxiang Zhang, and Lu Qianchun. Learning like humans: Advancing llm reasoning capabilities via adaptive difficulty curriculum learning and expert-guided self-reformulation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6630–6644, 2025a. Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi. Modeling future conversation turns to teach llms to ask clarifying questions. In The Thirteenth International Conference on Learn- ing Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025b. URL https://openreview.net/forum?id=cwuSAR7EKd. Zhihao Zhang, Qiaole Dong, Qi Zhang, Jun Zhao, Enyu Zhou, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Mingqi Wu, et al. Why reinforcement fine-tuning enables mllms preserve prior knowl- edge better: A data perspective. arXiv preprint arXiv:2506.23508, 2025c. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol., 15(2):20:1–20:38, 2024. doi: 10.1145/3639372. URL https://doi.org/10.1145/ 3639372. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building au- tonomous agents. arXiv preprint arXiv:2307.13854, 2023. 18 Can RL Improve Generalization of LLM Agents? An Empirical Study Appendix A. Limitations and Future Work This paper adopts a new perspective to investigate how reinforcement fine-tuning affects the gener- alization ability of LLM agents. Our experiments focus on the Qwen2.5 family and yield extensive empirical results together with substantive insights. Despite this progress, important challenges re- main. First, we rely on default training/evaluation protocols and hyperparameter choices, without exhaustive tuning; more careful optimization may bring new findings. Second, we adopt GRPO—the most commonly used RL algorithm for LLM agents at present—rather than other approaches such as REINFORCE++ (Hu, 2025) or DAPO (Yu et al., 2025); we leave these for future work. Additionally, due to computational constraints, our sequential multi-environment training study does not enumer- ate all possible environment orderings; instead, we evaluate a small set of representative sequence. Although these experiments already reveal important insights, a broader and more fine-grained in- vestigation of ordering effects is left for future work and may further substantiate our findings. B. Dataset Details We select five representative agent environments, including WebShop (Yao et al., 2022a), SearchQA (Dunn et al., 2017), TextCraft (Sanghi et al., 2022), Alf World (Shridhar et al., 2020), and BabyAI (Chevalier-Boisvert et al., 2018). The characteristics of each environment are shown in Table 1 in Section 3.2. Here, we provide their detailed types and action spaces in Table 4. Table 4 | Detailed action spaces for each environment. EnvironmentTypesAction Spaces WebShopWeb Navigation search, click SearchQAQ&A Searchsearch, answer TextCraftText-based Game get, craft, inventory Alf WorldHouseholdgo to, open, close, take from, put in/on, use, heat, cool, clean, slice, inven- tory, look, examine BabyAIEmbodiedturn right, turn left, move forward, go to, go through, toggle and go through, toggle, pickup, drop, check available actions Our data is sourced from AgentGym (Xi et al., 2025b). Following practice in previous work (Ben- gio et al., 2009; Mukherjee et al., 2023), we categorize the tasksU into easy and hard difficulty levels (denoted asU easy andU hard ) based on the avg@8 results of the Qwen2.5-7B-Instruct model, while ensuring a balanced distribution of data between the two difficulty levels. The same categorization is applied to the test set. Detailed statistics for each environment are provided in Table 5. Table 5 | Detailed data statistics for each environment. Environment Training datasetTesting dataset easy hardall easy hard all WebShop 2104 1826 3930 84 116 200 SearchQA 1960 2040 4000 120 280 400 TextCraft235 209 444 57 43 100 Alf World 1267 1153 2420 55 145 200 BabyAI398 412 810 52 38 90 19 Can RL Improve Generalization of LLM Agents? An Empirical Study C. Detailed algorithm of GRPO. Group Relative Policy Optimization (GRPO) is an efficient online reinforcement learning algorithm tailored for LLMs. It eliminates the need for a separate critic network typically required in PPO, thereby reducing computational overhead. Instead, GRPO estimates the baseline using group relative advantages to significantly reduce gradient variance. Specifically, for each input query 푞 (derived from 푢), GRPO samples a group of outputs 푦 1 , 푦 2 , . . . , 푦 퐺 from the old policy 휋 휃 표푙푑 and obtains their corresponding rewards 푅 1 , 푅 2 , . . . , 푅 퐺 . The advantage 퐴 푖 for each output is calculated by normalizing the rewards within the group: 퐴 푖 = 푟 푖 − mean(푅 1 , . . . , 푅 퐺 ) std(푅 1 , . . . , 푅 퐺 ) (4) Finally, GRPO updates the policy by maximizing the following surrogate objective, which incorpo- rates PPO-style clipping (Schulman et al., 2017) and a KL divergence penalty to ensure training stability: J GRPO (휃)= 피 " 1 퐺 퐺 ∑︁ 푖=1 min 푟 푖 (휃)퐴 푖 , clip 푟 푖 (휃), 1− 휖, 1+ 휖 퐴 푖 − 훽픻 KL # , (5) where 푟 푖 (휃)= 휋 휃 (푦 푖 |푞) 휋 휃 표푙푑 (푦 푖 |푞) denotes the probability ratio between the new and old policies, 푦 푖 퐺 푖=1 ∼ 휋 휃 old (푞) represents the outputs 푦 푖 sampled from the old policy 휋 휃 old given the query 푞, 휖 is the clipping parameter, and 훽 is the coefficient for the KL divergence term. D. Detailed Results of Sequential Cross-Environment Training 0 50 100 train w/ AWtrain w/ WS Eval on AW Eval on WS train w/ AWtrain w/ TC Eval on AW Eval on TC train w/ AWtrain w/ BA Eval on AW Eval on BA train w/ AWtrain w/ SQ Eval on AW Eval on SQ 0 50 100 train w/ BAtrain w/ WS Eval on BA Eval on WS train w/ BAtrain w/ TC Eval on BA Eval on TC train w/ BAtrain w/ AW Eval on BA Eval on AW train w/ BAtrain w/ SQ Eval on BA Eval on SQ 100200300400 0 50 100 train w/ SQtrain w/ WS Eval on SQ Eval on WS 100200300400 train w/ SQtrain w/ TC Eval on SQ Eval on TC 100200300400 train w/ SQtrain w/ AW Eval on SQ Eval on AW 100200300400 train w/ SQtrain w/ BA Eval on SQ Eval on BA Training Steps Performance (%) Figure 7 | Training dynamics of forgetting and transfer in other sequential two-stage cross- environment training with Qwen2.5-7B-Instruct, where blue and red denote the upstream environ- ment and the downstream environment, respectively. Figure 3 in Section 6 illustrates the training dynamics for 8 two-stage sequential training configu- rations. Here, we present the dynamics for the remaining 12 pairs in Figure 7. Additionally, detailed final results are reported in Table 6. 20 Can RL Improve Generalization of LLM Agents? An Empirical Study Table 6 | Results of sequential training across different environments with Qwen2.5-7B-Instruct model. Upstream Downstream WebShop SearchQA TextCraft AlfWorld BabyAI base model28.5931.1933.6326.5667.00 WebShop86.5033.2840.7524.1379.21 + SearchQA85.0841.4444.6317.8178.88 + TextCraft86.3236.8182.5026.6381.68 + AlfWorld85.9934.2243.7590.6976.34 + BabyAI86.3833.9749.2525.5087.74 SearchQA47.0746.1235.2516.7580.33 + WebShop80.3546.1634.5018.4478.59 + TextCraft49.3146.1978.6326.0678.03 + AlfWorld39.4242.9241.0094.0081.29 + BabyAI44.3645.4425.0014.0689.62 TextCraft38.3032.1980.8831.5077.95 + WebShop84.3334.5977.0018.3178.44 + SearchQA39.1743.1671.6319.6978.97 + AlfWorld42.0431.5676.5090.3862.19 + BabyAI23.0033.3774.8822.8187.55 AlfWorld34.3129.5936.1392.0072.91 + WebShop82.9531.5035.8887.7549.65 + SearchQA53.0444.6643.5090.6377.86 + TextCraft50.3130.8176.0091.1977.01 + BabyAI25.6828.3139.2585.5675.59 BabyAI10.2529.4139.2528.1388.79 + WebShop83.6030.1936.3833.3890.66 + SearchQA33.7138.8440.6324.0689.37 + TextCraft17.9533.3786.0029.3188.60 + AlfWorld13.6029.8841.2592.5089.32 E. Detailed Results of average turns and generated tokens in different environ- ment Figure 2 in section 4 reports the average number of interaction turns and generated tokens across different environments. The more detailed results are reported in Figure 8. F. Failure Mode Analysis Through detailed analysis, we summarize 8 common error types: 1. Instruction Misinterpretation: The agent fails to understand the prompt correctly, does not follow the instructions as indicated, or generates the wrong answer format. 2. Guessing or Fabrication: The agent relies too heavily on internal parameters or makes unsup- ported guesses, instead of utilizing environmental tools to gather necessary external informa- tion. 3. Action Execution Failure: The agent generates actions that cannot be interpreted by the en- vironment or do not correspond to specific objects within the environment. 21 Can RL Improve Generalization of LLM Agents? An Empirical Study 0 500 1000 Avg Generated Tokens WebShop 0 200 400 600 AlfWorld 0 500 1000 TextCraft 0 200 400 600 BabyAI 500 750 1000 1250 SearchQA Base U easy U hard U 0 5 10 15 Avg Turn Base U easy U hard U 0 5 10 15 Base U easy U hard U 0 5 10 15 Base U easy U hard U 0 5 10 Base U easy U hard U 5 10 15 Figure 8 | Average generated tokens and average turn across different environments for Qwen2.5- 3B-Instruct model trained with varying difficulties. 4. Constraint Prioritization Failure: The agent recognizes multiple constraints in the instruc- tions but fails to evaluate their relative importance correctly, resulting in the violation of core constraints in favor of secondary ones. 5. Confirmation Bias: The agent becomes confident that it has found the correct answer or com- pleted key steps, but does not proceed with further verification. Alternatively, it may forcefully apply other clues or validate erroneous information from the prompt. 6. State or Memory Inconsistency: The agent exhibits contradictions over time, forgetting tools it has recently invoked or results it has obtained. Completed steps may be repeated unneces- sarily. 7. Logic or Numerical Deficit: The agent makes errors in logical or numerical reasoning. 8. Termination Protocol Failure: The agent fails to issue the environment-specific termination command or prematurely terminates before completing the task. The results of both in-domain and out-of-domain evaluations are discussed in Section 7. Here, we present a detailed error classification in Figure 9. G. Case Study In this section, we present specific case studies. It is worth noting that while all interactions are in- herently text-based, we provide visualizations for selected cases to enhance clarity. Additionally, due to the excessive number of interaction turns, we omit the majority of intermediate steps, highlighting only the pivotal moments in the figures. Figure 10 illustrates a case where a model trained on SearchQA is evaluated on WebShop, com- paring its generated trajectory against that of the base model on the same task. As observed in the case, the base model tends to blindly input the entire content of the instruction into the search bar, resulting in the retrieval of numerous irrelevant items. Furthermore, the base model exhibits incoher- ent decision-making under multiple constraints and struggles to accurately extract key information from the voluminous HTML content returned by the environment, leading to the selection of items from incorrect categories. In contrast, the model trained on SearchQA learns to formulate more flex- ible search queries, as well as perform efficient information extraction from complex results, thereby enabling it to successfully retrieve key information and select the correct item. Figure 11 highlights a key reason why agents struggle to generalize to SearchQA, using model 22 Can RL Improve Generalization of LLM Agents? An Empirical Study Train on WebShop 26.1% 26.1% 17.4% 13.0% 8.7% 4.3% 4.3% Test on WebShop 34.1% 23.3% 20.5% 16.9% 0.4% 1.4% 3.3% Test on SearchQA 19.7% 19.3% 15.2% 15.2% 14.8% 14.5% 0.7% 0.7% Test on TextCraft 25.6% 22.6% 14.2% 12.8% 9.1% 6.6% 6.6% 2.5% Test on ALFWorld 24.4% 22.0% 17.1% 14.6% 12.2% 4.9% 4.9% Test on BabyAI Train on SearchQA 27.9% 27.0% 19.7% 10.7% 5.7% 0.8% 3.3% 4.9% 26.1% 23.8% 23.0% 17.5% 6.3% 0.1% 0.7% 2.4% 19.9% 19.5% 16.3% 16.3% 12.8% 12.4% 1.4% 1.4% 23.1% 22.9% 14.3% 13.0% 10.2% 8.3% 5.8% 2.4% 23.1% 19.2% 19.2% 15.4% 15.4% 1.9% 1.9% 3.8% Train on TextCraft 25.0% 20.2% 15.4% 14.4% 10.6% 6.4% 5.3% 2.7% 33.5% 21.2% 20.7% 15.7% 1.9% 3.5% 3.5% 19.1% 19.1% 19.1% 16.2% 16.2% 10.3% 25.7% 21.7% 12.5% 11.7% 11.4% 10.6% 1.9% 4.6% 27.7% 25.5% 19.1% 12.8% 8.5% 2.1% 2.1% 2.1% Train on ALFWorld 27.9% 24.8% 16.3% 16.3% 7.8% 1.7% 2.4% 2.7% 32.8% 22.0% 20.0% 16.8% 0.2% 2.0% 2.1% 4.1% 23.1% 17.4% 16.1% 14.5% 14.0% 12.4% 2.5% 24.4% 22.0% 17.1% 12.2% 12.2% 2.4% 4.9% 4.9% 25.0% 17.1% 15.8% 15.8% 10.5% 9.2% 6.6% Train on BabyAI 25.5% 24.0% 24.0% 10.9% 9.3% 1.2% 1.8% 3.2% 35.0% 21.1% 20.4% 14.4% 1.0% 1.8% 2.7% 3.6% 21.1% 20.2% 15.1% 14.7% 14.2% 11.5% 3.2% 27.3% 22.3% 14.9% 12.4% 11.8% 6.2% 2.3% 2.8% 33.1% 21.5% 12.4% 9.9% 7.4% 5.8% 5.0% 5.0% Instruction Misinterpretation Confirmation Bias Guessing or Fabrication State or Memory Inconsistency Action Execution Failure Logic or Numerical Deficit Constraint Prioritization Failure Termination Protocol Failure Figure 9 | Failure mode distribution across all train-test environment combinations. 23 Can RL Improve Generalization of LLM Agents? An Empirical Study Instruction: Find me hand wash women's sweaters with long sleeve, stretch fabric, polyester spandex for teen girls, daily wear with color: xnj-tshirt347-white, and size: small, and price lower than 50.00 dollars Thought:I think the best way to find the desired product is to use a search query that includes all the necessary detailssuch as gender, style, material, color, and size. Action: search[hand wash women's sweaters long sleeve stretch fabric polyester spandex teen girls daily wear color:xnj-tshirt347-white size:smallprice:<50.00] Thought:I think the search should include the key detailsprovided with specific criteria. Action: search[hand wash women's long sleeve sweaters polyester spandex teen girl white small daily wear under 50] WebShop WebShop base model base model model trained on SearchQA WebShop WebShop base model · repeated loop "search-click-prev" Reaching the maximum number of rounds! · WebShop Score: 0 WebShop · Thank you for shopping with us! Score: 1 click[B09M63B87V] click[small] click[Buy Now] model trained on SearchQA model trained on SearchQA click[B09QCVCYVY] click[< Prev] · search[...] Figure 10 | Comparison of trajectories on WebShop between the base model and the model trained on SearchQA. trained on Alf World as a case study. Although both models fail in their initial attempts, the model trained on SearchQA demonstrates the ability to refine its search queries to achieve greater preci- sion. This capability proves difficult to transfer from other environments, e.g., the model trained on Alf World falls into a repetitive loop of the same searches and answers, consequently failing to resolve the task. Figure 12 illustrates how training enhances exploration efficiency, using BabyAI as a case study. The base model, lacking strong spatial awareness, struggles to accurately pinpoint the target location based on textual descriptions. Consequently, it falls into a pattern of redundant exploration; although it eventually completes the task, it requires an excessive number of interaction turns. In contrast, the trained model is able to precisely locate the target, thereby completing the task via an optimal path and significantly improving exploration efficiency. 24 Can RL Improve Generalization of LLM Agents? An Empirical Study SearchQA Question: Where did the director of film The Power (1984 Film) graduate from? [Omit detailed intermediate steps]Search and findthat the director is Stephen Carpenter models model trained on AlfWorld Action: search[Stephen Carpenter graduated from] SearchQA Observation: Doc 1fishing, and hunting all come together in ecology, ... Doc 2 Stephen Russell Carpenter is an American lake ecologist ...Doc 3 She grew up in several Midwestern cities and then in Short Hills, New Jersey. model trained on SearchQA Thought:There seems to be a mix-up as the director in question here might be a different Stephen Carpenterfrom the one found ... it seems the query needs a more specificfocus to pinpoint the correct answer. Action: search[Stephen Carpenter film directoreducation] [Omit detailed intermediate steps]Try to answer "Amherst College” but the environment return "incorrect" ... ... Action: search[Stephen Carpenter education] SearchQA Observation: Doc 1 Stephen Carpenter ... is an American writer, director, and ... He graduated from the University of California, Los Angeles... model trained on SearchQA Action: answer[University of California, Los Angeles] Score: 0 Action: answer[Amherst College] SearchQA Sorry, your answer is incorrect. Please try again. Reaching the maximum number of rounds! Congratulations! You have answered the question correctly. ... Score: 1 [Omit detailed intermediate steps]Repeated loop model trained on AlfWorld Action: search[Stephen Carpenter education] Action: answer[Amherst College] SearchQA Sorry, your answer is incorrect. Please try again. ... ... Figure 11 | Comparison of trajectories on SearchQA between the model trained on SearchQA and the mode trained on Alf World. 25 Can RL Improve Generalization of LLM Agents? An Empirical Study BabyAI Instruction: go to the yellow ball In front of you in this room, you can see several objects: There is a red ball 1 0 steps in front of you and 2 steps to your left ... The room has walls around you. You are facing a wall 3 steps away. You are not carrying anything. Available actions: ["turn left", "turn right", "move forward", "pickup red ball 1", "pickup green key 1", "pickup red key 1", "pickup green ball 1", "toggle and go through red closed door 1", "go to red ball 1", "go to green key 1", "go to red key 1", "go to yellow locked door 1", "go to green ball 1", "go to red closed door 1", "check available actions"] Thought:The yellow ball is the target. It is located 2 steps in front of me and 1 step to my left. First, I need to get to the yellow locked door, then use the green key to unlock it. After that, I can proceed to pick up the yellow ball. Action:go to yellow locked door 1 base model Action: turn right Action:pickup green key 1 Action:go to green locked door 1 Action:toggle and go through green locked door 1 Action: turn left Action: turn right Action: turn left Action:check available actions Action: turn right Action: drop Action:pickup green key 1 Action: turn left Action: turn right Action:move forward Action: drop Action:pickup green key 1 Action: turn left Action:go through green open door 1 Action:go to yellow ball 1 Thought:I needtogoto theyellow ball behind me. Action:turn right model trained on BabyAI Thought:I need to go to the yellow ball. Action:go to yellow ball 1 model trained on BabyAI Figure 12| A case study in the BabyAI environment. Compared to the base model, the trained model demonstrates significantly improved exploration efficiency. 26