Paper deep dive
Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents
Ziang Ye, Wentao Shi, Lei Chen, Zhengzhou Cai, Yu Wang, Zhiyuan Yao, Zihan Wang, Xierui Song, Fuli Feng, Yixin Cao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/12/2026, 6:49:46 AM
Summary
The paper introduces ReTRAIL, a framework for long-horizon LLM agents that enables retrospective exploration by rewinding the environment to earlier states while preserving valid progress and carrying forward knowledge from abandoned exploration. ReTRAIL utilizes State-Anchored GRPO for execution improvement and a branch-advancement reward for reflection training, demonstrating superior performance on benchmarks like MineSweeper, Sokoban, ALFWorld, and ScienceWorld compared to baselines.
Entities (9)
Relation Signals (7)
ReTRAIL → evaluatedon → Sokoban
confidence 95% · Experiments across four interactive benchmark... Sokoban... show that... ReTRAIL enables agents to rewind
ReTRAIL → evaluatedon → MineSweeper
confidence 95% · Experiments across four interactive benchmark MineSweeper... show that... ReTRAIL enables agents to rewind
ReTRAIL → evaluatedon → ALFWorld
confidence 95% · Experiments across four interactive benchmark... ALFWorld... show that... ReTRAIL enables agents to rewind
ReTRAIL → evaluatedon → ScienceWorld
confidence 95% · Experiments across four interactive benchmark... ScienceWorld... show that... ReTRAIL enables agents to rewind
ReTRAIL → uses → branch-advancement reward
confidence 95% · while a branch-advancement reward guides reflection through the progress of subsequent exploration.
ReTRAIL → uses → State-Anchored GRPO
confidence 95% · ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO compares alternative continuations from revisited states to improve execution
Ziang Ye → affiliatedwith → University of Science and Technology of China
confidence 90% · Ziang Ye 1;2;6∗ ... 1 University of Science and Technology of China
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions. We introduce ReTRAIL, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration. The environment state moves backward while the agent's knowledge continues to move forward. ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO compares alternative continuations from revisited states to improve execution, while a branch-advancement reward guides reflection through the progress of subsequent exploration. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind primitive provides limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration. These results demonstrate the value of coupling environment rewinding with experience reuse within an ongoing task for long-horizon task solving. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.13058v1
- Canonical: https://arxiv.org/abs/2610.13058v1
Trouble viewing inline? Open PDF directly →
Full Text
129,908 characters extracted from source content.
Expand or collapse full text
TRAVELBACKTOMOVEFORWARD: RETROSPECTIVE EXPLORATION ANDEVOLVING FORLONGHORIZON AGENTS Ziang Ye 1;2;6∗ , Wentao Shi 3 , Lei Chen 1 , Zhengzhou Cai 1 , Yu Wang 1 , Zhiyuan Yao 4 , Zihan Wang 5 , Xierui Song 6† , Fuli Feng 1 , Yixin Cao 7† 1 University of Science and Technology of China 2 Shanghai Innovation Institute 3 Facebook 4 Zhejiang University 5 Tsinghua University 6 Meituan 7 Fudan University yza03@mail.ustc.edu.cn,guqi03@meituan.com, fulifeng93@gmail.com,caoyixin2011@gmail.com ABSTRACT Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions. We introduceReTRAIL, a framework that enables agents to rewind the environ- ment to earlier states while preserving valid progress and carrying forward knowl- edge learned from abandoned exploration. The environment state moves back- ward while the agent’s knowledge continues to move forward. ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO com- pares alternative continuations from revisited states to improve execution, while a branch-advancement reward guides reflection through the progress of subse- quent exploration. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind prim- itive provides limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration. These results demon- strate the value of coupling environment rewinding with experience reuse within an ongoing task for long-horizon task solving. The code is available at https: //github.com/whi497/ReTRAIL. 1 INTRODUCTION Large language model (LLM) agents are rapidly evolving from conversational systems into au- tonomous decision-making agents capable of solving long-horizon tasks through extended interac- tions, diverse tool use, and complex state transitions ( Jimenez et al.,2024;Sun et al.,2026b;He et al.,2026b). As task complexity increases, agents inevitably encounter unfamiliar constraints and make early mistakes that may alter the environment, trigger cascading failures, and accumulate irrel- evant information into the context ( Zhou et al.,2024;Ye et al.,2026). Recovering from such failures requires agents to revisit earlier viable environment states without discarding either valid progress or useful experience. Crucially, returning to an earlier state should not mean returning to the same decisions: agents must distill what they have learned from abandoned exploration and use it to guide a more informed continuation. Prior work has sought to guide LLMs to recover from errors and learn from unsuccessful history through reflection, repeated trials, and backtracking. Interleaving reasoning with acting lets an agent revise its plan at each step, but the revision stays local and the execution keeps moving forward ( Yao et al.,2023;Choi et al.,2026). Cross-episode adaptation uses accumulated experience to guide new ∗ Work done during internship at Meituan. † Corresponding authors 1 arXiv:2610.13058v1 [cs.AI] 9 Oct 2026 Preprint attempts (Shinn et al.,2023;Ouyang et al.,2026;Jiang et al.,2026), but applies it after an episode terminates and execution restarts from the initial state, requiring earlier progress to be repeated. Backtracking and trajectory-revision methods enable agents to reconsider earlier decisions through tree search, prompted controllers, or supervised correction traces (Yuan et al.,2025;Li et al.,2025; Zhang et al.,2026). A central challenge is to train agents to proactively recover within an ongoing task and use the experience gained from abandoned branches to improve subsequent exploration. This calls for learning signals that improve both execution after a rewind and the experience carried forward. Our key observation is that environment states can be rewound while the knowledge gained from ex- ploration should be accumulated. An unsuccessful branch may reveal hidden constraints, ineffective actions, or promising alternatives, allowing the agent to revisit an earlier state with a better-informed context. We introduceReTRAIL, a framework that exposes an environment-levelrewindinterface to the agent, allowing it to restore earlier environment states. Building on this interface, we propose Retrospective Exploration: the agent proactively leaves an unproductive branch, rewinds the envi- ronment to an earlier state, distills useful information from the abandoned branch into a compact re- flection, and resumes execution with the preserved execution prefix and accumulated experience. To jointly develop the agents ability to rewind and distill useful experience from abandoned branches, we useState-Anchored GRPOto improve execution by comparing alternative continuations from the same environment state, and abranch-advancement rewardto train reflection based on the relative progress of subsequent exploration. We evaluate ReTRAIL across four diverse interactive benchmarks: MineSweeper ( Li et al.,2024), Sokoban (Racanière et al.,2017), ALFWorld (Shridhar et al.,2021), and ScienceWorld (Wang et al., 2022). ReTRAIL achieves the highest success rates among the evaluated methods when rewind is enabled. Our experiments reveal that simply providing existing agents with a rewind operation does not consistently improve performance and can even degrade it. In contrast, ReTRAIL benefits from proactive rewinding across all four benchmarks and achieves further gains through reflection-guided retries. On MineSweeper, success increases from 58.8% without rewind to 63.5% with proactive rewinding, and reaches 95.3% with up to three reflection-guided attempts, compared with 70.8% for the strongest baseline under the same evaluation protocol. Further analysis shows that retry gains are larger on longer tasks and when abandoned branches reveal information useful for subsequent deci- sions. These findings highlight the value of learning to rewind and carry forward useful experience, enabling unsuccessful exploration to inform continued progress within an ongoing long-horizon task. Our main contributions are summarized as follows: •We introduceReTRAIL, a framework forRetrospective Explorationthrough an environment- level rewind interface. Agents proactively revisit earlier states while preserving valid progress and carrying forward experience distilled from abandoned branches to guide subsequent exploration. •We develop two complementary learning signals for a shared execution and reflection policy. State-Anchored GRPOcompares alternative continuations from the same environment state to improve execution, while abranch-advancement rewardtrains reflection based on the relative progress of subsequent exploration. •We evaluate ReTRAIL across four interactive benchmarks, demonstrating improved success through proactive rewinding and reflection-guided retries. Further experiments show larger gains on longer and harder tasks, highlighting the value of retaining experience to guide subsequent exploration. 2 RELATEDWORK 2.1 LONG-HORIZONEXECUTION ANDRECOVERY Sustaining reliable execution over long interaction sequences motivates research on planning, con- text management, and error recovery in LLM agents ( Yao et al.,2023;Zhang et al.,2023;Yuan et al.,2025). Search-based methods explore alternative action sequences within context (Zhou et al., 2024;Choi et al.,2026). Agent-R learns decision correction from revision traces (Yuan et al.,2025). However, revising decisions through context alone does not reverse changes to the environment, 2 Preprint Multi-episode adaptation Agent π θ s 0 s T · × s 0 s T · Restart from the beginning with retained experience ReTRAIL: Retrospective exploration (1) Explore Abandoned suffix Agent π θ s 0 s j s j+1 s k · Retained prefix s j Same state (2) Reflect, then restore Experience e m Distilled from suffix context I made a wrong decision. I am stuck and cannot make progress. s′ j+1 Goal · Knowledge forward (3) Continue with experience Figure 1:Retrospective exploration with retained experience.Multi-episode adaptation (top) restarts the task froms 0 after an unsuccessful attempt, carrying retained experience into a new episode. In contrast, ReTRAIL (bottom) proactively abandons an unproductive branch, distills its experience, rewinds to an earlier state, and resumes with the retained execution prefix and accumu- lated experience. Recovery can occur before terminal failure. which can leave agents trapped in unproductive states. WebRollback uses action, critique, and roll- back modules to select recovery points and carry forward experience summaries, with prompting and supervised fine-tuning from demonstrations (Zhang et al.,2026). GA-Rollback uses a prompted assistant to detect and localize errors during inference, restores earlier environment states through reset-and-replay, and retains error feedback to guide subsequent actions (Li et al.,2025). ReTRAIL take a step forward to train a shared execution and reflection policy that couples environment-level rewind with reinforcement learning, enabling more diverse exploration and more effective interac- tion budgets using. 2.2 TEST-TIMEADAPTATION ANDSELF-EVOLVINGAGENTS Adapting from interaction experience enables LLM agents to improve their behavior in unfamiliar tasks and environments. Reflection and memory mechanisms distill past trajectories into feedback and reusable strategies that guide subsequent decisions ( Shinn et al.,2023;Ouyang et al.,2026). Be- yond memory accumulation, EvoTest and Meta-TTL adapt prompts, memory, and tool-use routines from episode feedback (He et al.,2026c;Lou et al.,2026). Recent work further uses reinforcement learning to optimize how agents explore and adapt from experience. LaMer and MR-Search train exploration and post-episode reflection through meta-reinforcement learning, using discounted out- come reward to provide feedback for adaptation (Jiang et al.,2026;Xiao et al.,2026). Learning to Self-Evolve rewards context revisions according to downstream performance improvements (Chen et al.,2026b). ReTRAIL brings experience-informed adaptation into recovery within an ongoing task. By revisiting earlier environment states while preserving valid progress and distilled branch experience, agents can revise their exploration before the current attempt terminates, with subse- quent branch progress providing a learning signal for reflection. 3 RETROSPECTIVEEXPLORATION We formalizeRetrospective Exploration, in which an agent rewinds to an earlier environment state while preserving the execution prefix and retaining experience from abandoned branches. We first introduce agent interaction and multi-episode adaptation, then describe how an environment-level rewindinterface enables experience-informed recovery within an ongoing task. Agent Interaction and Multi-Episode Adaptation.Given a tasku, an LLM agent starts from environment states 0 . At stept, it observeso t and selects an actiona t ∈ Abased on the interaction 3 Preprint historyh t = (o 0 , a 0 , . . . , a t−1 , o t ). The environment transitions tos t+1 , returning rewardr t and observationo t+1 . An episode yields a trajectoryτ= (s 0 , a 0 , r 0 , . . . , a T−1 , r T−1 , s T ), ending upon task completion, environment termination, or exhaustion of its interaction budget. Multi-episode adaptation groups successive attempts into a trialT= (τ (0) , . . . , τ (N−1) )(Jiang et al.,2026). Each episode restarts froms 0 , while the policy conditions on experienceE (n) retained from preceding episodes: a (n) t ∼π · |u, h (n) t ,E (n) ,E (0) =∅.(1) The trial ends upon success or exhaustion of the attempt budget. Experience thus accumulates across episodes, but each restart requires rebuilding previously achieved environment progress. Retrospective Exploration.ReTRAIL provides an environment-levelrewindinterface that allows the agent to restore a previously visited state during an ongoing attempt. When the current branch is unlikely to yield further progress, the agent can proactively rewind, preserve the execution prefix, and use experience distilled from the abandoned branch to guide a new continuation. We call this processretrospective exploration. Execute.A forward segmentσ m is the contiguous interaction sequence from the initial state or a restored state to the next recovery boundary or trial termination. Rewinding ends the current branch and starts a new segment from the selected earlier state, while accumulated experience remains available. We index these segments bymand condition execution on the interaction historyh t and accumulated branch experienceE m = (e 0 , . . . , e m−1 ), wheree i is the reflection distilled at recovery iandE 0 =∅: a t ∼π (· |u, h t ,E m ).(2) Detect recovery.Ats k m , recovery may be initiated proactively once the agent judges that the current branch is unlikely to yield further progress, or triggered by a failure signal. The agent selects an earlier states j m , partitioning the current execution path into a retained prefix and an abandoned suffix: ρ m = rewind (s k m →s j m ),j m < k m , τ 0:k m =τ 0:j m |z retained prefix ⊕β m |z abandoned suffix ,β m =τ j m :k m . (3) Here,ρ m denotes the rewind decision, before restoration is executed. The indicesj m andk m refer to positions on the current execution path. The operator⊕denotes concatenation, with a shared boundary state included only once when joining trajectory segments. Reflect.Before discardingβ m , the agent distills its useful information into a compact reflection: e m ∼π (· |u, h j m , β m ,E m ).(4) The reflection may capture discovered constraints, failure causes, actions to avoid, or alternative directions. Restore and resume.Return tos j m , truncate history toh j m , and retain experience: E m+1 =E m ⊕e m .(5) s k m , h k m ,E m −→ s j m , h j m ,E m+1 .(6) Execution resumes with the retained prefix and updated experience, allowing subsequent decisions to use a better-informed context from the same environment state. A complete trial of retrospective exploration is written as Γ = σ 0 , ρ 0 , e 0 , σ 1 , ρ 1 , e 1 , . . . , σ M ,(7) whereσ m denotes a forward trajectory segment ending at a recovery boundary or trial termination. The first segment starts froms 0 ; each subsequent segment resumes from the selected earlier state and conditions on the updated experience. The abandoned suffixβ m is determined by the selected rewind point and may span portions of multiple earlier segments. 4 LEARNING FROMRETROSPECTIVEEXPLORATION Retrospective exploration allows agents to revisit earlier states and continue with accumulated expe- rience. Learning to use this capability effectively requires improving both execution decisions and 4 Preprint 1 Retrospective trial batch G trials for one task u σ: forward segmentendpoint S Γ 1 σ 0 e 0 σ 1 e 1 ⋯ e M−1 σ M S σM Γ 2 σ 0 e 0 σ 1 e 1 ⋯ e M−1 σ M S σM Γ G σ 0 e 0 σ 1 e 1 ⋯ e M−1 σ M S σM 2 Execution credit same state + retained prefix Return G ξ A state mean ξ old s j s j+1 s k − ξ 1 s j s′ j+1 s′ j+2 s′ j+3 s′ t + ξ 2 s j s′ j+1 s′ j+2 s′ ℓ − task-wide endpoint progress ⋯ ξ old ⋯ ξ 1 ⋯ ξ 2 ⋯ one score per segmentnormalize over S(u) A cont = A macro + λ state A state execution tokens + rewind J exec = GRPO(θ; A cont ) execution-token mask 3 Reflection credit selected old / resumed pair ξ old s j s j+1 s k ξ 1 s j s′ j+1 s′ j+2 s′ t with e m verified progress V ξ old ξ 1 r ref = v + − v − A ref = Norm(r ref ) J ref = GRPO(θ; A ref ) reflection-token mask: e_m J(θ) = J exec + λ ref J ref → shared π θ Figure 2:Learning from retrospective exploration.The top panel collects retrospective trials for one task. Left: State-Anchored GRPO combines task-wide endpoint progress with return compar- isons among continuations sharing an environment state and retained prefix, producingA cont for execution tokens including rewind. Right: the verified-progress difference between a matched aban- doned/resumed pair producesA ref for the reflection tokens that generatee m . The separate masked objectives jointly update one shared policy. the experience distilled from abandoned branches. It therefore yields two complementary learning signals. First, alternative continuations from the same environment state and retained prefix can be compared to identify more productive execution paths. Second, the progress of a resumed continu- ation can be compared with that of the abandoned suffix to assess the usefulness of the experience carried forward. We use these two signals to jointly train a shared execution and reflection policy.State-Anchored GRPOimplements the first comparison by assigning credit to continuations from the same state and retained prefix. Abranch-advancement rewardimplements the second by measuring the progress of the resumed continuation relative to the abandoned suffix. Figure2illustrates these two learning signals. 4.1 STATE-ANCHOREDGRPOOVERREVISITEDSTATES A final task outcome gives limited guidance about which decisions were productive. Local com- parisons, as used in GiGPO (Feng et al.,2025a), provide more focused feedback. Retrospective exploration creates the comparison groups needed for execution credit: after returning to an ear- lier point, the agent can generate alternative continuations from the same environment state while retaining the same execution prefix. Constructing continuation groups.For each tasku, we sample a group of retrospective trials Γ i B i=1 and collect their forward segments inS(u). Each recorded state visit defines a continuation ξending with its segment at success, failure, rewind, or budget exhaustion. A model-initiated rewind is included as the terminal execution decision when present. Lets andh denote the starting environment state and retained history ofξ. We group continuations by the same task, starting state, and retained history: H u (s, h) =ξ∈ C(u) :s =s, h =h,(8) 5 Preprint whereC(u)is the collection of continuations extracted from the sampled trials. Groups include earlier segment suffixes passing throughsand new continuations after rewind. Samples retain their original generation history and experience; the accumulated experience may differ within a group. Assigning execution credit.Each continuation has a discounted returnG from its starting state to the segment endpoint and a task-progress scoreS at that endpoint, shared by continuations from the same segment. Its advantage combines task-wide and state-relative feedback: A cont = Norm S(u) (S ) |z A macro ξ +λ state Norm H u (s ξ ;h ξ ) (G ) |z A state ξ ,(9) where Norm subtracts the corresponding group mean and divides by its standard deviation plus ε . The macro term normalizes endpoint progress overS(u), counting each segment once, to preserve the task-wide objective. The state-relative term favors higher-return continuations from the same state and prefix. For singleton groups, this term is zero and the macro signal remains. We applyA cont to the execution tokens inξ, including any terminal model-initiated rewind decision. The retained prefix serves as input context and is excluded from the continuation loss. Execution returns end at rewind; subsequent advancement supervises the reflection response. AppendixD.5 details the construction, and AppendixD.6illustrates this credit boundary. 4.2 LEARNINGREFLECTION FROMBRANCHADVANCEMENT Reflection is generated from an abandoned branch, but its value is realized only through the con- tinuation that follows recovery. The discarded suffix may contain useful information about invalid actions, hidden constraints, or promising alternatives; however, the final trial outcome is too coarse and too delayed to determine whether this information improved the resumed exploration. A suc- cessful trial may succeed despite an uninformative reflection, while a useful reflection may produce measurable progress before task completion. We therefore evaluate reflection by comparing the progress of the post-recovery continuation with that of the abandoned branch from the same re- stored state. At recoverym, the agent distills the abandoned suffixβ m into experiencee m and resumes from state s j m . The abandoned suffix and the new forward segmentσ m+1 share this starting state, while the latter conditions on the updated experienceE m+1 . Here,β m is the suffix discarded by the rewind and may contain portions of multiple earlier segments; it need not coincide with a single continuation used for execution grouping. Using an environment-verifiable progress measureV, we evaluate the two paths: v − m =V(β m |s j m ),v + m =V(σ m+1 |s j m ),(10) whereVmeasures task progress, such as completed subgoals, for exploration starting froms j m . The branch-advancement reward uses their difference as a proxy for experience utility; positive values indicate greater progress after recovery: r ref m =v + m −v − m .(11) Joint policy optimization.We jointly optimize execution and reflection through J(θ) =J exec GRPO θ;A cont +λ ref J ref GRPO θ;A ref ,(12) whereA ref is obtained by group-normalizing the branch-advancement rewardsr ref m , andλ ref con- trols the relative weight of reflection training. The two objectives updateπ through execution tokens (including model-initiated rewind) and reflection tokens, respectively. Appendix D.5gives sample weighting and GRPO details. 5 EXPERIMENTS Our experiments ask whether ReTRAIL improves recovery-enabled success while retaining com- petitive forward execution (Section 5.2), which rollout and learning components support these gains (Section 6), and how performance varies with instance difficulty, task length, and the number of recovery attempts (Sections5.3andB.1). 6 Preprint Table 1:Performance comparison on four long-horizon interactive benchmarks.P@1evaluates forward execution without rewind,T@1enables proactive rewind within one trial, andT@3permits up to three macro attempts with reflection-guided recovery. Every setting shares a 60-call budget per trial, including execution and reflection. Colored annotations show absolute changes relative to P@1; bold values indicate the best result in each column. All values are success rates (%). Method MineSweeperSokobanALFWorldScienceWorld P@1 T@1 T@3 P@1 T@1 T@3 P@1 T@1 T@3 P@1 T@1 T@3 Prompting Methods Zero-shot3.53.1 ↓0.4 5.2 ↑1.7 6.05.5 ↓0.5 9.0 ↑3.0 35.034.2 ↓0.8 40.1 ↑5.1 8.07.2 ↓0.8 11.0 ↑3.0 ReAct6.0 6.5 ↑0.5 10.2 ↑4.2 7.0 6.8 ↓0.2 11.8 ↑4.8 40.0 41.0 ↑1.0 47.5 ↑7.5 10.2 9.4 ↓0.8 15.1 ↑4.9 Reflexion5.57.0 ↑1.5 9.8 ↑4.3 6.56.2 ↓0.3 12.0 ↑5.5 38.042.1 ↑4.1 45.3 ↑7.3 9.013.0 ↑4.0 16.2 ↑7.2 Rollback 6.7 6.5 ↓0.2 13.2 ↑6.5 9.7 10.3 ↑0.6 15.0 ↑5.3 35.0 34.2 ↓0.8 43.1 ↑8.1 13.2 12.3 ↓0.9 20.0 ↑6.8 Fine-Tuning Methods GRPO54.651.6 ↓3.0 60.8 ↑6.2 70.368.9 ↓1.4 82.8 ↑12.5 91.390.8 ↓0.5 93.5 ↑2.2 48.947.5 ↓1.4 52.1 ↑3.2 RLOO53.2 50.4 ↓2.8 59.1 ↑5.9 69.1 67.8 ↓1.3 81.0 ↑11.9 90.7 90.2 ↓0.5 92.8 ↑2.1 47.6 46.3 ↓1.3 50.8 ↑3.2 GiGPO57.956.4 ↓1.5 65.5 ↑7.6 78.979.7 ↑0.8 85.3 ↑6.4 90.990.5 ↓0.4 94.3 ↑3.4 56.155.0 ↓1.1 58.3 ↑2.2 BEACON 58.5 57.2 ↓1.3 68.0 ↑9.5 77.0 76.1 ↓0.9 84.0 ↑7.0 91.8 91.2 ↓0.6 94.5 ↑2.7 56.5 55.1 ↓1.4 61.0 ↑4.5 LaMer55.457.2 ↑1.8 70.8 ↑15.4 72.170.5 ↓1.6 83.3 ↑11.2 89.790.1 ↑0.4 95.7 ↑6.0 53.554.2 ↑0.7 60.2 ↑6.7 ReTRAIL58.863.5 ↑4.7 95.3 ↑36.5 76.687.5 ↑10.9 93.7 ↑17.1 92.794.0 ↑1.3 98.5 ↑5.8 56.762.0 ↑5.3 71.2 ↑14.5 5.1 EXPERIMENTALSETUP Environments.The reported results cover four interactive benchmarks with complementary fail- ure structures.MineSweeper(Li et al.,2024) requires logical deduction under partial observabil- ity; unsuccessful exploration may reveal board constraints.Sokoban(Racanière et al.,2017) tests planning with irreversible forward transitions, such as pushing a box into a dead corner.ALF- World(Shridhar et al.,2021) evaluates multi-step instruction following, whileScienceWorld(Wang et al. ,2022) involves procedural reasoning with long dependency chains. All use text interaction and the state-restoration interface in Section3. Dataset partitions are listed in Table5. Baselines.We compare with zero-shot prompting, ReAct (Yao et al.,2023), Reflexion (Shinn et al. ,2023), adapted GA-Rollback (Li et al.,2025), and trained baselines GRPO (Shao et al.,2024), RLOO (Ahmadian et al.,2024), GiGPO (Feng et al.,2025a), BEACON (Wang et al.,2026c), and LaMer (Jiang et al.,2026). GiGPO uses local group comparisons; LaMer learns across episodes. Recovery-enabled evaluations share a rewind interface with corresponding execution and reflection prompts. Evaluation protocol and metrics. Pass@1 (P@1)measures single-pass execution without rewind.Trial@1 (T@1)permits model-initiated rewind but terminates at the first irreversible fail- ure; one macro-attempt may therefore explore multiple branches.∆ pro = T@1−P@1measures the net gain from enabling proactive recovery.Trial@3 (T@3)permits up to three macro-attempts, with reflection-guided recovery after irreversible failure. We denote∆ ref = T@3−T@1as the retry gain: it measures the combined benefit of additional attempts, recovery, and retained experi- ence. The configured step limit for all settings is 60. Details are provided in Appendix C.4. Training details.We use Qwen3-4B (Yang et al.,2025) as the base model for all methods. ReTRAIL samples groups of trials using retrospective exploration and is optimized using our State- Anchored GRPO objective (Section 4.1). For the standard RL baselines, we adjust the rollout group size to match the sampling budget of ReTRAIL at each gradient update. We otherwise keep the learning rate, batch size, prompt template, and task splits identical across methods. The contin- uation construction and execution objective are detailed in Appendix D.5. AppendixC.5reports equal-benchmark aggregation. Reported results are means over three evaluation seeds. 7 Preprint 5.2 MAINRESULTS ReTRAIL achieves stronger performance.ReTRAIL achieves the highest T@1 and T@3 on all four benchmarks (Table1), outperforming the strongest baseline in T@3 by24.5percentage points on MineSweeper and10.2on ScienceWorld. Its forward-only P@1 remains competitive, slightly exceeding the strongest baseline on MineSweeper, ALFWorld, and ScienceWorld, while trailing GiGPO by2.3points on Sokoban. Together, these results show that ReTRAIL’s advantage becomes more pronounced when recovery is enabled, while its forward-solving ability remains competitive. ReTRAIL makes effective use of test-time recovery.ReTRAIL benefits from recovery both within and across macro-attempts. Within a single attempt, proactive recovery yields positive gains on every benchmark, ranging from1.3percentage points on ALFWorld to10.9on Sokoban, whereas enabling rewind for baseline policies produces mixed changes. Across attempts, ReTRAIL further improves with reflection-guided recovery and retained experience. On MineSweeper, success rises from63.5at T@1 to95.3at T@3, a gain of31.8points, compared with9.1for GiGPO and13.6 for LaMer. These results demonstrate ReTRAIL’s ability to adapt through recovery, benefiting from both within-attempt course correction and continued exploration after failure. ReTRAIL improves exploration coverage and execution efficiency.Figure3compares the cov- erage of distinct environment states and the number of actions taken in successful trials. Relative to LaMer, ReTRAIL increases the median number of distinct states visited from11.0to14.0on MineSweeper and from8.0to11.5on Sokoban. Meanwhile, the median action count among suc- cessful trials decreases from53.0to40.0and from48.5to37.0, respectively. Together, these find- ings suggest that recovering early from unproductive branches helps ReTRAIL explore more broadly while completing tasks more efficiently. MineSweeperSokoban 5 10 15 Distinct states unique states (a) Distinct states visited MineSweeperSokoban 30 40 50 60 Actions to success successful trials (b) Exploration efficiency LaMerReTRAIL Figure 3:Exploration diversity and efficiency of LaMer and ReTRAIL on MineSweeper and Sokoban. (a) States visited, where higher values indicate broader exploration; (b) Actions to suc- cess, where lower values indicate more efficient solving. ReTRAIL visits more states while requiring fewer actions. 5.3 SCALINGINSTANCEDIFFICULTY ANDRECOVERYATTEMPTS We next examine performance as instance difficulty and inference-time computation increase. Scaling to harder instances.We increase the number of mines in MineSweeper and boxes in Sokoban to construct harder variants. Figure 4(a) shows that success decreases for all plotted methods, while ReTRAIL retains an advantage at every displayed setting. At the hardest setting, ReTRAIL reaches63.9on MineSweeper, compared with45.3for the strongest baseline, and55.2 on Sokoban, compared with45.5, yielding margins of18.6and9.7percentage points. These results show that ReTRAIL maintains its advantage on harder instances. The analysis by shortest-solution length is reported separately in Figure 5. Scaling the number of attempts.Allowing up to four macro-attempts provides additional op- portunities for recovery with accumulated experience. ReTRAIL achieves the highest success rate throughout Figure 4(b). On MineSweeper, success increases from63.5at T@1 to97.8at T@4, a gain of34.3percentage points, compared with10.8for GiGPO and16.3for LaMer. On Sokoban, 8 Preprint 345 # mines 40 60 80 100 T@3 success (%) MineSweeper 63.9 234 # boxes 40 60 80 100 Sokoban 55.2 T@1T@2T@3T@4 60 70 80 90 100 Success (%) MineSweeper 97.8 T@1T@2T@3T@4 70 80 90 100 Sokoban 95.1 (a) Difficulty (b) Permitted macro attempts GiGPOLaMerReTRAIL (Ours) Figure 4:Scaling instance difficulty and recovery attempts. (a) T@3 success with additional mines or boxes; these settings vary instance difficulty. (b) Success with one to four permitted macro- attempts. ReTRAIL’s endpoint value is annotated in each panel. Y-axis ranges differ by subplot and are explicitly labeled. Higher is better. Table 2:Ablation of retrospective exploration. Success rate (%). Variant MineSweeper Sokoban T@1 T@3 T@1 T@3 ReTRAIL (full)63.595.387.593.7 w/o model rewind 58.1 78.4 72.3 89.1 w/o reflection61.8 69.7 86.0 87.2 Table 3:Ablation of the learning objective. Suc- cess rate (%). Variant MineSweeper Sokoban T@1 T@3 T@1 T@3 ReTRAIL (full)63.595.387.593.7 w/o progress-diff reward 60.9 80.2 81.7 84.9 GiGPO objective57.6 90.1 84.2 90.3 ReTRAIL reaches95.1at T@4 and remains highest at each attempt budget. Thus, the learned execution and reflection policy continues to benefit from additional recovery opportunities. 6 ANALYSIS Ablation.Tables2and3evaluate the roles of retrospective exploration and its learning objectives. Removing model-initiated rewind lowers T@1 from63.5to58.1on MineSweeper and from87.5 to72.3on Sokoban, showing the benefit of proactive recovery. Replacing distilled reflection with the raw abandoned suffix has a smaller effect on T@1, but lowers T@3 to69.7and87.2, respec- tively. This suggests that distilling branch experience provides more useful guidance for subsequent attempts than retaining raw history alone. We next ablate the learning objective. Replacing the branch-advancement rewardr ref m =v + m − v − m with post-recovery progressv + m alone lowers T@3 from95.3to80.2on MineSweeper and from93.7to84.9on Sokoban, alongside smaller declines in T@1. These results support using subsequent branch progress to guide reflection. We also replace State-Anchored GRPO with GiGPO within the same retrospective-exploration framework; this variant is distinct from the standalone GiGPO baseline in Table 1. T@1/T@3 falls from 63 . 5 / 95 . 3 to 57 . 6 / 90 . 1 on MineSweeper and from87.5/93.7to84.2/90.3on Sokoban. Together, these results support the proposed execution and reflection objectives, with the GiGPO comparison evaluating the credit-assignment designs as implemented. Recovery Gains across Task Horizons.We stratify tasks by reference-solution length to examine how ReTRAIL’s recovery gains vary with task horizon (AppendixB.1). From short to long tasks, its reflection-guided retry gain increases from4.5to19.0percentage points, while its proactive gain rises from1.3to7.8points. Although LaMer achieves a slightly larger retry gain on short tasks, ReTRAIL surpasses it by5.8points on long tasks. These results suggest that retrospective exploration is particularly beneficial for long-horizon execution. 7 CONCLUSION Learning to recover from unproductive exploration is crucial for autonomous agents to make sus- tained progress on long-horizon tasks. We introducedReTRAIL, a framework that combines exe- 9 Preprint cution recovery with experiential learning through retrospective exploration. With an environment- levelrewindinterface, agents can revisit earlier states and explore alternative continuations while retaining experience from abandoned branches. ReTRAIL trains a shared execution and reflec- tion policy under this protocol, using state-anchored comparisons to improve execution and branch- advancement feedback to guide reflection. Across four interactive benchmarks, ReTRAIL consis- tently outperforms both methods without rewind and baselines equipped with the same interface, with ablations supporting the complementary roles of recovery, retained experience, and progress- based feedback. More broadly, these findings highlight retrospective exploration as a promising foundation for autonomous agents that can recover from setbacks, adapt through accumulated expe- rience, and sustain progress toward long-horizon goals. AI USESTATEMENT In preparing this manuscript, we used generative AI tools for language polishing and literature search and survey. Language assistance focused on improving grammar, sentence flow, clarity, and read- ability while preserving the intended technical meaning. Literature assistance supported the iden- tification of relevant prior work and the organization and summarization of existing research. The authors remain responsible for reviewing all AI-assisted material, checking literature summaries and citations against the original sources, and ensuring that the manuscript accurately represents the re- search. We take full responsibility for the final content of this work, including all text, claims, and artifacts produced with the assistance of generative AI. ETHICSSTATEMENT This work studies recovery and learning in autonomous agents through experiments on four con- trolled benchmark environments: MineSweeper, Sokoban, ALFWorld, and ScienceWorld. The framework aims to improve agents’ ability to recover from unproductive exploration, but state restoration in these environments should not be interpreted as reversing the consequences of actions in real-world systems. Deployment beyond these benchmarks should therefore include appropriate authorization, safeguards for external actions, and human oversight where actions may have con- sequential or irreversible effects. Retained interaction histories and reflection memories should be handled with appropriate privacy and access controls, and any reuse or release of data, models, and software should respect their applicable licenses and terms of use. REPRODUCIBILITYSTATEMENT To support reproducibility, we specify the retrospective-exploration framework and learning ob- jectives in Sections 3and4. AppendixDdetails environment restoration, recovery and context management, and provides pseudocode for rollout collection and policy updates. Section5.1and AppendixCdocument the benchmarks and data partitions, baseline configurations, training hyper- parameters, reward and progress signals, evaluation protocols, model-call accounting, and aggrega- tion across benchmarks and seeds. AppendixEprovides execution and reflection prompt templates and examples of observation formatting for the four benchmark environments. REFERENCES Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740. Anthropic. Introducing claude opus 4.5.https://w.anthropic.com/news/ claude-opus-4-5 , November 2025. Accessed: 2026-04-29. Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, and Bing Liu. Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers, 2026. URL https://arxiv.org/abs/2602.00933. 10 Preprint Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URLhttps://arxiv.org/abs/ 2506.07982. Adithya Bhaskar, Xi Ye, and Danqi Chen. Language models that think, chat better.CoRR, abs/2509.20357, 2025. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. InNeurIPS, 2020. Arthur Chen, Zuxin Liu, Jianguo Zhang, Akshara Prabhakar, Zhiwei Liu, Shelby Heinecke, Silvio Savarese, Victor Zhong, and Caiming Xiong. Test-time adaptation for llm agents via environment interaction, 2026a. URLhttps://arxiv.org/abs/2511.04847. Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. Automan- ual: Generating instruction manuals by LLM agents via interactive environmental learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Pwl9n4zlf5. Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, and Yuxiong He. Learning to self- evolve.arXiv preprint arXiv:2603.18620, 2026b. URLhttps://arxiv.org/abs/2603. 18620. Yuxin Chen, Zhengzhou Cai, Xiangtian Ji, Weixiang Zhao, An Zhang, Xiang Wang, and Tat- Seng Chua. Understanding multilingualism in mixture-of-experts llms: Routing mechanism, ex- pert specialization, and layerwise steering, 2026c. URL https://arxiv.org/abs/2601. 14050. Yuxin Chen, Yu Wang, Yi Zhang, Ziang Ye, Zhengzhou Cai, Yaorui Shi, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, An Zhang, and Tat-Seng Chua. Learning to self-verify makes language models better reasoners, 2026d. URLhttps://arxiv.org/abs/2602.07594. Yuxin Chen, Yi Zhang, Zhengzhou Cai, Yaorui Shi, Zhiyuan Yao, Chenhang Cui, Jingnan Zheng, Yaqi Huo, Xi Su, Qi Gu, Xunliang Cai, Xiang Wang, An Zhang, and Tat-Seng Chua. Vitabench 2.0: Evaluating personalized and proactive agents in long-term user interactions, 2026e. URL https://arxiv.org/abs/2605.27141. Jae-Woo Choi, Hyungmin Kim, Hyobin Ong, Youngwoo Yoon, Minsu Jang, Dohyung Kim, and Jaehong Kim. ReAcTree: Hierarchical LLM agent trees with control flow for long-horizon task planning. InInternational Conference on Autonomous Agents and Multiagent Systems, 2026. URL https://arxiv.org/abs/2511.02424. Victor de Lamo Castrillo, Habtom Kahsay Gidey, Alexander Lenz, and Alois Knoll. Fundamentals of building autonomous llm agents, 2025. URLhttps://arxiv.org/abs/2510.09244. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948, 2025. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoran Wei, Haowei Zhang, Haowen Luo, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, Jialiang Huang, Ji- ashi Li, Jiaxin Xu, Jiewen Hu, Jingchang Chen, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexin Huang, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Liang Zhao, Liangsheng Yin, Lihua Guo, Lingxiao Luo, Linwang Ma, Litong 11 Preprint Wang, Liyue Zhang, M. S. Di, M. Y Xu, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang, Peixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Songyang Zhou, Tao Ni, Tao Yun, Tian Pei, Tian Ye, Tianyuan Yue, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjun Gao, Wentao Zhang, Xi Gao, Xiangwen Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyuan Li, Xu Chen, Xuecheng Su, Xuehai Pan, Xuheng Lin, Xuwei Fu, Y. Q. Wang, Yang Zhang, Yanhong Xu, Yanru Ma, Yao Li, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yiliang Xiong, Ying He, Ying Zhou, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuduan Wang, Yue Gong, Yuhan Wu, Yuheng Zou, Yukun Li, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehua Zhao, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhiyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Zizheng Pan, Zongqing Yao, Bei Feng, Hui Li, J. L. Cai, Jiaqi Ni, Lei Xu, Meng Li, Ning Tian, R. J. Chen, R. L. Jin, S. S. Li, Shuang Zhou, Tianyu Sun, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xinnan Song, Xinyi Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Dongjie Ji, Jian Liang, Jianzhong Guo, Jin Chen, Leyi Xia, Miaojun Wang, Mingming Li, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, Shengfeng Ye, T. Wang, W. L. Xiao, Wei An, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Ying Tang, Yukun Zha, Zekai Zhang, Zhe Ju, Zhen Zhang, and Zihua Qu. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for LLM agent training. InAdvances in Neural Information Processing Systems, volume 38, 2025a. URL https://proceedings.neurips.c/paper_files/paper/2025/ hash/420c9f777c0b4f78d515e53cf74d58b2-Abstract-Conference.html. Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. Trans. Mach. Learn. Res., 2025, 2025b. Google. A new era of intelligence with gemini 3, 2025. URLhttps://blog.google/ products-and-platforms/products/gemini/gemini-3/#note-from-ceo. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.CoRR, abs/2507.17746, 2025. Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=9JQtrumvg8. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. InACL (1), p. 3828–3850. Association for Computational Lin- guistics, 2024. Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, and Bo An. Hierarchy-of-groups policy opti- mization for long-horizon agentic tasks. InThe Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview.net/forum?id=T8Dev99qnz. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi GU, Hui Su, and Xunliang Cai. Vitabench: Benchmarking LLM agents with versatile interactive tasks in real-world applications. InThe Fourteenth International Conference on Learning Representations, 2026b. URL https: //openreview.net/forum?id=rtcX9qOBaz. 12 Preprint Yufei He, Juncheng Liu, Yue Liu, Yibo Li, Tri Cao, Zhiyuan Hu, Xinxing Xu, and Bryan Hooi. EvoTest: Evolutionary test-time learning for self-improving agentic systems. InInternational Conference on Learning Representations, 2026c. URLhttps://arxiv.org/abs/2510. 13220. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Wese: Weak exploration to strong exploitation for llm agents, 2024. URL https://arxiv.org/abs/2404.07456. Xiangtian Ji, Yuxin Chen, Zhengzhou Cai, Xiang Wang, An Zhang, and Tat-Seng Chua. Tiny brains, giant impact: Uncovering the keystone neurons of llm with just a few prompts, 2026. URLhttps://arxiv.org/abs/2605.24846. Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055, 2025. Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, and Maria Brbi ́ c. Meta-RL induces exploration in language agents. InInternational Conference on Learning Representations, 2026. URLhttps://arxiv.org/abs/2512.16848. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview. net/forum?id=VTF8yNQM66. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training llms to reason and leverage search engines with reinforcement learning.CoRR, abs/2503.09516, 2025. Xingzuo Li, Kehai Chen, Yunfei Long, Xuefeng Bai, Yong Xu, and Min Zhang. Generator-assistant stepwise rollback framework for large language model agent, 2025. URLhttps://arxiv. org/abs/2503.02519. Yinghao Li, Haorui Wang, and Chao Zhang. Assessing logical puzzle solving in large language models: Insights from a minesweeper case study. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5981. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.4. URLhttp://dx.doi.org/10.18653/v1/ 2024.naacl-long.4. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InICLR. OpenReview.net, 2024. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ. Yuxin Liu, Ziang Ye, Yueqing Sun, Mingye Zhu, Jinwei Xiao, Zhuowen Han, Qi GU, Xunliang Cai, and Lei Zhang. Map: A map-then-act paradigm for long-horizon interactive agent reasoning, 2026. URL https://arxiv.org/abs/2605.13037. Zhanzhi Lou, Hui Chen, Yibo Li, Qian Wang, and Bryan Hooi. Learning to learn-at-test-time: Language agents with learnable adaptation policies.arXiv preprint arXiv:2604.00830, 2026. URL https://arxiv.org/abs/2604.00830. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025. URL https://arxiv.org/abs/2308.08747. 13 Preprint Shinji Mai, Yunpeng Zhai, Ziqian Chen, Cheng Chen, Anni Zou, Shuchang Tao, Zhaoyang Liu, and Bolin Ding. Cues: A curiosity-driven and environment-grounded synthesis framework for agentic rl, December 2025. URLhttps://arxiv.org/abs/2512.01311. MiniMax. Minimax m2 and agent: Ingenious in simplicity, 2025. URLhttps://w. minimax.io/news/minimax-m2. OpenAI. GPT-4 technical report.CoRR, abs/2303.08774, 2023. OpenAI.Introducing gpt-5.2, 2025.URLhttps://openai.com/index/ introducing-gpt-5-2/. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. InNeurIPS, 2022. Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with reasoning memory. InInternational Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.25140. Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan Awadallah. Explorer: Scaling exploration-driven web trajectory synthe- sis for multimodal web agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the Association for Computational Linguistics: ACL 2025, p. 6300–6323, Vienna, Austria, July 2025. Association for Computational Lin- guistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.326. URLhttps: //aclanthology.org/2025.findings-acl.326/. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=tBRNC6YemY. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=dHng2O0Jjr. Sébastien Racanière, Théophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pas- canu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-augmented agents for deep reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2017. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024. Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Deepseekmath-v2: Towards self-verifiable mathematical reasoning.CoRR, abs/2511.22570, 2025. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. InEuroSys, p. 1279–1297. ACM, 2025. 14 Preprint Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning.InAd- vances in Neural Information Processing Systems, volume 36, 2023.URL https://proceedings.neurips.c/paper_files/paper/2023/hash/ 1b44b878b782e6954cd888628510e90-Abstract-Conference.html. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations (ICLR), 2021. Xiaoshuai Song, Haofei Chang, Guanting Dong, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. En- vscaler: Scaling tool-interactive environments for llm agent via programmatic synthesis, 2026. URLhttps://arxiv.org/abs/2601.05808. Hongjin SU, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan O Arik. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=3UKOzGWCVY. Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains.CoRR, abs/2503.23829, 2025. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models.Trans. Mach. Learn. Res., 2025, 2025. Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. Scaling long-horizon agent via context folding. InProceedings of the 43rd International Conference on Machine Learning, 2026a. URLhttps://openreview.net/forum?id=lNRgWoGfYg. Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, Weishu Zhang, Tyler Zeng, Ying Yan, Bo Liu, Han- son Wen, Mingyang Xu, Xiaoyuan Liu, Zimeng Chen, Weiyan Shi, Amanda Dsouza, Vin- cent Sunn Chen, Patrick Bryant, Carl Boettiger, Yamini Rangan, Bradley Rothenberg, Kyle Ste- infeld, Arvind Rao, Tapio Schneider, Georgios Yannakakis, Laure Zanna, Kaan Ozbay, Ida Sim, Tarek Zohdi, George Em Karniadakis, Jack Gallant, Teresa Head-Gordon, Yushan Li, Wenxi Deng, Tao Sun, Huiqi Wang, Zhun Wang, Justin Xu, Chris Yuhao Liu, Yafei Cheng, Rongwang Hu, Aras Bacho, Shengcao Cao, Zengyi Qin, Yixiong Chen, Hengduan Fan, Hao Liu, Lin Zeng, Shashank Muralidhar Bharadwaj, Litian Gong, Yingxuan Yang, Maojia Song, Ruheng Wang, Zongzheng Zhang, Honglin Bao, Shuo Lu, Jianhong Tu, Zhonghua Wang, Zheng Zhang, Zi- jiao Chen, Yanqiong Jiang, Zhendong Li, Bohan Lyu, Chang Ma, Peiran Xu, Benran Zhang, Shangding Gu, Haoyue Hua, Haoyang Li, Wanzhe Liao, Chengzhi Liu, Junbo Peng, Haoran Sun, Zechen Xu, Bo Chen, Jiayi Cheng, Yi Jiang, Keying Kuang, Yuan Li, Youbang Pan, Ziyan Rao, Alexander Schubert, Yifan Shen, Vincent Siu, Xiatao Sun, Kangqi Zhang, Xiaopan Zhang, Yuchen Zhu, Ishaan Singh Chandok, Lei Ding, Jingxuan Fan, Andrew Glover, Jiaming Hu, Yiran Hu, Wenbo Huang, Zixin Jiang, Haoran Jin, Lukas Kim, Ming Liu, Yang Liu, Alireza Rafiei, Xuhuan Shen, Kunyang Sun, Sophia Sun, Ting Sun, Eric Wang, Yixin Wang, Hanwen Xing, Si- han Xu, Yuzheng Xu, Zhongxing Xu, Zhiling Yan, Boqin Yuan, Ruiqi Zhang, Yifan Zhang, Zibo Zhao, Liana, Santanu Bosu Antu, Haoyue Bai, Carlo Bosio, Joseph Cavanagh, Patricia Cavazos- Rehg, Tianxing Chen, Xuewen Chen, Yipu Chen, Chenyu Zhu, Chen Dai, Stefano De Castro, Yunfu Deng, Kaustubh Dhole, Jiayuan Ding, Chenchen Du, Zhehang Du, Hao Fan, Run-Ze Fan, Hengyu Fu, Shi Gu, Yifan Gu, Charlie Guo, Baihe Huang, Baixiang Huang, Rimika Jaiswal, Zhi- han Jiang, Ran Jin, Erin Kasson, Xin Lan, Joseph Lee, Deren Lei, Chenyu Li, Daofeng Li, Haitao Li, Hongwei Li, Jingyan Li, Xiao Li, Yi Li, Yinsheng Li, Yuangang Li, Zhixu Li, Wenyu Liang, Longtai Liao, Kevin Qinghong Lin, Andy Zeyi Liu, Che Liu, Jiaming Liu, Kaiyuan Liu, Xuan Liu, Pan Lu, Wenbo Lv, Yicheng Lyu, Qiuyang Mang, Kyle Montgomery, Yuzhou Nie, Ruoxi Ning, Jorin Overwiening, Xu Pan, Layna Paraboschi, Core Francisco Park, Justin Purnomo, Swati Ra- jwal, Scott Rankin, Bixuan Ren, Yiren Rong, HaoYang Shang, Ventus Shaw, Fiona Shen, Jiawei Shen, Minqi Shi, Shi Qiu, Huaxiu Yao, Tianneng Shi, Jonah So, Vladislav Susoy, Hannah Szlyk, Haocheng Wang, Jialu Wang, Wei Wang, Xinyu Wang, Zehao Wang, Dowling Wong, Angela Wu, 15 Preprint Dehao Wu, Fangyu Wu, Mengyuan "Millie" Wu, Yu Wu, Yuchen Wu, Yuhao Wu, Qingpo Wuwu, Weihang Xiao, Yongyi Xiong, Fan Xu, Ruiling Xu, Mingxuan Yan, Benjamin Yang, Jirong Yang, Sen Yang, Xiaoli Yang, Yushi Yang, Haoran Ye, Xiaohu Yu, Zhengming Yu, Chenlong Zhang, Chi Zhang, Hanning Zhang, Hanwen Zhang, Junge Zhang, Kunpeng Zhang, Song Zhang, Wen- jin Zhang, Wenshuo Zhang, Ying Zhang, Yizhi Zhang, Brian Zhao, Qijian Zhao, Yimin Zhao, Yuhaohua Zheng, Liwei Zhou, Tianyue Zhou, Sichen Zhu, Siqi Zhu, Yan Zhu, Yishu Zhu, Jierui Zuo, Chonghao Cai, Helena Casademunt, Wenjia Chen, Cheng Cheng, Nawen Deng, Rao Fu, Tianfu Fu, Yifan Han, He Ren, Zhenyu He, Qiao Jin, Langlang Li, Yuetai Li, Sylvia Liu, Lu Lu, Luqing Zhou, Subhabrata Mukherjee, Yunqi Ouyang, Yin Ren, Dawei Shi, Haoran Wu, Zhiyue Wu, Hannah Yao, Zhuoran Yi, Jenny Yu, Rhea Zhan, Hang Zhou, Blake Zhu, Junfan Zhu, Alan Yuille, Yang Liu, Russell Alan Poldrack, Jiachen Li, Zhenglu Li, Molei Tao, Jing Huang, Wenqi Shi, Costas Spanos, Lichao Sun, Chenguang Wang, Orson Xu, Zhen Dong, Hector Gomez, Aylin Caliskan, Ali Emami, Haimin Hu, Zhi Li, Lihui Liu, Murphy Niu, Yi Shao, Jianxin Sun, Mikko Tolonen, Ting Wang, Sanjiv Das, Yanjun Gao, Wenbo Guo, Erika J Schneider, Zhiyong Lu, Yian Ma, Mark Mueller, Radha Poovendran, Somayeh Sojoudi, Yinglun Zhu, and Dawn Song. Agents’ last exam, 2026b. URL https://arxiv.org/abs/2606.05405. Gemma Team. Gemma 3 technical report.CoRR, abs/2503.19786, 2025. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.CoRR, abs/2302.13971, 2023. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNIPS, p. 5998–6008, 2017. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language mod- els.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=ehfRiF0R3a. Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, and Kam-Fai Wong. Harnessing the reasoning economy: A survey of efficient reasoning for large language models.CoRR, abs/2503.24377, 2025a. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? InProceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), p. 11279–11298. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.emnlp-main.775. URLhttps: //aclanthology.org/2022.emnlp-main.775/. Yu Wang, Yi-Kai Zhang, Wentao Shi, Ziang Ye, Yuchun Miao, Yueqing Sun, Qi Gu, Xunliang Cai, Lan-Zhe Guo, Han-Jia Ye, and Fuli Feng. Cast: Game solvers as turn-level teachers for llm agents, 2026a. URLhttps://arxiv.org/abs/2607.25308. Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, and Yujiu Yang. Agentopsd: Recursive self-distillation for agentic reinforcement learning, 2026b. URLhttps://arxiv.org/abs/ 2608.05987 . Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning.CoRR, abs/2504.20073, 2025b. Zixuan Wang, Yuchen Yan, Hongxing Li, Teng Pan, Dingming Li, Ruiqing Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Milestone-guided policy learning for long-horizon language agents. InProceedings of the 43rd International Conference on Machine Learning, 2026c. URL https://arxiv.org/abs/2605.06078. 16 Preprint Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URLhttps://arxiv.org/abs/2504. 12516. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URLhttps://arxiv.org/abs/2410.23218. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large language model-based agents across diverse environments, 2024. Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Jun- jie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang. Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning, 2025. URLhttps://arxiv.org/abs/2509.08755. Teng Xiao, Yige Yuan, Hamish Ivison, Huaisheng Zhu, Faeze Brahman, Nathan Lambert, Pradeep Dasigi, Noah A. Smith, and Hannaneh Hajishirzi. Meta-reinforcement learning with self- reflection for agentic search.arXiv preprint arXiv:2603.11327, 2026. URLhttps://arxiv. org/abs/2603.11327. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 2024. doi: 10.48550/ARXIV.2404.07972. URLhttps://doi.org/10.48550/ arXiv.2404.07972. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report.CoRR, abs/2412.15115, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report.CoRR, abs/2505.09388, 2025. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URL https://arxiv.org/abs/2207.01206. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. Zhiyuan Yao, Yuxin Chen, Zhengxi Lu, Zishan Xu, Yueqing Sun, Yifu Guo, Yuquan Lu, Zhengzhou Cai, Kangning Zhang, Zhuowen Han, Zi-Han Wang, Ziang Ye, Qi Gu, Xunliang Cai, Weiwen Liu, and Yongliang Shen. Skillrise: Agentic reinforcement learning for cross-task skill evolution, 2026. URL https://arxiv.org/abs/2607.26784. 17 Preprint Ziang Ye, Yang Zhang, Wentao Shi, Xiaoyu You, Fuli Feng, and Tat-Seng Chua. Visualtrap: A stealthy backdoor attack on gui agents via visual grounding manipulation, 2025a. URLhttps: //arxiv.org/abs/2507.06899. Ziang Ye, Zhenru Zhang, Yang Zhang, Jianxin Ma, Junyang Lin, and Fuli Feng. Disentangling reasoning tokens and boilerplate tokens for language model fine-tuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the As- sociation for Computational Linguistics: ACL 2025, p. 20939–20957, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/ 2025.findings-acl.1078. URLhttps://aclanthology.org/2025.findings-acl. 1078/. Ziang Ye, Wentao Shi, Yuxin Liu, Yu Wang, Zhengzhou Cai, Yaorui Shi, Qi Gu, Xunliang Cai, and Fuli Feng. Look before you leap: Autonomous exploration for llm agents, 2026. URL https://arxiv.org/abs/2605.16143. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476, 2025a. Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. RLPR: extrapolating RLVR to general domains without verifiers.CoRR, abs/2506.18254, 2025b. Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-R: Training language model agents to reflect via iterative self-training.arXiv preprint arXiv:2501.11425, 2025. URLhttps://arxiv.org/abs/2501.11425. Z.AI. Glm-4.7: Advancing the coding capability, 2025. URLhttps://z.ai/blog/glm-4.7. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for llms, 2023. Yuyuan Zeng, Yufei Huang, Can Xu, Qingfeng Sun, Jianfeng Yan, Guanghui Xu, Tao Yang, and Fengzong Lian. Zero reinforcement learning towards general domains.CoRR, abs/2510.25528, 2025. Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan Xu, Rui Lu, Hongning Wang, Jie Tang, and Yuxiao Dong. Agentrl: Scaling agentic reinforcement learning with a multi-turn, multi-task framework, 2025. URLhttps: //arxiv.org/abs/2510.04206. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges, 2024. URLhttps: //arxiv.org/abs/2401.07339. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025. Zhisong Zhang, Tianqing Fang, Kaixin Ma, Wenhao Yu, Hongming Zhang, Haitao Mi, and Dong Yu. WebRollback: Enhancing web agents with explicit rollback mechanisms. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 2: Short Papers), p. 187–197. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.eacl-short.12. URL https://aclanthology.org/2026. eacl-short.12/. 18 Preprint Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=5NTt8GFjUHkr. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. InProceedings of the 41st International Conference on Machine Learning, volume 235, p. 62138–62160. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/zhou24r.html. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web en- vironment for building autonomous agents.ArXiv, abs/2307.13854, 2023. URLhttps: //api.semanticscholar.org/CorpusID:260164780. Siyu Zhou, Tianyi Zhou, Yijun Yang, Guodong Long, Deheng Ye, Jing Jiang, and Chengqi Zhang. WALL-e: World alignment by neurosymbolic learning improves world model-based LLM agents. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=DorAT49sxj. Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents, 2025. URLhttps://arxiv.org/abs/2506.01716. 19 Preprint A LIMITATIONS ANDFUTUREWORK Our results motivate two directions for extending retrospective exploration. (i) Extending recovery to broader environments.The current framework assumes access to state restoration through snapshots or reset-and-replay; the latter requires deterministic transitions and resettable external state (AppendixD.1). Extending recovery to environments with approximate restoration or irreversible side effects is an important next step. Future work could explore restora- tion controllers and compensating actions, together with checks on the recovered state, to determine when earlier progress can be safely reused. (i) Improving recovery credit and efficiency.ReTRAIL combines task-wide execution feedback with branch advancement, an environment-progress proxy for the utility of retained experience. The current execution return ends at each rewind boundary, while subsequent advancement supervises reflection (AppendixD.6). This design motivates studying credit assignment (Wang et al.,2026b;a) across recovery boundaries, so that rewind timing and target selection can receive direct feedback from the resumed branch. Accounting for restoration and reflection costs, and developing progress estimators for settings with sparse or noisy feedback, could further broaden the framework’s practi- cal use. B ADDITIONALRESULTS ANDANALYSIS B.1 RECOVERYGAINS BYREFERENCE-SOLUTIONLENGTH We group held-out instances by reference-solution length from the initial state. All methods use the same groups, independent of their observed rollout lengths. Reference-solution length and grouping.For Sokoban, the reference is the breadth-first-search solution lengthd 0 (AppendixC.2); for MineSweeper, it is the minimum reveal count under correct deduction. For ALFWorld and ScienceWorld, it is the length of the provided expert or gold action sequence. We use the termreference-solution lengthbecause expert sequences need not be shortest solutions. On ALFWorld, we follow the length boundaries ofWang et al.(2026c): Short (at most 4 actions), Medium (5–7), and Long (more than 7). For each remaining environment, we use the terciles of its held-out reference lengths. The bins therefore describe length within each environment, rather than a common absolute action count across environments. Metrics and aggregation.Within each length bin, we compute the proactive gain∆ pro = T@1− P@1and the reflection-guided retry gain∆ ref = T@3−T@1on the same instances for every method. We then average each gain across the four environments with equal weight; values are success-rate differences in percentage points. Observed pattern.ReTRAIL’s retry gain increases from+4.5on short tasks to+9.2on medium and+19.0on long tasks; LaMer’s corresponding gains are+5.6,+6.0, and+13.2. Thus, ReTRAIL’s retry-gain advantage over LaMer is−1.1,+3.2, and+5.8points in the three groups. Its proactive gain also increases from+1.3to+5.3and+7.8. These descriptive results show larger recovery gains for ReTRAIL on longer-reference tasks, while LaMer has the larger retry gain in the short group. They do not isolate length from other task properties or establish a causal effect of reflection alone. B.2 AVERAGEPERFORMANCEACROSSBENCHMARKS Table4summarizes Table1using the equal-benchmark rule in AppendixC.5. It is computed from the reported, rounded benchmark means, so minor rounding differences are possible. The reported spread is across evaluation seeds, not additional runs. The forward-only results support competitive performance, without establishing statistically significant superiority on each benchmark. 20 Preprint 0 5 10 15 20 Δ ref (p) +3.8 +3.3 +7.4 +5.6 +6.0 +13.2 +4.5 +9.2 +19.0 ShortMedium Long 0 4 8 Δ pro (p) -0.4 -1.1 -0.4 +0.4 +0.7 +0.1 +1.3 +5.3 +7.8 By reference-solution length GiGPOLaMerReTRAIL (Ours) Figure 5:Recovery gains by reference-solution length. The upper row shows reflection-guided retry gain∆ ref = T@3−T@1; the lower row shows proactive gain∆ pro = T@1−P@1. Values are equal-weight averages across the four environments, in percentage points. Short, Medium, and Long use the environment-specific grouping rules above. Table 4:Equal-weight averages across the four benchmarks, computed from the reported benchmark means in Table1and rounded to one decimal place.±is the standard deviation across the three evaluation seeds. These are aggregate point estimates, not additional runs. MethodP@1T@1T@3 Zero-shot 13.1±1.612.5±1.716.3±2.7 ReAct15.8±2.915.9±1.221.2±3.2 Reflexion 14.8±2.517.1±2.620.8±1.4 Rollback 16.2±2.715.8±1.722.8±3.1 GRPO66.3±1.264.7±1.472.3±2.0 RLOO65.2±2.363.7±2.370.9±2.1 GiGPO71.0±1.770.4±2.075.9±1.9 BEACON 71.0±2.569.9±1.176.9±2.0 LaMer67.7±0.968.0±2.277.5±2.4 ReTRAIL 71.2±1.576.8±1.989.7±2.2 C EXPERIMENTALSETUP C.1 BENCHMARKS ANDDATAPARTITIONS All four benchmarks use text interaction and an environment-restoration interface. Table5lists the registered task instances loaded by the training entrypoints. The registered Sokoban variants use 6×6or7×7rooms with two or three boxes; each has 256 training and 128 test entries. The difficulty experiment varies the box count separately. Dataset counts refer to task entries, not repeated rollouts or evaluation seeds. C.2 REWARDS ANDPROGRESSSIGNALS The optimizer uses three quantities: the per-decision rewardδ t , the segment-endpoint scoreS, and the branch scoreV(ξ|s). Table 6gives their definitions. If one decision executes several primitive actions,δ t sums their environment rewards. A decision with no environment transition, including a rewind request, hasδ t = 0. 21 Preprint Table 5:Registered task-instance partitions loaded by the training entrypoints. Counts describe distinct dataset entries, not generated continuations or repeated evaluations. ALFWorld’s selected test partition isvalid_seen; its separately registeredvalid_unseenpartition contains 134 entries. ScienceWorld uses the L0 partition. Environment Train Test Partition information MineSweeper2561286×6boards, 3 mines Sokoban256128 registered room variants share these sizes ALFWorld3,553140train/valid_seen ScienceWorld 3,322 1,661 L0 task/variation pairs Table 6:Training signals by environment.δ t supplies the execution return,Sthe task-wide endpoint score, andVthe branch score for reflection. Derived advantages and branch-score differences are not policy inputs. δ t (per step)S(segment endpoint)V(ξ|s)(branch) MineSweeper native board reward:+0.5 safe reveal,+2.0blank- region expansion,+10.0 board cleared,−0.1mine revealed,−1.0invalid or no-change action cumulativeδon the retained path froms 0 cumulativeδalong the branch Sokobanenvironment-returned transi- tion reward d 0 −d(endpoint)by breadth- first search d 0 −d end , withd end = 0 if solved andd end =d 0 if deadlocked ALFWorldenvironment-returned transi- tion reward fraction of satisfied goal con- ditions at the endpoint maximum fraction of satis- fied goal conditions along the branch ScienceWorld environment-returned transi- tion reward native completion score at the endpoint maximum native completion score along the branch Policy visibilityenvironment-dependent na- tive feedback ScienceWorld score is visible the branch comparison is not a prompt input MineSweeper.Completion and mine revelation end the attempt, with the corresponding terminal reward in Table6applied to that action.Saccumulates rewards along the retained path froms 0 ; restoring a checkpoint also restores this prefix score.Vsums rewards only after the selected anchor, so the common prefix cancels in the branch comparison. Sokoban.Letd 0 be the shortest-solution distance from the initial state andd end the remaining distance at a branch endpoint, both computed by breadth-first search. We useV=d 0 −d end , taking d end = 0for a solved state andd end =d 0 for a deadlocked or unsolvable state. The new-minus-old branch score is therefored − end −d + end . These search distances are evaluator-only and are not shown to the policy. ALFWorld and ScienceWorld.ALFWorld measures the fraction of satisfied goal conditions; Sci- enceWorld uses the native task-completion score. In each environment,Sis the endpoint score andVis the maximum score along the branch, including its starting state. These differ when progress can be undone. ScienceWorld’s native score is visible in its observations; the derived branch-advancement reward is not. C.3 TRAININGCONFIGURATION ANDSAMPLINGBUDGET All trained methods start from Qwen3-4B ( Yang et al.,2025). They share the optimizer, learning- rate schedule, task splits, and decoding settings. Table7lists the configuration. Prompt serialization follows each rollout interface (Appendix E); shared task instructions do not imply identical execution and reflection conversations. 22 Preprint Table 7:Training and evaluation configuration of ReTRAIL. Optimizer settings are shared with the trained baselines. The retrospective-exploration and objective blocks describe ReTRAIL; a trial can contain several forward segments and should not be equated with one baseline trajectory. SettingValue Policy and optimization Learning rate1×10 −6 LR SchedulerConstant Optimizer updates300 Batch size16 Advantage-normalizationε1×10 −6 Discountγ1.0 Retrospective exploration rollout Baseline trajectories per task/update 24 ReTRAIL trials per task/updateB8 Macro-attempts per trial (training)at most 3 Model calls per ReTRAIL trial, execution + reflection60 Model calls per baseline trajectory60 Maximum calls per task/update, baseline / ReTRAIL1440 / 480 Model-initiated rewinds per trialat most 3 Context length16,384 Objective weights State-anchored weightλ state 0.5 Reflection weightλ ref 0.5 Evaluation Evaluation seeds3, reported as the mean Table 8:Units used for rollout and budget accounting. UnitMeaning TrialThe full run under one global model-call budget. Macro-attemptOne attempt until success, irreversible failure, or budget exhaustion; proactive rewinds stay within it. Forward segment Execution between recovery boundaries, or until success, failure, or budget exhaustion. Model callOne execution or reflection generation; a rewind request is an execution call. Primitive action One environment action; a Sokoban execution call can contain several moves. Per task and update, baselines sample 24 trajectories, each allowing 60 model calls. ReTRAIL samples eight trials, each allowing 60 calls and at most three macro-attempts. This aligns nominal attempt slots, with call ceilings of 1,440 for the baseline and 480 for ReTRAIL. These ceilings do not measure realized calls, tokens, or compute. Experience and advantage-normalization groups remain within each task. C.4 EVALUATIONPROTOCOL ANDCALLACCOUNTING Table8distinguishes the units used throughout the paper. Rewinding restores an active path but never refunds calls already made. P@1 disables recovery. T@1 allows proactive recovery within one macro-attempt and stops at the first irreversible failure. T@3 also permits failure-triggered recovery, up to three macro-attempts. Training uses the same attempt cap. Model-initiated rewinds are limited to three per trial. The attempt-scaling experiment varies the macro-attempt cap from one to four. Training and evaluation impose a global ceiling of 60 model calls per trial:p+f≤60, wherep counts execution andfreflection. A proactive recovery uses an execution call to request rewind and a separate reflection call. Forced recovery uses a reflection call after the preceding execution. Recovery starts only when its attempt/rewind allowance permits it and at least two calls remain for 23 Preprint reflection and continuation. Reset-and-replay without model generation adds environment actions but no model calls. The prompt backends also track a branch allowance. MineSweeper and Sokoban restore its saved consumption at the checkpoint; the supplied ALFWorld flow resets its segment-turn counter. These backend rules are distinct from the global call ceiling. A forward-segment cap counts both recovery types, whereas the macro-attempt count increases only after failure-triggered recovery. Algorithm1 takes these rules as configuration inputs. C.5 AGGREGATIONACROSSBENCHMARKS ANDSEEDS Letp (k) s;e be success on benchmarke, under evaluation seedsand protocolk∈ P@1,T@1,T@3. We average equally across the four benchmarks and three seeds: M (k) s = 1 4 4 X e=1 p (k) s;e , M (k) = 1 3 3 X s=1 M (k) s = 1 4 4 X e=1 p (k) e .(13) Each benchmark has equal weight regardless of its instance count. This differs from pooling all test instances. Table 4reports the resulting means together with the standard deviation across the three evaluation seeds. Individual seed values cannot be recovered from these means. D METHODDETAILS This section follows the data flow: restore an environment state, resume execution with branch memory, record the rollout, and construct the training objective. D.1 ENVIRONMENTRESTORATION ANDCHECKPOINTS The environment harness implements rewind. MineSweeper and Sokoban store and restore serializ- able state snapshots. ALFWorld and ScienceWorld reset the task and replay the recorded actions up to the selected point, since their engines do not expose a general snapshot interface. Replay requires deterministic transitions and resettable external state. The active path stores ordered checkpointsK= (C 0 , . . . , C k ), each containing its state, path history, and restoration information. A targetj < kidentifies a path position: identical observations can occur at different positions. RestoringC j truncates the active path but preserves the chronological training archiveΓ. This archive stores all outputs with their original input contexts, segment bound- aries, feedback, and abandoned/resumed branch pairs. Thus an abandoned branch remains available for training after leaving the live prompt. D.2 RECOVERYTRIGGERS ANDREFLECTION A model-initiated recovery begins with an execution output requesting rewind. A forced recovery begins when the controller detects a configured failure. Both generate reflection before restoration, while the abandoned branch is still available. The reflection returns branch memory in<remark> tags. For MineSweeper and Sokoban, a valid model-selected target uses a known-target reflection prompt; forced recovery asks the model to select a checkpoint as well. ALFWorld and ScienceWorld also request a checkpoint in their reflection responses. Thusforcedidentifies who triggers recovery, not a fixed destination immediately before failure. Table 9summarizes the two paths; AppendixC.4 specifies their budget limits. D.3 CONTEXTAFTERRECOVERY The logical historyhis the retained execution path. The actual inputcis the conversation produced by the environment’s message builder. The distinction matters because the backends serialize history differently. 24 Preprint Table 9:Recovery triggers and model calls.Restoration follows reflection, so the abandoned branch remains available when its memory is generated. Checkpoint selection and fallback parsing follow the environment-specific prompt implementation. Model-initiated recoveryForced recovery Triggerexecution output requests an earlier check- point controller observes a configured failure con- dition Targetrequested by the policy; validated before restoration selected through the checkpoint-choice re- flection interface Reflectiona separate generation returning<remark>a separate generation returning<remark> and a target Call accounting one execution call for the request, plus one reflection call one reflection call in addition to preceding execution calls Main-text metric available under T@1 and T@3failure-triggered retries available under T@3 MineSweeper and Sokoban save a conversation at each checkpoint. Recovery restores this prefix, removes the abandoned continuation, and appends a user message with the new branch memory. A previous trailing reflection at the same checkpoint is first made available to the reflection call, then replaced. The next observation is appended as another user message. ScienceWorld also restores a message prefix, but includes branch memories in its subsequent obser- vation. ALFWorld rebuilds the resumed conversation from the system message, restored observa- tion, admissible commands, and branch memories; it does not copy the earlier conversation prefix. Appendix Egives the literal templates and formatting examples. Training always uses the recorded inputc, without adding later reflections to earlier samples. D.4 ROLLOUTALGORITHM Algorithm1records one trial under a frozen behavior policy. Its four blocks follow Section3. The active historyh, checkpointsK, and memoryEguide execution; the archiveΓretains every generation with its original context and feedback. A segmentσgroups execution decisions between recovery boundaries, andn=p+fcounts all model calls. Invalid outputs include unparseable actions and disabled or out-of-range rewind requests; they con- sume a call and receive feedback without changing the environment. Checkpoints include states reached on failure. Recovery requires allowance for reflection and continuation (Appendix C.4). Target parsing follows AppendixD.2; memory and context updates follow AppendixD.3. Restor- ing a checkpoint never truncatesΓor refunds calls. D.5 STATE-ANCHOREDGRPO Each execution decision is one training sample. Its remaining segment supplies the return used to score that decision, while the loss applies only to its own generated output. D.5.1 DECISIONSAMPLES ANDRETURNS For tasku, collectBtrialsΓ i B i=1 and their forward segmentsS(u). Index decisiontin segment σ i;m byq= (i, m, t), and letT i;m be the segment length: I(u) =(i, m, t) :σ i;m ∈ S(u),0≤t < T i;m ,ξ q =σ i;m [t:T i;m ].(14) The continuationξ q ends with its original segment, at success, failure, rewind, or budget exhaustion. A proactive rewind is the segment’s final execution decision. Record its starting states q , logical retained historyh q , and actual input context. LetY q contain only the generated reasoning and action or rewind request for decisionq. The indexed collectionC(u) = (ξ q ) q∈I(u) contains return records, not extra sequences to train again. For example, an old segments 0 →s 1 →s 2 →s 3 that ends in a rewind tos 1 supplies a continuation froms 1 , even though the segment began ats 0 . This can be compared with the new continuation from 25 Preprint Algorithm 1Retrospective exploration for one task Require:tasku, frozen policyπ old , protocolB, call limitC max = 60 1:reset task; initializeh,K= (C 0 ),E=∅and protocol counters 2:Γ←∅;σ←∅;n←0 3:whilen < C max andBpermits executiondo 1. ExecuteEquation2 4:c←BUILDCONTEXT(u, h,E,K);y∼π old (·|c);n←n+ 1 5:record(c, y)inΓ; append the execution decision toσ 6:ifyis invalidthenrecord feedback;continue 2. Detect recoveryEquation3 7:ifyrequests rewindthen 8:retain the requested target▷model-initiated recovery 9:else 10:execute the parsed action; updateh,K; record rewards and observations inΓ 11:iftask solvedthen break 12:ifno failure triggerthen continue▷otherwise, forced recovery 13:end if 14:closeσinΓ 15:ifn+ 2> C max orBdisallows recovery/continuationthen break 3. ReflectEquation4 16:buildc ref from the active branch, memory, and any requested target 17:z ∼ π old ( ·| c ref ) ; n ← n + 1 ; record ( c ref , z ) in Γ 18:resolve(j, e)using any requested target,z, and the configured fallback 19:ifno valid target0≤j <|K|−1remainsthen break 4. Restore and resumeEquation6 20:record abandoned suffixβ=τ j:k and recovery link inΓ, wherek=|K|−1 21:restoreC j ; truncateh,Kto the retained prefix 22:updateEand resumed context withe; update branch/protocol counters 23:start a new emptyσlinked to this recovery 24:end while 25:close any nonempty open segment inΓ;returnΓ s 1 . The two decisions share a state and logical prefix, but retain their original generation contexts and potentially different branch memories. D.5.2 EXECUTIONADVANTAGES The return from decisionqand its comparison group are G q = T i,m −1 X `=t γ `−t δ i;m;` ,H u (s, h) =q∈ I(u) :s q =s, h q =h,(15) Hereδ i;m;` sums environment rewards produced by execution decisionℓ. A rewind request has zero transition reward; reflection calls do not enter this return. Group members share a task, environment state, and retained history. Their accumulated experience and remaining global budget may differ, so the comparison does not hold the full policy input fixed. The macro signal uses endpoint progressS i;m , with each segment contributing once to normalization regardless of its length. For any decision in that segment, the main text’sS q equalsS i;m . The combined advantage is A state q = Norm H u (s q ;h q ) (G q ), A cont q = Norm S(u) (S i;m ) | z A macro i,m +λ state A state q . (16) Normsubtracts the group mean and divides by its standard deviation plusε. Singleton and constant- score groups have zero advantage; empty groups contribute no samples. These rules apply separately to macro, state, and reflection groups within each task. When a decision has no local comparison, its macro term remains available. 26 Preprint All decisions in a segment share its macro advantage but have their own local advantages. Each advantage applies only to that decision’s output. A reward can contribute to several earlier returns without duplicating output tokens in the loss. Repeated visits are separate generation events. D.5.3 REFLECTIONREWARDS ANDELIGIBILITY At a recovery point, reflection is scored by the improvement of the resumed branch over the aban- doned suffix: r ref m =V(σ m+1 |s j m )−V(β m |s j m ).(17) The evaluator computesVusing the environment-specific definitions in AppendixC.2. LetY ref i;m be the reflection output. Eligible recoveries are R(u) = (i, m) :Y ref i;m ̸=∅, T i;m+1 >0, v − i;m , v + i;m are defined .(18) Herev − i;m andv + i;m are the abandoned and resumed branch scores. A missing resumed segment has length zero. Eligibility requires an observed target, not positive advancement or eventual success. Both recovery types are normalized together overR(u)within the same task. If no continuation is recorded, the reflection tokens receive no loss; earlier execution outputs keep their usual contribu- tion. D.5.4 LOSSMASKS ANDPOLICYUPDATE Execution and reflection use disjoint output masks. A rewind request made during execution be- longs toY q ; a checkpoint choice generated within a reflection belongs to that reflection. Prompts, observations, copied memories, and retained-prefix tokens are masked out. LetY ref (u)contain the eligible reflection tokens, with advantagesA ref normalized overR(u). The task objective is J u (θ) = 1 max1,|I(u)| X q∈I(u) J GRPO θ;Y q , A cont q +λ ref J GRPO θ;Y ref (u), A ref , (19) J GRPO is the token-averaged clipped policy surrogate. The execution term first averages tokens within a decision, then averages over decisions. An empty reflection set or a term with no contribut- ing tokens gives zero loss. Both current and behavior policies use each token’s original context. The batch combines task objectives without pooling their normalization groups. Algorithm2summarizes the update. All rollouts are collected before optimization; the behavior policy, data, and advantages remain fixed throughout the corresponding update. Algorithm 2 One State-Anchored GRPO update with task-local groups Require:nonempty task batchU, current parametersθ,Btrials per task, weightsλ state , λ ref 1:θ old ←θ▷freeze the behavior policy for this batch 2:foreach tasku∈Udo 3:sampleΓ i B i=1 withπ θ old using Algorithm1 4:collect segmentsS(u), decisionsI(u), and return recordsC(u) 5:retain each decision’s context, behavior log-probabilities, and output mask 6:computeS i,m once per segment and return-to-goG q for each decision 7:normalizeS i,m σ i,m ∈S(u) to obtainA macro i,m 8:group visits by(s q , h q )withinu; computeA state q andA cont q by Equation16 9:form the eligible recovery setR(u)in Equation18 10:foreach(i, m)∈R(u)do 11:r ref i,m ←V(σ i,m+1 |s j i,m )−V(β i,m |s j i,m ) 12:end for 13:normalizer ref i,m (i,m)∈R(u) to obtainA ref i,m 14:formJ u (θ)with each execution output once (Equation19) 15:end for 16:updateθfrom task objectives; keepθ old , data, and advantages fixed 27 Preprint D.6 CREDITASSIGNMENT AT AREWIND: ANANALYTICEXAMPLE Consider actionawith rewardd, followed by proactive rewindρ. Their decision indices areq a , q , and their shared macro advantage isM= Norm S(u) (S i;m ). Rewind ends the segment with zero transition reward: G q a =d+γ0 =d,G q ρ = 0, A cont q a =M+λ state Norm H u (s q a ;h q a ) (d),A cont q ρ =M+λ state Norm H u (s q ρ ;h q ρ ) (0). (20) Each advantage trains its own output. A singleton rewind group gives total advantageM. With group returns0, g,g >0, its local contribution is−λ state (g/2)/(Std0, g+ε). The total remains positive ifMis large enough. Comparison with negative returns can instead give a positive local term. With abandoned scoreband resumed scoreb+ ∆, reflection receives r ref i;m = (b+ ∆)−b= ∆,A ref i;m = Norm R(u) (∆).(21) This trains the whole reflection, without separating target and memory credit. Normalization can give positive advancement a negative advantage if other reflections improve more; a singleton group gives zero. Execution returns stop at rewind. With the pre-rewind segment and group statistics fixed, different proactive targets receive the same execution advantage regardless of later outcomes. Post-recovery feedback applies to the separate reflection response. E PROMPTTEMPLATES This section groups the execution, reflection, and observation templates by environment. Reflection is a separate model call. Appendix D.3explains how each backend assembles the resumed context; AppendixC.4defines the global call ceiling. Literal references to “model action steps” below do not allow extra calls. The observation boxes are formatting examples, not measured trajectories. They use checkpointC 1 , one action used, a global action allowance of 60, and a branch allowance of 20. Sokoban allows three moves per turn; ScienceWorld uses score zero. Invalid-action feedback, prior memories, and recent-event blocks are omitted from these examples and are added by the builders when present. E.1 MINESWEEPER Therefacted,undiscounted_final, andgigpoflows share the following system and re- flection templates. MineSweeper Rewind System Prompt You are an expert agent operating in the Minesweeper game with REWIND capability. You will be given a board_size by board_size board, with n_mines hidden mines. Rows and columns are indexed from 1 to board_size. The task is guaranteed to be solvable. You must never conclude that the task is unsolvable, ,→hopeless, or impossible, and you must never give up. Even if some branches appear ,→unpromising or lead to dead ends, you can still learn from them, rewind when necessary, ,→and continue exploring. Your environment state may move backward through rewind, but your knowledge must keep ,→moving forward. Use each failed attempt to refine your understanding, avoid repeating ,→the same mistakes, and make better decisions in future branches until the task is solved ,→. # Cell States - Unopened cells (?): hidden cells that may contain mines. - Blank cells (.): opened safe cells with no neighboring mines. - Numbered cells (1-8): opened safe cells showing how many neighboring cells contain mines. - Mine cells ( * ): opened mine cells; revealing one loses the game. # Goal Reveal all Unopened and non-mine cells without hitting any mine. 28 Preprint # Reveal Rules - Choose exactly ONE unopened cell (?) to reveal per model action step. - Blank cells auto-cascade to reveal contiguous blanks and bordering numbered cells, you ,→did not need to reveal the blank cells. - Numbered cells reveal only that single cell. - Mine cells end the current branch in failure. # Checkpoints and Rewind Capability You operate under two budgets, and you should manage them differently. The first is a global action budget: step_budget model action steps for the entire ,→episode. Each model action step reveals exactly one cell. This is a hard ceiling shared ,→across the whole episode. You may use it for exploration, trial and error, and ,→ultimately solving the task. This budget is accumulated globally and is never refunded ,→by rewinding. The second is a task-solve budget: branch_attempt_budget model action steps for making ,→progress along the current active path. You should aim to solve the task within this ,→allowance. Unlike the global action budget, this budget is tied to the checkpoint path. ,→When you rewind to checkpoint C_j, the task-solve budget is rolled back to the amount ,→already consumed at C_j. In other words, rewinding to C_j refunds the task-solve steps ,→spent after C_j, while everything consumed up to C_j remains spent. Use the global action budget and the task-solve budget together to balance exploration, ,→learning from failed branches, trial and error, and direct progress toward solving the ,→task. Checkpoints track the current path as C_0, C_1, ..., C_k: - C_0 is the initial board state. - C_i is the state reached after i executed reveal actions on the current path. Once you have executed at least one reveal action, you may travel back to any earlier checkpoint: -`<action>rewind to C_j</action>`, where 0 <= j < the current checkpoint index. -`<action>rewind to j</action>`is equivalent. Rewinding restores the environment to checkpoint C_j and rolls your branch attempt budget back to its value at C_j, but what you learned on the abandoned branch is not lost it persists as memory. you can rewind when you find * a better plan to start at C_j * recognize the current branch is stuck(an action trap, an invalid loop, a dead end, or ,→unproductive exploration) * You are in good path to success but find insufficient budget to make progress(the task- ,→solve budget is exhausted) and want to move to current state in less turns and context. # Response Format - First reason step-by-step about numbered clues, candidate mines, and safe cells. - Then reveal one unopened (?) cell(number cell and Blank cells is safe but make no ,→progress):`<action>(row, col)</action>`. - OR rewind to a previous checkpoint:`<action>rewind to C_j</action>`. Only the final`<action>...</action>`tag will be executed. MineSweeper Known-Target Reflection Prompt You are an expert Minesweeper player reflecting on a branch attempt. Board: board_size by board_size, n_mines mines. Rows/columns indexed 1 to board_size ,→. # Cell States - Unopened cells (?): hidden cells that may contain mines. - Blank cells (.): opened safe cells with no neighboring mines. - Numbered cells (1-8): opened safe cells showing how many neighboring cells contain mines. - Mine cells ( * ): opened mine cells; revealing one loses the game. # Reveal Rules - Choose exactly ONE unopened cell (?) to reveal per model action step. - Blank cells auto-cascade to reveal contiguous blanks and bordering numbered cells, you ,→did not need to reveal the blank cells. - Numbered cells reveal only that single cell. - Mine cells end the current branch in failure. # Core rollback semantics The game was at step C_rewind_from_step when you decided to rewind back to step C_ ,→rewind_to_step. 29 Preprint Thus the env state returns to step C_rewind_to_step, Your goal is to push the knowledge ,→state forward. compress the branch attempt into a small branch memory that record what have you done that ,→lead to what result, what have you learned, what you should avoid and what you should do next. Do not preserve the full trajectory. Preserve only information useful for future decisions the core target is to help move closer to the target during the next new attempt or try ,→explore other states. # Context rewind_target_instruction # Rewind trigger rewind_reason # Relevant checkpoint states checkpoint_context # State at C_rewind_from_step before rewinding current_observation # Branch history from C_rewind_to_step to C_rewind_from_step history_after_rewind # Your task Reflect on what went wrong and produce a concise branch memory for the next attempt based ,→on the actions you have done and the environment feedback. Focus on: - revealed mine cells that should not be repeated; - invalid/already-open/out-of-range reveals; - numbered clue constraints and specific cells that are likely mines or safe; - whether the branch should restart before or after a useful clue reveal; - a concrete improved plan from the checkpoint you return to. you could move faster as you ,→already know some of the state and the knowledges. Include a compact branch memory inside <remark> </remark> tags. Use this structure, replacing C_j with the checkpoint you choose or confirm: <remark> # History Attempt from C_rewind_to_step to C_rewind_from_step State at C_rewind_to_step: rewind_to_observation State at C_rewind_from_step: rewind_from_observation Actions I have done: ... What's bad about the actions: ... How to move forward: ... Next new plan from C_j that can move faster and avoid the same mistakes: ... </remark> MineSweeper Observation-Message Example (user role) Current observation at checkpoint C_1 (segment turn 2): observation Global action budget used: 1/60 (remaining: 59). Task-solve budget used on this branch: 1/20 (remaining: 19). Current checkpoint: C_1. Valid model rewind targets: C_0. Choose exactly one unopened cell to reveal inside one final action tag, e.g. <action>(2, 3) ,→</action>, or travel back: <action>rewind to C_j</action>. E.2 SOKOBAN Therefacted,undiscounted_final, andgigpoflows share the following system and re- flection templates. Sokoban Rewind System Prompt You are an expert Sokoban agent operating with REWIND capability. The task is guaranteed to be solvable. You must never conclude that the task is unsolvable, ,→hopeless, or impossible, and you must never give up. Even if some branches appear ,→unpromising or lead to dead ends, you can still learn from them, rewind when necessary, ,→and continue exploring. 30 Preprint Your environment state may move backward through rewind, but your knowledge must keep ,→moving forward. Use each failed attempt to refine your understanding, avoid repeating ,→the same mistakes, and make better decisions in future branches until the task is solved ,→. # Symbols and Their Meaning - Walls (`#`): These block movement. - Floor (`_`): Open spaces where you can walk and move boxes. - Targets (`O`): The spots where boxes need to go. - Boxes (`X`): These are what you need to push onto the targets. - Player (`P`): That's you! - Box on Target (``): A box successfully placed on a target. - Player on Target (`S`): You standing on a target. # Goal Push all boxes (`X`) onto targets (`O`). # Rules - Admissible actions: ["up", "down", "left", "right"] - You can only push one box at a time. You can't pull boxes. - You can't walk through or push boxes into walls or other boxes. - Avoid pushing boxes into corners or against walls where they can't be moved. - A blocked move may consume budget without improving the state, so avoid repeating it. # Checkpoints and Rewind Capability You operate under two budgets, and you should manage them differently. The first is a global action budget: step_budget model action steps for the entire ,→episode. Each model action step may contain up to num_actions_per_turn primitive ,→Sokoban moves. This is a hard ceiling shared across the whole episode. You may use it ,→for exploration, trial and error, and ultimately solving the task. This budget is ,→accumulated globally and is never refunded by rewinding. The second is a task-solve budget: branch_attempt_budget model action steps for making ,→progress along the current active path. You should aim to solve the task within this ,→allowance. Unlike the global action budget, this budget is tied to the checkpoint path. ,→When you rewind to checkpoint C_j, the task-solve budget is rolled back to the amount ,→already consumed at C_j. In other words, rewinding to C_j refunds the task-solve steps ,→spent after C_j, while everything consumed up to C_j remains spent. Use the global action budget and the task-solve budget together to balance exploration, ,→learning from failed branches, trial and error, and direct progress toward solving the ,→task. Checkpoints track your progress along the current path as C_0, C_1, ..., C_k: - C_0 is the initial state. - C_i is the state reached after i executed model action steps on this path. Once you have executed at least one model action step, you may travel back to any earlier checkpoint: -`<action>rewind to C_j</action>`, where 0 <= j < the current checkpoint index. -`<action>rewind to j</action>`is equivalent. Rewinding restores the environment to checkpoint C_j and rolls your branch attempt budget back to its value at C_j, but what you learned on the abandoned branch is not lost it persists as memory. you can rewind when you find * a better plan to start at C_j * recognize the current branch is stuck(an action trap, an invalid loop, a dead end, or ,→unproductive exploration) * You are in good path to success but find insufficient budget to make progress(the task- ,→solve budget is exhausted) and want to move to current state in less turns and context( ,→more actions each turn) # Response Format - First reason step-by-step about the current state and box/target geometry. - Then choose up to num_actions_per_turn moves inside one final action tag, e.g. `<action>up, right, right</action>`. - OR rewind to a previous checkpoint:`<action>rewind to C_j</action>`. Only the final`<action>...</action>`tag will be executed. Sokoban Known-Target Reflection Prompt You are an expert Sokoban player independently reflecting on a branch attempt. # Symbols and Their Meaning - Walls (`#`): These block movement. - Floor (`_`): Open spaces where you can walk and move boxes. - Targets (`O`): The spots where boxes need to go. 31 Preprint - Boxes (`X`): These are what you need to push onto the targets. - Player (`P`): The agent. - Box on Target (``): A box successfully placed on a target. - Player on Target (`S`): Agent standing on a target. # Rules - Admissible actions: ["up", "down", "left", "right"] - You can only push one box at a time. You can't pull boxes. - You can't walk through or push boxes into walls or other boxes. - Avoid pushing boxes into corners or against walls where they can't be moved. - A blocked move may consume budget without improving the state, so avoid repeating it. # Core rollback semantics The game was at step C_rewind_from_step when you decided to rewind back to step C_ ,→rewind_to_step. Thus the env state returns to step C_rewind_to_step, Your goal is to push the knowledge ,→state forward. compress the branch attempt into a small branch memory that record what have you done that ,→lead to what result, what have you learned, what you should avoid and what you should do next. Do not preserve the full trajectory. Preserve only information useful for future decisions the core target is to help move closer to the target during the next new attempt or try ,→explore other states. # Context rewind_target_context # Rewind trigger rewind_reason # Relevant checkpoint states checkpoint_context # State at C_rewind_from_step before rewinding current_observation # Branch history from C_rewind_to_step to C_rewind_from_step history_after_rewind # Prior reflection outputs reflection_history_context # Your task Reflect on the branch attempt and produce a concise branch memory for the next new attempt ,→direction from the checkpoint you return to. Focus on: - Invaild actions that environment not accepted or skipped by the environment or stuck by ,→the walls or boxes; - failure reason or likely bad push/walk sequence; - boxes pushed into corners, walls, dead zones, or away from reachable targets; - blocked/no-op moves or invalid loops that should not be repeated; - useful spatial facts about player access, box positions, target positions, and required ,→order; - the task is always solvable, so you should always find a concrete improved plan from the ,→checkpoint you return to. Include a compact branch memory inside <remark> </remark> tags. Use this structure, replacing C_j with the selected checkpoint: <remark> # History Attempt from C_rewind_to_step to C_rewind_from_step State at C_rewind_to_step: rewind_target_context State at C_rewind_from_step: current_observation Actions I have done: ... What's bad about the actions: ... How to move forward: ... Next new Plan from C_j that Must be different from the actions I have done and independently and carefully think about and forward towards the goal and to a different ,→state or to a good observed state but in less turns and context(more actions each turn) ,→: ... </remark> 32 Preprint Sokoban Observation-Message Example (user role) Current observation at checkpoint C_1 (segment turn 2): observation Global action budget used: 1/60 (remaining: 59). Task-solve budget used on this branch: 1/20 (remaining: 19). Current checkpoint: C_1. Valid model rewind targets: C_0. Choose up to THREE moves inside one final action tag, e.g. <action>up, right</action>, or ,→travel back: <action>rewind to C_j</action>. E.3 ALFWORLD Reward variants wrap the same rewind-choice flow. The system prompt below extends the house- hold prompt with explicit rewind instructions. The user-message builder adds checkpoints, budgets, admissible commands, and branch memories. Actions and rewind requests use anactioncode block. ALFWorld Rewind System Prompt You are an expert agent controlling a robot in the ALFWorld household environment with ,→REWIND capability. Your task is to complete household tasks by navigating rooms and ,→interacting with objects. The task is guaranteed to be solvable. You must never conclude that the task is unsolvable, ,→hopeless, or impossible, and you must never give up. Even if some branches appear ,→unpromising or lead to dead ends, you can still learn from them, rewind when necessary, ,→and continue exploring. Your environment state may move backward through rewind, but your knowledge must keep ,→moving forward. Use each failed attempt to refine your understanding, avoid repeating ,→the same mistakes, and make better decisions in future branches until the task is solved ,→. Action Commands: For an environment action, choose one of the following, strictly following the command ( ,→argument) format. To rewind instead, use the checkpoint command described below. ## Navigation & Observation: - look: Look around your current location to get more details. - inventory: Check the object you are currently holding (you can only hold one). - go to (receptacle): Move to a receptacle (e.g., table, fridge, sink). ## Interacting with Receptacles: - open (receptacle): Open a receptacle. - close (receptacle): Close a receptacle. ## Interacting with Objects: - take (object) from (receptacle): Pick up an object from a receptacle. - put (object) in/on (receptacle): Place the object you are holding into or onto a ,→receptacle. - examine (object): Examine an object closely to learn its properties. ## Changing Object States: - heat (object) with (receptacle): Heat an object with a device (e.g., microwave). - cool (object) with (receptacle): Cool an object with a device (e.g., fridge). - clean (object) with (receptacle): Clean an object with a device (e.g., sink). - slice (object) with (object): Slice an object using a sharp object (e.g., knife). Important Rules 1. You must first navigate to a location before interacting with objects there 2. You can only hold one object at a time 3. Some objects need to be heated, cooled, or cleaned before placing them 4. Always check admissible commands to see what actions are currently valid # Checkpoints and Rewind Capability You operate under two budgets, both shown in each observation message. The first is a global action budget for the entire episode. Each model action step requests ,→exactly one environment action or one rewind. This is a hard ceiling shared across ,→exploration, trial and error, and solving the task. This budget is accumulated globally ,→and is never refunded by rewinding. The second is a current-branch budget for the current execution segment. You should aim to ,→complete the task within the remaining allowance. After a rewind, a new segment starts ,→from the restored checkpoint with a fresh current-branch allowance. The global action 33 Preprint ,→budget remains spent. Always use the remaining budgets reported in the latest ,→observation. Use these budgets together to balance exploration, learning from failed branches, and ,→direct progress toward solving the task. Checkpoints track the current path as C_0, C_1, ..., C_k: - C_0 is the initial household state. - C_i is the state reached after i executed environment actions on the current path. Once you have executed at least one environment action, you may travel back to an earlier ,→checkpoint listed under Valid rewind targets: ```action rewind to C_j ``` where 0 <= j < the current checkpoint index. Rewinding restores the environment to checkpoint C_j, including your location, inventory, ,→object locations, and object states. Changes made after C_j are undone, but what you ,→learned on the abandoned branch persists as branch memory. Use this memory to guide ,→future actions; do not assume that an object remains held, moved, heated, cooled, or ,→cleaned unless that was already true at C_j. You can rewind when: - You find a better plan that starts at an earlier checkpoint. - The current branch is stuck in repeated invalid actions, no-effect actions, a dead end, ,→or unproductive exploration. - You chose the wrong object or receptacle, or performed navigation, open/take/put, or ,→state-change steps in an unhelpful order. - The current-branch budget is nearly exhausted and knowledge from this branch can help you ,→complete the task more efficiently from an earlier checkpoint. Retain useful progress when choosing a checkpoint. Use learned object locations, ,→prerequisites, and failed-action outcomes to form an improved plan from the restored ,→state. Response Format Think through your approach step by step, then provide exactly one environment action or ,→rewind command in the following format: <think> Your reasoning and analysis here... </think> ```action <your action here> ``` you must adhere to the response format strictly or your action will not be executed: For example: <think> I need to go to the countertop to get the knife. </think> ```action go to countertop 1 ``` To rewind instead: <think> I chose the wrong object. I will return to C_2 and use what I learned to choose the correct ,→object. </think> ```action rewind to C_2 ``` Use this rewind example only when C_2 is listed as a valid target. Always provide exactly ONE action code block at a time, containing either one environment ,→action or one rewind command. Do not combine them in the same response. ALFWorld Rewind Reflection Prompt You are reflecting on a failed ALFWorld branch. # Rewind trigger rewind_reason # Current checkpoint The branch is currently at C_current_position. 34 Preprint # Valid rewind targets valid_targets # Branch history branch_history Write a compact memory that helps the next attempt avoid the failed branch. Include wrong locations, wrong object choices, missed open/take/put/state-change steps, and useful facts learned from observations. Return memory inside <remark>...</remark>, then end with exactly one action block: ```action rewind to C_j ``` ALFWorld Observation-Message Example (user role) Current observation at checkpoint C_1: observation Admissible commands: - admissible_command Global action budget used: 1/60 (remaining: 59). Current-branch budget used: 1/20 (remaining: 19). Valid rewind targets: C_0. Choose exactly one environment action in```action ...```, or travel back with```action rewind to C_j ```. E.4 SCIENCEWORLD Reward variants share the following system and reflection templates. The initial user message in- cludes the task description. Later observations include admissible actions, budgets, branch memo- ries, and the native score. ScienceWorld Rewind System Prompt You are an expert autonomous agent operating in the ScienceWorld environment, a text-based ,→virtual world for accomplishing tasks from the elementary science curriculum, with ,→REWIND capability. You interact by issuing one free-text command per turn. Each turn you are shown the current ,→observation and the admissible actions (an action template list plus the objects that , → can replace OBJ). Typical actions include: look around, open OBJ, close OBJ, activate ,→OBJ, deactivate OBJ, go to LOC, teleport to LOC, pick up OBJ, move OBJ to OBJ, focus on ,→OBJ, read OBJ, wait, task. Important rules: 1. You must navigate to a location before interacting with the objects there. 2. Replace OBJ / LOC placeholders with concrete objects and locations from the observation. 3.`focus on OBJ`signals the object the task is about use it deliberately, it can end the ,→task. # Checkpoints and Rewind Capability You operate under two budgets. The first is a global action budget: step_budget model action steps for the entire episode. This budget is never refunded by rewinding. The second is a task-solve budget: branch_attempt_budget model action steps for progress along the current active checkpoint path. When you rewind to checkpoint C_j, this task-solve budget rolls back to the amount already consumed at C_j. Knowledge from abandoned branches persists as memory. Checkpoints track the current path as C_0, C_1, ..., C_k: - C_0 is the initial environment state. - C_i is the state reached after i executed actions on the current path. Once you have executed at least one action, you may travel back to any earlier checkpoint: -`<action>rewind to C_j</action>`, where 0 <= j < the current checkpoint index. -`<action>rewind to j</action>`is equivalent. 35 Preprint Use rewind when a branch reaches a dead end, becomes inconsistent with earlier observations, repeats ineffective actions, or when a better plan should restart from an earlier state. # Response Format - First reason step-by-step about the task, the observation, and the admissible actions. - Then give exactly ONE action:`<action>focus on door</action>`. - OR rewind to a previous checkpoint:`<action>rewind to C_j</action>`. Only the final`<action>...</action>`tag will be executed. ScienceWorld Rewind Reflection Prompt You are an expert ScienceWorld agent reflecting on a branch attempt. # Core rollback semantics The current branch is at C_rewind_from_step. The model must choose or confirm the rewind target before the environment is restored. The knowledge state must move forward: compress the branch attempt into a small branch ,→memory. Do not preserve the full trajectory. Preserve only information useful for future decisions. rewind_target_instruction # Rewind trigger rewind_reason # Valid rollback targets valid_targets # Relevant checkpoint states checkpoint_context # State at C_rewind_from_step before rewinding current_observation # Branch history from C_history_start_step to C_rewind_from_step history_after_rewind # Prior reflection outputs reflection_history_context # Your task Reflect on what went wrong and produce a concise branch memory for the next attempt. Focus on: - wrong locations, wrong object choices, or ineffective actions that should not be repeated ,→; - missed navigation, open/activate/move/focus, or state-change steps; - useful facts learned from observations about objects and locations; - whether the branch should restart before or after a useful action; - a concrete improved plan from the checkpoint you return to. Include a compact branch memory inside <remark> </remark> tags. Use this structure, replacing C_j with the checkpoint you choose or confirm: <remark> Failure reason: ... Useful facts learned: ... Avoid repeating: ... Next new plan from C_j: ... </remark> End with exactly one final action tag selecting the checkpoint to restore: <action>rewind to C_j</action> ScienceWorld Observation-Message Example (user role) Current observation at checkpoint C_1 (segment turn 2): observation Score so far: 0/100. Admissible actions: available_actions Global action budget used: 1/60 (remaining: 59). Task-solve budget used on this branch: 1/20 (remaining: 19). Current checkpoint: C_1. Valid model rewind targets: C_0. 36 Preprint Choose exactly one action inside one final action tag, e.g. <action>focus on door</action>, ,→or travel back: <action>rewind to C_j</action>. 37 Preprint F CASESTUDIES These cases illustrate failure-triggered recovery in MineSweeper and proactive recovery in Sokoban. Each trace follows the abandoned branch, the retained reflection, and the resumed actions. Logged cases and protocol.The cases are selected validation rollouts from archived training vari- ants, not runs underlying the final evaluation tables. Their prompts allow 60 action turns globally and 20 on the active path; the main protocol instead counts execution and reflection together within 60 model calls. We therefore report action turns, primitive actions, and reflection calls separately. Checkpoint indices can be reused after rewind. Boxed excerpts preserve the logged output, with omissions marked[...]; surrounding text is our analysis. Outcomes were checked against rollout metrics and independent state replay. F.1 MINESWEEPER: RECOVERINGAFTER AFAILEDREVEAL The task is to reveal 33 safe cells on a6×6board with three mines. Reveals at(4,1),(5,1), and (6,1)produce clues 2, 2, and 1. AtC 3 , 32 safe cells are open. The policy then reveals(6,2), hits a mine, and ends the branch. It had acknowledged that(5,2)was unopened but incorrectly treated(6,2)as the only remaining cell. Valid prefix Checkpoint C3 123456 1 ...... 2 11.... 3 ?21... 4 2?1... 5 2?2... 6 1?1... 32 safe cells open Three reveals retained Failed reveal Checkpoint C4 123456 1 ...... 2 11.... 3 ?21... 4 2?1... 5 2?2... 6 1*1... Mine hit at (6,2) Branch terminates State restored; knowledge kept Restored checkpoint C3 123456 1 ...... 2 11.... 3 ?21... 4 2?1... 5 2?2... 6 1?1... Board restored Mine knowledge retained Successful continuation Replayed final board 123456 1 ...... 2 11.... 3 ?21... 4 2?1... 5 222... 6 1?1... All 33 safe cells open Reveal (5,2) completes the board reveal (6,2) restore C3 reveal (5,2) Blue outlines: retained reveals Dashed outline: mine known from memory Figure 6:MineSweeper: restoring the board while retaining the failed reveal’s outcome.Re- vealing(6,2)ends the branch. The policy selectsC 3 , preserving all three preceding safe reveals, and carries the mine location into the resumed execution. It then reveals(5,2), completing the board. Coordinates are one-indexed. Observed branch and recovery Valid prefix,C 0 →C 3 (4,1)→2;(5,1)→2;(6,1)→1. Erroneous reveal,C 3 →C 4 (6,2)→mine hit; the environment recordsdone=True, won=False. RecoveryThe failure triggers reflection. The policy selectsC 3 , restoring the board before the mine hit while preserving the three safe reveals. Policy reflection and selected checkpoint (excerpts) <remark> [...] But (6,2) was revealed and it's a mine that means (5,2) is safe. [...] Revealing (5,2) will complete all reveals. Final action: <action>(5,2)</action> 38 Preprint [...] </remark> <action>rewind to C_3</action> The next execution prompt contains the entire remark. Although restoration hides(6,2)again, the policy remembers the mine and chooses the other cell: Resumed policy output (excerpts) So (6,2) is a mine [...] [...] Therefore, the only cell left to reveal is (5,2). Revealing (5,2) will complete all safe reveals. Final action: <action>(5,2)</action> The final reveal completes the board. The run uses five reveals and one reflection call, without repeating a reveal after restoration. A single-episode rollout on the same instance follows the same four-action prefix and ends at the mine hit. What this case shows.The board returns toC 3 while the policy retains knowledge from the failed branch and preserves the three safe reveals. The original error was avoidable: the visible clues already determined the mine layout. Although parts of the reflection are redundant or inconsistent about completion, its final mine identification and recommended action agree with the successful continuation. F.2 SOKOBAN: LEAVING ALOOPWHILEPRESERVING ASUBGOAL The task is to place two boxes on targets in a7×7room. AtC 4 , one box is on target at(4,1), the player is at(4,4), and the other box must move from(3,4)to(3,2). Four successiveRight, Leftcommands return the player to the same position without moving either box. Two further Leftcommands also make no box progress. AtC 10 , the policy requestsrewind to C_4before any terminal failure. Valid prefix Checkpoint C4 0123456 0 1 2 3 4 5 6 First box placed Four execution turns retained Unproductive suffix Checkpoint C10 0123456 0 1 2 3 4 5 6 No box progress 4 x (Right, Left); Left; Left Prefix preserved Restored checkpoint C4 0123456 0 1 2 3 4 5 6 First box retained Revised route from C4 Successful continuation Replayed final board 0123456 0 1 2 3 4 5 6 Both boxes placed Right, Up, Left; then Left six turns loop rewind to C4 two turns 4 moves Circle: player Outline circle: target Filled square: box Blue frame: retained subgoal Figure 7:Sokoban: replacing an unproductive continuation while preserving a solved subgoal. The policy returns fromC 10 toC 4 , retaining the first box on its target. The final reflection identifies how to reach the right side of the remaining box; the next two action turns execute that route and complete the task. Coordinates follow the zero-indexed environment rendering. 39 Preprint Observed branch and model-initiated rewind Retained prefix,C 0 →C 4 Four action turns place the first box on(4,1)and leave the player at (4,4). Repeated loop, C 4 →C 8 Four Right, Left commands each return to the same board state; neither box moves. Further detour,C 8 →C 10 TwoLeftcommands move the player to(4,2). Both box positions remain unchanged. Policy decision atC 10 <action>rewind to C_4</action>. The restored state keeps the first box on target and the four-turn prefix. After initially misreading the grid, the reflection identifies a route to the right side of the remaining box: move from(4,4)through(4,5)to(3,5), then push left twice: Final part of the policy reflection (excerpts) <remark> [...] So the correct plan from C_4 must: - Right to (4,5) - Up to (3,5) - Left to push (3,4) to (3,3) - Left again to push (3,3) to (3,2) [...] Next new Plan from C_4: Right, Up, Left, Left </remark> The runner retains the last 4,800 of the remark’s 6,706 characters, prefixed by[truncated]. Both quoted plan blocks survive. The next two calls follow the plan: The next two policy actions (excerpts) <action>right, up, left</action> [...] <action>left</action> The calls move the box through(3,3)onto(3,2), while the first box stays on target at(4,1). The run uses 13 action turns, including one rewind request, plus one reflection call and 20 primitive movement attempts. No forced rewind occurs. What this case shows.The policy leaves a loop, preserves a solved subgoal, and follows the retained plan. The pre-rewind room was still solvable. This case does not show an efficiency advan- tage: a single-episode rollout on the same instance succeeds in three action turns. In both cases, the logs confirm that reflection was supplied and its final recommendation followed; they do not isolate its effect from that of state restoration. 40