Paper deep dive
Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agents
Hanlin Wang, Chak Tou Leong, Jian Wang, Wenjie Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 9:29:26 AM
Summary
The paper identifies 'belief inertia' in embodied agents, where agents fail to update internal beliefs despite receiving contradictory environmental observations, leading to 'observational neglect'. To mitigate this, the authors propose the Estimate-Verify-Update (EVU) mechanism. EVU is an active belief intervention that explicitly generates textual belief states through a three-step process: estimating expected outcomes, verifying them against actual observations (generating a 'surprise signal'), and updating the belief. The mechanism is designed to be compatible with both prompting-based and training-based (SFT, PPO, GRPO) agent reasoning methods. Experiments on ALFWorld, VirtualHome, and ScienceWorld benchmarks demonstrate that EVU significantly improves task success rates by effectively mitigating belief inertia.
Entities (10)
Relation Signals (4)
EVU → improves → ALFWorld
confidence 100% · Extensive experiments across three embodied benchmarks demonstrate that EVU consistently yields substantial gains in task success rates.
EVU → mitigates → Belief Inertia
confidence 100% · Further in-depth analysis confirms that our EVU mitigates belief inertia effectively.
Belief Inertia → causes → Observational Neglect
confidence 90% · We refer to this behavior as 'observational neglect', where the agent observes but fails to integrate observed information into its internal reasoning process. ... We term this failure to update belief inertia.
LLM → exhibits → Belief Inertia
confidence 90% · Recent advancements in large language models (LLMs) have enabled agents to tackle complex embodied tasks... However, these agents still make suboptimal decisions... as they often overlook critical environmental feedback... we characterize this as belief inertia.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advancements in large language models (LLMs) have enabled agents to tackle complex embodied tasks through environmental interaction. However, these agents still make suboptimal decisions and perform ineffective actions, as they often overlook critical environmental feedback that differs from their internal beliefs. Through a formal probing analysis, we characterize this as belief inertia, a phenomenon where agents stubbornly adhere to prior beliefs despite explicit observations. To address this, we advocate active belief intervention, moving from passive understanding to active management. We introduce the Estimate-Verify-Update (EVU) mechanism, which empowers agents to predict expected outcomes, verify them against observations through explicit reasoning, and actively update prior beliefs based on the verification evidence. EVU is designed as a unified intervention mechanism that generates textual belief states explicitly, and can be integrated into both prompting-based and training-based agent reasoning methods. Extensive experiments across three embodied benchmarks demonstrate that EVU consistently yields substantial gains in task success rates. Further analyses validate that our approach effectively mitigates belief inertia, advancing the development of more robust embodied agents. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.17252v1
- Canonical: https://arxiv.org/abs/2604.17252v1
Trouble viewing inline? Open PDF directly →
Full Text
72,846 characters extracted from source content.
Expand or collapse full text
Seeing Isn’t Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agents Hanlin Wang 1 , Chak Tou Leong 1 , Jian Wang 1,2† , Wenjie Li 1 1 Department of Computing, The Hong Kong Polytechnic University 2 College of Computer Science, Sichuan University hanlin-henry.wang, chak-tou.leong@connect.polyu.hk jian51.wang@polyu.edu.hk cswjli@comp.polyu.edu.hk Abstract Recent advancements in large language mod- els (LLMs) have enabled agents to tackle com- plex embodied tasks through environmental in- teraction. However, these agents still make suboptimal decisions and perform ineffective actions, as they often overlook critical environ- mental feedback that differs from their internal beliefs. Through a formal probing analysis, we characterize this as belief inertia, a phe- nomenon where agents stubbornly adhere to prior beliefs despite explicit observations. To address this, we advocate active belief inter- vention, moving from passive understanding to active management. We introduce the Estimate- Verify-Update (EVU) mechanism, which em- powers agents to predict expected outcomes, verify them against observations through ex- plicit reasoning, and actively update prior be- liefs based on the verification evidence.EVU is designed as a unified intervention mecha- nism that generates textual belief states explic- itly, and can be integrated into both prompting- based and training-based agent reasoning meth- ods. Extensive experiments across three em- bodied benchmarks demonstrate thatEVUcon- sistently yields substantial gains in task suc- cess rates. Further analyses validate that our approach effectively mitigates belief inertia, ad- vancing the development of more robust em- bodied agents. Our code is available athttps: //github.com/WangHanLinHenry/EVU. 1 Introduction Large language models (LLMs) have revolution- ized embodied AI, enabling agents to solve increas- ingly complex, long-horizon tasks (Huang et al., 2022; Wang et al., 2023; Li et al., 2024). Effec- tive task-solving requires not only sophisticated reasoning, but also continuous interaction with the embodied environment. To this end, prior work † Corresponding author. This work was mainly conducted at PolyU, while the author is now at Sichuan University. Put a clean knife into the countertop There are many items on the table, but I still need to find a knife... Interaction Step T: Move to the table and search for the knife Step T+1: Move to the drawer and search for the knife . . . . . Figure 1: Illustrative example of observational neglect in embodied agents. While the agent observes a knife on the target table, its subsequent internal belief (“I still need to find a knife”) fails to integrate the observed information, leading to an unnecessary search action. VanillaSFTRL 0.0 0.2 0.4 0.6 0.8 1.0 Proportion of Trajectories with Observational Neglect 0.10 0.070.07 0.20 0.77 0.70 ×2.00 ×11.00 ×10.00 Success Traj. Failure Traj. Figure 2: Statistical results of observational neglect on the ALFWorld benchmark. has explored a variety of techniques, including inference-time iteration (Yao et al., 2022; Shinn et al., 2023), imitation learning (Chen et al., 2023), and reinforcement learning (Wu et al., 2025). The key to these methods is a tight feedback loop, in which the agent perceives the environment, inter- prets observations to internal states, reasons, and executes actions accordingly. Since an agent’s en- tire decision-making process critically depends on environmental feedback, integrating the observed information into its reasoning is crucial for task success (Wang et al., 2024; Fung et al., 2025). However, we observe a significant gap between receiving observations and effectively utilizing them. As illustrated in Figure 1, an agent may observe a knife on a countertop, yet its subse- arXiv:2604.17252v1 [cs.CL] 19 Apr 2026 quent reasoning behaves as if the knife were still missing, initiating a redundant search action. We refer to this behavior as “observational neglect”, where the agent observes but fails to integrate ob- served information into its internal reasoning pro- cess. Our statistical analysis (see Figure 2) on the ALFWorld (Shridhar et al., 2020) benchmark re- veals that such neglect is not a marginal error but a predominant failure mode in unsuccessful tra- jectories. Moreover, this behavior is widespread across various learning paradigms, from vanilla prompting to RL-tuned models, indicating a criti- cal bottleneck in how embodied agents transform observations into their own beliefs, i.e., the internal understandings of the environment states. To uncover the root cause of observational ne- glect, we conduct probing experiments to analyze the agent’s belief dynamics (see Section 3). Our analysis identifies a critical cognitive bias which we term belief inertia, a phenomenon that agents tend to stubbornly adhere to their prior expecta- tions of action outcomes, even when faced with contradictory evidence. This inertia results in a belief-observation misalignment, where the agent’s internal belief remains unchanged despite observ- ing a changing environment. While recent stud- ies have explored belief modeling (Zhang et al., 2024; Lidayan et al., 2025), they largely rely on implicit belief dynamics, where beliefs are updated passively and latently. Without an effective inter- vention when necessary, such strategies leave the agent’s reasoning prone to being “blinded” by bi- ased priors that overshadow its observations. To address this, we advocate active belief inter- vention, shifting the paradigm from passive update to active cognitive management (see Section 4). We introduce the Estimate-Verify-Update (EVU), a simple yet effective mechanism for belief inter- vention. Unlike previous works,EVUdecouples belief management from action generation by pro- ducing explicit belief states in a textual form. With EVU, the agent first estimates an expected outcome, verifies it against actual observations through LLM- based reasoning, and finally updates its prior belief to a grounded posterior. Crucially, we integrateEVU seamlessly into both prompting- and training-based agent learning methods. Extensive experiments across diverse embodied benchmarks demonstrate thatEVUconsistently yields substantial gains. Fur- ther in-depth analysis confirms that ourEVUmiti- gates belief inertia effectively. In summary, our contributions are as follows: •We identify and formalize belief inertia, a critical phenomenon in embodied agents where internal beliefs overshadow actual observations, leading to widespread observational neglect. •We propose active belief intervention, imple- mented via the Estimate-Verify-Update (EVU) mechanism. It provides a unified way to actively manage belief states and can be seamlessly inte- grated with various agent learning methods. • We demonstrate the superiority and generalizabil- ity ofEVUthrough extensive experiments across multiple embodied benchmarks. Further analysis validates thatEVUsignificantly mitigates belief in- ertia, providing valuable insights into developing robust embodied agents. 2 Preliminary Problem Formulation.The reasoning process of embodied agents is often formulated as a Partially Observable Markov Decision Process (POMDP), denoted as(U,S,A,O,T ,Z,R), whereUis the instruction space,Sthe hidden state space,Athe action space,Othe observation space,Tthe tran- sition function,Zthe observation function, and Rthe reward function. To isolate the cognitive aspects of agent–environment interaction from low- level perception, we focus on text-only settings whereU,A, andOare all expressed in natural lan- guage. Accordingly, we model an embodied agent as an LLM policyπ θ that generates textual actions. s t s t+1 a t−1 r t−1 o t a t r t o t+1 b t−1 b t b t+1 :Environment State :Observation :Reward:Agent’s Belief :Action Figure 3: POMDP formulation in embodied agents. Agent Beliefs. In a POMDP, the true environ- ment states t is never observed directly, so the agent must maintain an internal belief stateb t that sum- marizes its estimate ofs t and serves as the basis for decision making. As shown in Figure 3, at each step the agent integrates new observationo t with its prior beliefb t−1 to obtain an updated beliefb t , and then reasons the next action conditioned on b t , which in turn shapes future observations and rewards. In many LLM-based agents, such be- lief dynamics are handled implicitly. A common practice, i.e., ReAct-style (Yao et al., 2022) agent, appends all past actions and observations into an interaction historyh t = (u,o 1 ,a 1 ,o 2 ,...,o t )and relies on LLMs to latently inferb t when reasoning the next action, without explicitly representing the belief state. 3 Belief Inertia in Embodied Agents In this section, we investigate why ReAct-style em- bodied agents exhibit observational neglect. Since this phenomenon manifests as a misalignment be- tween the agent’s implicit beliefs and the external environment, we employ a probing-based method to explicitly elicit and track these beliefs. 3.1 Probing Agent Beliefs To probe an agent’s beliefs, we append probing questions to the interaction history and utilize the agent’s responses to decode its internal understand- ing of the environment. Specifically, given the interaction historyh t at timet, we define a set of task-relevant environment variablesV, such as whether an object has been acquired or whether it is currently inside a con- tainer. For each variablev ∈ V, we construct a corresponding yes–no probe questionq v (e.g., “Is the key currently in the box?”). We then construct a probing prompt by concatenatingh t andq v and feeding it into the agent policy. Finally, we read the first-token logits for the candidate answers “yes” and “no”, denoted byℓ yes (h t ,q v )andℓ no (h t ,q v ). We then define the raw belief value, the question- induced bias, and the debiased belief value as s(h t ,q v ) = ℓ yes (h t ,q v )− ℓ no (h t ,q v ), b(q v ) = ℓ yes (h ∅ ,q v )− ℓ no (h ∅ ,q v ), β(h t ,q v ) = s(h t ,q v )− b(q v ). (1) A positiveβ(h t ,q v )indicates an inclination to answer “yes” forvunderh t after filtering out question-induced bias, while a negative value indi- cates an inclination to answer “no”. For each variablev, lety v ∈ +1,−1denote the ground-truth answer in the current environment state, wherey v = +1corresponds to “yes” and y v =−1to “no”. We define the True Belief Value A(h t ,v) = y v β(h t ,q v ),(2) whose sign indicates whether the probed belief agrees with the true state (A(h t ,v) > 0) or not (A(h t ,v) < 0), and whose magnitude|A(h t ,v)| serves as a proxy for the confidence of this belief. Correlation analyses between probing results and the agent’s behavior support the reliability of our probing method (see Appendix A). Current BeliefEstimated Belief Subsequent Belief 0.4 0.3 0.2 0.1 0.0 0.1 True Belief Value SFTRLSFTRL Belief Boundary Figure 4: Probing results of belief dynamics across three stages. The belief boundary separates positive (correct) beliefs from negative (incorrect) ones. 3.2 Belief Inertia Phenomenon To analyze the agent’s decision-making process when the observational neglect occurs, we apply the above probing method to examine how its be- liefs are evolved. We first collect 100 observational-neglect cases from both SFT-trained and RL-trained agents in ALFWorld (Shridhar et al., 2020), where the agent’s internal reasoning for the next action a t neglects the newly received observationo t . We then extracto t fromvthat captures the critical feedback and construct the corresponding probe questionq v . To examine belief updating for this variable, we analyze how the agent’s beliefs evolve between the previous actiona t−1 that produces the new observationo t and the subsequent actiona t . Specifically, we probe the agent’s belief about v at three stages: (1) Current belief, representing the belief before taking a t−1 , probed under(u,o 1 ,a 1 ,o 2 ,..., o t−1 ); (2) Estimated belief, representing the belief after takinga t−1 but before observingo t , probedunder(u,o 1 ,a 1 ,o 2 ,..., o t−1 , a t−1 ); and(3)Subsequentbelief,representing the belief after receivingo t , probed under (u,o 1 ,a 1 ,o 2 ,..., o t−1 , a t−1 , o t ). Figure 4 visualizes the probed True Belief Value across the three stages for both SFT- and RL- trained agents. At the initial stage,A(c t ,v)lies near the belief boundary, indicating that the agent does not hold a strong prior about the query. After taking an action, the agent forms a strong but incor- rect belief aboutv. Although the subsequent belief increases slightly after receivingo t , it remains nega- tive for both agent types. This persistence suggests that the new environmental feedback fails to up- date the agent’s internal state. Consequently, the agent reasons the next action based on a stale be- lief formed immediately after the previous action, 0.40.30.20.10.00.10.20.30.4 True Belief Value 40 50 60 70 80 Success Rate (%) SFT (w/o BI). SFT (w/ BI). RL (w/o BI). RL (w/ BI). Belief Boundary Figure 5: Impact of oracle belief intervention (BI). which remains inconsistent with the environment state. We term this failure to update belief iner- tia, a phenomenon that manifests as observational neglect during subsequent reasoning. 3.3 Belief Intervention To examine whether the belief inertial causes obser- vational neglect, we conduct a Belief Intervention (BI) experiment: we manually correct the agent’s belief to match the true environment state and then observe whether it can reason and act correctly. Specifically, we adopt the same observational- neglect cases as mentioned above and evaluate both the SFT-trained and RL-trained agents under two distinct settings. In the typical setting (w/o BI), the agent generatesa t conditioned on the standard interaction historyh t . In the intervention setting (w/ BI), we explicitly append a description of the oracle environment states ∗ t to the interaction his- tory, yielding(u,o 1 ,a 1 ,...,o t ,s ∗ t ). In both set- tings, we apply our probing method to assess the agent’s belief about the environment variables rele- vant too t and report the task success rate on these observational-neglect cases. As illustrated in Figure 5, belief intervention shifts the agents’ internal states across the Be- lief Boundary, correlating directly with improved task performance. Without intervention, both SFT- trained and RL-trained agents linger in the negative True Belief Value region, indicating a persistence of incorrect beliefs that corresponds to lower suc- cess rates. Upon intervention, the True Belief Value becomes positive, signifying the adoption of the correct environmental state. Crucially, this belief correction translates into a marked increase in suc- cess rates for both agents. These results confirm that the primary bottleneck in observational neglect cases is the failure to update beliefs. By manually aligning the belief state with the actual environ- ment, we mitigate the downstream consequences of this failure, demonstrating that the agents pos- sess the necessary reasoning capabilities to succeed once the belief barrier is removed. 4 Method: Active Belief Intervention Drawing upon the crucial findings in Section 3, we advocate active belief intervention and introduce a simple yet effective Estimate-Verify-Update (EVU) mechanism that estimates, verifies, and updates beliefs actively through a unified perspective. Fig- ure 6 shows the overview of our approach. 4.1 Estimate-Verify-Update Mechanism In contrast to typical ReAct-style agents, which passively encode the entire interaction history as an implicit belief about the environment within the latent model parameters, ourEVUmechanism maintains an explicit belief stateB t , a natural lan- guage summary that sufficiently represents the un- derstanding of the environment. Crucially,EVU recursively takes the previous belief state as input and evolves it through a structured loop of estima- tion, verification, and update by the agent itself. Estimation. Initially, the agent attempts to pre- dict the immediate consequence of its previous ac- tion before processing the actual new observation. In this step, the agent establishes a baseline expec- tation by estimating action outcomes E t as: E t ∼ π θ (·| B t−1 ,a t−1 ,o t ),(3) whereE t describes what the agent expects to ob- serve, explicitly modeling its expectation. Verification.The agent then processes the actual observationo t from the environment. Instead of updating the belief directly, the agent first generates a verification evidenceV t to compare its estimation against the actual observation: V t ∼ π θ (·| B t−1 ,a t−1 ,o t ,E t ).(4) Here,V t serves as a structured “surprise signal” that explicitly captures whether the observation confirms or contradicts the expectation, preventing the agent from hallucinating success or overlooking contradictory evidence. Belief Update.Finally, the agent synthesizes the reasoning chain to transition from the previous be- lief stateB t−1 to the current belief stateB t . This Task Belief State (t-1) Action (t-1) Observation (t-1) Estimation (t) TaskBelief State (t-1) Action (t-1) Observation (t-1) Estimation (t) Verification (t) TaskBelief State (t-1)Action (t-1) Observation (t-1) Estimation (t) Verification (t)Belief state (t) Task Belief State (t-1)Action (t-1) Observation (t-1)Estimation (t)Verification (t)Belief state (t)Action (t) Task Belief State (t) Action (t) Observation (t) Estimation (t+1) Go to coffee table 2The agent moved to the coffee table 2 expecting to see if a knife there. Put a clean knife into the countertop On the coffee table 2, you see a cup 1, a glass bottle 1, a knife 1, a plate 2... The observation confirms the expectation, there is a knife there. ... There isn’t a knife on the desk, shelf, side table... ... There isn’t a knife on the desk, shelf, side table. There is a knife on the coffee table... Estimate Observation Verify Expectation Update Belief Step (t) Step (t+1) Task Action 1 Observation 1 Action 2 Observation 2 Action (t-1) Observation (t-1) . . .. . . Action (t) Belief State (t-1) : Context : Generated tokens : Latent calculation Reason Next Action a) b) Figure 6: Overview of our proposed active belief intervention method. Compared to typical belief modeling methods (top), we introduce a unified Estimate-Verify-Update (EVU) mechanism (bottom). process takes the prior belief, the initial estima- tionE t , and the verification evidenceV t (i.e., the surprise signal) as inputs: B t ∼ π θ (·| B t−1 ,a t−1 ,o t ,E t ,V t ).(5) By leveraging this surprise-aware verificationV t , the model ensures that the new belief stateB t (e.g., “There is a knife on the coffee table. . . ”) accurately reflects the latest environmental changes while re- taining valid historical information about the en- vironment. This updated belief then serves as the foundation for subsequent reasoning. 4.2 A Unified Intervention Perspective OurEVUis a general mechanism that decouples state maintenance from action generation. This separation allows for a unified formulation that is agnostic to both the prompting-based methods and the training-based algorithms. 4.2.1 Prompting-based Belief Intervention Standard prompting-based methods typically em- ploy a prompting strategySto directly map inter- action historyh t to an actiona t . In our approach, we realize belief intervention by augmenting the original strategySwith specific instructions de- signed to enable active belief dynamics. We denote this belief-enhanced strategy asS ∗ , which explic- itly guides the agent to perform active belief inter- vention before decision-making. This process is formulated as: (E t ,V t ,B t ,a t )∼ π θ (·|S ∗ (B t−1 ,a t−1 ,o t )). (6) By requiring the agent to explicitly model its belief dynamics prior to the actiona t , this inter- vention ensures that the agent’s decision-making is grounded in a structured and updated understand- ing of the environment, rather than relying solely on implicit patterns within the raw history. 4.2.2 Training-based Belief Intervention In training-based approaches, we update the model parametersθto internalize the belief update mech- anism. Unlike typical training methods that focus solely on optimizing action generation, we unify belief update and action generation into an autore- gressive process and optimize them jointly. Formally, at each time stept, the model takes the previous belief stateB t−1 and the recent interac- tion history(a t−1 ,o t−1 )as inputs to generate the current reasoning chain and action: (E t ,V t ,B t ,a t )∼ π θ (·| B t−1 ,a t−1 ,o t ).(7) To optimize this process, we define a general objective functionJ (θ), which represents the ex- pected utility of the generated trajectory. Depend- ing on the training paradigm,J (θ)can be flexibly instantiated as the negative log-likelihood in Super- vised Fine-Tuning (SFT) or the expected reward in Reinforcement Learning (e.g., PPO, GRPO). This is formulated as: θ ∗ = argmax θ J (θ) = argmax θ E τ∼π θ [U (τ )], (8) whereτ = (B 0 ,a 0 ,o 0 ,E 1 ,V 1 ,B 1 ,a 1 ,... )repre- sents the augmented trajectory containing both cog- nitive states and external actions. By maximizing Method ALFWorldVirtualHomeScienceWorld SeenUnseenSeenUnseenSeenUnseen Prompting DeepSeek V3.2 NoThinking50.7–42.3–8.0–7.2–47.0–46.0– ,→ w/ EVU (Ours)55.0(↑4.3)47.6(↑4.3)12.8(↑4.8)12.8(↑5.6)55.0(↑8.0)52.2(↑6.2) Plan-and-Act52.1–44.8–12.8–12.8–55.0–50.9– ,→ w/ EVU (Ours)53.6(↑1.5)46.3(↑1.5)15.2(↑2.4)14.4(↑1.6)58.3(↑3.3)55.3(↑4.4) ReAct55.7–47.6–13.6–12.8–60.3–57.8– ,→ w/ EVU (Ours)56.4(↑0.7)49.8(↑2.2)16.0(↑2.4)13.6(↑0.8)62.3(↑2.0)60.9(↑3.1) Training Qwen3-1.7B-Instruct SFT37.1–20.1–7.2–8.0–7.3–11.2– ,→ w/ EVU (Ours)41.4(↑4.3)33.6(↑13.5)16.0(↑8.8)25.6(↑17.6)23.2(↑15.9)24.8(↑13.6) PPO42.1–32.0–10.4–22.4–37.0–41.0– ,→ w/ EVU (Ours)47.1(↑5.0)40.3(↑8.3)17.6(↑7.2)28.8(↑6.4)62.9(↑25.9)54.0(↑13.0) GRPO47.0–44.0–15.7–19.4–41.7–42.9– ,→ w/ EVU (Ours)52.1(↑5.1)49.3(↑5.3)20.0(↑4.3)36.9(↑17.5)47.7(↑6.0)50.3(↑7.4) Qwen2.5-3B-Instruct SFT65.7–50.7–20.0–20.0–19.9–13.7– ,→ w/ EVU (Ours)70.0(↑4.3)56.7(↑6.0)27.2(↑7.2)34.4(↑14.4)49.0(↑29.1)45.3(↑31.6) PPO77.8–54.4–24.0–23.2–53.6–51.6– ,→ w/ EVU (Ours)79.3(↑1.5)58.2(↑3.8)28.8(↑4.8)35.2(↑12.0)60.3(↑6.7)62.7(↑11.1) GRPO83.6–70.8–25.6–24.8–49.7–52.8– ,→ w/ EVU (Ours)85.7(↑2.1)79.1(↑8.3)31.2(↑5.6)36.0(↑11.2)70.9(↑10.6)70.8(↑18.0) Table 1: Main results of success rates (%) on three representative embodied benchmarks. “Seen” and “Unseen” denote held-out test sets with tasks seen and unseen during training, respectively. “w/EVU” denotes plugging our EVUmechanism into base methods. Bold values represent the best performance within each backbone model group. 58 60 62 64 Success Rate (%) Prompting (Seen) 58 60 62 64 Prompting (Unseen) 20 40 60 SFT (Seen) 20 40 60 SFT (Unseen) 40 50 60 70 GRPO (Seen) 50 60 70 GRPO (Unseen) ReflActVAGENNoneBIDBIOBIEVU (Ours) Figure 7: Success rates (%) of different methods with belief intervention variants. J (θ), the optimization algorithm adjusts the proba- bility mass not just for the final actiona t , but for the entire reasoning chain (E t ,V t ,B t ). This ensures that the model learns to maintain high-quality be- liefs that causally lead to optimal actions, allowing gradients (or reward signals) to propagate through the belief update process. 5 Experiments 5.1 Experimental Setup Benchmarks. We evaluate our method on three representative embodied agent benchmarks: ALF- World (Shridhar et al., 2020), VirtualHome (Puig et al., 2018), and ScienceWorld (Wang et al., 2022). Following prior studies (Song et al., 2024; Wang et al., 2025a), we adopt Success Rate (SR) as our primary evaluation metric and evaluate agents on both seen and unseen scenarios. Appendix B pro- vides more details of these datasets. Baseline Methods. We evaluate our method by integrating it into two categories of base- lines and measuring the resulting performance gains: (1) prompting-based methods, including No- Thinking (Ma et al., 2025), Plan-and-Act (Kim et al., 2025), and ReAct (Yao et al., 2022); and (2) training-based methods, including SFT (Chen et al., 2023), PPO (Schulman et al., 2017), and GRPO (Shao et al., 2024). Additional details about these baselines are provided in Appendix C. Implementation Details. We conduct experi- ments on DeepSeek V3.2 (Liu et al., 2025) for prompting-based evaluations, as well as Qwen2.5- 3B-Instruct (Yang et al., 2025a) and Qwen3-1.7B- Instruct (Yang et al., 2025a) for training-based eval- uations. For the SFT phase, the training epochs are set to 3. For the RL phase, the training process consists of 250 steps, and we select the checkpoint with the best performance on the validation set for SFTGRPOPPO 0.3 0.2 0.1 0.0 0.1 0.2 0.3 True Belief Value NoneBIDBIOBIOurs Belief Boundary Figure 8: Quantitative probing results of different belief intervention methods in mitigating belief inertia. final testing. During inference, the decoding tem- perature of the LLMs is set to 0.0 for deterministic generation. Detailed hyperparameters and prompt designs are provided in the Appendix E. 5.2 Main Results Table 1 presents a comprehensive evaluation of different methods across three benchmarks. We summarize key findings of EVU below: Consistent improvement across benchmarks and backbone models. As shown in Table 1, EVUconsistently outperforms all baseline meth- ods across all three benchmarks, demonstrating its effectiveness and robustness. Notably, we ob- serve that the average performance gain on Unseen splits (+9.21) is higher than that on Seen splits (+6.8). This indicates that our method effectively grounds the agent even when facing novel obser- vations in OOD scenarios, thereby substantially enhancing generalization capabilities. Additional comparisons with advanced context-management and search-based agentic baselines are provided in Appendix D, where EVU remains consistently beneficial. Robustness in both prompting- and training- based settings. Our method demonstrates re- markable flexibility by seamlessly integrating with both prompting-based and training-based methods. As shown in Table 1,EVUconsistently enhances performance across these distinct modes. Notably, the average improvement in training-based settings (+10.37) significantly exceeds that in prompting- based settings (+3.28). This disparity suggests that whileEVUserves as an effective inference-time guidance, its full potential is unleashed when the backbone model is allowed to internalize the active belief update process, leading to more substantial performance gains. Current Belief Estimated Belief Verification Belief Updated Belief 0.35 0.25 0.15 0.05 0.05 0.15 0.25 True Belief Value SFTPPOGRPO Belief Boundary Figure 9: Quantitative probing results of different phase in mitigating belief inertia. 6 Analyses and Discussions 6.1 Variant Analysis To investigate the efficacy of belief intervention (BI), we examined distinct intervention strategies injected before the thought-action generation pro- cess. These include: NoneBI, which serves as the baseline without any intervention; DBI (Di- rect Belief Intervention), where the belief state is generated directly; OBI (Observation-based Belief Intervention), which compels the agent to reiterate the recent observation prior to forming a belief; Re- flAct, which encourages the agent to reflect on its progress relative to the goal; and VAGEN (Wang et al., 2025c), which tasks the agent with predicting the environmental state following a potential action. Please refer to Appendix F for more details. The comparative results are presented in Fig- ure 7. First, we observe that methods incorpo- rating belief intervention consistently outperform the baseline (NoneBI) across the majority of set- tings. This trend underscores the fundamental effi- cacy of explicit belief modeling in enhancing task performance. Second, and more importantly, our proposedEVUachieves superior performance com- pared to all other intervention variants across di- verse paradigms and evaluation splits. This consis- tent dominance suggests that theEVUmechanism provides a more robust and accurate strategy for belief generation than simple repetition or reflec- tion, thereby serving as an effective intervention strategy. 6.2 Analysis on Belief Inertia Mitigation CanEVUeffectively mitigate the belief inertia phenomenon? To investigate this, we conduct a dedicated analysis to examine how our method be- haves on the observational neglect cases collected in Section 3.2. Specifically, we employ our prob- ing method to detect the agent’s belief immedi- EasyMediumHardVery Hard Task Difficulty Level 0% 5% 10% 15% 20% 25% 30% Success Rate Rising SFTGRPOPPO Figure 10: Relative improvement of our method com- pared to baselines (SFT, GRPO, PPO) across different levels of task difficulty. ately prior to the decision-making phase. (See Ap- pendix G for detailed experimental configurations.) As illustrated in Figure 8, we observe that the true belief values for our method are consistently posi- tive. This indicates that the agent’s internal belief state aligns with the actual environmental state, suc- cessfully overcoming belief inertia. Furthermore, compared to other variants,EVUexhibits the highest true belief values. This superiority demonstrates that our approach not only corrects the belief, but also achieves the highest level of confidence in the true state of the environment. How doesEVUtake effect to mitigate the belief inertia? To understand the internal mechanism, we conduct further analysis to observe the agent’s belief dynamic evolution using the same set of ob- servational neglect cases mentioned above. We probe the agent’s belief at different stages within ourEVUframework. Please refer to the Appendix G for more details. As illustrated in Figure 9, the agent’s belief value starts in the negative region and gradually ascends. While the estimation and veri- fication phases push the belief towards and across the boundary, respectively, it is the final update phase that significantly boosts the value to fully align with the ground truth. This indicates that the update stage is the decisive factor for synchroniza- tion, effectively building upon the foundations laid by the preceding phases. 6.3 Impact on Task Difficulty We further investigate the necessity of active belief intervention as the task difficulty increases. Dif- ficult tasks inherently involve longer interaction horizons, requiring the agent to process more ob- servations. Consequently, the ability to maintain accurate and synchronized belief states becomes increasingly critical during these interactions. To ReActOurs 800 1000 1200 1213.3 892.9 Input Tokens / Turn ReActOurs 0 100 200 41.1 152.9 Output Tokens / Turn ReActOurs 9 10 11 10.6 9.3 Reasoning Time / Turn ReActOurs 150 200 250 231.8 175.4 Reasoning Time / Task ReActOurs 0.8 0.9 1.0 0.89 0.98 Grounding Rate ReActOurs 18 20 20.4 18.2 Number of Turns / Task Figure 11: Comparison between ReAct and Ours in terms of computational overhead. verify this, we evaluate the relative success rate improvement (∆Success Rate) of our method com- pared to three training baselines (SFT, GRPO, and PPO) across four difficulty levels (see Appendix H for details). As shown in Figure 10, our approach consis- tently outperforms the baselines across all settings. More importantly, the performance gap exhibits a clear rising trend: as the difficulty escalates from “Easy” to “Very Hard,” the relative improvement becomes significantly more pronounced. This val- idates that active belief intervention enables the agent to understand the environment more deeply, thereby preventing compounding errors. 7 Analysis on Computational Overhead Do the performance gains of EVU come with substantial additional computational overhead? To investigate this, we compare the standard Re- Act baseline and ReAct+EVU on the Qwen2.5-3B backbone. We report the average input/output to- kens, reasoning latency, grounding rate, and aver- age number of turns per task. As illustrated in Figure 11, although EVU nat- urally increases the number of output tokens per turn due to the generation of the Estimate, Verify, and Update components, it substantially reduces the input context length by compressing the ver- bose interaction history into a concise belief state. As a result, the average total token consumption per task decreases from 1213.3 to 892.9, while the average reasoning latency per task also decreases from 231.8 to 175.4. In addition, EVU improves the grounding rate from 0.89 to 0.98 and reduces the average number of turns from 20.4 to 18.2, in- dicating that the agent avoids invalid exploration and plans more effectively. These results show that EVU does not incur a larger overall computational burden; instead, it improves efficiency by replacing Method# Reasoning TokensSR (%) Multiple Reflection153.346.3 EVU (Ours)138.049.8 Table 2: Comparison between ourEVUand the Multiple Reflection method. low-quality exploration with more grounded and directed reasoning. Are the gains of EVU simply due to increased inference computation (test-time scaling)? We further conducted a controlled experiment on ALF- World with DeepSeek V3.2 in the prompting-based setting. Specifically, we compare EVU against a Multiple Reflection baseline, where the agent is prompted to reflect multiple times to deliberately increase the reasoning token budget. This setting allows us to examine whether the gain of EVU comes merely from consuming more reasoning to- kens, or from the specific structure of the EVU process itself. As shown in Table 2, EVU achieves a higher suc- cess rate than Multiple Reflection while using fewer reasoning tokens, which first rules out the expla- nation that the gain simply comes from spending more computation at inference time. More impor- tantly, this comparison also highlights a key distinc- tion between EVU and standard reflection. Mul- tiple Reflection is retrospective: it asks the agent to reconsider whether it may have made a mistake after the fact. In contrast, EVU is predictive and discrepancy-driven. The Estimate step makes the agent explicitly predict the expected outcome be- fore observing the new state, giving the Verify step a concrete reference point. This allows EVU to detect a “surprise signal,” i.e., the mismatch be- tween expectation and observation. The Update step then revises the belief state accordingly, rather than merely triggering another round of generic self-correction. This predictive mechanism is par- ticularly important for overcoming belief inertia: without an explicit prior estimate, standard reflec- tion may remain trapped in the agent’s previous be- lief and fail to recognize the reality gap. Therefore, EVU improves performance not by encouraging longer reasoning chains, but by enforcing a simple yet effective belief-correction process that is more targeted than standard reflection. 8 Related Work Embodied Planning. Recent advancements in Large Language Models (LLMs) have empowered embodied agents to engage in complex embod- ied planning (Li et al., 2025; Yang et al., 2025b; Liao et al., 2025). To facilitate effective decision- making, existing studies employ diverse strategies: Prompting methods (Yao et al., 2022; Shinn et al., 2023; Yao et al., 2023) structure reasoning at infer- ence time, supervised finetuning (Chen et al., 2023; Wang et al., 2025b; Qiao et al., 2024) internalize expert priors, and reinforcement learning refines policy from reward signals (Song et al., 2024; Chen et al., 2025; Wang et al., 2025a; Zhang et al., 2025). However, these methods primarily prioritize action optimization, often neglecting the critical need to maintain a reliable internal world model amidst dy- namic environmental changes (Huang et al., 2023; Wang et al., 2024; Kim et al., 2025). Agent Beliefs. To achieve goals, agents are re- quired to maintain and update their internal be- liefs during interaction. Prior work has highlighted multiple complementary facets of such beliefs. First, agents often rely on task belief—estimates of progress and knowledge—to support long-horizon planning (Qiao et al., 2024; Wang et al., 2024; Zhang et al., 2024). Second, in human-agent in- teraction, agents capture users’ latent intent to in- terpret ambiguous instructions (Lin et al., 2025; Ramrakhya et al., 2025). Third, in cooperative multi-agent settings, agents track others’ capabili- ties, objectives, and likely future actions to enable coordination (Fan et al., 2025; Lic ̆ a et al., 2024; Wang et al., 2026). In this work, we focus on agent beliefs regarding the evolving environment, which grounds reasoning under environment state changes during interactions. 9 Conclusion In this work, we identify and formalize belief in- ertia as a key failure mode of LLM-based embod- ied agents, where they stubbornly adhere to prior beliefs despite explicit observations. To address this issue, we advocate active belief intervention and instantiate it with the Estimate-Verify-Update (EVU) mechanism. By integratingEVUinto both prompting-based and training-based methods, we mitigate belief inertia effectively, thereby obtain- ing consistent improvements in task performance across multiple embodied agent benchmarks. Limitations While our approach demonstrates superior perfor- mance compared to baseline methods, it is impor- tant to acknowledge the limitations of our current work as follows: (1) Dependency on Observation Quality: Our method relies on the quality and granularity of en- vironmental observations to update its belief dy- namics. In scenarios with extremely sparse, noisy, or ambiguous feedback, where the ground truth is difficult to discern even with active reasoning, the agent’s belief updates may become unstable. Fu- ture work could explore more robust active belief dynamics that can better handle uncertainty and noise. (2) Limited Exploration of Model Variants: Due to computational resource constraints, our experi- ments on prompting methods were primarily con- ducted using DeepSeek V3.2, and we did not ex- tensively evaluate the approach across a broader range of LLM backbones. Furthermore, while our work addresses the fundamental challenge of belief updating and is expected to be compatible with var- ious methods, we have not yet explored alternative designs to further facilitate accurate belief genera- tion, such as integrating dense reward shaping or auxiliary supervision signals. Future work could incorporate these advanced designs to refine the belief intervention process. Ethics Statement This work aims to develop LLM-based embodied agents within simulated environments. The Vir- tualHome and ALFWorld environment setup and related data strictly follow the specifications of Vir- tualHome (Puig et al., 2018), ALFWorld (Shrid- har et al., 2020), and ScienceWorld (Wang et al., 2022). We utilize VirtualHome v2.3.0 1 (MIT li- cense 2 ), ALFWorld 3 (MIT license 4 ) and Science- World 5 (MIT license 6 ) to conduct our experiments. All the LLMs we use for fine-tuning are open- source, and we strictly follow the protocols for the academic use of these models. Additionally, 1 https://github.com/xavierpuigf/virtualhome/ tree/master 2 https://github.com/xavierpuigf/virtualhome/ blob/master/LICENSE 3 https://github.com/alfworld/alfworld 4 https://github.com/alfworld/alfworld/blob/ master/LICENSE 5 https://github.com/allenai/ScienceWorld 6 https://github.com/allenai/ScienceWorld/blob/ main/LICENSE while AI assistants (e.g., Cursor and ChatGPT) were partially utilized for code optimization and linguistic refinement, we affirm that all core con- tent and findings in this paper are the original work of the authors. Acknowledgements This work was supported by the Research Grants Council of Hong Kong (15209724, 15205325), and also in part by the PolyU Postdoc Matching Fund Scheme (4-W40Z). The authors would like to thank the anonymous reviewers for their valuable feed- back and constructive suggestions. References Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. Hanyang Chen, Mark Zhao, Rui Yang, Qinwei Ma, Ke Yang, Jiarui Yao, Kangrui Wang, Hao Bai, Zhen- hailong Wang, Rui Pan, and 1 others. 2025. Era: Transforming vlms into embodied agents via embod- ied prior learning and online reinforcement learning. arXiv preprint arXiv:2510.12693. Xianzhe Fan, Xuhui Zhou, Chuanyang Jin, Kolby Not- tingham, Hao Zhu, and Maarten Sap. 2025. Somi- tom: Evaluating multi-perspective theory of mind in embodied social interactions.arXiv preprint arXiv:2506.23046. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978. Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hongyu Gong, Hervé Jégou, Alessandro Lazaric, and 1 others. 2025. Embod- ied ai agents: Modeling the world. arXiv preprint arXiv:2506.22355. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International conference on ma- chine learning, pages 9118–9147. PMLR. Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and 1 others. 2023. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances in Neural Information Processing Systems, 36:59636– 59661. Jeonghye Kim, Sojeong Rhee, Minbeom Kim, Dohyung Kim, Sangmook Lee, Youngchul Sung, and Kyomin Jung. 2025. Reflact: World-grounded decision mak- ing in llm agents via goal-state reflection. In Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33421–33453. Manling Li, Yunzhu Li, Jiayuan Mao, and Wenlong Huang. 2025. Foundation models meet embodied agents. In Proceedings of the 2025 Annual Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 5: Tutorial Abstracts), pages 15–24, Albuquerque, New Mexico. Association for Computational Linguistics. Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, and 1 others. 2024. Embodied agent interface: Benchmarking llms for embodied decision making. Advances in Neural In- formation Processing Systems, 37:100428–100534. Yi Liao, Yu Gu, Yuan Sui, Zining Zhu, Yifan Lu, Guo- hua Tang, Zhongqian Sun, and Wei Yang. 2025. Think in games: Learning to reason in games via reinforcement learning with large language models. arXiv preprint arXiv:2508.21365. Mircea Lic ̆ a, Ojas Shirekar, Baptiste Colle, and Chirag Raman. 2024. Mindforge: Empowering embodied agents with theory of mind for lifelong collaborative learning. arXiv preprint arXiv:2411.12977. Aly Lidayan, Jakob Bjorner, Satvik Golechha, Kartik Goyal, and Alane Suhr. 2025. Abbel: Llm agents act- ing through belief bottlenecks expressed in language. arXiv preprint arXiv:2512.20111. Xingyao Lin, Xinghao Zhu, Tianyi Lu, Sicheng Xie, Hui Zhang, Xipeng Qiu, Zuxuan Wu, and Yu-Gang Jiang. 2025. Ask-to-clarify: Resolving instruction ambi- guity through multi-turn dialogue. arXiv preprint arXiv:2509.15061. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingx- uan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. 2025. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.09858. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502. Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2024. Agent planning with world knowledge model. Advances in Neural Information Processing Systems, 37:114843–114871. Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, and Roozbeh Mottaghi. 2025. Grounding multimodal llms to embodied agents that ask for help with reinforcement learning. arXiv preprint arXiv:2504.00907. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Process- ing Systems, 36:8634–8652. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and em- bodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization of llm agents. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7584–7600. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Hanlin Wang, Chak Tou Leong, Jian Wang, and Wenjie Li. 2024. E2cl: exploration-based error correction learning for embodied agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7626–7639. Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. 2025a. Spa-rl: Reinforcing llm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732. Hanlin Wang, Jian Wang, Chak Tou Leong, and Wenjie Li. 2025b. Steca: Step-level trajectory calibration for llm agent learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11597– 11614. Jiashuo Wang, Jiawen Duan, Jian Wang, Kaitao Song, Chunpu Xu, Johnny K. W. Ho, Fenggang Yu, Wenjie Li, and Johan F. Hoorn. 2026. Foresight optimiza- tion for strategic reasoning in large language models. Preprint, arXiv:2604.13592. Kangrui Wang, Pingyue Zhang, Zihan Wang, Yan- ing Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, and 1 oth- ers. 2025c. Vagen: Reinforcing world model rea- soning for multi-turn vlm agents. arXiv preprint arXiv:2510.16907. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298. Di Wu, Jiaxin Fan, Junzhe Zang, Guanbo Wang, Wei Yin, Wenhao Li, and Bo Jin. 2025. Reinforced reasoning for embodied planning. arXiv preprint arXiv:2505.22050. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, and 1 others. 2025b. Embodiedbench: Compre- hensive benchmarking multi-modal large language models for vision-driven embodied agents. In In- ternational Conference on Machine Learning, pages 70576–70631. PMLR. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. 2024. Agent-pro: Learning to evolve via policy-level reflection and op- timization. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5348–5375. Wenqi Zhang, Mengna Wang, Gangao Liu, Xu Huixin, Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, and 1 others. 2025. Embodied-reasoner: Synergizing visual search, rea- soning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.21696. Zirui Zhao, Wee Sun Lee, and David Hsu. 2023. Large language models as commonsense knowledge for large-scale task planning. Advances in neural infor- mation processing systems, 36:31967–31987. Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. 2025. Mem1: Learning to synergize memory and reason- ing for efficient long-horizon agents. arXiv preprint arXiv:2506.15841. A Reliability Evaluation To verify that our probing method accurately re- flects the agent’s true internal belief state—rather than generating hallucinations or unrelated out- puts—we conducted a consistency analysis. We hypothesize that if the probing method is reliable, the belief elicited by the probe at timesteptshould be highly consistent with the agent’s explicit rea- soning (Thought) manifested in the subsequent timestep. We validate the reliability by comparing the probing results against the agent’s subsequent be- havior. The specific procedure is as follows: •Probing att: At a given timestept, we focus on a specific object variableo t . We query the agent’s belief upon receiving obervationo t using our probing method to obtain a binary answer (Yes or No), denoted as the Predicted Label. •Action at t: We then allow the agent to pro- cess observationo t and conduct next step rea- soning, denoted asa t . We extract the agent’s explicit understanding regarding the specific object variableo t from the thought component of a t , denoted as the True Label. •Comparison: We compare the consistency between the probed answer and the subse- quent thought across 100 sampled cases. We visualized the consistency between the probed beliefs and the agent’s thoughts using a confusion matrix. As illustrated in Figure 12, our probing method demonstrates a high degree of fi- delity with the agent’s internal reasoning at next step: •When the agent’s subsequent thought indi- cates a negative state (No), the probe correctly identifies this belief 96.0% of the time. •When the agent’s subsequent thought indi- cates a positive state (Yes), the probe correctly aligns with this belief 92.0% of the time. The low off-diagonal error rates (4.0% and 8.0%) indicate minimal discrepancy. This strong align- ment confirms that our method is reliable: it suc- cessfully externalizes the agent’s latent beliefs with- out significant distortion, validating its utility for interpreting the agent’s decision-making process. NoYes Predicted Label (Probe) No Yes True Label (Thought) 96.0%4.0% 8.0%92.0% 20 40 60 80 Figure 12: Confusion Matrix evaluating the consistency between the Probed Answer (Predicted Label) and the Agent’s Subsequent Thought (True Label). The high values on the diagonal indicate that the probing method reliably reflects the agent’s internal beliefs. B Datasets and Preprocessing ALFWorld is an interactive text-based environ- ment that parallels the embodied worlds found in the ALFRED dataset. In this domain, agents are tasked with exploring a simulated household to complete high-level instructions, such as put a clean apple in the fridge.” The dataset includes both seen” splits for in-distribution evaluation and “unseen” splits to test out-of-distribution general- ization. For the ReAct baseline, we utilize the SFT data generated by Song et al. (2024). ScienceWorld is a complex text-based virtual environment designed to simulate elementary sci- ence experiments. It encompasses various distinct task types, such as thermodynamics and electrical circuits, requiring agents to ground their under- standing of scientific concepts through practical, embodied interaction. Similar to ALFWorld, we adopt the training trajectories for the ReAct base- line provided by the Song et al. (2024). To ensure computational efficiency, we Task-8, Task-9, Task- 1 (boil/freeze), Task-4 (grow fruit/plant), Task-5 (clean), Task-7 (paint), and Task-10 (decorate) due to their excessively long task-solving trajectories. VirtualHome is a platform that simulates com- plex daily household activities, where agents exe- cute programs to interact with objects and the en- vironment. Unlike the previous datasets, we align the SFT data for the ReAct baseline and the experi- mental setup with the methodology established in STeCa (Wang et al., 2025b). Consistent with the STeCA setting, we filter the dataset significantly; specifically, we remove approximately half of the training and testing data instances. This reduction is performed because many tasks in the original dataset are highly similar, ensuring a more efficient evaluation. Across all three domains, the environments pro- vide binary final rewards, where a reward of 1 in- dicates successful task completion and 0 indicates failure. Consequently, we report the average re- ward across the tested tasks as the success rate. Dataset Statistics We summarize the detailed statistics of the three datasets in Table 3. The table reports the number of instances in the training set, as well as the Test-Seen” (in-distribution) and Test- Unseen” (out-of-distribution) evaluation sets. It also lists the average number of interaction turns required for expert trajectories, which serves as an indicator of task complexity across the different environments. Dataset#Train#Test-Seen#Test-Unseen#Turns ALFWorld28511401347.97 VirtualHome24601251258.79 ScienceWorld12531511619.64 Table 3: Statistics of datasets. “Test-Seen” and “Test- Unseen” are test set with seen and unseen scenarios respectively. “#Turns” denotes the average number of interaction turns for the expert trajectories. C Additional Details about Baselines In this section, we provide additional implementa- tion details for the baseline methods, categorized by their prompting strategies and training method- ologies. Prompting Settings. We consider three frame- works: (1) No-Thinking(Ma et al., 2025): The agent generates an action directly at each time step without any reasoning step. (2) Plan-and-Act(Kim et al., 2025): The agent conducts reasoning only at the first step and outputs actions without further thoughts in subsequent steps. and (3) ReAct (Yao et al., 2022): The agent first reasons about the next action at each time step and then generates an ac- tion. Training Settings. We employ three distinct ap- proaches: (1) SFT(Chen et al., 2023): The model is fine-tuned using standard supervised learning on a dataset of expert trajectories. (2) PPO(Schulman et al., 2017): A proximal policy optimization algo- rithm that utilizes a separate value network (critic) to reduce variance and stabilize training. and (3) GRPO (Shao et al., 2024): A group relative policy optimization method that eliminates the need for a critic model by estimating the baseline from the average reward of a group of sampled outputs for the same input. D Additional Comparisons with Advanced Baselines we provide additional comparisons with more ad- vanced baselines to further clarify the role of EVU. Specifically, we aim to answer two questions: (1) whether the gain of EVU mainly comes from bet- ter context management or from its active belief intervention mechanism, and (2) whether EVU can also benefit stronger agentic frameworks beyond a standard ReAct-style policy. To this end, we con- duct three additional experiments: a short-context control experiment, a comparison against a history- summarization baseline, and an integration with a search-based agentic planner. Is EVU more than context management? To disentangle belief inertia from long-context crowd- ing, we first revisit the failure cases identified in our analysis and truncate the interaction history to retain only the most recent two turns, so that the critical observation remains explicitly visible and no long-context retrieval is required. Even under this short-context setting, 95% of the cases still exhibit belief inertia, where the agent ignores the immediate contradictory observation and con- tinues to follow its prior belief. This result sug- gests that belief inertia is not merely a symptom of crowded context, but a distinct failure to integrate contradictory evidence. We further compare EVU against MEM1 (Zhou et al., 2025), a representa- tive history-summarization baseline that manages context through memory compression but does not include an explicit estimate-verify-update loop. As shown in Table 4, EVU consistently outperforms MEM1 on ScienceWorld, improving performance from 67.1 to 70.9 on seen tasks and from 64.9 to 70.8 on unseen tasks. These results indicate that passive context management alone is insufficient to overcome belief inertia; the key benefit comes from actively forcing the agent to estimate, verify, and update its belief state. Can EVU benefit stronger agentic planners? To evaluate whether EVU is compatible with more advanced agentic systems, we integrate EVU with LLM-MCTS (Zhao et al., 2023) on VirtualHome. This experiment examines whether EVU is com- MethodSeenUnseen MEM167.164.9 EVU (Ours)70.970.8 Table 4: Comparison with the history-summarization baseline MEM1 on ScienceWorld. EVU consistently outperforms passive context management on both seen and unseen tasks. plementary to search-based planning rather than being tied to a simple action-generation policy. As shown in Table 5, incorporating EVU consistently improves over the strong LLM-MCTS baseline, raising performance from 28.8 to 31.2 on seen tasks and from 25.6 to 28.8 on unseen tasks. This result shows that EVU acts as a generalizable cognitive module that complements advanced planning algo- rithms by helping the agent maintain a more accu- rate belief state during search and execution, rather than being useful only in a standalone intervention setting. MethodSeenUnseen LLM-MCTS28.825.6 LLM-MCTS + EVU (Ours)31.228.8 Table 5: Comparison with the search-based agentic plan- ner LLM-MCTS on VirtualHome. EVU consistently improves over a stronger planning baseline. E Additional Implementation Details Our training infrastructure for both Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is built upon the verl-agent framework (Feng et al., 2025). All key hyperparameters for both phases are summarized in Table 6. The templates corresponding to the ALFWorld, ScienceWorld, and VirtualHome environments are illustrated in Figure 13, Figure 14, and Figure 15, respectively. FExperimental Setup for Different Belief Intervention Strategies In this section, we provide a detailed formulation of the belief intervention strategies employed in our experiments. We first define a general framework for belief intervention and then describe how each specific strategy instantiates this framework. F.1 General Formulation We consider an agent interacting with an environ- ment to achieve a goalG. At any time stept, given HyperparameterValue SFT Phase Learning Rate1e−5 SchedulerCosine Epochs3 RL Phase Group Size (G)6 Learning Rate1× 10 −6 Total Steps250 KL Coefficient (β)0.01 Temperature (rollout)0.8 Temperature (validation)0.0 Common / Other Max Prompt Length5120 Max Response Length512 Micro Batch Size4 Mini Batch Size32 Invalid Action Penalty0.1 Gradient CheckpointingTrue Tensor Parallel Size2 Temperature (eval)0.0 Table 6: Key hyperparameters for SFT and RL training. the interaction history and current observation (col- lectively denoted as ContextC t ), the standard agent directly generates a thought chain and an action. To investigate the role of environmental reason- ing, we introduce an intermediate Belief Interven- tion (BI) module. The decision-making process is decomposed into two phases: 1. Belief Generation: The agent first generates a belief state contentBbased on a specific intervention strategyS . 2. Action Generation: The agent then generates the thought and actionA t conditioned on both the context and the generated belief. Formally, this process can be represented as: C t StrategyS −→B t −→ A t (9) whereB t varies depending on the definition of the strategyS . F.2 Strategy Instantiations Below, we detail the five strategies compared in the main text. To illustrate the differences, we provide a running example where the agent’s goal is to "Put the apple in the fridge", and the agent is currently standing in front of a closed fridge holding an apple. NoneBI (Baseline).This represents the standard setting without any explicit belief intervention. The agent proceeds directly from observation to action. • Formulation: B t =∅ (Empty set). • Mechanism: The thought-action generation is solely dependent on the context C t . Example Output: Thought: The fridge is closed. I need to open it to put the apple inside. Action: Open(Fridge) DBI (Direct Belief Intervention).DBI requires the agent to explicitly generate the current belief state regarding the environment before planning. •Formulation:B t = S t , whereS t is the de- scription of the current environmental state relevant to the goal. Example Output: [Belief]: The fridge is currently closed. I am holding the apple. Thought: Since the fridge is closed, I must open it first. Action: Open(Fridge) OBI (Observation-based Belief Intervention). OBI compels the agent to ground its belief in raw perception. It forces the agent to reiterate the recent observation text explicitly before summarizing the belief state. •Formulation:B t =O t ⊕ S t , whereO t is the reiteration of the observation andS t is the inferred state. Example Output: [Belief]: Observation: I see a white fridge door in front of me. State: The fridge is closed and I cannot see inside. Thought: I need to access the interior of the fridge. Action: Open(Fridge) ReflAct. ReflAct (Kim et al., 2025) encourages the agent to reflect on its dynamic progress rather than just the static state. It prompts the agent to evaluate what has been accomplished and what remains to be done relative to the goal G. • Formulation:B t = Reflect(S t ,G), focusing on the gap between current state and goal. Example Output: [Belief]: I have successfully picked up the apple. However, the goal is to put it in the fridge, and the fridge is currently closed. Thought: The next logical step is to open the fridge. Action: Open(Fridge) VAGEN.VAGEN (Wang et al., 2025c) tasks the agent with a predictive "look-ahead" process. The agent must identify the current belief state, propose a hypothetical action, and then predict the resulting environmental state. •Formulation:B t = S t Hypothetical Action ̃a −→ S t+1 , representing a mental simulation of the immediate future. Example Output: [Belief]: Current State: The fridge is closed. Hypothetical Action: Open the fridge. Predicted Future State: The fridge door will be open, revealing the contents inside. Thought: This action will allow me to place the apple inside. Action: Open(Fridge) G Experimental Setup for Belief Inertia Mitigation Analysis To investigate the internal mechanism ofEVUin mitigating belief inertia, we conduct a fine-grained probing analysis using the identical set of observa- tional neglect cases described in Section 3.2. These cases represent critical moments where the agent’s actiona t fails to reflect the current observationo t , indicating a strong inertia from prior beliefs. We track the belief evolution by probing the agent at four distinct stages of our active belief dynamic process. For each stage, we construct the input prompt by progressively accumulating the intermediate reasoning outputs. Specifically, the probing stages are defined as follows: (1) Cur- rent Belief: This represents the baseline state prior to any active intervention, probed using the his- torical context(B t−1 ,a t−1 ,o t−1 ); (2) Estimated Belief: This captures the belief state immediately after the estimation phase, where the context is augmented with the generated estimation to form (B t−1 ,a t−1 ,o t−1 ,E t ); (3) Verification Belief: This reflects the state after the agent validates the estimation against the observation, probed under the context(B t−1 ,a t−1 ,o t−1 ,E t ,V t ); and (4) Up- dated Belief: This represents the final consolidated state after the update phase, probed using the com- plete context(B t−1 ,a t−1 ,o t−1 ,E t ,V t ,B t ). By comparing the belief values across these stages, we quantify the contribution of each component in correcting the belief inertia. H Experimental Setup for Task Difficulty Analysis To systematically evaluate the robustness of active belief dynamics as tasks become more challeng- ing, we conduct a difficulty analysis within the ALFWorld environment. We classify task diffi- culty based on the minimum number of subgoals required to achieve the final objective, as tasks with more subgoals necessitate longer interaction hori- zons. We obtain four distinct difficulty levels: Easy (0–4 subgoals), Medium (5–8 subgoals), Hard (9– 12 subgoals), and Very Hard (13–16 subgoals). We randomly sampled a total of 200 tasks to ensure diverse coverage across these difficulty levels. For each level, we compare the success rate of our pro- posed method against three training baselines: SFT, GRPO, and PPO. To quantify the advantage, we calculate the relative success rate improvement (∆ Success Rate) of our method over each baseline. ALFWorld Prompt Template Interact with a household to solve a task. Imagine you are an intelligent agent in a household environment and your target is to perform actions to complete the task goal. At each step, you will be given task goal, action history and the last turn’s information (Reason, Belief State, Thought, and Action). You need to process the information in a specific order: 1. Reason: Analyze the last action and the observation in one or two concise sentences. What did you expect to see? What did you actually see? Does this confirm or contradict your previous belief? 2.Belief State: State where the agent is, what it is holding, and the known status of goal-related objects. Do NOT list irrelevant objects. 3. Thought: Plan your future actions based on the updated belief. 4. Action: Output your next action. The available actions are: 1. go to (recep) 2. task (obj) from (recep) 3. put (obj) in/on (recep) 4. open (recep) 5. close (recep) 6. toggle (obj) (recep) 7. clean (obj) with (recep) 8. heat (obj) with (recep) 9. cool (obj) with (recep) where (obj) and (recep) correspond to objects and receptacles. After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. If the environment output “Nothing happened”, that means the previous action is invalid and you should try more options. Your response should use the following format: Reason: <Analyze expectation vs. actual observation to update your understanding> Belief State: <your belief state> Thought: <your thoughts> Action: <your next action> Your task is to complete the task goal: task_goal Below is the action history and the last turn’s information: Action History: action_history Last Turn’s Information: last_turn_information Figure 13: Prompt template of our method on the ALFWorld benchmark. VirtualHome Prompt Template Interact with a household to solve a task. Imagine you are an intelligent agent in a household environment and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given the detailed description of the current environment and your goal to accomplish. At each step, you will be given task goal, action history and the last turn’s information (Reason, Belief State, Thought, and Action). You need to process the information in a specific order: 1.Reason: Analyze the last action and the observation in one or two concise sentences. What did you expect to see? What did you actually see? Does this confirm or contradict your previous belief? 2.Belief State: State where the agent is, what it is holding, and the known status of goal-related objects. Do NOT list irrelevant objects. 3. Thought: Plan your future actions based on the updated belief. 4. Action: Output your next action. The available actions are: 1. walk to (obj) 2. run to (obj) 3. grab (obj) 4. open (obj) 5. close (obj) 6. put (obj) on (recep) 7. put (obj) in (recep) 8. switch on (obj) 9. switch off (obj) 10. drink (obj) 11. look at (obj) 12. sit on (obj) 13. stand up 14. watch (obj) 15. wipe (obj) 16. type on (obj) 17. wash (obj) 18. cut (obj) 19. eat (obj) 20. sleep 21. wake up 22. plug in (obj) 23. plug out (obj) 24. pour (obj) into (recep) 25. move (obj) 26. release 27. turn to (obj) After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. If the environment output “Nothing happened”, that means the previous action is invalid and you should try more options. Your response should use the following format: Reason: <Analyze expectation vs. actual observation to update your understanding> Belief State: <your belief state> Thought: <your thoughts> Action: <your next action> Your task is to complete the task goal: task_goal Below is the action history and the last turn’s information: Action History: action_history Last Turn’s Information: last_turn_information Figure 14: Prompt template of our method on the VirtualHome benchmark. ScienceWorld Prompt Template You are a helpful assistant to do some scientific experiment in an environment. In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, hallway. You should explore the environment and find the items you need to complete the experiment. You can teleport to any room in one step. All containers in the environment have already been opened, you can directly get items from the containers. At each step, you will be given task goal, action history and the last turn’s information (Reason, Belief State, Thought, and Action). You need to process the information in a specific order: 1. Reason: Analyze the last action and the observation in one or two concise sentences. What did you expect to see? What did you actually see? Does this confirm or contradict your previous belief? 2.Belief State: State where the agent is, what it is holding, and the known status of goal-related objects. Do NOT list irrelevant objects. 3. Thought: Plan your future actions based on the updated belief. 4. Action: Output your next action. The available actions are: • open OBJ: open a container • close OBJ: close a container • activate OBJ: activate a device • deactivate OBJ: deactivate a device • connect OBJ to OBJ: connect electrical components • disconnect OBJ: disconnect electrical components • use OBJ [on OBJ]: use a device/item • look around: describe the current room • examine OBJ: describe an object in detail • look at OBJ: describe a container’s contents • read OBJ: read a note or book • move OBJ to OBJ: move an object to a container • pick up OBJ: move an object to the inventory • pour OBJ into OBJ: pour a liquid into a container • mix OBJ: chemically mix a container • teleport to LOC: teleport to a specific room • focus on OBJ: signal intent on a task object • wait: task no action for 10 steps • wait1: task no action for a step Your response should use the following format: Reason: <Analyze expectation vs. actual observation to update your understanding> Belief State: <your belief state> Thought: <your thoughts> Action: <your next action> Your task is to complete the task goal: task_goal Below is the action history and the last turn’s information: Action History: action_history Last Turn’s Information: last_turn_information Figure 15: Prompt template of our method on the ScienceWorld benchmark.