Paper deep dive
World Potential Model: Pretrained World Knowledge as Progress Potentials
Jun Zhao, Jixin Tang, Yang Shu, Jinyang Wu, Yuyang Lu, Jingqi Tong, Hao Xu, Weifeng Ge, Qi Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/11/2026, 2:44:23 AM
Summary
The paper introduces the World Potential Model (WPM), a framework that leverages pretrained world knowledge to evaluate task-relative realized progress in long-horizon language agents without task-specific fine-tuning. WPMs provide ordinal progress judgments and can be anchored to milestones to create scalar potentials. These potentials offer process-sensitive step-level credit for policy optimization, significantly improving success rates over outcome-only methods like GRPO in environments such as ALFWorld and ScienceWorld.
Entities (6)
Relation Signals (5)
Jun Zhao → authored → World Potential Model
confidence 95% · World Potential Model: Pretrained World Knowledge as Progress Potentials Jun Zhao1,*,†
World Potential Model → evaluatedin → AlfWorld
confidence 95% · In ALFWorld and ScienceWorld, off-the-shelf pretrained models substantially outperform chance
World Potential Model → evaluatedin → ScienceWorld
confidence 95% · In ALFWorld and ScienceWorld, off-the-shelf pretrained models substantially outperform chance
World Potential Model → uses → Pretrained Models
confidence 95% · We formalize this capability with a World Potential Model (WPM)... off-the-shelf pretrained models substantially outperform chance
World Potential Model → improves → GRPO
confidence 90% · WPM-guided optimization improves success over outcome-only GRPO across all evaluated configurations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-horizon language agents often receive supervision only from terminal task outcomes, leaving little signal for distinguishing productive intermediate behavior from stagnation or even regression. Rather than learning a separate value function or process reward model for every task, we ask whether pretrained models can recognize task progress from their existing world knowledge. We formalize this capability with a World Potential Model (WPM), a goal-conditioned evaluator of task-relative realized progress in agent contexts. In ALFWorld and ScienceWorld, off-the-shelf pretrained models substantially outperform chance at recovering realized-progress structure without task-specific evaluator fine-tuning. We further anchor these progress judgments to task-specific milestones to obtain scalar world potentials, whose temporal differences provide process-sensitive step-level credit for policy optimization. Under matched comparisons, WPM-guided optimization improves success over outcome-only GRPO across all evaluated configurations. Together, these results provide initial evidence that pretrained world knowledge can support reusable realized-progress evaluation and provide useful supervision for long-horizon agents.
Tags
Links
- Source: https://arxiv.org/abs/2610.09560v1
- Canonical: https://arxiv.org/abs/2610.09560v1
Trouble viewing inline? Open PDF directly →
Full Text
77,444 characters extracted from source content.
Expand or collapse full text
jzhao.nlp@gmail.com ://github.com/Gabrile166/world-potential-model World Potential Model: Pretrained World Knowledge as Progress Potentials Jun Zhao1,*,† , Jixin Tang2,*, Yang Shu2, Jinyang Wu3, Yuyang Lu2, Jingqi Tong2, Hao Xu4, Weifeng Ge2, Qi Zhang2 * Equal contribution. † Corresponding author. Affiliation: National University of Singapore Affiliation: Fudan University Affiliation: Tsinghua University Affiliation: The University of Sydney Abstract Long-horizon language agents often receive supervision only from terminal task outcomes, leaving little signal for distinguishing productive intermediate behavior from stagnation or even regression. Rather than learning a separate value function or process reward model for every task, we ask whether pretrained models can recognize task progress from their existing world knowledge. We formalize this capability with a World Potential Model (WPM), a goal-conditioned evaluator of task-relative realized progress in agent contexts. In ALFWorld and ScienceWorld, off-the-shelf pretrained models substantially outperform chance at recovering realized-progress structure without task-specific evaluator fine-tuning. We further anchor these progress judgments to task-specific milestones to obtain scalar world potentials, whose temporal differences provide process-sensitive step-level credit for policy optimization. Under matched comparisons, WPM-guided optimization improves success over outcome-only GRPO across all evaluated configurations. Together, these results provide initial evidence that pretrained world knowledge can support reusable realized-progress evaluation and provide useful supervision for long-horizon agents. Figure 1: Overview of the World Potential Model (WPM) and its downstream scalarization. A WPM elicits task-relative realized-progress structure from pretrained world knowledge, with ordinal progress judgments as its primary interface. For downstream use, these judgments can be referenced to task-specific progress anchors to obtain scalar world potentials, whose temporal differences provide process-sensitive credit for policy optimization. 1 Introduction Large language models (LLMs) are rapidly becoming the cognitive core of agentic systems Wang et al. (2024b), where they interpret environments Huang et al. (2023), invoke tools Schick et al. (2023); Paranjape et al. (2023), and drive multi-step reasoning Yao et al. (2023); Zhao et al. (2024); Wang et al. (2024a) across increasingly complex tasks. As these tasks become longer-horizon, however, terminal outcomes alone provide too little structure for evaluating intermediate processes Rajasekaran (2026); Young (2025); Zhou et al. (2026); Ju et al. (2026); Merrill et al. (2026); Li et al. (2026); Chandwani and Gupta (2026); Tan et al. (2026). A failed trajectory may have made substantial progress before a late mistake, while a successful trajectory may contain redundant exploration, prolonged stagnation, or temporary regression. Long-horizon agents therefore need a way to distinguish productive intermediate behavior before the final outcome is known. Value functions and process reward models provide one route to denser supervision by estimating future return or scoring intermediate behavior Ouyang et al. (2022); Lightman et al. (2023). These evaluators can be effective, but typically rely on task-specific annotation and training. For agents expected to operate across heterogeneous tasks and environments, repeatedly constructing a new evaluator may be expensive and difficult to scale. This raises a complementary question: must progress evaluation itself require learning a task-specific evaluator, or can it be elicited from knowledge already acquired during pretraining? Pretrained foundation models encounter broad descriptions of procedures, physical interactions, causal relationships, and the structure of complex tasks (Huang et al., 2022; ichter et al., 2023; Rocamonde et al., 2024; Ruis et al., 2025). Such knowledge provides a plausible basis for recognizing not only whether an individual action is plausible, but also what has already been accomplished in an evolving agent context. For example, acquiring a required object, resolving a constraint, or completing a necessary subprocedure may represent meaningful progress, whereas undoing a satisfied condition may represent regression. This suggests that pretrained models may support a reusable capability for recognizing realized task progress without task-specific evaluator training. We formalize this hypothesized capability through the abstraction of a World Potential Model (WPM). Given a task goal, a WPM evaluates the goal-relevant progress already realized in an agent context using pretrained world knowledge. We call the resulting task-relative progress structure world potential. Because globally calibrated scalar progress is difficult to define across open-ended tasks, we take ordinal judgments of relative progress as the primary interface to this structure. To obtain a scalar potential for downstream use, we introduce one concrete instantiation that anchors these judgments to ordered task-specific milestones. The resulting task-local potentials can then provide step-level progress signals through their temporal differences. We study the WPM abstraction and its downstream utility in ALFWorld Shridhar et al. (2021) and ScienceWorld Wang et al. (2022), two structured long-horizon environments in which privileged progress signals enable controlled evaluation. Across both environments, off-the-shelf pretrained models substantially outperform chance at recovering realized-progress structure without task-specific evaluator fine-tuning, providing evidence for the existence of the progress-recognition capability underlying WPMs. We further find that, when these judgments are anchored to task-specific milestones, the resulting scalar potentials provide useful downstream supervision: under matched comparisons, WPM-guided optimization improves success over outcome-only GRPO across all evaluated configurations. This paper makes three contributions. First, we introduce the World Potential Model, a general abstraction for eliciting task-relative realized progress from pretrained world knowledge. Second, we develop a practical framework for turning realized-progress judgments into task-local scalar potentials for long-horizon agents. Third, we show how world potentials can provide process-sensitive supervision for sparse-reward policy optimization. Together, our results provide initial evidence that realized-progress evaluation is a reusable capability of pretrained models and a useful source of supervision for long-horizon agents. 2 Problem Setup and World Potential Models 2.1 Long-Horizon Sparse-Reward Agent Tasks We consider a finite-horizon episodic task specified by a goal or instruction g. The environment may be partially observable. At step t, let xtx_t denote the agent-visible interaction context, comprising the information available from the interaction history, including observations, previous actions, tool outputs, intermediate reasoning, memory, and other task-relevant information. The policy conditions on both the task goal and the current context, at∼π(⋅∣g,xt).a_t π(· g,x_t). (1) After the action is executed, the environment returns new feedback or observations, which are incorporated into the interaction context to form xt+1x_t+1. A trajectory is therefore represented as τ=(x0,a0,x1,a1,…,xT).τ=(x_0,a_0,x_1,a_1,…,x_T). (2) This formulation accommodates embodied, web, software, and tool-using agents whose decisions depend on an evolving interaction context rather than a compact fully observed environment state. We do not require the agent-visible context itself to be Markov. We focus on sparse, outcome-only supervision: rtenv=0,t<T−1,R(τ),t=T−1,r_t^env= cases0,&t<T-1,\\ R(τ),&t=T-1, cases (3) where R(τ)R(τ) denotes the terminal task outcome and is binary in our experiments. Such feedback determines whether the episode ultimately succeeds, but provides little direct information about the goal-relevant progress realized at intermediate contexts. 2.2 World Potential Models World potential. For a task goal g, a World Potential Model (WPM) is an evaluator that uses pretrained world knowledge to assess an agent context x in terms of the goal-relevant progress already realized toward g. We refer to the resulting task-relative progress structure as world potential. The target is retrospective: it characterizes what has already been accomplished in the current context, rather than predicting future return. A WPM need not directly produce a globally calibrated numerical progress score. Instead, we take ordinal judgments of relative realized progress as its primary interface, and introduce a task-local scalar representation only when needed for downstream use. Ordinal progress interface. Given two contexts xix_i and xjx_j from the same task instance, a WPM compares their relative realized progress. We write xi⪰gxj,x_i _gx_j, (4) when the WPM judges xix_i to embody at least as much goal-relevant realized progress toward g as xjx_j. Relative progress can be meaningful even when an absolute progress score is not. For example, after an agent acquires a necessary clue, it may be natural to judge the resulting context as representing greater realized progress than the preceding one, while assigning either context a globally meaningful score such as “0.63 complete” is considerably less well-defined. The relation is task-relative: a WPM need not place unrelated tasks on a common numerical scale, but only compare realized progress within the task being evaluated. Listing in Appendix A.1 gives the pairwise prompt used in our implementation. 2.3 Anchored Scalar World Potentials The ordinal WPM interface in Section 2.2 does not require a numerical notion of progress. For downstream applications that require a scalar signal, we introduce a task-local reference frame consisting of ordered progress anchors, Mg=(mk,ϕk)k=0K,M_g=\(m_k, _k)\_k=0^K, (5) where each mkm_k describes a verifiable goal-relevant accomplishment and ϕk _k assigns it a task-local scalar coordinate. The anchors range from task initiation to completion, with m0m_0 representing no meaningful progress and mKm_K representing task completion, and 0=ϕ0<ϕ1<⋯<ϕK=1.0= _0< _1<·s< _K=1. (6) These coordinates need only be meaningful within the task; no numerical calibration across unrelated tasks is assumed. Given an intermediate context xtx_t, the WPM localizes its realized progress relative to the ordered anchors. We define zt=maxk∣xt⪰gmk,z_t= \k x_t _gm_k\, (7) where ztz_t denotes the most advanced anchor relative to which xtx_t is judged to embody at least as much goal-relevant realized progress. The corresponding scalar world potential is Φg(xt)=ϕzt. _g(x_t)= _z_t. (8) This construction separates task-local calibration from progress evaluation. The anchors specify a sparse reference frame for the task, while the frozen WPM supplies the evaluative mapping that places arbitrary agent contexts onto that frame using pretrained knowledge. The formulation itself is agnostic to how the reference frame is obtained. In our experiments, we instantiate MgM_g once per task from the task description and a single successful expert trajectory. The resulting anchors are then fixed, and the successful trajectory is not provided when the WPM evaluates new agent contexts. Appendix A.2 describes this construction in detail. 2.4 Relation to Existing Abstractions World Potential Models are related to value functions, reward models, and world models, but differ in what they evaluate and how that evaluation is obtained. Value functions. Both value functions and WPMs evaluate an agent context or state, but their targets differ fundamentally. A policy-conditioned value function Vπ(x)V^π(x) is prospective: it estimates the expected future return obtainable from x under policy π, and therefore generally depends on the policy being evaluated. A WPM is retrospective: it evaluates the goal-relevant progress already realized in the current context rather than predicting future return. This progress judgment is defined independently of the policy that produced the context. Two contexts may therefore embody similar realized progress even if their future prospects under a particular policy differ substantially. Reward and process-reward models. World potentials can be converted into reward-like signals, for example through temporal differences such as γΦg(xt+1)−Φg(xt)γ _g(x_t+1)- _g(x_t), but the WPM itself is an evaluator of realized progress rather than a reward model trained for a particular task. Conventional reward or process-reward models are typically fitted from task-specific labels, preferences, or rollouts. In our setting, the evaluator parameters receive no task-specific fine-tuning; instead, progress judgments are elicited from a pretrained model and, for the scalar instantiation above, referenced to task-specific milestone anchors. Thus, task-specific structure enters through the scalarization reference frame rather than through evaluator parameter training. World models. Despite the shared term world, a WPM is not a model of environment dynamics. Conventional world models predict transitions, future observations, or other consequences of actions, for example through p(st+1∣st,at)p(s_t+1 s_t,a_t). A WPM neither predicts transitions nor simulates future states. It evaluates the realized progress represented in the current context. World instead refers to the pretrained world knowledge from which these progress judgments are elicited. 3 Do Pretrained Models Expose World-Potential Structure? The central empirical question behind the WPM abstraction is whether pretrained models can recover meaningful task-relative progress structure from agent contexts without task-specific evaluator fine-tuning. We evaluate this question at the two levels introduced in Section 2. Pairwise progress comparison directly tests whether pretrained models can make the relative progress judgments that constitute the WPM’s primary ordinal interface. Anchored progress localization further tests whether these judgments can be referenced to a fixed task-specific milestone ladder to obtain the task-local localization used in the scalar construction. Table 1: Accuracy (%) on two realized-progress evaluations. Pairwise comparison directly evaluates the ordinal progress interface of a WPM, while Anchored localization evaluates context localization relative to a task-specific milestone ladder. Random Guess denotes uniform selection among the valid candidate labels for each instance and is averaged over the corresponding evaluation set. Model ALFWorld ScienceWorld Avg. Pairwise Anchored Pairwise Anchored Random Guess 50.00 20.00 50.00 15.27 33.82 MiMo-VL-7B XiaomiMiMo (2025) 92.67 58.00 97.33 63.33 77.83 Qwen2.5-VL-7B Qwen Team (2025b) 80.00 60.00 81.33 48.00 67.33 Qwen2.5-32B Qwen Team (2024) 95.33 80.67 96.67 76.00 87.17 Qwen2.5-VL-72B Qwen Team (2025a) 93.33 82.00 94.67 81.33 87.83 Qwen3-30B-A3B Qwen Team (2025c) 91.33 86.67 97.33 80.00 88.83 Qwen3-VL-30B-A3B Qwen Team (2025d) 90.00 88.00 98.00 85.33 90.33 Qwen3-VL-32B Qwen Team (2025e) 92.67 86.67 94.67 70.00 86.00 3.1 Evaluation Tasks Pairwise progress comparison. Given two agent contexts xix_i and xjx_j associated with the same task goal g, the model predicts which context embodies greater goal-relevant progress already realized toward task completion. The two contexts may arise from different interaction trajectories and need not share a common prefix. The task therefore requires comparing the realized progress represented in the contexts rather than relying on their temporal positions within a trajectory. No milestone ladder is provided, so this evaluation directly probes the ordinal progress interface defined in Section 2.2. Anchored progress localization. Given an agent context xtx_t together with its fixed task-specific milestone ladder MgM_g, the model predicts the anchor index ztz_t corresponding to the context’s realized-progress level. The number of anchors is determined during task-specific milestone construction and may vary across tasks. The corresponding scalar world potential is then given by the fixed anchor coordinate ϕzt _z_t. This evaluation therefore tests the anchored scalar representation introduced in Section 2.3 rather than requiring the model to assign an unconstrained numerical progress score directly to the context. Both tasks are formulated as categorical prediction problems. The exact evaluation prompts are provided in Appendix A.1. 3.2 Experimental Setting Benchmarks and labels. We instantiate both evaluation tasks in ALFWorld Shridhar et al. (2021) and ScienceWorld Wang et al. (2022), two long-horizon interactive environments in which successful task completion requires multiple stages of meaningful intermediate progress. Both environments provide privileged signals for constructing environment-grounded progress labels. In ALFWorld, we use normalized remaining expert-plan length as a planner-based proxy: although formally a cost-to-go measure, manual inspection indicates that its ordering tracks relative realized progress, and we filter near-tied states. In ScienceWorld, the native task score directly awards partial credit for satisfied goals and subgoals. These signals are used only for evaluation-label construction and are never provided to the models; Appendix A.3 gives full details. Models and evaluation protocol. We evaluate seven language and vision-language models from three model families: MiMo, Qwen2.5, and Qwen3. All models are tested under the same prompting protocol and without task-specific evaluator fine-tuning. We measure accuracy against environment-grounded labels. For an evaluation instance with CiC_i candidate labels, uniform random guessing has expected accuracy 1/Ci1/C_i. Pairwise comparison therefore has a fixed chance accuracy of 50%50\%, whereas anchored progress localization has a task-dependent chance accuracy because the number of milestone anchors varies across tasks. The mean random-guess accuracies are 20.00%20.00\% in ALFWorld and 15.27%15.27\% in ScienceWorld. 3.3 Results Table 1 reports performance on pairwise progress comparison and anchored progress localization. Relative realized progress is strongly recoverable in these settings. All evaluated models substantially outperform the 50%50\% random baseline on pairwise progress comparison. Averaged across models and environments, pairwise accuracy reaches 92.52%92.52\%, with the best results reaching 95.33%95.33\% on ALFWorld and 98.00%98.00\% on ScienceWorld. These results provide direct evidence that pretrained models can often determine which of two partially completed agent contexts embodies greater goal-relevant realized progress, even when the contexts arise from different interaction paths. The result is consistent across all three evaluated model families, both language-only and vision-language models, and both environments, suggesting that it is not specific to a single model or benchmark. Anchored progress localization is harder but remains informative. Average localization accuracy across models and environments reaches 74.71%74.71\%, with all evaluated models exceeding the corresponding random baselines of 20.00%20.00\% in ALFWorld and 15.27%15.27\% in ScienceWorld. The best results reach 88.00%88.00\% and 85.33%85.33\%, respectively. Although localization accuracy is lower than pairwise accuracy, this gap is consistent with the different structure of the two evaluations: pairwise comparison requires resolving a single relative-progress relation, whereas anchored localization requires placing a context among multiple ordered, task-dependent progress anchors. These results show that pretrained models can meaningfully localize agent contexts relative to fixed task-specific milestone ladders, providing the localization step required by the scalar construction in Section 2.3. Takeaway. The pairwise results support the core premise of the WPM abstraction: in two controlled long-horizon environments, off-the-shelf pretrained models recover substantial task-relative ordinal realized-progress structure without task-specific evaluator fine-tuning. Anchored localization is less accurate but remains substantially above chance, showing that these judgments can also be referenced to task-specific progress anchors for the scalar construction introduced in Section 2.3. We next ask whether world potentials constructed from these imperfect judgments provide useful downstream supervision. 4 Progress-Sensitive Policy Optimization with World Potentials The anchored construction in Section 2.3 provides a task-local scalar potential Φg(x) _g(x) over agent contexts. We now consider one downstream use of this signal: converting temporal changes in world potential into process-sensitive step-level credit for critic-free policy optimization. Section 5 uses this construction to test whether imperfect world-potential estimates provide useful supervision beyond sparse terminal outcomes. 4.1 World-Potential Transition Signal Under the sparse-reward setting of Section 2.1, the environment reward is zero at intermediate steps and is determined by the task outcome on the terminal transition. We augment this sparse signal with the temporal difference in world potential: rtWP=rtenv+γΦg(xt+1)−Φg(xt),r_t^WP=r_t^env+γ _g(x_t+1)- _g(x_t), (9) where γ∈(0,1]γ∈(0,1] is the base temporal discount. A transition to a context with higher world potential receives a positive potential contribution, whereas a transition to a lower-potential context receives a negative contribution. When the potential remains unchanged (i.e., Φg(xt+1)=Φg(xt)>0 _g(x_t+1)= _g(x_t)>0) and γ<1γ<1, the potential contribution becomes −(1−γ)Φg(xt)<0-(1-γ) _g(x_t)<0, yielding a small negative signal and mildly discouraging interaction that fails to advance the task. 4.2 Process-Sensitive Returns We next aggregate the transition signals in Eq. 9 into step-level returns. To control how future intermediate contexts contribute to the credit assigned at step t, we introduce a process-sensitivity parameter ρ∈[0,1]ρ∈[0,1] and define the effective return discount as β=γρ.β=γρ. (10) The resulting world-potential return from step t is GtWP,ρ=∑ℓ=0T−t−1βℓrt+ℓWP=∑ℓ=0T−t−1(γρ)ℓrt+ℓWP.G_t^WP,ρ= _ =0^T-t-1β r_t+ ^WP= _ =0^T-t-1(γρ) r_t+ ^WP. (11) Let H=T−tH=T-t. Substituting Eq. 9 and collecting the potential terms gives GtWP,ρ= G_t^WP,ρ= ∑ℓ=0H−1βℓrt+ℓenv⏟environment return−Φg(xt)⏟current-context potential+γ(1−ρ)∑k=1H−1βk−1Φg(xt+k)⏟intermediate world potentials+γβH−1Φg(xT)⏟terminal-context potential. _ =0^H-1β r_t+ ^env_environment return- _g(x_t)_current-context potential+ γ(1-ρ) _k=1^H-1β^k-1 _g(x_t+k)_intermediate world potentials+ γβ^H-1 _g(x_T)_terminal-context potential. (12) For 0<ρ<10<ρ<1, each future intermediate context contributes explicitly to the return with weight wk(ρ)=γ(1−ρ)(γρ)k−1=γk(1−ρ)ρk−1,w_k(ρ)=γ(1-ρ)(γρ)^k-1=γ^k(1-ρ)ρ^k-1, (13) where 1≤k<H1≤ k<H and adjacent weights satisfy wk+1(ρ)wk(ρ)=γρ<1, w_k+1(ρ)w_k(ρ)=γρ<1, (14) so the contribution of intermediate world potentials decays geometrically with temporal distance. Smaller values of ρ concentrate this contribution more strongly on nearer-term contexts, whereas larger values extend credit over a longer portion of the remaining trajectory. The return can therefore remain sensitive to when progress is realized and how it evolves over subsequent intermediate contexts. The boundary cases further clarify the role of ρ. At ρ=0ρ=0, we have β=0β=0, and the return reduces to the one-step transition signal, GtWP,0=rtWP.G_t^WP,0=r_t^WP. (15) At ρ=1ρ=1, by contrast, β=γβ=γ and the coefficient on every intermediate potential in Eq. 12 becomes zero. In this case, using the same discount γ for both the potential difference and its temporal accumulation causes the intermediate potential terms to cancel telescopically, leaving GtWP,1=∑ℓ=0H−1γℓrt+ℓenv−Φg(xt)+γHΦg(xT).G_t^WP,1= _ =0^H-1γ r_t+ ^env- _g(x_t)+γ^H _g(x_T). (16) Thus, for 0<ρ<10<ρ<1, intermediate world potentials remain explicitly represented in the return rather than canceling out. The resulting return is therefore intentionally process-sensitive: it uses intermediate realized-progress estimates to distinguish trajectories that may share the same terminal outcome but differ in how progress unfolds along the way. 4.3 Step-Level Group-Relative Advantages We retain the group-relative standardization principle of GRPO, but apply it to step-level world-potential returns rather than trajectory-level outcome rewards. Consider a rollout group of N trajectories sampled for the same task g. For every valid environment step, we compute the world-potential return Gi,tWP,ρG^WP,ρ_i,t and standardize these returns across all valid environment steps from the N trajectories in the task group: A^i,t=Gi,tWP,ρ−μgstepσgstep+ϵnorm, A_i,t= G^WP,ρ_i,t-μ^step_gσ^step_g+ _norm, (17) where μgstepμ^step_g and σgstepσ^step_g denote the mean and standard deviation over the corresponding set of valid step-level returns. This changes the granularity of credit assignment relative to our outcome-only GRPO baseline. In that baseline, each trajectory receives a single group-relative advantage derived from its terminal outcome, which is shared by all actions in the trajectory. WPM-guided optimization instead assigns a distinct advantage to each valid environment step according to its process-sensitive return. Actions followed by different patterns of progress, stagnation, or regression can therefore receive different optimization weights even within the same trajectory. 5 A Utility Case Study in Sparse-Reward Policy Optimization Section 3 shows that pretrained models can recover meaningful realized-progress structure from agent contexts, while Section 4 converts anchored world potentials into process-sensitive step-level credit. We now ask whether this credit provides useful supervision for policy learning under sparse terminal rewards. We compare WPM-guided optimization with outcome-only GRPO under matched actor-training configurations, using task success as the downstream evaluation criterion. This experiment is intended as a controlled test of the utility of WPM-derived process-sensitive credit rather than as a comprehensive comparison of reinforcement learning algorithms. 5.1 Experimental Setup Environments and data splits. We evaluate on the same two long-horizon interactive environments used in Section 3: ALFWorld and ScienceWorld. For ALFWorld, policies are trained on the official training split and evaluated on valid_unseen. For ScienceWorld, policies are trained on the official training variations and evaluated on the corresponding test variations. Both environments provide sparse terminal supervision. Policies and controlled comparison. We optimize Qwen2.5-Instruct models at 3B and 7B scale and Qwen3 at 4B scale. For reference, we also report the corresponding vanilla checkpoints. Our primary comparison is between outcome-only GRPO and WPM-guided optimization. For each actor model, the two optimization methods use matched task sampling, rollout construction, and actor-update configurations; they differ in the learning signal and resulting credit construction. Outcome-only GRPO derives trajectory-level advantages from terminal task outcomes, whereas WPM-guided optimization uses the process-sensitive step-level returns defined in Section 4. World potentials are produced by a frozen Qwen3-VL-32B evaluator using the fixed task-specific milestone ladders described in Section 2.3. The evaluator parameters remain fixed throughout actor training. For each actor–environment configuration, we run both methods with the same three random seeds and report mean performance across the three runs. Evaluation metric. We report task success rate, defined as the percentage of evaluation episodes whose final environment state satisfies the task goal. Full details of task sampling, rollout construction, optimization hyperparameters, evaluator settings, and metric computation are provided in Appendix C. Table 2: Task success rate (%) for vanilla policies, outcome-only GRPO, and WPM-guided optimization in ALFWorld and ScienceWorld. Results for GRPO and WPM are averaged over three independent random seeds. Model Method Alf World Science World Qwen2.5‑3B Vanilla 10.93 8.59 GRPO 53.90 51.56 WPM 60.50 55.39 Qwen2.5‑7B Vanilla 15.62 13.28 GRPO 68.70 66.40 WPM 72.10 74.73 Qwen3‑4B Vanilla 22.65 19.53 GRPO 48.43 42.96 WPM 53.64 46.09 5.2 Results WPM-guided optimization consistently improves task success. As shown in Table 2, WPM-guided optimization outperforms the matched outcome-only GRPO baseline in all six actor–environment configurations. The improvements range from 3.133.13 to 8.338.33 percentage points, with an average gain of 5.085.08 points. The largest improvement occurs for Qwen2.5-7B on ScienceWorld, where success increases from 66.40%66.40\% to 74.73%74.73\%. Improvements are observed for all three actor models in both environments. These results provide evidence that the realized-progress information recovered by the WPM can be converted into useful supervision for sparse-reward policy optimization. In particular, under the matched training configurations studied here, the process-sensitive credit construction of Section 4 provides more useful learning signal than assigning credit solely from terminal trajectory outcomes. The experiment therefore establishes a downstream utility result for world potentials in the evaluated settings, without requiring the WPM evaluator itself to be trained on task-specific progress labels. 6 Related Work Process supervision and progress-based credit. Process supervision provides intermediate learning signals beyond terminal correctness Lightman et al. (2023). Recent work extends this idea to interactive agents through learned process reward models, rollout-derived credit, and subgoal- or progress-based supervision Tan et al. (2026); Choudhury (2025); Xi et al. (2025); Zhang et al. (2025a); Auzina et al. (2026); Wang et al. (2026). These approaches reinforce the importance of intermediate credit in long-horizon tasks, but obtain such credit from different sources, including policy rollouts, learned evaluators, intrinsic beliefs, or explicit subgoal structure. Our focus is complementary: we ask whether a frozen pretrained model already exposes a task-relative notion of realized progress that can be elicited without task-specific evaluator fine-tuning. A WPM therefore treats realized-progress evaluation as the object of study itself, with ordinal comparison as its primary interface, rather than defining progress through predicted future return. Foundation models for reward and progress construction. Pretrained language and vision-language models have been used to generate reward functions, provide reward judgments, and construct task-progress signals Xie et al. (2024); Ma et al. (2024); Rocamonde et al. (2024); Wang et al. (2024c); Sarukkai et al. (2025); Zhang et al. (2025b); Müller and Kudenko (2026). These methods share with WPMs the premise that pretrained models contain task knowledge that can support downstream learning. Our formulation isolates a different interface to that knowledge: the pretrained evaluator makes ordinal judgments about progress already realized in an agent context, which can then be referenced to task-specific anchors when a scalar potential is needed. This separates the general progress-evaluation capability from the particular scalarization and optimization scheme used downstream. Our policy-optimization construction is also related to potential-based reward shaping Ng et al. (1999); however, for 0<ρ<10<ρ<1, intermediate world potentials remain explicitly represented in the process-sensitive return, so the resulting objective is not the classical policy-invariant shaping construction. 7 Conclusion We introduced the World Potential Model (WPM), an abstraction for eliciting task-relative realized progress from pretrained world knowledge. In ALFWorld and ScienceWorld, off-the-shelf pretrained models recover substantial realized-progress structure without task-specific evaluator fine-tuning. We further showed that these judgments can be anchored into scalar world potentials and converted into process-sensitive step-level credit. Under matched actor-training configurations, WPM-guided optimization improves task success over outcome-only GRPO across all evaluated settings. Together, these results provide initial evidence that realized-progress evaluation can serve as a reusable interface between pretrained world knowledge and long-horizon agent learning. Evaluating world potentials across broader tasks and agent settings remains an important direction. References Auzina et al. (2026) I. A. Auzina, J. Strüber, S. Hernández-Gutiérrez, S. Goel, A. Prabhu, and M. Bethge Intrinsic credit assignment for long horizon interaction. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §6. Chandwani and Gupta (2026) A. Chandwani and I. Gupta Beyond binary correctness: scaling evaluation of long-horizon agents on subjective enterprise tasks. External Links: 2603.22744, Link Cited by: §1. Choudhury (2025) S. Choudhury Process reward models for llm agents: practical framework and directions. External Links: 2502.10325, Link Cited by: §6. Huang et al. (2022) W. Huang, P. Abbeel, D. Pathak, and I. Mordatch Language models as zero-shot planners: extracting actionable knowledge for embodied agents. External Links: Link Cited by: §1. Huang et al. (2023) W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, T. Jackson, N. Brown, L. Luu, S. Levine, K. Hausman, and b. ichter Inner monologue: embodied reasoning through planning with language models. In Proceedings of The 6th Conference on Robot Learning, K. Liu, D. Kulic, and J. Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, p. 1769–1782. External Links: Link Cited by: §1. ichter et al. (2023) b. ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K. Lee, Y. Kuang, S. Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu Do as i can, not as i say: grounding language in robotic affordances. In Proceedings of The 6th Conference on Robot Learning, K. Liu, D. Kulic, and J. Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, p. 287–318. External Links: Link Cited by: §1. Ju et al. (2026) L. Ju, J. Zhao, M. Chai, Z. Shen, X. Wang, Y. Geng, C. Ma, H. Peng, G. Li, T. Li, C. Liao, F. Wang, X. Wang, J. Chen, R. Gong, S. Liang, F. Li, M. Zhang, K. Tan, J. Ye, Z. Xi, S. Dou, T. Gui, Y. Ying, Y. Shi, Y. Zhang, and Q. Zhang WisPaper: your ai scholar search engine. External Links: 2512.06879, Link Cited by: §1. Li et al. (2026) K. Li, J. Shi, Y. Xiao, M. Jiang, J. Sun, Y. Wu, D. Fu, S. Xia, X. Cai, T. Xu, W. Si, W. Li, D. Wang, and P. Liu AgencyBench: benchmarking the frontiers of autonomous agents in 1m-token real-world contexts. External Links: 2601.11044, Link Cited by: §1. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. External Links: 2305.20050, Link Cited by: §1, §6. Ma et al. (2024) Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar Eureka: human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §6. Merrill et al. (2026) M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V. Sharma, K. Sun, S. Dillmann, A. Anand, A. Lanpouthakoun, B. Koopah, C. Hu, E. K. Guha, G. H. S. Dreiman, J. Zhu, K. Krauth, L. Zhong, N. Muennighoff, R. K. Amanfu, S. Tan, S. Pimpalgaonkar, T. Aggarwal, X. Lin, X. Lan, X. Zhao, Y. Liang, Y. Wang, Z. Wang, C. Zhou, D. Heineman, H. Liu, H. Trivedi, J. Yang, J. Lin, M. Shetty, M. Yang, N. Omi, N. Raoof, S. Li, T. Y. Zhuo, W. Lin, Y. Dai, Y. Wang, W. Chai, S. Zhou, D. Wahdany, Z. She, J. Hu, Z. Dong, Y. Zhu, S. Cui, A. Saiyed, A. Kolbeinsson, C. M. Rytting, R. Marten, Y. Wang, J. Jitsev, A. Dimakis, A. Konwinski, and L. Schmidt Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1. Müller and Kudenko (2026) H. Müller and D. Kudenko Automating potential-based reward shaping with vision language model guidance. External Links: 2606.27180, Link Cited by: §6. Ng et al. (1999) A. Y. Ng, D. Harada, and S. J. Russell Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, San Francisco, CA, USA, p. 278–287. External Links: ISBN 1558606122 Cited by: §6. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §1. Paranjape et al. (2023) B. Paranjape, S. M. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro ART: automatic multi-step reasoning and tool-use for large language models. ArXiv abs/2303.09014. External Links: Link Cited by: §1. Qwen Team (2024) Qwen Team Qwen2.5-32b-instruct-awq. Hugging Face. Note: https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQAWQ 4-bit quantized model checkpoint. Accessed: 2026-05-18 Cited by: Table 1. Qwen Team (2025a) Qwen Team Qwen2.5-vl-72b-instruct-awq. Hugging Face. Note: https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct-AWQAWQ quantized instruction-tuned 72B Qwen2.5-VL model checkpoint. Accessed: 2026-05-18 Cited by: Table 1. Qwen Team (2025b) Qwen Team Qwen2.5-vl-7b-instruct. Hugging Face. Note: https://huggingface.co/Qwen/Qwen2.5-VL-7B-InstructInstruction-tuned 7B Qwen2.5-VL model checkpoint. Accessed: 2026-05-18 Cited by: Table 1. Qwen Team (2025c) Qwen Team Qwen3-30b-a3b-instruct-2507-fp8. Hugging Face. Note: https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8FP8 quantized checkpoint of Qwen3-30B-A3B-Instruct-2507. Accessed: 2026-05-18 Cited by: Table 1. Qwen Team (2025d) Qwen Team Qwen3-vl-30b-a3b-instruct. Hugging Face. Note: https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-InstructMoE instruction-tuned Qwen3-VL model checkpoint. Accessed: 2026-05-18 Cited by: Table 1. Qwen Team (2025e) Qwen Team Qwen3-vl-32b-thinking-fp8. Hugging Face. Note: https://huggingface.co/Qwen/Qwen3-VL-32B-Thinking-FP8FP8 quantized checkpoint of Qwen3-VL-32B-Thinking. Accessed: 2026-05-18 Cited by: Table 1. Rajasekaran (2026) P. Rajasekaran Harness design for long-running application development. Anthropic. External Links: Link Cited by: §1. Rocamonde et al. (2024) J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner Vision-language models are zero-shot reward models for reinforcement learning. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §6. Ruis et al. (2025) L. Ruis, M. Mozes, J. Bae, S. R. Kamalakara, D. Gnaneshwar, A. Locatelli, R. Kirk, T. Rocktaeschel, E. Grefenstette, and M. Bartolo Procedural knowledge in pretraining drives reasoning in large language models. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 29367–29429. External Links: Link Cited by: §1. Sarukkai et al. (2025) V. Sarukkai, B. Shacklett, Z. Majercik, K. Bhatia, C. Re, and K. Fatahalian Automated rewards via LLM-generated progress functions. External Links: Link Cited by: §6. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. Shridhar et al. (2021) M. Shridhar, X. Yuan, M. Cote, Y. Bisk, A. Trischler, and M. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, External Links: Link Cited by: §1, §3.2. Tan et al. (2026) H. Tan, X. Yang, H. Chen, J. Shao, Y. Wen, Y. Shen, W. Luo, X. Du, L. Guo, and Y. Li Hindsight credit assignment for long-horizon llm agents. External Links: 2603.08754, Link Cited by: §1, §6. Wang et al. (2024a) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar Voyager: an open-ended embodied agent with large language models. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §1. Wang et al. (2024b) L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. External Links: ISSN 2095-2236, Document, Link Cited by: §1. Wang et al. (2022) R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu ScienceWorld: is your agent smarter than a 5th grader?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, p. 11279–11298. External Links: Link, Document Cited by: §1, §3.2. Wang et al. (2026) T. Wang, S. Gooding, F. Hartmann, O. Riva, and E. Grefenstette A subgoal-driven framework for improving long-horizon llm agents. External Links: 2603.19685, Link Cited by: §6. Wang et al. (2024c) Y. Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson RL-VLM-f: reinforcement learning from vision language foundation model feedback. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 51484–51501. External Links: Link Cited by: §6. Xi et al. (2025) Z. Xi, C. Liao, G. Li, Y. Yang, W. Chen, Z. Zhang, B. Wang, S. Jin, Y. Zhou, J. Guan, W. Wu, T. Ji, T. Gui, Q. Zhang, and X. Huang AgentPRM: process reward models for llm agents via step-wise promise and progress. External Links: 2511.08325, Link Cited by: §6. XiaomiMiMo (2025) XiaomiMiMo MiMo-vl-7b-rl-2508. Hugging Face. Note: https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL-2508Model checkpoint. Accessed: 2026-05-18 Cited by: Table 1. Xie et al. (2024) T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu Text2Reward: reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §6. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1. Young (2025) J. Young Effective harnesses for long-running agents. Anthropic. External Links: Link Cited by: §1. Zhang et al. (2025a) D. Zhang, S. Zhang, Z. Yang, Z. Zhu, Z. Zhao, R. Cao, L. Chen, and K. Yu ProgRM: build better gui agents with progress rewards. External Links: 2505.18121, Link Cited by: §6. Zhang et al. (2025b) X. Zhang, N. Gao, X. Jiang, Y. Chen, Y. Pan, M. Zhang, and Y. Deng Progress reward model for reinforcement learning via large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §6. Zhao et al. (2024) J. Zhao, C. Zu, X. Hao, Y. Lu, W. He, Y. Ding, T. Gui, Q. Zhang, and X. Huang LONGAGENT: achieving question answering for 128k-token-long documents through multi-agent collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 16310–16324. External Links: Link, Document Cited by: §1. Zhou et al. (2026) C. Zhou, H. Chai, W. Chen, Z. Guo, R. Shan, Y. Song, T. Xu, Y. Yang, A. Yu, W. Zhang, C. Zheng, J. Zhu, Z. Zheng, Z. Zhang, X. Lou, C. Zhang, Z. Fu, J. Wang, W. Liu, J. Lin, and W. Zhang Externalization in llm agents: a unified review of memory, skills, protocols and harness engineering. External Links: 2604.08224, Link Cited by: §1. Appendix A World Potential Model Instantiation and Capability Evaluation Details This appendix provides implementation details for the World Potential Model (WPM) instantiation and the capability evaluations in Section 3. We first present the prompt interfaces used to elicit realized-progress judgments, then describe the construction of task-specific milestone ladders, and finally detail the environment-grounded labels used for capability evaluation. A.1 WPM Progress-Judgment Interfaces As described in Section 2, we access world potential primarily through ordinal judgments rather than asking the model to directly produce a globally calibrated scalar score. Our implementation uses two prompt interfaces: pairwise progress comparison between agent contexts and anchored progress localization of a single context. Pairwise progress comparison. Given two agent contexts associated with the same task goal, the WPM identifies which context embodies greater goal-relevant realized progress toward task completion. Listing shows the prompt used for this comparison. Prompt for comparing the progress of two agent contexts. ⬇ You are evaluating the progress of two agent contexts on the same task. ## Task Description sample.task_description ## Context A context_a ## Context B context_b ## Question Which context has made MORE progress toward completing the task? Provide your reasoning first, then give your final answer. Output format: Reasoning: <your analysis> Answer: <A or B> Anchored progress localization. For downstream scalarization, we additionally provide the WPM with the fixed task-specific milestone ladder and ask it to localize the current context relative to the ordered progress anchors. The output is the anchor index corresponding to the context’s realized-progress level, whose associated coordinate determines the scalar world potential. Listing shows the prompt used in our implementation. Prompt for anchored progress localization. ⬇ You are evaluating the realized progress of an embodied agent on a task. ## Task sample.task_description ## Ordered Progress Anchors generated_formatted_milestones The anchors represent increasing levels of realized progress toward task completion. ## Current Context formatted_context ## Question Which anchor best represents the realized-progress level of the current context? Do not assign progress that has not yet been realized. Provide your reasoning first, then give your final answer. Output format: Reasoning: <your analysis> Answer: <one anchor id, e.g. answer_options> Pairwise comparison directly probes the ordinal progress relation underlying a WPM, while anchored localization implements the task-local scalarization used in our experiments. In Section 3, both outputs are evaluated against independently constructed environment-grounded labels described in Appendix A.3. A.2 Task-Specific Anchors Construction The scalar construction in Section 2.3 requires a task-local set of ordered progress anchors MgM_g. The WPM formulation does not require a particular source for these anchors; here we describe the construction used in our experiments. For each task instance, we provide a pretrained model with the task description and one successful expert trajectory. The purpose of the trajectory is to expose a valid realization of the task from which the model can identify salient intermediate accomplishments. It is used only to construct the reference frame: it does not train or fine-tune the WPM evaluator, and it is not provided when the WPM subsequently evaluates agent-generated contexts. Importantly, the generated anchors describe task states rather than steps of the demonstration. We instruct the model to identify verifiable accomplishments that characterize increasing levels of goal-relevant realized progress, rather than reproducing the demonstrated action sequence. Consequently, the resulting ladder provides a sparse set of reference points rather than progress labels for the arbitrary contexts that the policy may later visit. Localizing such contexts relative to these anchors remains the responsibility of the frozen WPM. Concretely, the model generates between 4 and 10 intermediate anchors according to task complexity. Each anchor must (i) describe a salient accomplishment relevant to task completion, (i) be state-based rather than tied to a particular action or trajectory position, and (i) be verifiable from the agent-visible context. The model also assigns each anchor a scalar coordinate satisfying 0=ϕ0<ϕ1<⋯<ϕK=1,0= _0< _1<·s< _K=1, where m0m_0 denotes no meaningful task progress and mKm_K denotes task completion. Coordinates are intended to reflect the relative amount of goal-relevant progress represented by each accomplishment rather than being determined solely by its ordinal index. The resulting anchor set is generated once and fixed throughout capability evaluation and policy optimization. Thus, this instantiation uses a single successful demonstration to establish task-local calibration, while relying on pretrained knowledge to interpret previously unseen and potentially off-demonstration agent contexts relative to that calibration. Listing gives the prompt used to construct the anchor set, and Listing shows an example. Prompt template for constructing a task-specific milestone ladder. ⬇ You are a task decomposition expert. ## Task Description task_description ## Expert Successful Trajectory expert_trajectory ## Instructions Analyze the task and successful expert trajectory and construct an ordered set of milestones representing increasing levels of realized progress from task initiation to completion. Requirements: 1. Generate between 4 and 10 intermediate milestones depending on task complexity: simpler tasks need fewer milestones, while complex multi-stage tasks need more. 2. Each milestone should describe a salient accomplishment or state relevant to completing the task. 3. Criteria must be state-based rather than step-based. Do not refer to step numbers or positions in the expert trajectory. 4. Milestones should be verifiable from the agent context and ordered according to the logical progression of the task. 5. Assign each milestone a progress coordinate between 0 and 1 that reflects how much goal-relevant progress the milestone represents. Coordinates must increase strictly with the milestone order; do not assign them solely from the milestone index. Use 0 for the initial anchor and 1 for task completion. Output format (strict JSON): "milestones": [ "id": "M0", "name": "Not started", "criteria": "No meaningful task progress has been made.", "progress": 0.0 , "id": "M1", "name": "Milestone name", "criteria": "Verifiable state-based accomplishment", "progress": <value between 0 and 1> , ... "id": "M<K>", "name": "Task complete", "criteria": "The task objective has been completed.", "progress": 1.0 ], "reasoning": "Brief explanation of the milestone decomposition and progress coordinates." Example of a generated task-specific milestone ladder. ⬇ "milestones": [ "id": "M0", "name": "Not started", "criteria": "No meaningful task progress has been made.", "progress": 0.0 , "id": "M1", "name": "Locate the cup", "criteria": "The target cup is explicitly visible in the environment observation.", "progress": 0.10 , "id": "M2", "name": "Pick up the cup", "criteria": "The agent has successfully picked up the target cup.", "progress": 0.25 , "id": "M3", "name": "Reach the sink with the cup", "criteria": "The agent is carrying the target cup and has reached the sink basin.", "progress": 0.40 , "id": "M4", "name": "Clean the cup", "criteria": "The target cup has been successfully cleaned.", "progress": 0.65 , "id": "M5", "name": "Reach the cabinet with the clean cup", "criteria": "The agent is carrying the cleaned cup and has reached the target cabinet.", "progress": 0.80 , "id": "M6", "name": "Place the clean cup in the cabinet", "criteria": "The cleaned target cup has been placed in the specified cabinet, completing the task.", "progress": 1.0 ] A.3 Environment-Grounded Capability Evaluation Labels We construct capability-evaluation labels from privileged progress signals available in the two environments. These signals are used only to construct evaluation targets and are never provided to the evaluated models. For each environment, let P(x)P(x) denote the privileged progress score associated with an agent context x, with larger values indicating greater environment-grounded progress toward task completion. Environment-grounded progress scores. For ALFWorld, we use the environment’s symbolic planner to measure remaining distance to task completion. Let LD(x)L_D(x) denote the length of an expert plan from the restored environment state associated with context x to a goal-satisfying state. We define PALF(x)=1−LD(x)LD(x0),P_ALF(x)=1- L_D(x)L_D(x_0), (18) where x0x_0 is the initial context of the task instance. PALFP_ALF is formally a normalized cost-to-go-based proxy, rather than a definition of retrospective realized progress itself. We use it because ALFWorld consists of structured navigation and manipulation tasks in which realizing additional goal-relevant prerequisites typically reduces the remaining expert-plan distance, while losing previously satisfied conditions can increase it. We additionally exclude near-tied states when constructing pairwise labels, as described below, to avoid treating small and potentially ambiguous differences in planner distance as reliable progress distinctions. As a qualitative sanity check, we manually inspected 100100 randomly sampled context pairs and found that the planner-induced ordering was consistent with relative realized-progress judgments in 100%100\% of the inspected pairs. For ScienceWorld, we use the benchmark’s native normalized task score, PSW(x)=Senv(x),P_SW(x)=S_env(x), (19) which provides partial credit according to the task’s environment-defined goals and subgoals. Because this score directly reflects which task components have already been satisfied, it provides a more direct operationalization of realized task progress than the planner-distance proxy used in ALFWorld. Pairwise progress labels. For two contexts xix_i and xjx_j from the same task instance, we label the context with the larger privileged progress score as having greater realized progress. Pairs with equal scores are discarded. For ALFWorld, we additionally retain a pair only when |PALF(xi)−PALF(xj)|>δ,δ=0.12, |P_ALF(x_i)-P_ALF(x_j) |>δ, δ=0.12, (20) to exclude near-tied states for which differences in planner distance provide an ambiguous progress ordering. Anchored-localization labels. Each generated milestone anchor mkm_k is associated with a corresponding expert-trajectory context x¯k x_k. We map this context onto the privileged environment progress scale, sk=P(x¯k),s_k=P( x_k), (21) and use sks_k as the environment-grounded progress level associated with anchor k. For a sampled context x, the localization target is yloc(x)=maxk:sk≤P(x).y_loc(x)= \k:s_k≤ P(x)\. (22) Thus, the target identifies the most advanced anchor whose environment-grounded progress level has been reached by the current context. Importantly, this construction does not require the natural-language criterion of each milestone to be independently evaluated when constructing the target. We use PALFP_ALF in ALFWorld and PSWP_SW in ScienceWorld. Pairwise comparison and Anchored localization therefore use the same underlying environment-grounded progress signal for evaluation: the former tests the induced ordering between contexts, while the latter tests localization relative to the fixed task-specific progress anchors. Appendix B Process-Sensitive Return Derivation This appendix derives the process-sensitive return introduced in Section 4 and characterizes how the parameter ρ controls the temporal weighting of world potentials. We first derive the return decomposition, then examine its boundary cases and induced trajectory preferences. B.1 Return Decomposition Recall the world-potential transition signal rtWP=rtenv+γΦg(xt+1)−Φg(xt),r_t^WP=r_t^env+γ _g(x_t+1)- _g(x_t), (23) and the effective return discount β=γρ,ρ∈[0,1].β=γρ, ρ∈[0,1]. (24) For a trajectory terminating at context xTx_T, let H=T−tH=T-t. The process-sensitive return from step t is GtWP,ρ=∑ℓ=0H−1βℓrt+ℓWP.G_t^WP,ρ= _ =0^H-1β r_t+ ^WP. (25) Substituting the transition signal gives GtWP,ρ G_t^WP,ρ =∑ℓ=0H−1βℓrt+ℓenv+γ∑ℓ=0H−1βℓΦg(xt+ℓ+1)−∑ℓ=0H−1βℓΦg(xt+ℓ). = _ =0^H-1β r_t+ ^env+γ _ =0^H-1β _g(x_t+ +1)- _ =0^H-1β _g(x_t+ ). (26) Re-indexing the second potential sum yields GtWP,ρ G_t^WP,ρ =∑ℓ=0H−1βℓrt+ℓenv−Φg(xt)+∑k=1H−1(γβk−1−βk)Φg(xt+k)+γβH−1Φg(xT). = _ =0^H-1β r_t+ ^env- _g(x_t)+ _k=1^H-1 (γβ^k-1-β^k ) _g(x_t+k)+γβ^H-1 _g(x_T). (27) Since β=γρβ=γρ, γβk−1−βk=γ(1−ρ)βk−1.γβ^k-1-β^k=γ(1-ρ)β^k-1. (28) Therefore, GtWP,ρ G_t^WP,ρ =∑ℓ=0H−1βℓrt+ℓenv−Φg(xt) = _ =0^H-1β r_t+ ^env- _g(x_t) +γ(1−ρ)∑k=1H−1βk−1Φg(xt+k)+γβH−1Φg(xT). +γ(1-ρ) _k=1^H-1β^k-1 _g(x_t+k)+γβ^H-1 _g(x_T). (29) Equation (29) separates the return into four components: discounted environment reward, the current-context potential, temporally weighted intermediate potentials, and the terminal-context potential. For 0<ρ<10<ρ<1, the intermediate-potential term is nonzero, allowing realized progress along the continuation to directly affect step-level credit. B.2 Temporal Weighting Controlled by ρ Equation (29) makes explicit how ρ controls the temporal weighting of world potentials. The k-th future intermediate context receives weight wk(ρ)=γ(1−ρ)(γρ)k−1=γk(1−ρ)ρk−1,1≤k≤H−1,w_k(ρ)=γ(1-ρ)(γρ)^k-1=γ^k(1-ρ)ρ^k-1, 1≤ k≤ H-1, (30) while the terminal-context potential receives weight wT(ρ)=γ(γρ)H−1=γHρH−1.w_T(ρ)=γ(γρ)^H-1=γ^Hρ^H-1. (31) The environment reward at temporal offset ℓ receives weight uℓ(ρ)=(γρ)ℓ.u_ (ρ)=(γρ) . (32) For two adjacent intermediate contexts, wk+1(ρ)wk(ρ)=γρ. w_k+1(ρ)w_k(ρ)=γρ. (33) Thus, γργρ determines how rapidly the influence of future intermediate contexts decays with temporal distance. Smaller values of ρ concentrate credit on near-term realized progress, whereas larger values allow progress and regression farther in the future to influence the current action. At the same time, increasing ρ reduces the factor 1−ρ1-ρ multiplying intermediate potentials while increasing the relative influence of longer-range environment rewards and the terminal potential. Thus, ρ controls both the temporal range of credit and the relative emphasis on intermediate versus terminal signals. Boundary cases. When ρ=0ρ=0, we have β=0β=0, and therefore GtWP,0=rtenv+γΦg(xt+1)−Φg(xt)=rtWP.G_t^WP,0=r_t^env+γ _g(x_t+1)- _g(x_t)=r_t^WP. (34) The current action is evaluated only through its immediate environment signal and one-step change in realized progress. For 0<ρ<10<ρ<1, future intermediate contexts receive nonzero weight, so the return can distinguish continuations according to when progress is realized and whether it is subsequently preserved. When ρ=1ρ=1, the intermediate-potential coefficient vanishes: γ(1−ρ)∑k=1H−1(γρ)k−1Φg(xt+k)=0.γ(1-ρ) _k=1^H-1(γρ)^k-1 _g(x_t+k)=0. (35) The potential-dependent component then reduces to −Φg(xt)+γHΦg(xT).- _g(x_t)+γ^H _g(x_T). (36) Thus, when the environment return and the current and terminal potentials are fixed, intermediate realized-progress differences no longer affect the return. Temporal scale of an individual intermediate context. For a fixed temporal position k≥2k≥ 2, the dependence of Eq. (30) on ρ, up to the factor γkγ^k, is fk(ρ)=(1−ρ)ρk−1.f_k(ρ)=(1-ρ)ρ^k-1. (37) Differentiating gives dρ[(1−ρ)ρk−1]=ρk−2[(k−1)−kρ]. ddρ [(1-ρ)ρ^k-1 ]=ρ^k-2 [(k-1)-kρ ]. (38) Thus, the weight assigned to the k-th future intermediate context is maximized at ρk⋆=k−1k=1−1k. _k = k-1k=1- 1k. (39) For k=1k=1, w1(ρ)=γ(1−ρ)w_1(ρ)=γ(1-ρ) is instead maximized at ρ=0ρ=0. Equation (39) provides a temporal-scale interpretation: sensitivity to progress or regression occurring farther in the future requires a larger value of ρ. It should not be interpreted as an optimal parameter formula for the complete learning objective, since the appropriate choice also depends on the task horizon, environment rewards, and evaluator noise. Geometric-weighting interpretation. When γ=1γ=1, the potential component of Eq. (29) becomes −Φg(xt)+(1−ρ)∑k=1H−1ρk−1Φg(xt+k)+ρH−1Φg(xT).- _g(x_t)+(1-ρ) _k=1^H-1ρ^k-1 _g(x_t+k)+ρ^H-1 _g(x_T). (40) The coefficients on all noncurrent contexts satisfy (1−ρ)∑k=1H−1ρk−1+ρH−1=1.(1-ρ) _k=1^H-1ρ^k-1+ρ^H-1=1. (41) They can therefore be viewed as a truncated geometric weighting over a future evaluation time. For the corresponding untruncated geometric distribution, Pr(K=k)=(1−ρ)ρk−1,[K]=11−ρ. (K=k)=(1-ρ)ρ^k-1, [K]= 11-ρ. (42) When γ<1γ<1, both environment and potential contributions additionally decay according to β=γρβ=γρ. In this case, Heff≈11−γρH_eff≈ 11-γρ (43) provides a rough temporal decay scale. Because γ also changes the total coefficient mass, Eq. (43) should be interpreted as an influence range rather than the exact mean of a normalized distribution. B.3 Illustrative Trajectory Preferences The decomposition in the preceding subsection makes the induced trajectory preferences explicit. We illustrate these preferences with two simple examples, setting γ=1γ=1 and omitting the current-context term and environment return, which are identical across the compared trajectories. For γ=1γ=1, the potential-dependent contribution to the return can be written as G~=(1−ρ)∑k=1H−1ρk−1Φg(xk)+ρH−1Φg(xH), G=(1-ρ) _k=1^H-1ρ^k-1 _g(x_k)+ρ^H-1 _g(x_H), (44) where G~ G contains only the terms that can differ between the trajectories considered below. B.3.1 Earlier Versus Later Realized Progress Consider two trajectories that reach the same terminal potential: Trajectory A: Φg(x1)=1,Φg(x2)=1,Φg(x3)=1, _g(x_1)=1, _g(x_2)=1, _g(x_3)=1, Trajectory B: Φg(x1)=0,Φg(x2)=0,Φg(x3)=1. _g(x_1)=0, _g(x_2)=0, _g(x_3)=1. Trajectory A realizes high potential earlier and preserves it, whereas Trajectory B reaches the same potential only at termination. Here H=3H=3. For Trajectory A, G~A G_A =(1−ρ)[Φg(x1)+ρΦg(x2)]+ρ2Φg(x3) =(1-ρ) [ _g(x_1)+ρ _g(x_2) ]+ρ^2 _g(x_3) =(1−ρ)(1+ρ)+ρ2 =(1-ρ)(1+ρ)+ρ^2 =1. =1. (45) For Trajectory B, G~B G_B =(1−ρ)[Φg(x1)+ρΦg(x2)]+ρ2Φg(x3) =(1-ρ) [ _g(x_1)+ρ _g(x_2) ]+ρ^2 _g(x_3) =ρ2. =ρ^2. (46) Therefore, ΔGearly G_early =G~A−G~B = G_A- G_B =1−ρ2 =1-ρ^2 (47) Thus, for any 0<ρ<10<ρ<1, the objective prefers the trajectory that realizes the same terminal progress earlier and preserves it. The strength of this preference decreases monotonically with ρ and vanishes at ρ=1ρ=1, where intermediate-context potentials no longer contribute directly to the return. B.3.2 Preserved Progress Versus Intermediate Regression Now consider two trajectories with the same next-context and terminal potentials: Trajectory A: Φg(x1)=1,Φg(x2)=1,Φg(x3)=1, _g(x_1)=1, _g(x_2)=1, _g(x_3)=1, Trajectory B: Φg(x1)=1,Φg(x2)=0,Φg(x3)=1. _g(x_1)=1, _g(x_2)=0, _g(x_3)=1. Both trajectories make the same immediate progress and reach the same terminal potential, but Trajectory B temporarily regresses at the second future context. Again, H=3H=3. For Trajectory A, G~A G_A =(1−ρ)[Φg(x1)+ρΦg(x2)]+ρ2Φg(x3) =(1-ρ) [ _g(x_1)+ρ _g(x_2) ]+ρ^2 _g(x_3) =(1−ρ)(1+ρ)+ρ2 =(1-ρ)(1+ρ)+ρ^2 =1. =1. (48) For Trajectory B, G~B G_B =(1−ρ)[Φg(x1)+ρΦg(x2)]+ρ2Φg(x3) =(1-ρ) [ _g(x_1)+ρ _g(x_2) ]+ρ^2 _g(x_3) =(1−ρ)+ρ2. =(1-ρ)+ρ^2. (49) Their return difference is therefore ΔGpreserve G_preserve =G~A−G~B = G_A- G_B =1−[(1−ρ)+ρ2] =1- [(1-ρ)+ρ^2 ] =ρ−ρ2 =ρ-ρ^2 =(1−ρ)ρ. =(1-ρ)ρ. (50) Thus, for any 0<ρ<10<ρ<1, Trajectory A is preferred over Trajectory B: preserving realized progress yields a higher process-sensitive return than temporarily regressing and later recovering to the same terminal potential. At ρ=0ρ=0, this difference is zero because a one-step objective cannot observe a regression occurring two transitions later. At ρ=1ρ=1, it is also zero because intermediate-context potentials no longer contribute. The preference is maximized at ρ⋆=12.ρ = 12. (51) More generally, suppose two trajectories differ in potential by ΔΦ only at the k-th future context. From Eq. (30), the induced return difference is ΔGk=γk(1−ρ)ρk−1ΔΦ. G_k=γ^k(1-ρ)ρ^k-1 . (52) For k≥2k≥ 2, this difference is maximized with respect to ρ at ρk⋆=k−1k. _k = k-1k. (53) This illustrates that sensitivity to progress or regression farther in the future requires a larger value of ρ. B.4 Experimental Setting We use the same parameter setting in all policy-optimization experiments: γ=0.99,ρ=0.95.γ=0.99, ρ=0.95. (54) The corresponding effective return discount is β=γρ=0.9405.β=γρ=0.9405. (55) For intuition, Eq. (43) gives the approximate decay scale Heff≈11−0.9405≈16.8H_eff≈ 11-0.9405≈ 16.8 (56) environment steps. This setting allows substantially delayed progress and terminal outcomes to influence earlier actions while retaining a nonzero intermediate-potential coefficient 1−ρ=0.051-ρ=0.05. We use ρ=0.95ρ=0.95 uniformly across tasks without per-task tuning or a systematic sensitivity analysis. We therefore treat it as a fixed algorithmic choice rather than claiming that it is empirically optimal or universally preferable for long-horizon tasks. Appendix C Policy Optimization Implementation Details This appendix provides implementation details for the policy-optimization experiments in Section 5. The experiments compare outcome-only GRPO with WPM-guided optimization under a shared actor-training pipeline. The two methods use matched task sampling, rollout construction, policy models, and actor-update configurations, while differing in the learning signal and resulting credit construction. C.1 Environments and Task Sampling Benchmark documentation. We study two English-language interactive agent benchmarks with complementary task domains. ALFWorld consists of compositional household navigation and object-manipulation tasks, including object placement, examination, cleaning, heating, and cooling. ScienceWorld is a text-based environment grounded in an elementary-school science curriculum and contains 30 task types spanning scientific procedures and reasoning involving topics such as changes of state, measurement, electrical conductivity, biology, chemistry, and forces. Both benchmarks require multi-step interaction with simulated environments rather than static language prediction. They contain no demographic groups or human-subject data relevant to our experiments. Data partitions and statistics. We use the official data partitions provided by ALFWorld and ScienceWorld and do not construct additional random train–evaluation splits. ALFWorld contains 3,5533,553 training games, 140140 valid_seen games, and 134134 valid_unseen games. We sample policy-training tasks from the 3,5533,553-game training split and evaluate on all 134134 valid_unseen games, which contain held-out environments and task instances. ScienceWorld contains 7,2077,207 task variations across 3030 task types. Its official partitions contain 3,5893,589 training, 1,7991,799 development, and 1,8191,819 test variations. We sample policy-training tasks from the official training variations and evaluate on the official test variations; the development variations are not used in our policy-optimization experiments. The ScienceWorld splits are exposed by the environment through get_variations_train(), get_variations_dev(), and get_variations_test(). The dataset records supplied to our trainer serve only as rollout slots and should not be interpreted as additional benchmark examples or a new data split. Interactive task instances are sampled from the corresponding official benchmark partition when the environment is initialized or reset. C.2 Training Configuration and Software Training Configuration. We optimize Qwen2.5-Instruct-3B, Qwen2.5-Instruct-7B, and Qwen3-4B policies. Outcome-only GRPO and WPM-guided optimization use the same actor-training configuration. Each update samples 16 task groups with 8 policy rollouts per task, yielding 128 trajectories. Each rollout is capped at 30 environment actions, and the policy context retains the two most recent interaction steps. We use PPO-style clipped actor updates with KL regularization and the same invalid-action treatment for both methods. The learning rate is 1×10−61× 10^-6, and training runs for 100 epochs. Software and implementations. Our released implementation targets Linux with NVIDIA GPUs and Python 3.12. The reproduction environment uses vLLM 0.11.0, FlashAttention 2.7.4.post1, Gymnasium 0.29.1, and Stable-Baselines3 2.6.0. We use the ALFWorld environment through its official Python package and the ScienceWorld implementation included in our released codebase. The policy checkpoints are specified through the exact Hugging Face model identifiers. World-potential evaluation and milestone generation use OpenAI-compatible vLLM endpoints serving Qwen/Qwen3-VL-32B-Thinking-FP8. The released implementation and experiment entry points are provided in our public repository. Our computational experiments were conducted using 8 NVIDIA H100 GPUs. C.3 Learning-Signal Construction Outcome-only GRPO. Let R(τ)=10⋅wonR(τ)=10·1_won denote the binary task outcome. In the policy-optimization implementation, we use the scaled terminal environment reward rtenv=R(τ),t=T−1,0,t<T−1.r_t^env= casesR(τ),&t=T-1,\\ 0,&t<T-1. cases (57) Outcome-only GRPO constructs trajectory-level group-relative advantages from this terminal outcome signal. WPM-guided optimization. World-potential supervision is produced by the frozen evaluator described in Section 5. Milestone-ladder construction and anchored progress localization follow Appendix A. The process-sensitive transition signal and return follow Section 4 and Appendix B, using γ=0.99,ρ=0.95.γ=0.99, ρ=0.95. For successful episodes, the terminal world potential is set to 11; otherwise, the final evaluated potential is retained. World potentials and process-sensitive returns are computed at the environment-action level. Step-level group-relative advantages are then constructed as described in Section 4. Because a single environment action may contain multiple generated tokens, all valid tokens belonging to the same action share the corresponding step-level advantage. The remaining actor-update procedure is shared with outcome-only GRPO. C.4 Evaluation We evaluate policies using episode-level task success rate. An episode is counted as successful when its final environment output reports info["won"] = True. We report the percentage of successful episodes over the evaluation set. The same evaluation protocol is used for vanilla, outcome-only GRPO, and WPM-guided policies. C.5 Artifact Licenses and Terms of Use We use publicly released benchmark environments and pretrained model checkpoints in accordance with their respective licenses and terms of use. The ALFWorld software is released under the MIT License; its official repository additionally lists TextWorld under the MIT License and Fast Downward under GPL v3.0. ScienceWorld is released under the Apache License 2.0. For the capability evaluations in Table 1, MiMo-VL-7B-RL-2508 is released under the MIT License; Qwen2.5-VL-7B-Instruct, Qwen2.5-32B-Instruct-AWQ, Qwen3-30B-A3B-Instruct-2507-FP8, Qwen3-VL-30B-A3B-Instruct, and Qwen3-VL-32B-Thinking-FP8 are released under the Apache License 2.0; and Qwen2.5-VL-72B-Instruct-AWQ is distributed under the Qwen License Agreement. For policy optimization, Qwen2.5-3B-Instruct is distributed under the Qwen Research License, which permits non-commercial research and evaluation use, while Qwen2.5-7B-Instruct and Qwen3-4B are released under the Apache License 2.0. The Qwen3-VL-32B evaluator used for world-potential estimation is also released under the Apache License 2.0. Our use of these artifacts is limited to research-oriented evaluation, inference, and policy optimization. We do not assume broader licensing rights for benchmark assets distributed separately from their software repositories, and any redistribution of third-party artifacts should retain the notices and conditions required by the corresponding upstream licenses.