Paper deep dive
Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agents
Jiacheng Qiu, Christopher E. Mower, Jan Peters, Haitham Bou-Ammar, Matthieu Zimmer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/1/2026, 8:32:46 AM
Summary
The paper identifies a 'task-blind corruption' in diffusion-based large language models (dLLMs) where agents fall into retry loops by over-weighting contextually salient actions regardless of task success. It proposes 'Reflect Reverse', a training-free method that scores candidate actions by evaluating the reverse conditional likelihood of the task given the state and action, effectively canceling out the corruption bias. Experiments on four embodied benchmarks show improved success and progression rates compared to forward-scoring baselines.
Entities (10)
Relation Signals (9)
Reflect Reverse → evaluatedon → BabyAI
confidence 95% · We evaluate our proposed method on ... BabyAI
Reflect Reverse → evaluatedon → Jericho
confidence 95% · We evaluate our proposed method on ... Jericho
Reflect Reverse → evaluatedon → ALFWorld
confidence 95% · We instantiate the rule as Reflect Reverse and evaluate it on four multi-turn embodied benchmarks ... ALFWorld
Reflect Reverse → evaluatedon → ScienceWorld
confidence 95% · We evaluate our proposed method on ... ScienceWorld
Reflect Reverse → mitigates → task-blind corruption
confidence 95% · Under this model, we analyse an invariance proposition: the task-blind factor cancels exactly from the reverse conditional
Reflect Reverse → improves → task success
confidence 90% · Reflect Reverse ... improves task success and progression rates over forward-scoring baselines.
dLLMs → suffersfrom → retry loops
confidence 90% · dLLM-backed agents repeatedly fall into retry loops, re-issuing an action long after it has failed.
LLaDA-8B-Instruct → usedwith → Reflect Reverse
confidence 90% · We use LLaDA-8B-Instruct ... as the backbone models for our dLLM-based agents.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion-based large language models (dLLMs) promise to break the sequential latency bottleneck of autoregressive agents through parallel decoding, but recent evaluations show this efficiency does not transfer to embodied agentic competence: dLLM-backed agents repeatedly fall into retry loops, re-issuing an action long after it has failed. We give a mechanistic account of this failure and a training-free remedy. We trace the retry loop to the adaptivity of masked decoding: the sampler commits the positions it is most confident about and defers the uncertain ones, and at a failure state the context already offers a confident fill for the deferred decision, i.e. the failed action itself, so the retry is committed without the failure feedback ever being confronted. We model the resulting distortion of the action distribution as a task-blind corruption: contextually salient actions (e.g., the action just taken) receive inflated probability by a factor that depends on the state and the action but not on the task. Under this model, we analyse an invariance proposition: the task-blind factor cancels exactly from the reverse conditional, i.e. the likelihood of the task given the state and a candidate action, which coincides with the task posterior of an idealized uncorrupted model. Masked dLLMs evaluate the reverse conditional natively, unlike autoregressive models, by masking the task tokens and denoising, at the cost of a few parallel passes per candidate. We instantiate the rule as Reflect Reverse and evaluate it on four multi-turn embodied benchmarks, where it improves task success and progression rates over forward-scoring baselines.
Tags
Links
- Source: https://arxiv.org/abs/2609.38536v1
- Canonical: https://arxiv.org/abs/2609.38536v1
Trouble viewing inline? Open PDF directly →
Full Text
75,891 characters extracted from source content.
Expand or collapse full text
Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agents Jiacheng Qiu1,2 Christopher E. Mower1 Jan Peters2 Haitham Bou-Ammar1,3 Matthieu Zimmer1 1 Huawei Noah’s Ark Lab 2 Technical University of Darmstadt 3 UCL Centre for AI Abstract Diffusion-based large language models (dLLMs) promise to break the sequential latency bottleneck of autoregressive agents through parallel decoding, but recent evaluations show this efficiency does not transfer to embodied agentic competence: dLLM-backed agents repeatedly fall into retry loops, re-issuing an action long after it has failed. We give a mechanistic account of this failure and a training-free remedy. We trace the retry loop to the adaptivity of masked decoding: the sampler commits the positions it is most confident about and defers the uncertain ones, and at a failure state the context already offers a confident fill for the deferred decision, i.e. the failed action itself, so the retry is committed without the failure feedback ever being confronted. We model the resulting distortion of the action distribution as a task-blind corruption: contextually salient actions (e.g., the action just taken) receive inflated probability by a factor that depends on the state and the action but not on the task. Under this model, we analyse an invariance proposition: the task-blind factor cancels exactly from the reverse conditional, i.e. the likelihood of the task given the state and a candidate action, which coincides with the task posterior of an idealized uncorrupted model. Masked dLLMs evaluate the reverse conditional natively, unlike autoregressive models, by masking the task tokens and denoising, at the cost of a few parallel passes per candidate. We instantiate the rule as Reflect Reverse and evaluate it on four multi-turn embodied benchmarks, where it improves task success and progression rates over forward-scoring baselines. 1 Introduction An agent that samples its next action from a language model inherits the pathologies of the distribution it samples from. Diffusion language models (dLLMs) (Li et al., 2022; Lou et al., 2023; Gong et al., 2025) are attractive agent backbones: they generate entire spans in a handful of parallel denoising steps, promising to break the sequential latency bottleneck of autoregressive decoding (Christianos et al., 2023). But the distribution a dLLM samples from has a distinctive, structural pathology in embodied settings: it over-weights actions that are already salient in the context, by an amount that depends on the state and the action but not on the task. The bias is present at every decision point, however it is clearly visible on failure states: at a state where the previous action has just failed, the model keeps proposing that same action, verbatim, as if the most re-usable prediction were whatever already sits in context (Holtzman et al., 2019). Why should this be structural rather than a matter of knowledge? Masked decoding is adaptive: the sampler commits the masked positions it is most confident about and defers the rest (Nie et al., 2025), and Ni et al. (2026) show that the deferred positions are the “forking” ones where several continuations remain viable: by the time the sampler returns to them, the committed context has already resolved the choice. At a failure state, which action now? is such a fork, and the context already offers a confident fill for it: the failed action itself, verbatim in s. The retry is thus committed without the open decision ever being sampled: the failure feedback is bridged, not confronted (Section 5 develops the mechanism). Resampling at a stuck state shows the knowledge is there: the candidates include plausible alternatives, yet the distribution still concentrates on the contextually salient one. The bias does not care which task is being attempted. Figure 1: Task-blind corruption analyzed over 208 failed states in ALFWorld. We make this precise and turn it into a correction with a provable guarantee. Let u denote the task, s the interaction history, and a a candidate next action. Write πθ(a∣s,u) _θ(a s,u) for the observable model conditional, and μ(a∣s,u)μ(a s,u) for the faithful reference conditional: the distribution the model would produce if it had absorbed s correctly. Our hypothesis is that the gap between the two is a multiplicative distortion that does not depend on the task (Assumption 2.1): the same state–action inflation must explain the model’s behavior under every goal. To verify it empirically, we record the log-probability of the retried (”repeated”) and goal-directed (”alternative”) actions under the real goal u and ten swapped goals u1,…,u10u_1,…,u_10, and plot the mean change relative to the real-task level (Figure 1). Full details of the task-corruption procedure are provided in Appendix A. For both actions, LLaDA (Nie et al., 2025) is nearly insensitive to goal swaps, and always prefers the failed repeated action, in contrary to the faithful reference represented by Qwen here. If the corruption is task-blind, it should be invisible to any score that compares candidates by their task evidence alone. This suggests inverting the question. The standard rule asks the model the forward question: which action for this task? — and the corruption survives, because it inflates an action’s probability independently of the task. We instead ask the reverse question: does the state, together with this candidate action, still explain the task? Every candidate is then scored on the task evidence for it, πθ(u∣s,a) _θ(u s,a); under the coherence idealization of Section 2.2, a factor that carries no task information cannot move this score. For a masked dLLM the reverse question is a native operation unavailable easily in autoregressive models: mask the task tokens, make the state and the candidate action visible and denoise. Our main theoretical analysis makes this precise (Proposition 3.1): under the corruption and coherence assumptions (Section 2.2), the reverse score πθ(u∣s,a) _θ(u s,a) equals, for every action, the faithful model’s task posterior under a task prior that the corruption reweights. The posterior is proportional to a task-evidence ratio μ(a∣s,u)/m(a)μ(a s,u) /m(a): the faithful likelihood of the action under the actual task, divided by its likelihood m(a)m(a) under a mixture of tasks. The corruption cannot survive this comparison: it varies across actions, which is how it distorts the forward ranking, but at each action it is constant across tasks, which is why it cancels from a posterior over tasks. We instantiate our method as Reflect Reverse: sample candidates with chain of thoughts, score each by the reverse conditional, sample the next action proportionally to the scores. On four multi-turn embodied benchmarks and two dLLMs, Reflect Reverse reduces retry-loops and improves task success and progression rates over forward-scoring baselines. Contributions. • A task-blind corruption hypothesis for dLLM action distributions, stated formally and empirically verified (Section 1, Figure 1). • A recovery proposition: the reverse score equals the faithful model’s task posterior (Proposition 3.1); we also analyze the reranking’s residual bias (Theorem 3.1). • Reflect Reverse, a training-free action-selection rule that scores candidates by the model’s reverse conditional, requiring only a few forward passes per candidate (Section 3). • Experiments on ALFWorld, ScienceWorld, BabyAI, and Jericho showing improved success rates and progression rates with LLaDA and iLLaDA (Section 4). 2 Background 2.1 Masked diffusion language models Masked dLLMs (Nie et al., 2025; Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024) define a forward process that progressively masks tokens of a sequence x=(x1,…,xn)x=(x_1,…,x_n) and a reverse process that restores them. At each reverse step, a Transformer with bidirectional attention observes the currently unmasked tokens xox^o and predicts every masked position simultaneously pθ(xm∣xo)=∏j∈mpθ(xj∣xo),p_θ(x^m x^o)\;=\; _j∈ mp_θ(x_j x^o), i.e., the per-step joint over masked tokens is factorized given the context. Any span of a sequence can be scored against any other span by masking the first and clamping the second, in multiple denoising steps; there is no privileged direction of conditioning (Schiff et al., 2025). 2.2 Task-blind corruption We model the observed conditional as a faithful conditional distorted by a state–action factor. Assumption 2.1 (Task-blind multiplicative corruption). There exists a function ε(s,a) (s,a), depending on the state and the action but not on the task, such that for every task u in the support of p(⋅∣s)p(· s): πθ(a∣s,u)=μ(a∣s,u)eε(s,a)Z(s,u),Z(s,u):=∑a′μ(a′∣s,u)eε(s,a′). _θ(a s,u)\;=\; μ(a s,u)\,e (s,a)Z(s,u), Z(s,u):= _a μ(a s,u)\,e (s,a ). (1) The content of the assumption is the clause that ε does not depend on the task: for any two conditionals one can always fit some ε . Figure 1 tests it: at fixed states, the log-ratio logπθ(a∣s,u)−logμ _θ(a s,u)- μ-proxy varies with (s,a)(s,a) but is flat in u. The factor ε(s,a) (s,a) is the structural bias itself: an action whose tokens are salient in s is inflated by the same amount whatever the goal. 3 Method 3.1 Reflect Reverse Standard decoding samples an action from πθ(a∣s,u) _θ(a s,u), i.e. the forward question, but inherits the corruption of Assumption 2.1. Reflect Reverse inverts the direction of conditioning: 1. Propose. Sample K candidate actions a1,…,aK∼πθ(⋅∣s,u)a_1,…,a_K _θ(· s,u) with chain of thoughts (Wei et al., 2022) and deduplicate, yielding a candidate set (s)A(s). 2. Reflect. Score each candidate by the reverse conditional, estimated by Monte Carlo rollouts (Gulrajani and Hashimoto, 2023; Ou et al., 2025): provide s and a, sample M random masks of the task tokens, denoise, and read off the task-token log-likelihoods at the masked positions: ϕ(a):=1M∑m=1M1|Bm|∑j∈Bmlogπθ(uj∣s,a,u~(m)),φ(a)\;:=\; 1M _m=1^M 1|B_m| _j∈ B_m _θ(u_j s,a, u^(m)), (2) where Bm⊆1,…,|u|B_m \1,…,|u|\ is the random subset of task-token positions masked in rollout m and u~(m) u^(m) is the masked task-token. Each term of Eq.2 is a masked-estimate of the task log-likelihood logπθ(u∣s,a) _θ(u s,a), so ϕ(a)φ(a) is a Monte Carlo estimate of the same reverse score. 3. Act. Sample the next action from the corrected policy ρ(a∣s,u):=exp(βϕ(a))∑a′∈(s)exp(βϕ(a′)),a∈(s),ρ(a s,u)\;:=\; (β\,φ(a)) _a (s) (β\,φ(a )), a (s), (3) with inverse temperature β>0β>0. The whole procedure is training-free and adds |(s)|×M|A(s)|× M denoising passes per decision. Pseudocode is in Appendix B. 3.2 Invariance of the reverse score to task-blind corruption The reverse score is not an arbitrary heuristic: under the corruption model of Section 2.2, it equals the faithful model μ’s own verdict. Proposition 3.1 (Recovery of the faithful task posterior). Let Assumptions 2.1 and C.1 hold. Then, for every state s, task u in the support of p(⋅∣s)p(· s), and action a, πθ(u∣s,a)=ω(u∣s)μ(a∣s,u)∑u′ω(u′∣s)μ(a∣s,u′), _θ(u s,a)\;=\; ω(u s)\,μ(a s,u) _u ω(u s)\,μ(a s,u ), (4) where ω(u∣s):=p(u∣s)/Z(s,u)ω(u s):=p(u s)/Z(s,u) is a positive task weight that does not depend on the action a. The proof is in Appendix C.1. The cancellation is the pointwise mutual information (PMI) invariance identified by Holtzman et al. (2021). What is new here is not the identity but its agentic instantiation. The corruption is not an input-level bias corrected by an external signal (Holtzman et al., 2021; Chae et al., 2024) but a decoding-level mechanism. An exact score for comparing candidates. Identity Eq.4 holds pointwise in a, so the pairwise log-margins of the score πθ(u∣s,⋅) _θ(u s,·) coincide with those of the faithful task-evidence ratio μ(a∣s,u)/∑u′ω(u′∣s)μ(a∣s,u′)μ(a s,u) / _u ω(u s)\,μ(a s,u ): the score orders and ties candidates exactly as the ratio does. The remaining factor ω(u∣s)ω(u s) is shared by all candidates, so softmax normalization removes it at any temperature (Corollary C.1): the corrected policy ρ is exactly the softmax of the faithful task-evidence ratio over the candidate set, at any β. What the guarantee does and does not cover. The guarantee governs the comparison among the candidates on the table, not the table itself: the proposal step still draws candidates from the corrupted πθ(⋅∣s,u) _θ(· s,u), so the support (s)A(s) need not coincide with the faithful model’s support. Moreover, ρ matches the faithful task-evidence ratio, not the normalized faithful conditional: writing m(a):=∑u′ω(u′∣s)μ(a∣s,u′)m(a):= _u ω(u s)\,μ(a s,u ) for the task-marginal likelihood of a, we have ρ(a)∝μ(a∣s,u)β/m(a)βρ(a) μ(a s,u)^β /m(a)^β, and the factor m(a)m(a) depends on the candidate and is not removed by normalization. ρ therefore coincides with sampling from μ(⋅∣s,u)μ(· s,u) restricted to (s)A(s) only when the marginal is flat across the candidate set (Remark C.1); what holds unconditionally is the exact recovery of the ratio’s ordering and ties. Why the forward score cannot say this. Raw probability has no such invariance: πθ(a∣s,u)∝μ(a∣s,u)eε(s,a) _θ(a s,u) μ(a s,u)\,e (s,a) reorders freely as ε varies across candidates. The reverse question is precisely the one in which a task-blind factor is invisible under our assumptions. At a failure state, the retried action carries a large ε(s,a) (s,a), because it is already in context, so it wins the forward comparison; but its faithful task evidence is low, because it just failed, so it loses the reverse comparison. 3.3 How large is the residual bias? How far is ρ from the faithful reference μ(⋅∣s,u)μ(· s,u) on the candidate set, and how does that residual bias compare with the bias of forward selection from the corrupted πθ(⋅∣s,u) _θ(· s,u) on the same set? Theorem 3.1 below answers both questions in total variation, dTV(P,Q):=12∑a|P(a)−Q(a)|d_TV(P,Q):= 12 _a|P(a)-Q(a)|; the proof, and the sharp range lemma it rests on, are in Appendix C.2. Fix a state s, a task u in the support of p(⋅∣s)p(· s), and a finite candidate set ⊆suppμ(⋅∣s,u)A μ(· s,u), abbreviating =(s)A=A(s) when the candidate set is the proposed one. Write μ _A and π _A for the faithful and corrupted conditionals renormalized to A and measure the heterogeneity of the candidates through the log-spreads: Λμ _μ :=maxa∈logμ(a∣s,u)−mina∈logμ(a∣s,u), \;:=\; _a μ(a s,u)- _a μ(a s,u), (5) Λm _m :=maxa∈logm(a)−mina∈logm(a), \;:=\; _a m(a)- _a m(a), (6) Λtask _task :=supu′∈suppp(⋅∣s)(maxa∈logμ(a∣s,u′)−mina∈logμ(a∣s,u′))∈[0,+∞], \;:=\; _u ∈ p(· s) ( _a μ(a s,u )- _a μ(a s,u ) )\;∈\;[0,+∞], (7) where the supremum runs over tasks u′u under which μ(⋅∣s,u′)μ(· s,u ) is not identically zero on A, with the convention log0=−∞ 0=-∞. The spreads Λμ _μ and Λm _m are always finite, and Λtask _task is finite in particular whenever no candidate is ruled out by any task in the support. Theorem 3.1 (Reverse selection is bounded by the faithful spread; forward selection by the corruption). Fix a state s, a task u in the support of p(⋅∣s)p(· s), and a finite candidate set ⊆suppμ(⋅∣s,u)A μ(· s,u) with ||≥2|A|≥ 2. Let Assumptions 2.1 and C.1 hold and let β>0β>0. Let a+:=argmaxa∈ε(s,a)a^+:= _a (s,a) be the most inflated candidate, γ:=μ(a+)γ:= _A(a^+) its faithful mass, and Δ2:=ε(s,a+)−maxa∈∖a+ε(s,a)≥ 0 _2\;:=\; (s,a^+)- _a \a^+\ (s,a)\;≥\;0 (8) the inflation gap between the most inflated candidate and the rest. Then: (i) The reverse bias is bounded independently of the corruption. dTV(ρ,μ)≤tanh(|β−1|Λμ+βΛm4)≤tanh(|β−1|Λμ+βΛtask4),d_TV(ρ, _A)\;≤\; \! ( |β-1|\, _μ+β _m4 )\;≤\; \! ( |β-1|\, _μ+β _task4 ), (9) and the right-most bound does not involve ε at all; at β=1β=1 it reads dTV(ρ,μ)≤tanh(Λm/4)≤tanh(Λtask/4)d_TV(ρ, _A)≤ ( _m/4)≤ ( _task/4). (i) The forward bias grows with the corruption. dTV(μ,π)≥γ(1−γ)(1−e−Δ2)γ+(1−γ)e−Δ2,d_TV( _A, _A)\;≥\; γ(1-γ) (1-e^- _2 )γ+(1-γ)e^- _2, (10) with equality when ||=2|A|=2; the right-hand side is increasing in Δ2 _2 and tends to 1−γ1-γ as Δ2→∞ _2→∞. Part (i) controls the residual bias of the corrected policy by the spread of the faithful log-probabilities over the candidate set: at β=1β=1 the bound is tanh(Λtask/4) ( _task/4), a quantity of the uncorrupted model, whatever the size of ε . Part (i) shows that the forward-selection bias is a function of the corruption itself: it grows with the inflation gap Δ2 _2 between the most inflated candidate and the rest, toward 1−γ1-γ. Where the corruption is flat, forward selection is exact and the corrected policy retains only the m-bias; the retry loop is the regime in which the two are farthest apart, with forward selection’s bias growing while the corrected policy’s stays capped by a corruption-free quantity. 4 Experiments 4.1 Experimental setup Benchmarks. We evaluate our proposed method on four multi-turn agent benchmarks spanning diverse task domains: ALFWorld (Shridhar et al., 2020) for household task execution, ScienceWorld (Wang et al., 2022) for interactive scientific reasoning, BabyAI (Chevalier-Boisvert et al., 2018) for instruction following in gridworld environments, and Jericho (Hausknecht et al., 2019) for text-based adventure games. Backbone models and inference. We use LLaDA-8B-Instruct (Nie et al., 2025) and iLLaDA-8B-Instruct (Nie et al., 2026) as the backbone models for our dLLM-based agents. iLLaDA follows LLaDA’s masked diffusion formulation and is trained from scratch with larger-scale pre-training and supervised fine-tuning. For efficient inference, we use the Fast-dLLM (Wu et al., 2025) implementation for LLaDA and adapt its block-wise approximate key–value (KV) caching and parallel decoding to iLLaDA. Baseline. We compare our proposed method, Reflect Reverse, with One Pass. Following the action selection procedure of Lu et al. (2026), this baseline executes the first valid action extracted from the generated output. Evaluation protocol. We evaluate both methods at two sampling temperatures T∈0.8,0.9T∈\0.8,0.9\ for LLaDA and T∈0.9,1.0T∈\0.9,1.0\ for iLLaDA, with three random seeds 12, 22 and 32. Each evaluation episode is limited to 30 interaction steps and terminates early if the task is completed. At each step, we generate 20 candidate actions and apply the corresponding action-selection method. Generated actions are mapped to valid environment actions through similarity-based matching, with a match accepted only when its similarity score exceeds 0.5. Complete hyperparameters for the experiments are summarized in Appendix D. Evaluation metrics. For each benchmark, we report success rate (SR) and progress rate (PR) (Ma et al., 2024) to capture complementary aspects of agent performance. Success rate measures the fraction of evaluation episodes in which the agent completes the task within the interaction budget. Progress rate measures advancement toward the task objective, averaged across evaluation episodes. 4.2 Main results Figure 2 compares Reflect Reverse with One Pass across the four benchmarks and two backbone models. Reflect Reverse achieves higher mean SR in 13 of the 16 evaluation groups and higher mean PR in 14. In particular, SR improves on ALFWorld, ScienceWorld, and BabyAI for both backbones at both evaluated temperatures, while PR improves in all eight configurations for iLLaDA. Notably, SR improves on ALFWorld, ScienceWorld, and BabyAI for both backbones at both evaluated temperatures. For iLLaDA, PR also improves across all four benchmarks at both temperatures, demonstrating consistent gains in advancement toward task objectives. The improvements are substantial in several settings. For LLaDA, the largest absolute SR gain occurs on ALFWorld at T=0.9T=0.9, where SR increases from 8.71% to 16.17%, accompanied by an increase in PR from 29.33% to 36.24%. For iLLaDA, the largest absolute SR gain occurs on BabyAI at T=1.0T=1.0, where SR rises from 11.61% to 20.24% and PR increases from 25.13% to 35.40%. The benefits also extend to partial progress on Jericho, where PR improves for both backbones at both temperatures, although SR decreases at the lower temperature evaluated for each backbone. Across all configurations, the only PR decreases occur in two LLaDA settings and are smaller than one percentage point in both cases. Taken together, these results support reverse reflection as an effective approach to improving multi-turn agent performance, with gains spanning both backbone models and diverse task domains. Figure 2: Success rate and progress rate for LLaDA-8B-Instruct and iLLaDA-8B-Instruct. 4.3 Ablations 4.3.1 Rerank on forward score To evaluate the contribution of reverse conditional scoring, we compare Reflect Reverse with Reflect Forward. Rather than masking task description tokens u as in Eq.2, Reflect Forward masks candidate action tokens a and scores their reconstruction conditioned on s, u, and the unmasked action tokens: ψ(a):=1M∑m=1M1|Bm|∑j∈Bmlogπθ(aj∣s,u,a~(m)),ψ(a):= 1M _m=1^M 1|B_m| _j∈ B_m _θ(a_j s,u, a^(m)), (11) where Bm⊆1,…,|a|B_m \1,…,|a|\ is a randomly sampled, nonempty set of masked positions, and a~(m) a^(m) denotes the candidate action with the tokens at those positions masked. The Act step retains the same softmax action-selection policy, replacing ϕ(a)φ(a) with ψ(a)ψ(a) when sampling the next action. Table 1: Comparison of success rate (SR) and progress rate (PR) for Reflect Reverse and Reflect Forward, averaged across the two sampling temperatures. Reverse and Forward results are reported as mean ± standard deviation (%). The relative change is computed as Δ=(Reverse−Forward)/Forward×100% =(Reverse-Forward)/Forward× 100\%. Positive relative changes are shown in bold. When the Forward value is zero, the relative change is undefined and reported as N/A. Model Benchmark Success Rate (SR) Progress Rate (PR) Reverse Forward (%) Reverse Forward (%) LLaDA ALFWorld 14.18±2.9914.18± 2.99 8.21±1.338.21± 1.33 +72.72+72.72 35.26±2.2935.26± 2.29 30.59±1.7930.59± 1.79 +15.27+15.27 ScienceWorld 6.85±1.786.85± 1.78 4.63±1.784.63± 1.78 +47.95+47.95 22.82±2.0622.82± 2.06 21.15±2.4321.15± 2.43 +7.90+7.90 BabyAI 17.56±2.5717.56± 2.57 17.56±3.3217.56± 3.32 0.000.00 26.83±1.3526.83± 1.35 29.40±2.3529.40± 2.35 −8.74-8.74 Jericho 1.67±4.081.67± 4.08 1.67±2.581.67± 2.58 0.000.00 23.63±3.9923.63± 3.99 19.23±2.6119.23± 2.61 +22.88+22.88 iLLaDA ALFWorld 10.07±1.4010.07± 1.40 0.75±0.940.75± 0.94 +1242.67+1242.67 28.20±2.1328.20± 2.13 16.08±1.5616.08± 1.56 +75.37+75.37 ScienceWorld 3.89±1.693.89± 1.69 0.19±0.450.19± 0.45 +1947.37+1947.37 25.67±1.6625.67± 1.66 8.76±1.688.76± 1.68 +193.04+193.04 BabyAI 18.01±2.9618.01± 2.96 8.48±1.238.48± 1.23 +112.38+112.38 33.07±2.7333.07± 2.73 19.98±2.4219.98± 2.42 +65.52+65.52 Jericho 0.83±2.040.83± 2.04 0.00±0.000.00± 0.00 N/A 16.65±5.5016.65± 5.50 10.57±1.4910.57± 1.49 +57.52+57.52 Table 1 compares the mean success rate and progress rate across four benchmarks, averaged over two sampling temperatures, and reports the relative change (Δ ) between Reflect Reverse and Reflect Forward. Consistent with these averaged results, the detailed results in Tables 3 and 4 in Appendix E show that Reflect Reverse generally outperforms Reflect Forward. Across the 16 configurations, reverse scoring achieves higher SR in 12 and matches it at the reported precision in three others. It also achieves higher PR in 14 configurations. For LLaDA, reverse scoring improves both SR and PR on ALFWorld and ScienceWorld at both T=0.8T=0.8 and T=0.9T=0.9. On ALFWorld at T=0.9T=0.9, SR increases from 8.96% to 16.17% and PR rises from 31.45% to 36.24%. On Jericho, reverse scoring matches forward scoring in SR at both temperatures while improving PR from 17.52% to 20.53% at T=0.8T=0.8 and from 20.94% to 26.72% at T=0.9T=0.9. The gains are particularly consistent for iLLaDA, where reverse scoring improves SR in seven of eight configurations and PR in all eight. On BabyAI, SR increases from 8.93% to 15.77% at T=0.9T=0.9 and from 8.04% to 20.24% at T=1.0T=1.0, while PR rises from 21.52% to 30.73% and from 18.44% to 35.40%, respectively. For LLaDA on BabyAI, reverse scoring improves SR at T=0.9T=0.9 but is less consistent on PR. Overall, these results support reverse conditional scoring as an effective candidate-selection criterion, with advantages over forward scoring across both backbones and the sampling temperatures evaluated for each. 4.3.2 Retry State Loops The dominant retry-loop patterns consist of repeatedly executing the same action (e.g., A) or alternating between two actions (e.g., ABABABAB). Examples of both pattern can be found in Appendix F. We count each additional complete occurrence beyond the initial action or action pair as one retry, without double-counting overlaps. Thus, A contains five retries, while ABABABAB contains three. The retry percentage is defined as the total number of retries divided by the interaction budget. As shown in Figure 3, repetitive behavior is strongly associated with sampling temperature: lower temperatures yield higher mean retry percentages in 21 of the 24 pairwise temperature comparisons, suggesting that reduced action diversity increases agent’s repetitive behavior. Despite this general trend, Reflect Reverse consistently mitigates repetition, achieving the lowest mean retry percentage in all 16 evaluated settings. By contrast, Reflect Forward increases repetition relative to One Pass in five of the eight iLLaDA settings. While these results show that Reflect Reverse is effective at mitigating repetitive behavior, lower retry frequency does not necessarily translate directly into better task performance. For example, on ALFWorld at T=0.9T=0.9, iLLaDA with Reflect Forward exhibits fewer retries than One Pass, yet achieves lower success and progress rates. This indicates that avoiding repetitive action loops is only one component of effective agent behavior: successful task execution also depends on the model’s ability to generate correct and task-appropriate actions. Figure 3: Retry state percentage of LLaDA-8B-Instruct and iLLaDA-8B-Instruct. 5 Related Work Diffusion language models and adaptive decoding. Masked dLLMs such as LLaDA (Nie et al., 2025) and iLLaDA (Nie et al., 2026) generate by iteratively denoising masked spans; the standard inference recipe couples confidence-based remasking with semi-autoregressive block decoding (Nie et al., 2025; Wu et al., 2025; Arriola et al., 2025). Ni et al. (2026) identify a failure mode of this recipe in reasoning tasks: because the sampler commits the positions it is most confident about and defers the rest, it systematically bypasses the high-entropy “forking” tokens at which reasoning paths diverge; by the time these positions are filled, the committed context has already resolved them, so an open decision degenerates into a retrospective alignment with a pre-determined gap (entropy degradation). Forcing left-to-right order restores the confrontation of forks and enlarges the reachable solution space, with the effect growing monotonically in the degree of order freedom. The retry loops we study are this trap in a sharper form. At a failure state, which action now? is a fork, and unlike in reasoning tasks the context already contains a confident fill for it: the failed action, verbatim in s. Adaptive decoding commits the copy without the open decision ever being sampled, and the deferred positions are finally filled in alignment with the retry. LLM agents in embodied environments. Reasoning-and-acting agent frameworks (Yao et al., 2023; Shinn et al., 2023; Mower et al., 2026) couple chain-of-thought traces with environment actions and are standardly evaluated (Liu et al., 2024; Ma et al., 2024) on ALFWorld (Shridhar et al., 2020), ScienceWorld (Wang et al., 2022), BabyAI (Chevalier-Boisvert et al., 2018), and Jericho (Hausknecht et al., 2019). Recent evaluations of dLLM-backed agents (Lu et al., 2026) report that the latency gains of parallel decoding do not transfer to agentic competence, with retry loops as a dominant failure mode. We give this failure a mechanistic account, i.e. adaptive decoding bridging the failure feedback instead of confronting it, and a training-free correction. Sample-then-rerank action selection. Reflect Reverse belongs to the family of decoding-time corrections: best-of-N selection with verifiers (Lightman et al., 2024; Ji et al., 2025; Kang et al., 2026), self-consistency (Wang et al., 2023), and PMI-style rescoring of raw likelihoods (Holtzman et al., 2021; Ji et al., 2026; Nguyen et al., 2026). These methods share our observation that raw model probability is an unreliable score, but they correct it with an external signal: a learned verifier, a majority vote, or a domain-conditional baseline. Our score instead is the model’s own reverse conditional πθ(u∣s,a) _θ(u s,a). The same reverse-conditional identity appears, in a different role, in Lian et al. (2026), who train vision–language–action policies by maximizing the log-likelihood ratio logp(ℓ∣a,v)−logp(ℓ∣v) p( a,v)- p( v), the conditional PMI between action and instruction, as a training objective; our correction instead is a training-free, decoding-time rule. 6 Conclusion And Future Works We traced the retry loops that stall diffusion language model agents to a structural property of masked decoding: because the sampler commits the positions it is most confident about and defers the rest, the open decision at a failure state is postponed until the committed context resolves it; and the context already offers a confident fill, the failed action itself. We formalized the resulting distortion of the action distribution as a task-blind corruption, i.e. a state–action factor that inflates contextually salient actions independently of the task, and verified it through goal-swap interventions. The formalization is what motivates the correction: a factor that carries no task information cannot survive a posterior over tasks, so it cancels from the reverse conditional πθ(u∣s,a) _θ(u s,a), which masked dLLMs evaluate natively by masking the task tokens and denoising. The resulting rule, Reflect Reverse, is training-free, requires no learned critic, value function, or environment rollouts, adds only a few parallel denoising passes per candidate, and improves success and progress rates over forward-scoring baselines on four embodied benchmarks and two backbones. Future work. Firstly, Proposition 3.1 governs the comparison among the candidates on the table but proposals are still drawn from the corrupted forward conditional, so the faithful model’s support may go uncovered. Extending the invariance to the proposal step (e.g. dividing out an estimated ε(s,a) (s,a)) would close this gap. Secondly, the corruption is ultimately a training artifact of adaptive decoding; decoding-time correction could be complemented by fine-tuning at failure states or by regularizing forward–reverse consistency, removing the pathology at its source. Finally, the retry loop is the agentic form of entropy degradation, and the reverse question, does this continuation still explain the prompt?, applies to reasoning chains, suggesting reverse scoring as a general decoding-time instrument for masked diffusion models. References Arriola et al. (2025) M. Arriola, A. Gokaslan, J. Chiu, Z. Yang, Z. Qi, J. Han, S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, Vol. 2025, p. 50726–50753. Cited by: §5. Austin et al. (2021) J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems 34, p. 17981–17993. Cited by: §2.1. Binette (2019) O. Binette A note on reverse pinsker inequalities. IEEE Transactions on Information Theory 65 (7), p. 4094–4096. External Links: ISSN 1557-9654, Link, Document Cited by: §C.2, §C.2. Chae et al. (2024) K. Chae, J. Choi, Y. Jo, and T. Kim Mitigating hallucination in abstractive summarization with domain-conditional mutual information. In Findings of the association for computational linguistics: NAACL 2024, p. 1809–1820. Cited by: §3.2. Chevalier-Boisvert et al. (2018) M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio BabyAI: a platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, External Links: Link Cited by: §4.1, §5. Christianos et al. (2023) F. Christianos, G. Papoudakis, M. Zimmer, T. Coste, Z. Wu, J. Chen, K. Khandelwal, J. Doran, X. Feng, J. Liu, Z. Xiong, Y. Luo, J. Hao, K. Shao, H. Bou-Ammar, and J. Wang Pangu-agent: a fine-tunable generalist agent with structured reasoning. External Links: 2312.14878, Link Cited by: §1. Gong et al. (2025) S. Gong, S. Agarwal, Y. Zhang, J. Ye, L. Zheng, M. Li, C. An, P. Zhao, W. Bi, J. Han, et al. Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations, Vol. 2025, p. 5046–5073. Cited by: §1. Gulrajani and Hashimoto (2023) I. Gulrajani and T. B. Hashimoto Likelihood-based diffusion language models. Advances in Neural Information Processing Systems 36, p. 16693–16715. Cited by: item 2. Hausknecht et al. (2019) M. J. Hausknecht, P. Ammanabrolu, M. Côté, and X. Yuan Interactive fiction games: a colossal adventure. In AAAI Conference on Artificial Intelligence, External Links: Link Cited by: §4.1, §5. Holtzman et al. (2019) A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Cited by: §1. Holtzman et al. (2021) A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer Surface form competition: why the highest probability answer isn’t always right. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7038–7051. Cited by: §3.2, §5. Ji et al. (2025) X. Ji, S. S. Ramesh, M. Zimmer, I. Bogunovic, J. Wang, and H. B. Ammar On almost surely safe alignment of large language models at inference-time. External Links: 2502.01208, Link Cited by: §5. Ji et al. (2026) X. Ji, R. Tutunov, M. Zimmer, and H. B. Ammar Scalable power sampling: unlocking efficient, training-free reasoning for llms via distribution sharpening. External Links: 2601.21590, Link Cited by: §5. Kang et al. (2026) Z. Kang, X. Zhao, and D. Song Scalable best-of-n selection for large language models via self-certainty. Advances in neural information processing systems 38, p. 19720–19745. Cited by: §5. Li et al. (2022) X. L. Li, J. Thickstun, I. Gulrajani, P. Liang, and T. B. Hashimoto Diffusion-lm improves controllable text generation. External Links: 2205.14217, Link Cited by: §1. Lian et al. (2026) S. Lian, B. Yu, X. Lin, L. T. Yang, Z. Shen, C. Wu, Y. Miao, C. Huang, and K. Chen LangForce: bayesian decomposition of vision language action models via latent action queries. External Links: 2601.15197, Link Cited by: §5. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: §5. Liu et al. (2024) X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: evaluating llms as agents. In International Conference on Learning Representations, Vol. 2024, p. 52989–53046. Cited by: §5. Lou et al. (2023) A. Lou, C. Meng, and S. Ermon Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834. Cited by: §1. Lu et al. (2026) Q. Lu, L. Ding, K. Zhang, J. Zhang, and D. Tao The bitter lesson of diffusion language models for agentic workflows: a comprehensive reality check. ArXiv abs/2601.12979. External Links: Link Cited by: §4.1, §5. Ma et al. (2024) C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong, and J. He Agentboard: an analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems 37, p. 74325–74362. Cited by: §4.1, §5. Mower et al. (2026) C. E. Mower, Y. Wan, H. Yu, A. Grosnit, J. Gonzalez-Billandon, M. Zimmer, P. Liu, D. Palenicek, D. Tateo, J. Peters, K. Qu, M. Zhang, G. Lan, A. Cramariuc, C. Cadena, M. Hutter, G. Tian, Y. Zhuang, K. Shao, X. Quan, J. Hao, J. Wang, and H. Bou-Ammar A robot operating system framework for using large language models in embodied AI. Nature Machine Intelligence 8 (3), p. 313–325. External Links: ISSN 2522-5839, Document, Link Cited by: §5. Nguyen et al. (2026) T. Nguyen, M. Zimmer, R. Tutunov, X. Ji, and H. B. Ammar The model knows, the decoder finds: future value guided particle power sampling. External Links: 2605.02427, Link Cited by: §5. Ni et al. (2026) Z. Ni, S. Wang, Y. Yue, T. Yu, W. Zhao, Y. Hua, T. Chen, J. Song, C. Yu, B. Zheng, and G. Huang The flexibility trap: rethinking the value of arbitrary order in diffusion language models. External Links: 2601.15165, Link Cited by: §1, §5. Nie et al. (2026) S. Nie, Q. Min, S. Xu, Z. Huang, Y. Song, S. Yong, Y. Lin, W. X. Zhao, C. Li, and J. Wen Improved large language diffusion models. ArXiv abs/2606.25331. External Links: Link Cited by: §4.1, §5. Nie et al. (2025) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. ArXiv abs/2502.09992. External Links: Link Cited by: §1, §1, §2.1, §4.1, §5. Ou et al. (2025) J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, Vol. 2025, p. 64972–65009. Cited by: item 2. Sahoo et al. (2024) S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems 37, p. 130136–130184. Cited by: §2.1. Schiff et al. (2025) Y. Schiff, S. Sahoo, H. Phung, G. Wang, S. Boshar, H. Dalla-Torre, B. Almeida, A. Rush, T. Pierrot, and V. Kuleshov Simple guidance mechanisms for discrete diffusion models. In International Conference on Learning Representations, Vol. 2025, p. 43776–43821. Cited by: §2.1. Shi et al. (2024) J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems 37, p. 103131–103167. Cited by: §2.1. Shinn et al. (2023) N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. External Links: 2303.11366, Link Cited by: §5. Shridhar et al. (2020) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. J. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. ArXiv abs/2010.03768. External Links: Link Cited by: §4.1, §5. Wang et al. (2022) R. Wang, P. A. Jansen, M. Côté, and P. Ammanabrolu ScienceWorld: is your agent smarter than a 5th grader?. In Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §4.1, §5. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. External Links: 2203.11171, Link Cited by: §5. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: item 1. Wu et al. (2025) C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie Fast-dllm: training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. ArXiv abs/2505.22618. External Links: Link Cited by: §4.1, §5. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. External Links: 2210.03629, Link Cited by: §5. Appendix A Task corruption verification We evaluate task sensitivity on 208 retry states from ALFWorld benchmark. For each state sis_i, we retain the interaction history and compare the repeated action airepa_i^rep with a distinct admissible alternative aialta_i^alt, selected by Qwen2.5-7B-Instruct. Both actions remain fixed while the original task ui,0u_i,0 is replaced by K=10K=10 randomly sampled task descriptions. Let pm(a∣s,u)p_m(a s,u) denote the action score for model m: summed autoregressive token log likelihood for Qwen, or a Monte Carlo diffusion variational score estimate for LLaDA. LLaDA uses identical sampled masks across task conditions. For visualization, both actions are centered on the repeated action’s original-task score: Di,jm,a=pm(a∣si,ui,j)−pm(airep∣si,ui,0),a∈arep,aalt.D_i,j^m,a=p_m(a s_i,u_i,j)-p_m(a_i^rep s_i,u_i,0), a∈\a^rep,a^alt\. (12) The plotted curves are the state averages, D¯jm,a=N−1∑i=1NDi,jm,a D_j^m,a=N^-1 _i=1^ND_i,j^m,a. This common baseline preserves the relative preference between the two actions; the alternative curve is therefore not centered on its own original-task score. We quantify task sensitivity using the action-preference margin: Δi,jm=pm(aialt∣si,ui,j)−pm(airep∣si,ui,j). _i,j^m=p_m(a_i^alt s_i,u_i,j)-p_m(a_i^rep s_i,u_i,j). (13) For each state, we compute its population standard deviation across the replacement tasks, then average over states: σm=1N∑i=1N1K∑j=1K(Δi,jm−Δ¯im)2,Δ¯im=1K∑j=1KΔi,jm. _m= 1N _i=1^N 1K _j=1^K ( _i,j^m- _i^m )^2, _i^m= 1K _j=1^K _i,j^m. (14) The original task is excluded from this standard deviation. Smaller σm _m indicates less variation in action preference across replacement goals. We additionally measure how frequently task replacement reverses the preferred action. Defining bi,jm=[Δi,jm>0]b_i,j^m=1[ _i,j^m>0], the preference flip rate is Fm=1NK∑i=1N∑j=1K|bi,jm−bi,0m|.F_m= 1NK _i=1^N _j=1^K |b_i,j^m-b_i,0^m |. (15) Each replacement is compared with the original task, yielding 2,080 comparisons per model. These calculations give σLLaDA=1.01 _LLaDA=1.01 and σQwen=4.71 _Qwen=4.71 nats, with flip rates of 6.6%6.6\% and 44.4%44.4\%, respectively. The shaded bands use 2,000 episode-level bootstrap resamples to obtain pointwise 95% intervals for each action’s change from its own original-task score. For the alternative curve, these intervals are shifted by the observed mean original-task preference margin to match the plotted baseline; they therefore represent uncertainty in the task-induced change, conditional on that baseline offset. Appendix B Reflect reverse algorithm We summarize the method introduced in Section 3.1 in the following pseudocode. Algorithm 1 Reflect Reverse 0: State s, task u, dLLM πθ _θ, number of proposals K, number of reflection rollouts M, inverse temperature β 0: Selected action a⋆a 1: Propose: 2: Sample K action candidates a1,…,aK∼πθ(⋅∣s,u)a_1,…,a_K _θ(· s,u) with ReAct 3: Deduplicate candidates to obtain (s)A(s) 4: Reflect: 5: for each a∈(s)a (s) do 6: ϕ(a)←0φ(a)← 0 7: for m=1,…,Mm=1,…,M do 8: Sample a subset of task-token positions Bm⊆1,…,|u|B_m \1,…,|u|\ 9: Construct u~(m) u^(m) by masking positions BmB_m in u 10: Condition the dLLM on (s,a,u~(m))(s,a, u^(m)) 11: ℓm(a)←1|Bm|∑j∈Bmlogπθ(uj∣s,a,u~(m)) _m(a)← 1|B_m| _j∈ B_m _θ (u_j s,a, u^(m) ) 12: ϕ(a)←ϕ(a)+ℓm(a)/Mφ(a)←φ(a)+ _m(a)/M 13: end for 14: end for 15: Act: 16: ρ(a∣s,u)←exp(βϕ(a))∑a′∈(s)exp(βϕ(a′))∀a∈(s) ρ(a s,u)← (βφ(a)) _a (s) (βφ(a )) ∀ a (s) 17: Sample a⋆∼ρ(⋅∣s,u)a ρ(· s,u) 18: return a⋆a Appendix C Proof and theory Our method uses the model’s reverse conditional πθ(u∣s,a) _θ(u s,a), which masked dLLMs evaluate natively. We assume the two conditionals are consistent views of one underlying joint. Assumption C.1 (Coherent conditionals). The model’s forward and reverse conditionals satisfy Bayes’ rule with respect to the task prior p(u∣s)p(u s): for every a and every u in the support of p(⋅∣s)p(· s), πθ(u∣s,a)=p(u∣s)πθ(a∣s,u)∑u′p(u′∣s)πθ(a∣s,u′). _θ(u s,a)\;=\; p(u s)\, _θ(a s,u) _u p(u s)\, _θ(a s,u ). (16) Assumption C.1 is an idealization of the same kind used whenever a single model’s conditionals are composed; it is what licenses reading the reverse score as a posterior. A stronger variant, in which πθ(u∣s,a) _θ(u s,a) is coherent with the task posterior p(u∣s,a)p(u s,a) as well, removes the task prior from the proposition entirely (Remark C.3). C.1 Proof of the recovery proposition This appendix proves Proposition 3.1. The argument has two steps. The first step (Lemma C.1) isolates the only property of the task prior that matters: it can be changed arbitrarily without moving the induced task posterior, provided the corruption is re-absorbed accordingly. The second step substitutes Assumption 2.1 and lets the corruption cancel. Lemma C.1 (Prior rescaling leaves the posterior untouched). Let q(u∣s)q(u s) and q′(u∣s)q (u s) be two task priors with the same support, and let ν(a∣s,u)ν(a s,u) be any conditional. Define νq(u∣s,a):=q(u∣s)ν(a∣s,u)∑u′q(u′∣s)ν(a∣s,u′), _q(u s,a)\;:=\; q(u s)\,ν(a s,u) _u q(u s)\,ν(a s,u ), (17) and νq′(u∣s,a) _q (u s,a) analogously with q′q in place of q. Then νq(u∣s,a)=νq′(u∣s,a) _q(u s,a)= _q (u s,a) for every a if q′(u∣s)/q(u∣s)q (u s)/q(u s) is constant in u on the support; in particular the induced posterior is unchanged whenever q′q is replaced by q itself. Proof. Immediate from Eq.17: replacing q by q′q multiplies the numerator at u by q′(u∣s)/q(u∣s)q (u s)/q(u s) and the denominator by the same factor averaged over u′u ; the two coincide for every a exactly when the ratio does not depend on u. The trivial case q′=q =q gives the invariance used below. ∎ Proposition (Proposition 3.1 restated; exact recovery of the faithful task posterior). Let Assumptions 2.1 and C.1 hold. Then, for every state s, task u in the support of p(⋅∣s)p(· s), and action a, πθ(u∣s,a)=ω(u∣s)μ(a∣s,u)∑u′ω(u′∣s)μ(a∣s,u′), _θ(u s,a)\;=\; ω(u s)\,μ(a s,u) _u ω(u s)\,μ(a s,u ), (18) where ω(u∣s):=p(u∣s)Z(s,u)ω(u s)\;:=\; p(u s)Z(s,u) (19) is a positive task weight that does not depend on the candidate action a, and Z(s,u)=∑a′μ(a′∣s,u)eε(s,a′)Z(s,u)= _a μ(a s,u)\,e (s,a ) is the normalizer of Assumption 2.1. Proof. By Assumption C.1, the model’s reverse conditional satisfies πθ(u∣s,a)=p(u∣s)πθ(a∣s,u)∑u′p(u′∣s)πθ(a∣s,u′). _θ(u s,a)\;=\; p(u s)\, _θ(a s,u) _u p(u s)\, _θ(a s,u ). (20) Apply Assumption 2.1 to every term. The numerator at u becomes p(u∣s)πθ(a∣s,u)=p(u∣s)μ(a∣s,u)eε(s,a)Z(s,u),p(u s)\, _θ(a s,u)\;=\;p(u s)\, μ(a s,u)\,e (s,a)Z(s,u), (21) and the denominator becomes ∑u′p(u′∣s)πθ(a∣s,u′)=∑u′p(u′∣s)μ(a∣s,u′)eε(s,a)Z(s,u′)=eε(s,a)∑u′p(u′∣s)Z(s,u′)μ(a∣s,u′), _u p(u s)\, _θ(a s,u )\;=\; _u p(u s)\, μ(a s,u )\,e (s,a)Z(s,u )\;=\;e (s,a) _u p(u s)Z(s,u )\,μ(a s,u ), (22) where the factor eε(s,a)e (s,a) carries no task index u′u and factors out of the sum. Substituting Eq.21 and Eq.22 into Eq.20, the factor eε(s,a)e (s,a) appears once in the numerator and once in the denominator and cancels exactly: πθ(u∣s,a)=(p(u∣s)/Z(s,u))μ(a∣s,u)∑u′(p(u′∣s)/Z(s,u′))μ(a∣s,u′)=ω(u∣s)μ(a∣s,u)∑u′ω(u′∣s)μ(a∣s,u′), _θ(u s,a)\;=\; (p(u s)/Z(s,u) )\,μ(a s,u) _u (p(u s)/Z(s,u ) )\,μ(a s,u )\;=\; ω(u s)\,μ(a s,u) _u ω(u s)\,μ(a s,u ), which is Eq.18. Lemma C.1 is what licenses reading the right-hand side as a task posterior: the prior has been rescaled from p to ω, but the induced posterior remains a genuine Bayes posterior of μ, simply under the prior ω. ∎ Identity Eq.18 holds for every action individually, so it preserves every feature of the comparison of actions: for any a1,a2a_1,a_2, logπθ(u∣s,a1)−logπθ(u∣s,a2)= _θ(u s,a_1)- _θ(u s,a_2)\;=\; (23) logμ(a1∣s,u)∑u′ω(u′∣s)μ(a1∣s,u′)−logμ(a2∣s,u)∑u′ω(u′∣s)μ(a2∣s,u′), μ(a_1 s,u) _u ω(u s)\,μ(a_1 s,u )\;-\; μ(a_2 s,u) _u ω(u s)\,μ(a_2 s,u ), (24) with equal margins, equal ties, and hence the same ordering, top-k set, and argmax as the faithful model’s task-evidence ratio; the margins are those of the ratio, and need not coincide with the margins of μ(⋅∣s,u)μ(· s,u) itself (Remark C.1). Corollary C.1 (The corrected policy is the softmax of the faithful task-evidence ratio, at any temperature). Let (s)A(s) be a finite candidate set and β>0β>0 an inverse temperature. Under Assumptions 2.1 and C.1, πθ(u∣s,a)β∑a′∈(s)πθ(u∣s,a′)β=(μ(a∣s,u)/∑u′ω(u′∣s)μ(a∣s,u′))β∑a′∈(s)(μ(a′∣s,u)/∑u′ω(u′∣s)μ(a′∣s,u′))β _θ(u s,a)^β _a (s) _θ(u s,a )^β\;=\; (μ(a s,u) / _u ω(u s)\,μ(a s,u ) )^β _a (s) (μ(a s,u) / _u ω(u s)\,μ(a s,u ) )^β (25) for every a∈(s)a (s): an exact equality of probability distributions over candidates. The limit β→∞β→∞ recovers the argmax statement; β→0β→ 0 recovers uniform sampling. Proof. By Proposition 3.1, for each a the score πθ(u∣s,a) _θ(u s,a) equals the faithful task-evidence ratio times the factor ω(u∣s)ω(u s), which does not depend on a. Raising to β and normalizing over (s)A(s) removes this shared factor from both sides. ∎ Remark C.1 (Ratio versus conditional). The right-hand side of Eq.25 is the softmax of the faithful task-evidence ratio μ(a∣s,u)/∑u′ω(u′∣s)μ(a∣s,u′)μ(a s,u) / _u ω(u s)\,μ(a s,u ), not of the faithful conditional μ(a∣s,u)μ(a s,u) itself: the task-marginal likelihood m(a)=∑u′ω(u′∣s)μ(a∣s,u′)m(a)= _u ω(u s)\,μ(a s,u ) depends on the candidate and is not removed by normalization. The two policies coincide when m is constant over (s)A(s), e.g. when the candidates are equally plausible under the task mixture, or in the argmax limit β→∞β→∞, where only the ordering survives. Unconditionally, what is recovered exactly is the faithful task-evidence ratio’s comparison of candidates: ordering and ties. Remark C.2 (On the weights ω). The faithful baseline on the right of Eq.18 mixes tasks with weights ω(u∣s)=p(u∣s)/Z(s,u)ω(u s)=p(u s)/Z(s,u) rather than with the prior p(u∣s)p(u s) itself, because the corruption rescales each task-conditional by its own normalizer Z(s,u)Z(s,u). Only two properties of ω are used: it is positive and it does not depend on the candidate action a. If Z(s,u)Z(s,u) happens to be constant in u, i.e. the corruption rescales every task-conditional by the same total mass, then ω∝pω p and the baseline is exactly the task-marginal of μ under the original prior. Remark C.3 (Dropping the task prior). If the model’s conditionals are coherent with the task posterior as well, i.e. πθ(u∣s,a) _θ(u s,a) directly proportional to p(u∣s,a)πθ(a∣s,u)p(u s,a)\, _θ(a s,u), then the same cancellation applies to p(u∣s,a)p(u s,a) itself, and Proposition 3.1 holds with ω(u∣s)ω(u s) replaced by p(u∣s,a)/Z(s,u)p(u s,a)/Z(s,u). In this case the method requires only the single reverse score πθ(u∣s,a) _θ(u s,a): no explicit task prior and no baseline forward pass enter at all. C.2 Residual bias of the corrected policy Remark C.1 shows that the corrected policy ρ is exactly the softmax of the faithful task-evidence ratio, and asks how far it remains from the faithful conditional itself. This appendix answers the question: it proves Theorem 3.1 of Section 3.3, via a sharp range lemma (Lemma C.2). Throughout, fix a state s, a task u in the support of p(⋅∣s)p(· s), and a finite candidate set ⊆suppμ(⋅∣s,u)A μ(· s,u) with ||≥2|A|≥ 2; recall ρ from Eq.3, the positive task weights ω from Proposition 3.1, the task-marginal likelihood m(a)=∑u′ω(u′∣s)μ(a∣s,u′)m(a)= _u ω(u s)\,μ(a s,u ), and the restricted distributions and spreads of Eq.7. Support identity. Since eε(s,a)>0e (s,a)>0, Assumption 2.1 gives πθ(a∣s,u)>0 _θ(a s,u)>0 if and only if μ(a∣s,u)>0μ(a s,u)>0. Hence a candidate set proposed from πθ(⋅∣s,u) _θ(· s,u) automatically satisfies ⊆suppμ(⋅∣s,u)A μ(· s,u), and on A we have μ(a∣s,u)>0μ(a s,u)>0 as well as m(a)≥ω(u∣s)μ(a∣s,u)>0m(a)≥ω(u s)\,μ(a s,u)>0. Moreover, by Eq.1, π(a)=μ(a)eε(s,a)∑a′∈μ(a′)eε(s,a′),a∈, _A(a)\;=\; _A(a)\,e (s,a) _a _A(a )\,e (s,a ), a , (26) since the normalizer Z(s,u)Z(s,u) of Eq.1 cancels between numerator and denominator: the corrupted restriction is the faithful restriction tilted by the corruption. A sharp range bound. Lemma C.2 below quantifies a simple fact: two distributions whose likelihood ratio is confined to a band cannot be far apart in total variation, and the worst case for a band of logarithmic width Δ is exactly tanh(Δ/4) ( /4). The key input is a sharp result of Binette (2019) (Corollary 5): if the likelihood ratio dP/dQdP/dQ has essential infimum m and essential supremum M, then dTV(P,Q)≤(M−1)(1−m)M−md_TV(P,Q)≤ (M-1)(1-m)M-m, with the best possible constant for fixed (m,M)(m,M). Lemma C.2 is what this bound becomes under the scale-invariant constraint that only the range of log(dQ/dP) (dQ/dP), and not the location of the band, is known. Lemma C.2 (Range bound, sharp). Let P and Q be strictly positive probability distributions on a finite set S, and suppose the log-likelihood ratio has range at most Δ : maxx∈SlogQ(x)P(x)−minx∈SlogQ(x)P(x)≤Δ. _x∈ S Q(x)P(x)\;-\; _x∈ S Q(x)P(x)\;≤\; . (27) Then dTV(P,Q)≤tanh(Δ/4),wheredTV(P,Q):=12∑x∈S|P(x)−Q(x)|.d_TV(P,Q)\;≤\; \! ( /4 ), d_TV(P,Q):= 12 _x∈ S|P(x)-Q(x)|. (28) The constant is sharp: for every Δ there is a two-point family attaining equality. Lemma C.2 is obtained by Corollary 5 of Binette (2019), followed by optimizing over the location of the bounded likelihood-ratio interval. Proof of Theorem 3.1. (i) By Eq.25, ρ(a)=R(a)/∑a′∈R(a′)ρ(a)=R(a) / _a R(a ) with R(a)=(μ(a∣s,u)/m(a))βR(a)=(μ(a s,u)/m(a))^β. Hence for every a∈a , logρ(a)μ(a)=(β−1)logμ(a∣s,u)−βlogm(a)+(logμ(∣s,u)−log∑a′∈R(a′))⏟constant in a. ρ(a) _A(a)=(β-1) μ(a s,u)-β m(a)+ ( μ(A s,u)- _a R(a ) )_constant in a. (29) The range of log(ρ/μ) (ρ/ _A) over A is therefore the range of (β−1)logμ(⋅∣s,u)−βlogm(β-1) μ(· s,u)-β m, which is at most |β−1|Λμ+βΛm|β-1|\, _μ+β _m. Lemma C.2, applied to P=μP= _A and Q=ρQ=ρ (both strictly positive on A by the support identity), gives the first inequality in Eq.9. For the second inequality we show Λm≤Λtask _m≤ _task. Fix a1,a2∈a_1,a_2 and let S2:=u′∈suppp(⋅∣s):μ(a2∣s,u′)>0S_2:=\u ∈ p(· s):μ(a_2 s,u )>0\. If some u′∉S2u ∉ S_2 has μ(a1∣s,u′)>0μ(a_1 s,u )>0, then logμ(a1∣s,u′)−logμ(a2∣s,u′)=+∞ μ(a_1 s,u )- μ(a_2 s,u )=+∞, so Λtask=+∞ _task=+∞ and the claim is vacuous. Otherwise μ(a1∣s,u′)=0μ(a_1 s,u )=0 off S2S_2, and the mediant inequality (a weighted mean of ratios lies below the largest ratio) gives m(a1)m(a2)=∑u′∈S2ω(u′∣s)μ(a2∣s,u′)[μ(a1∣s,u′)/μ(a2∣s,u′)]∑u′∈S2ω(u′∣s)μ(a2∣s,u′)≤maxu′∈S2μ(a1∣s,u′)μ(a2∣s,u′), m(a_1)m(a_2)= _u ∈ S_2ω(u s)\,μ(a_2 s,u )\, [μ(a_1 s,u )/μ(a_2 s,u ) ] _u ∈ S_2ω(u s)\,μ(a_2 s,u )\;≤\; _u ∈ S_2 μ(a_1 s,u )μ(a_2 s,u ), so logm(a1)−logm(a2)≤maxu′[logμ(a1∣s,u′)−logμ(a2∣s,u′)]≤Λtask m(a_1)- m(a_2)≤ _u [ μ(a_1 s,u )- μ(a_2 s,u ) ]≤ _task. Taking the maximum over a1,a2∈a_1,a_2 yields Λm≤Λtask _m≤ _task, and tanh is increasing. (i) By Eq.26, writing ε(a):=ε(s,a) (a):= (s,a) and ε+:=ε(a+) ^+:= (a^+), π(a+)=γeε+∑a∈μ(a)eε(a)≥γeε+γeε++(1−γ)eε+−Δ2=γ+(1−γ)e−Δ2, _A(a^+)= γ\,e ^+ _a _A(a)e (a)\;≥\; γ\,e ^+γ e ^++(1-γ)e ^+- _2= γ+(1-γ)e^- _2, since ε(a)≤ε+−Δ2 (a)≤ ^+- _2 for all a≠a+a≠ a^+ by definition of Δ2 _2; equality holds when ||=2|A|=2, as there is only one other candidate. Moreover π(a+)≥γ=μ(a+) _A(a^+)≥γ= _A(a^+), because ∑aμ(a)eε(a)≤eε+ _a _A(a)e (a)≤ e ^+. Taking the event E=a+E=\a^+\ in dTV(μ,π)=supE(π(E)−μ(E))d_TV( _A, _A)= _E ( _A(E)- _A(E) ), dTV(μ,π)≥π(a+)−γ≥γ+(1−γ)e−Δ2−γ=γ(1−γ)(1−e−Δ2)γ+(1−γ)e−Δ2.d_TV( _A, _A)\;≥\; _A(a^+)-γ\;≥\; γ+(1-γ)e^- _2-γ= γ(1-γ) (1-e^- _2 )γ+(1-γ)e^- _2. For the monotonicity claim, write x=e−Δ2x=e^- _2: the right-hand side is γ(1−γ)(1−x)/(γ+(1−γ)x)γ(1-γ)(1-x)/(γ+(1-γ)x), with derivative −γ(1−γ)/(γ+(1−γ)x)2<0-γ(1-γ)/(γ+(1-γ)x)^2<0 in x; hence it increases in Δ2 _2 and tends to γ(1−γ)/γ=1−γ(1-γ)/γ=1-γ as x→0x→ 0. ∎ Remark C.4 (Finite-M estimation enters additively). Theorem 3.1 analyzes the exact reverse score ϕ(a)=logπθ(u∣s,a)φ(a)= _θ(u s,a), as does Proposition 3.1; the algorithm uses the Monte Carlo estimate ϕ φ of Eq.2. If the estimation error δ(a):=ϕ^(a)−ϕ(a)δ(a):= φ(a)-φ(a) satisfies supa∈|δ(a)|≤δmax _a |δ(a)|≤ _ , then for the estimated policy ρ^(a)∝eβϕ^(a) ρ(a) e^β φ(a), logρ^(a)ρ(a)=βδ(a)−logρ[eβδ]⟹dTV(ρ^,ρ)≤tanh(βδmax/2), ρ(a)ρ(a)=β\,δ(a)- _ρ [e^βδ ] d_TV( ρ,ρ)≤ \! (β _ /2 ), by Lemma C.2, since the range of βδβδ over A is at most 2βδmax2β _ . Hence dTV(ρ^,μ)≤tanh(βδmax/2)+Bd_TV( ρ, _A)≤ (β _ /2)+B: estimation noise degrades the corruption-free guarantee gracefully and vanishes as the estimator becomes consistent. Appendix D Hyperparameters Table 2 summarizes the hyperparameters used for action generation and reflection. On ScienceWorld and Jericho, Reflect Reverse uses more Monte Carlo sampling steps than Reflect Forward because the task description is longer than the candidate actions at each interaction step, resulting in more tokens to evaluate during reverse reflection. Due to iLLaDA’s pre-training configuration, the assistant prefill and end-of-thinking sequence is </think> followed by a newline and Thought:, while the visible response begins with Thought: followed by a space. Table 2: Evaluation hyperparameters for LLaDA and iLLaDA on ALFWorld, ScienceWorld, BabyAI, and Jericho. The upper table reports model settings and the lower table reports forward and reverse likelihood-scoring parameters Hyperparameters LLaDA iLLaDA Generation length (tokens) 128 128 Requested diffusion steps 128 128 Block size (tokens) 32 32 Threshold 0.9 0.9 Dual cache On Off Configured context length (tokens) 4,000 8,192 Temperature 0.8,0.9\0.8,0.9\ 0.9,1.0\0.9,1.0\ Seeds 12,22,32\12,22,32\ 12,22,32\12,22,32\ Maximum agent steps per episode 30 30 Generation attempt limit 20 20 Reflect Forward Reflect Reverse Benchmark MC rollouts Batch size MC rollouts Batch size ALFWorld 64 16 64 16 ScienceWorld 64 16 128 16 BabyAI 64 16 64 16 Jericho 64 16 128 16 Appendix E Supplementary experiment results We provide the full experimental results omitted from the main paper for space. Tables 3 and 4 report performance across all benchmarks, model variants, methods, and sampling temperatures, offering a more detailed view of the results summarized in the main text. Table 3: Success rate (SR) and progress rate (PR) evaluated by LLaDA-8B-Instruct. Entries are mean ± sample standard deviation. Boldface marks the best method within each benchmark and temperature category (including Mean), with ties included. Benchmark Method Temperature SR (%) PR (%) ALFWorld One Pass 0.8 6.47±2.406.47± 2.40 26.87±1.4626.87± 1.46 0.9 8.71±1.558.71± 1.55 29.33±0.7929.33± 0.79 Mean 7.59±2.187.59± 2.18 28.10±1.7128.10± 1.71 Reflect Forward 0.8 7.46±1.497.46± 1.49 29.73±2.1829.73± 2.18 0.9 8.96±0.758.96± 0.75 31.45±1.0231.45± 1.02 Mean 8.21±1.338.21± 1.33 30.59±1.7930.59± 1.79 Reflect Reverse 0.8 12.19±2.2812.19± 2.28 34.29±3.0334.29± 3.03 0.9 16.17±2.2816.17± 2.28 36.24±1.0136.24± 1.01 Mean 14.18±2.9914.18± 2.99 35.26±2.2935.26± 2.29 ScienceWorld One Pass 0.8 3.70±1.703.70± 1.70 21.69±4.4221.69± 4.42 0.9 4.07±0.644.07± 0.64 23.02±1.2123.02± 1.21 Mean 3.89±1.173.89± 1.17 22.36±2.9922.36± 2.99 Reflect Forward 0.8 3.70±1.283.70± 1.28 20.03±2.3720.03± 2.37 0.9 5.56±1.925.56± 1.92 22.26±2.3422.26± 2.34 Mean 4.63±1.784.63± 1.78 21.15±2.4321.15± 2.43 Reflect Reverse 0.8 7.78±1.927.78± 1.92 23.24±2.9123.24± 2.91 0.9 5.93±1.285.93± 1.28 22.41±1.2622.41± 1.26 Mean 6.85±1.786.85± 1.78 22.82±2.0622.82± 2.06 BabyAI One Pass 0.8 14.88±0.5214.88± 0.52 25.97±0.6625.97± 0.66 0.9 14.29±0.8914.29± 0.89 27.46±1.0227.46± 1.02 Mean 14.58±0.7314.58± 0.73 26.71±1.1226.71± 1.12 Reflect Forward 0.8 18.75±4.7218.75± 4.72 30.41±3.1930.41± 3.19 0.9 16.37±1.0316.37± 1.03 28.39±0.7728.39± 0.77 Mean 17.56±3.3217.56± 3.32 29.40±2.3529.40± 2.35 Reflect Reverse 0.8 17.86±1.7917.86± 1.79 26.55±1.1726.55± 1.17 0.9 17.26±3.6117.26± 3.61 27.12±1.7227.12± 1.72 Mean 17.56±2.5717.56± 2.57 26.83±1.3526.83± 1.35 Jericho One Pass 0.8 1.67±2.891.67± 2.89 19.59±6.3719.59± 6.37 0.9 0.00±0.000.00± 0.00 20.35±3.1720.35± 3.17 Mean 0.83±2.040.83± 2.04 19.97±4.5219.97± 4.52 Reflect Forward 0.8 0.00±0.000.00± 0.00 17.52±2.5017.52± 2.50 0.9 3.33±2.893.33± 2.89 20.94±1.3920.94± 1.39 Mean 1.67±2.581.67± 2.58 19.23±2.6119.23± 2.61 Reflect Reverse 0.8 0.00±0.000.00± 0.00 20.53±2.2220.53± 2.22 0.9 3.33±5.773.33± 5.77 26.72±2.4726.72± 2.47 Mean 1.67±4.081.67± 4.08 23.63±3.9923.63± 3.99 Table 4: Success rate (SR) and progress rate (PR) evaluated by iLLaDA-8B-Instruct. Entries are mean ± sample standard deviation. Boldface marks the best method within each benchmark and temperature category (including Mean), with ties included. Benchmark Method Temperature SR (%) PR (%) ALFWorld One Pass 0.9 5.47±0.435.47± 0.43 19.57±1.7619.57± 1.76 1.0 4.73±0.434.73± 0.43 21.06±1.8121.06± 1.81 Mean 5.10±0.565.10± 0.56 20.32±1.7920.32± 1.79 Reflect Forward 0.9 0.75±0.750.75± 0.75 15.80±1.3915.80± 1.39 1.0 0.75±1.290.75± 1.29 16.36±1.9716.36± 1.97 Mean 0.75±0.940.75± 0.94 16.08±1.5616.08± 1.56 Reflect Reverse 0.9 9.95±1.889.95± 1.88 27.69±2.8627.69± 2.86 1.0 10.20±1.1410.20± 1.14 28.71±1.5528.71± 1.55 Mean 10.07±1.4010.07± 1.40 28.20±2.1328.20± 2.13 ScienceWorld One Pass 0.9 0.00±0.000.00± 0.00 15.77±0.2215.77± 0.22 1.0 0.37±0.640.37± 0.64 16.47±1.7316.47± 1.73 Mean 0.19±0.450.19± 0.45 16.12±1.1716.12± 1.17 Reflect Forward 0.9 0.37±0.640.37± 0.64 7.70±1.217.70± 1.21 1.0 0.00±0.000.00± 0.00 9.82±1.489.82± 1.48 Mean 0.19±0.450.19± 0.45 8.76±1.688.76± 1.68 Reflect Reverse 0.9 3.70±1.283.70± 1.28 25.95±2.4325.95± 2.43 1.0 4.07±2.314.07± 2.31 25.39±0.8425.39± 0.84 Mean 3.89±1.693.89± 1.69 25.67±1.6625.67± 1.66 BabyAI One Pass 0.9 9.23±2.259.23± 2.25 22.65±2.0722.65± 2.07 1.0 11.61±2.3611.61± 2.36 25.13±2.8225.13± 2.82 Mean 10.42±2.4410.42± 2.44 23.89±2.6023.89± 2.60 Reflect Forward 0.9 8.93±0.898.93± 0.89 21.52±1.1521.52± 1.15 1.0 8.04±1.558.04± 1.55 18.44±2.4918.44± 2.49 Mean 8.48±1.238.48± 1.23 19.98±2.4219.98± 2.42 Reflect Reverse 0.9 15.77±2.2515.77± 2.25 30.73±1.4730.73± 1.47 1.0 20.24±1.3620.24± 1.36 35.40±0.3635.40± 0.36 Mean 18.01±2.9618.01± 2.96 33.07±2.7333.07± 2.73 Jericho One Pass 0.9 1.67±2.891.67± 2.89 11.58±2.3911.58± 2.39 1.0 1.67±2.891.67± 2.89 11.62±2.6111.62± 2.61 Mean 1.67±2.581.67± 2.58 11.60±2.2411.60± 2.24 Reflect Forward 0.9 0.00±0.000.00± 0.00 9.63±1.319.63± 1.31 1.0 0.00±0.000.00± 0.00 11.51±1.0811.51± 1.08 Mean 0.00±0.000.00± 0.00 10.57±1.4910.57± 1.49 Reflect Reverse 0.9 0.00±0.000.00± 0.00 11.93±1.2511.93± 1.25 1.0 1.67±2.891.67± 2.89 21.37±2.6921.37± 2.69 Mean 0.83±2.040.83± 2.04 16.65±5.5016.65± 5.50 Appendix F Retry state examples Tables 5 and 6 present two representative retry-state patterns from ALFWorld: single-action repetition and two-action alternation. Similar retry patterns are also frequently observed across other multi-turn benchmarks. Table 5: Single-action example. A and O denote the action and its resulting observation. The repetition loop is marked in red including the initial action of the loop. Look at pencil under the desklamp. A: go to desk 1 O: On the desk 1, you see a bowl 2, a creditcard 2, a desklamp 1, a laptop 1, a mug 1, and a pen 2. A: examine desk 1 O: On the desk 1, you see a bowl 2, a creditcard 2, a desklamp 1, a laptop 1, a mug 1, and a pen 2. A: go to desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: examine desk 2 O: On the desk 2, you see a alarmclock 1, a bowl 3, and a pencil 1. A: take pencil 1 from desk 2 O: You pick up the pencil 1 from the desk 2. A: go to desk 1 O: On the desk 1, you see a bowl 2, a creditcard 2, a desklamp 1, a laptop 1, a mug 1, and a pen 2. A: use desklamp 1 O: You turn on the desklamp 1. Table 6: Alternating-action example. A and O denote the action and its resulting observation. The repetition loop is marked in red including the pair of the loop. Put a clean egg in microwave. A: go to fridge 1 O: The fridge 1 is closed. A: open fridge 1 O: You open the fridge 1. The fridge 1 is open. In it, you see a apple 2, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: take egg 1 from fridge 1 O: You pick up the egg 1 from the fridge 1. A: put egg 1 in/on fridge 1 O: You put the egg 1 in/on the fridge 1. A: close fridge 1 O: You close the fridge 1. A: open fridge 1 O: You open the fridge 1. The fridge 1 is open. In it, you see a apple 2, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: close fridge 1 O: You close the fridge 1. A: open fridge 1 O: You open the fridge 1. The fridge 1 is open. In it, you see a apple 2, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: close fridge 1 O: You close the fridge 1. A: open fridge 1 O: You open the fridge 1. The fridge 1 is open. In it, you see a apple 2, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: take egg 1 from fridge 1 O: You pick up the egg 1 from the fridge 1. A: put egg 1 in/on fridge 1 O: You put the egg 1 in/on the fridge 1. A: go to microwave 1 O: The microwave 1 is closed. A: open microwave 1 O: You open the microwave 1. The microwave 1 is open. In it, you see a apple 1, and a potato 1. A: take apple 1 from microwave 1 O: You pick up the apple 1 from the microwave 1. A: go to fridge 1 O: The fridge 1 is open. In it, you see a apple 2, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: put apple 1 in/on fridge 1 O: You put the apple 1 in/on the fridge 1. A: take egg 1 from fridge 1 O: You pick up the egg 1 from the fridge 1. A: put egg 1 in/on fridge 1 O: You put the egg 1 in/on the fridge 1. A: go to microwave 1 O: The microwave 1 is open. In it, you see a potato 1. A: take potato 1 from microwave 1 O: You pick up the potato 1 from the microwave 1. A: put potato 1 in/on microwave 1 O: You put the potato 1 in/on the microwave 1. A: examine microwave 1 O: The microwave 1 is open. In it, you see a potato 1. A: go to fridge 1 O: The fridge 1 is open. In it, you see a apple 2, a apple 1, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: take egg 1 from fridge 1 O: You pick up the egg 1 from the fridge 1. A: put egg 1 in/on fridge 1 O: You put the egg 1 in/on the fridge 1. A: go to microwave 1 O: The microwave 1 is open. In it, you see a potato 1. A: close microwave 1 O: You close the microwave 1. A: go to fridge 1 O: The fridge 1 is open. In it, you see a apple 2, a apple 1, a egg 1, a lettuce 1, a pan 2, a plate 1, and a tomato 1. A: take egg 1 from fridge 1 O: You pick up the egg 1 from the fridge 1.