Paper deep dive
On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents
Deyu Zou, Yongqiang Chen, Fan Feng, Mufei Li, Pan Li, Yu Gong, James Cheng
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 6:31:31 AM
Summary
The paper identifies 'Information Self-Locking' (SeL) as a failure mode in LLM agents trained with reinforcement learning for active reasoning. SeL occurs due to a bidirectional coupling between deficient Action Selection (AS) and Belief Tracking (BT), where weak BT masks the value of informative queries, and poor AS limits the information available for BT improvement. The authors propose AReW, a framework that uses directional critiques to reallocate learning signals, effectively mitigating SeL and improving reasoning performance.
Entities (5)
Relation Signals (3)
Action Selection → coupledwith → Belief Tracking
confidence 95% · SeL emerges from a bidirectional coupling between AS and BT.
AReW → mitigates → Information Self-Locking
confidence 95% · AReW consistently mitigates SeL across tasks, algorithms, and model families.
Information Self-Locking → limits → Reinforcement Learning
confidence 90% · Information Self-Locking of RL training for active reasoning.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) with outcome-based rewards has achieved significant success in training large language model (LLM) agents for complex reasoning tasks. However, in active reasoning where agents need to strategically ask questions to acquire task-relevant information, we find that LLM agents trained with RL often suffer from information self-locking: the agent ceases to ask informative questions and struggles to internalize already-obtained information. To understand the phenomenon, we decompose active reasoning into two core capabilities: Action Selection (AS), which determines the observation stream through queries, and Belief Tracking (BT), which updates the agent's belief based on collected evidence. We show that deficient AS and BT capabilities will limit the information exploration during RL training. Furthermore, insufficient exploration in turn hinders the improvement of AS and BT, creating a feedback loop that locks the agent in a low-information regime. To resolve the issue, we propose a simple yet effective approach that reallocates the learning signal by injecting easy- to-obtain directional critiques to help the agent escape self-locking. Extensive experiments with 7 datasets show that our approach significantly mitigates the information self-locking, bringing up to 60% improvements.
Tags
Links
- Source: https://arxiv.org/abs/2603.12109v1
- Canonical: https://arxiv.org/abs/2603.12109v1
Trouble viewing inline? Open PDF directly →
Full Text
123,471 characters extracted from source content.
Expand or collapse full text
On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents Deyu Zou Yongqiang Chen Fan Feng Mufei Li Pan Li Yu Gong James Cheng Abstract Reinforcement learning (RL) with outcome-based rewards has achieved significant success in training large language model (LLM) agents for complex reasoning tasks. However, in active reasoning where agents need to strategically ask questions to acquire task-relevant information, we find that LLM agents trained with RL often suffer from information self-locking: the agent ceases to ask informative questions and struggles to internalize already-obtained information. To understand the phenomenon, we decompose active reasoning into two core capabilities: Action Selection (AS), which determines the observation stream through queries, and Belief Tracking (BT), which updates the agent’s belief based on collected evidence. We show that deficient AS and BT capabilities will limit the information exploration during RL training. Furthermore, insufficient exploration in turn hinders the improvement of AS and BT, creating a feedback loop that locks the agent in a low-information regime. To resolve the issue, we propose a simple yet effective approach that reallocates the learning signal by injecting easy-to-obtain directional critiques to help the agent escape self-locking. Extensive experiments with 77 datasets show that our approach significantly mitigates the information self-locking, bringing up to 60% improvements. Machine Learning, ICML 1 Introduction Reinforcement learning (RL) with outcome-based rewards has demonstrated great success in improving the reasoning capabilities of Large language models (LLMs) (Wang et al., 2024; Srivastava and Aggarwal, 2025; Xu et al., 2025; Guo et al., 2025). Recently, it has received increasing attention in building agents based on LLMs, where the agent needs to interact with the environment and resolve tasks beyond single-turn interactions (Zhang et al., 2025a; Plaat et al., 2025). Usually, the tasks do not provide clear statements, e.g., underspecified user queries, and hence the agent needs to ask questions to acquire the missing information strategically, i.e., multi-turn active reasoning (Zhou et al., 2025; Wu et al., 2025; Laban et al., 2025; Li et al., 2025). Despite the success, we find that training an LLM agent with outcome-based RL suffers from information self-locking (SeL). Under SeL, agents often get stuck in low-information interaction patterns, where the agent ceases to ask informative questions and struggles to internalize already-obtained information. This aligns with existing failure modes of agents in the real-world use (Wang et al., 2025b). To understand more about the failure modes of agents, we propose to decompose agentic behaviors in active reasoning into action selection (AS), which determines what information is queried, and belief tracking (BT), which governs how acquired evidence is internalized and affects the final outcome. Across two multi-turn active reasoning benchmarks in Sec. 2, we show that the two capabilities can not get effectively improved even when the task rewards increase. Hence, it raises a challenging research question: Why does SeL happen and how to mitigate it? Figure 1: Overall illustration of information self-locking (SeL) and its mitigation. (b,a,o)(b,a,o) denote the agent’s internal belief, its chosen action, and the resulting feedback at each turn. Under vanilla outcome-based RL (top), the agent can become trapped in a self-locking regime: deficient belief tracking masks contributions of informative queries, leading to misaligned credit assignment. Our AReW (bottom) introduces advantage reweighting via directional critiques, correcting the learning signal and assisting to mitigate SeL in active reasoning. To answer the question, beyond the empirical evidence in Sec. 2, we develop a theoretical framework to characterize the dynamics of AS and BT during the outcome-based RL training (Sec. 3). Under the low AS and BT regime, we prove that weak belief tracking can mask the learning signal for informative actions, while the low information budget by deficient AS in turn intrinsically limits belief improvement. Together, these effects induce a negative confounding effect in which neither capability can improve the other and thus the agent struggles to escape from SeL (Thm. 3.4). Built upon our theoretical understanding, we propose AReW (Sec. 4), a lightweight framework that exploits easy-to-obtain diagnostic signals in active reasoning, e.g., whether a query is informative depends on whether the user reveals new evidence in response to the query. These diagnostic signals provide binary directional critiques for both AS and BT of agents. AReW further injecting these critiques into policy-gradient via reweighing the advantages of each step in standard policy optimization. This design preserves a non-degenerate and stable learning signal in the SeL regime to effectively improve AS and BT to break SeL. Empirically, we demonstrate that AReW consistently mitigates SeL across tasks, algorithms, and model families, and is robust to critique design of different noise levels. Beyond improving final performance, AReW fundamentally alters training dynamics: agents recover information-seeking interaction patterns and exhibit sustained growth in both action selection and belief tracking capabilities (Sec. 5). (a) (b) (c) (d) Figure 2: (a)/(b): the training dynamics of outcome reward, per-turn AS, and per-turn BT proxies in PE-GS=2 and MediQ datasets (Qwen-2.5-7B-Instruct). (c)/(d): correlation between the reward and AS proxies in PE-GS=2 and MediQ (Qwen-2.5-7B-Instruct). In strong BT patterns (by human-defined rules or frontier LLMs), the same AS sequence exhibits stronger correlation with the final reward. 2 Self-Locking in Active Reasoning 2.1 Preliminaries In active reasoning, the LLM agent interacts with an external environment with partial information, and needs to ask strategic questions a∈a to acquire the missing information for solving the task (Zhou et al., 2025; Badola et al., 2025). Active reasoning can be modeled as a Partially Observable Markov Decision Process (POMDP) (,,,T,O,R,γ)(S,Q,O,T,O,R,γ) (Kaelbling et al., 1998), where S is the space of unobservable latent states, Q the action space O the observation space, T(s′∣s,a)T(s s,a) the transition dynamics, O(o∣s,a)O(o s,a) the observation model, R the reward function, and γ the discount factor. Belief tracking is essential to the success of active reasoning. Essentially, the agent needs to model its belief btM∈Δ()b_t^M∈ (S) about the progress of the problem solving and what information remains missing throughout turn t∈0,…,Ht∈\0,…,H\. The behaviors of an agent with parameters ω in active reasoning can be decomposed into two coupled processes: Action Selection (AS): The agent selects an action (e.g., a question) according to a belief-conditioned policy at∼πωQ(⋅∣btM)a_t _ω^Q(· b_t^M) aiming to elicit informative observations ot∼O(⋅|s⋆,at)o_t O(·|s ,a_t) from the environment; Belief Tracking (BT): After receiving an observation ot∈o_t , the agent updates its belief via an internal update operator bt+1M=πωU(btM,at,ot)b_t+1^M= _ω^U(b_t^M,a_t,o_t), integrating information accumulated over previous interaction rounds. 2.2 Testbeds and decomposed proxies With the previous decomposition, we can track the fine-grained behaviors of agents trained with outcome-based RL. We consider two interactive benchmarks, where we introduce proxies that separately track AS and BT dynamics. The full details can be found in Appendix C.1. Preference Estimation (PE-G). Adapted from Badola et al. (2025), PE-G is an interactive preference inference task under constrained information acquisition. The agent is given a finite set of items =x1,…,xNX=\x_1,…,x_N\, where each item xix_i is represented by a known attribute vector i∈ℝDa_i ^D. The user has an unknown latent preference vector ⋆∈[0,1]Dw ∈[0,1]^D. Through interaction, the agent maintains and iteratively refines an estimate t∈[0,1]Dw_t∈[0,1]^D of the user preference. At each round, the agent actively selects a low-dimensional attribute subspace St⊆1,…,DS_t \1,…,D\ and a pair of items (xi,xj)∈×(x_i,x_j) ×X designed to elicit the user’s preference feedback restricted to StS_t. Based on the feedback, the agent updates its belief state. The objective is to accurately recover ⋆w under sparse, outcome-based supervision. We additionally consider PE-F where |St|=D|S_t|=D, i.e., all attributes can be considered in each round. Proxies in PE-G. As it’s hard to precisely quantify the informativeness, we introduce a binary proxy that is simple to implement and effective in tracking the AS behavior. Specifically, for a queried attribute subspace StS_t and item pair (xi,xj)(x_i,x_j), we define the AS indicator ASt=[∃k1,k2∈Sts.t.,ai(k1)>aj(k1)∧ai(k2)<aj(k2)]AS_t=I\! [∃\,k_1,k_2∈ S_t\;s.t.,\;a_i^(k_1)>a_j^(k_1) a_i^(k_2)<a_j^(k_2) ], which means neither item strictly dominates the other on StS_t, ensuring that the resulting feedback is informative. BT evaluates whether the agent can incorporate such informative feedback. We measure BT by the improvement in similarity between the estimate and the ground-truth preference, BTt=sim(t+1,⋆)−sim(t,⋆)BT_t=sim(w_t+1,w )-sim(w_t,w ), where sim(⋅,⋅)sim(·,·) denotes cosine similarity. Positive BTtBT_t indicates effective absorption of newly acquired information. MediQ. Adapted from Li et al. (2024), agents in MediQ require asking the patient questions to identify the best hypothesis for the patient’s symptoms. The agent is provided with a clinical vignette and an associated medical question whose answer lies in a finite hypothesis set of size D. The agent maintains a belief estimate t∈[0,1]Dw_t∈[0,1]^D for D candidate hypotheses. Through interaction, the agent actively queries the LLM-simulated user for diagnostic information, receives structured feedback, and updates each hypothesis score accordingly. The learning objective is to progressively concentrate belief mass onto the correct hypothesis. Proxies in MediQ. AS is quantified by the amount of novel diagnostic evidence elicited by the queries. Let ℰtE_t denote the set of atomic clinical facts revealed at turn t, we define ASt=|ℰt∖⋃τ<tℰτ|AS_t= |E_t _τ<tE_τ |, to capture the information gain of each query. BT measures whether newly observed evidence sharpens hypothesis discrimination. Let gtgt denote the ground-truth hypothesis index. We define BT via the change in belief margin BTt=Δ(t(gt)−maxj≠gtt(j)),BT_t= \! (w_t^(gt)- _j w_t^(j) ), aggregated across turns, where larger positive values indicate more effective belief refinement. 2.3 Failure modes in reinforcement learning training Despite the success of RL with outcome-based rewards, interestingly, we find that LLM agents exhibit several failure modes across both active-reasoning testbeds during training. Observation 1: Reward improvements do not translate into increased information acquisition. Fig. LABEL:fig:3.1.1 (PE-G) and Fig. LABEL:fig:3.1.2 (MediQ) report the training dynamics of episode reward, per-turn AS, and per-turn BT. Across both datasets, we observe a pronounced decoupling: while the reward can be improved over training, BT exhibits only limited gains, and AS fails to improve, often plateauing or even degrading. This observation raises interesting questions about the confounding behaviors of AS and BT, as AS can not improve even with an improved BT. To isolate the effect, we analyze the relationship between AS and reward under different BT capabilities. Specifically, we fix identical action sequences and compare outcomes when the observation stream is processed by (i) the agent’s internal BT versus (i) stronger belief-update mechanisms, e.g., human-defined update rules or frontier reasoning models. Since the environment dynamics and action sequences are identical across conditions, the difference can be attributed solely to the belief update mechanism. Observation 2: Weak belief tracking masks the contribution of informative actions. As shown in Fig. LABEL:fig:3.1.3 and LABEL:fig:3.1.4, the correlation between AS and reward is substantially higher under strong BT, but remains weak when using the agent’s own BT. This indicates that the contribution of AS to the reward is masked when belief updates are unreliable: even high-information actions yield little reward improvement if their information is not incorporated into internal belief. As a result, policy optimization cannot yield stable learning signals to reinforce informative AS choices. Observation 3: Conservative action selection limits belief refinement and induces interaction-insensitive shortcuts. Complementary to Obs. 2, we now examine the reverse direction of the coupling. When AS gets conservative and yields little informative evidence, BT is deprived of meaningful signals to learn from. Under outcome-only supervision, this even incentivizes shortcut behaviors that reduce reliance on interaction, reinforcing a low-information training regime. We observe that as training progresses, agents become less sensitive to informative observations and increasingly rely on early-stage context. In MediQ, we intervene by replacing all patient feedback with Unknown while keeping all other configurations unchanged. Notably, the induced performance drop becomes smaller after RL training (41.25→30.5041.25\!→\!30.50 w/o RL versus 61.00→55.5061.00\!→\!55.50 with RL; see Fig. LABEL:fig:4.2.1), suggesting that interaction-derived evidence has a weaker causal effect on the final decision. Crucially, this reduced sensitivity is accompanied by an increase in belief consistency (Fig. LABEL:fig:4.2.1; 78.778.7 w/o RL versus 92.892.8 with RL): the agent increasingly adheres to its initial judgment instead of revising beliefs in response to interaction, which reflects a more “stubborn” belief update pattern. Together, these form interaction under-utilization: once conservative AS restricts information exposure and weak BT struggles to internalize evidence, RL pressure favors non-interactive heuristics that stabilize outcomes while further suppressing exploration and evidence usage. Information Self-Locking of RL training for active reasoning. Taken together, these observations indicate SeL emerges from a bidirectional coupling between AS and BT. The reward-relevant value of AS is mediated by the agent’s ability to absorb information through BT, while BT is in turn constrained by the information budget induced by AS. This mutual dependence can trap training dynamics in a low-information regime, giving rise to a self-locking behavior. 3 Understanding Self-Locking To formally understand the SeL behaviors of AS and BT, we present a theoretical framework for SeL. Due to the space constraints, we defer the full details to Appendix B. AS and BT capabilities. We compare the trajectories produced by the oracle bayesian belief updates τB=(b0B,q0B,o0,…,bHB)τ^B=(b_0^B,q_0^B,o_0,…,b_H^B), and those by LLM agents τM=(b0M,q0M,o0,…,bHM)τ^M=(b_0^M,q_0^M,o_0,…,b_H^M), and focus on the potential differences ΔΨt:=Ψ(bt+1)−Ψ(bt) _t:= (b_t+1)- (b_t), where Ψ(b):=b(s∗)∈[0,1] (b):=b(s^*)∈[0,1] that measures the agent’s confidence about the true latent state s∗s^* across turns. Then, we are able to quantify the deviations of AS and BT from the oracle: Definition 3.1 (AS Informativeness). Given an action-observation trajectory induced by the query policy πωQ _ω^Q with oracle belief update, the theoretical AS informativeness of a query policy πωQ _ω^Q as the expected total improvement in oracle belief quality, ℐth(ω):=τB∼πωQ[∑t=0H−1ItB],I_th(ω)\;:=\;E_τ^B _ω^Q [ _t=0^H-1I^B_t ], where ItB:=Ψ(bt+1B)−Ψ(btB)I_t^B:= (b_t+1^B)- (b_t^B) is one-step oracle belief progress. We next characterize how much of the theoretical supplied information is actually absorbed by the agent model’s belief-tracking dynamics. Definition 3.2 (Belief drift). For an on-policy trajectory τMτ^M generated under πω _ω, define the absorbed belief progress ItM,+:=(ΔΨtM)+I_t^M,+:=( _t^M)_+, where ΔΨtM:=Ψ(bt+1M)−Ψ(btM) _t^M:= (b_t+1^M)- (b_t^M) is the one-step potential change at turn t and (⋅)+:=max(⋅,0)(·)_+:= (·,0). The training-level belief-tracking (BT) index is defined as CBT(ω):=τ∼πω[∑t=0H−1ItM,+],C_BT(ω):=E_τ _ω\! [ _t=0^H-1I_t^M,+ ], Self-Locking regime. We formalize the notion of self-locking via a two-dimensional low-AS and low-BT region: Definition 3.3 (Self-Locking Regime). With δ,ε>0δ, >0, we define the locking regime as the subset of parameter space with low-AS and low-BT capabilities: ℛδ,ε:=ω∈Ω:ℐth(ω)≤δ,CBT(ω)≤ε.R_δ, \;:=\; \ω∈ :\ I_th(ω)≤δ,\ \ C_BT(ω)≤ \. This region represents a two-dimensional low-information plus low-BT regime. Policy-gradient decomposition of the outcome reward. Given the agent ω with query policy πωQ _ω^Q and belief update kernel πωU _ω^U, when trained with outcome-based reward J(ω):=τ∼πω[R(τ)]J(ω):=E_τ _ω[R(τ)] for trajectory τ, the policy gradients can be obtained as ∇ωJ(ω)=τ∼πω[R(τ)∇ωlogpω(τ)] _ωJ(ω)=E_τ _ω\! [R(τ)\, _ω p_ω(τ) ], then, we have the expanded policy gradient as ∇ωlogpω(τ) _ω p_ω(τ) =∑t=0H∇ωlogπωU(btM∣ct) = _t=0^H _ω _ω^U(b_t^M c_t) (1) +∑t=0H−1∇ωlogπωQ(qt∣btM), + _t=0^H-1 _ω _ω^Q(q_t b_t^M), where ct=(bt−1M,qt−1,ot−1)c_t=(b^M_t-1,q_t-1,o_t-1) with c0=(b0M)c_0=(b^M_0) when t=0t=0. Furthermore, we can derive the update directions of AS and BT as gJ,Qg_J,Q and gJ,Ug_J,U, respectively. gJ,Q(ω)=τ∼πω[∑t=0H−1∇ωlogπωQ(qt∣btM)AtJ,Q(btM,qt)],g_J,Q(ω)\!=\!E_τ _ω\! [ _t=0^H-1 _ω _ω^Q(q_t b_t^M)A^J,Q_t(b_t^M,q_t) ], gJ,U(ω)=τ∼πω[∑t=0H∇ωlogπωU(btM∣ct)AtJ,U(ct,btM)],g_J,U(ω)\!=\!E_τ _ω\! [ _t=0^H _ω _ω^U(b_t^M c_t)A^J,U_t(c_t,b_t^M) ], where AJ,QA^J,Q and AJ,UA^J,U are the advantages of AS and BT given by the policy gradients, respectively. Naturally, we have the query-projected update to the parameter as Q(ω):=ω+ηgJ,Q(ω),T_Q(ω)\ :=\ ω+η\,g_J,Q(ω), and the BT-channel projected update U(ω):=ω+ηgJ,U(ω),T_U(ω)\ :=\ ω+η\,g_J,U(ω), respectively. The coupling effects between AS and BT can be further characterized as ΔQℐth(ω):=ℐth(Q(ω))−ℐth(ω) _QI_th(ω):=I_th(T_Q(ω))-I_th(ω), and ΔUCBT(ω):=CBT(U(ω))−CBT(ω) _UC_BT(ω):=C_BT(T_U(ω))-C_BT(ω), respectively. With the quantities defined for the progress of AS and BT from policy gradient updates, we draw the following result: Theorem 3.4 (Informal). Fix δ,ε>0δ, >0, assume (i) LR-Lipschitz reward-belief updates, (i) updates to ℐth(ω)I_th(ω) and CBT(ω)C_BT(ω) are bounded by some constant G<∞G<∞, (i) self-destructive drift to belief is invariant to the query choice, (iv) absorbed belief update is also bounded by CmaxCBT(ω)C_maxC_BT(ω) and κU⋅ItB _U· I^B_t with Cmax≥1,κU>0C_max≥ 1, _U>0, then we have (a), for any ω∈ℛδ,εω _δ, , the one-sided projected drifts satisfy the following componentwise inequality: (ΔQℐth(ω)ΔUCBT(ω))⪯η(0αβIβC)(ℐth(ω)CBT(ω))+o(η), pmatrix _QI_th(ω)\\[2.0pt] _UC_BT(ω) pmatrix\ \ η pmatrix0&α\\ _I& _C pmatrix pmatrixI_th(ω)\\[2.0pt] C_BT(ω) pmatrix\ +\ o(η), where ⪯ denotes elementwise inequality, βI=2(H+1)GLRκU _I=2(H+1)GL_R _U, βC=2(H+1)GLR _C=2(H+1)GL_R and α=HGLRκcCmaxα=HGL_R _cC_max; Furthermore, we have (b) if the agent’s parameter is initialized as ω0∈ℛδ,ε _0 _δ, , then it can not leave ℛδ,εR_δ, within the first K steps, with K:=⌊1ηmlog(ε+Cηℐth(ω0)+Cη)⌋+,K\ :=\ 1η m\, \! ( +CηI_th( _0)+Cη ) _+, where m:=maxα,βI+βCm:= \α,\ _I+ _C\. The formal version of Thm. 3.4 along with the proof are given in Appendix B.4. Intuitively, Thm. 3.4 shows that, when under SeL regime, the learning signals from outcome reward are weakened by the limited AS and BT capabilities, scaling linearly with the current levels of ℐthI_th and CBTC_BT. Consequently, when the model is initialized within SeL regime, then it requires significant policy update steps to escape the SeL regime. In practice, this indicates that once training enters the SeL regime, it is unlikely to recover without explicit interventions that enhance exploration. 4 Breaking Self-Locking with Directions The previous section reveals the root cause of the SeL is the confounding behaviors of limited AS and BT capabilities. When belief tracking is weak, different queries lead to nearly indistinguishable outcome advantages in the AS channel. The limited AS, in turn, further limits the improvement of BT itself. Breaking SeL therefore requires directional learning signals that bring effective improvements to AS and BT under the SeL regime. Luckily, in multi-turn active reasoning, we can find easy-to-obtain diagnostic signals at each step, e.g., whether a query is informative depends on whether the user reveals new evidence in response to the query. This observation motivates AReW that exploits the lightweight directional signals to build critiques and reallocate the training credits. 4.1 Stepwise directional critiques The previous decomposition of agentic behaviors allows us to exploit the respective directional critiques to improve the query policy πωQ _ω^Q (AS) and the belief-update kernel πωU _ω^U (BT). Without loss of generality, the reasoning process of the agent can be considered as alternating between (i) an Action Round, in which the agent outputs only the query, and (i) an Update Round, in which the agent receives the new observation and updates its belief about the problem. AS directional critique. Specifically, for the AS channel, we assign a directional critique ztQ∈−1,0,+1z_t^Q∈\-1,0,+1\ to each executed query, where +1+1 indicates that the query elicits informative feedback from the environment or user, −1-1 for an uninformative query, and 0 for an abstention. Intuitively, ztQz_t^Q encourages the agent to strategically propose queries that induce information helpful to reasoning. BT directional critique. For the BT channel, the critique reflects whether newly acquired information is effectively incorporated into the agent’s internal belief state. While we can not directly access the agent’s belief state, we can still acquire a scalar readout Ψ^t∈[0,1] _t∈[0,1] that tracks task-relevant confidence over turns from the agent, such as through prompting.111Importantly, Ψ^t _t is used purely as instrumentation: it is neither assumed to coincide with, nor to recover, the latent belief btMb_t^M. Using the readout Ψ^t _t, we define ztU:=Sign(Ψ^t+1−Ψ^t)∈−1,0,+1,z_t^U:=Sign\! ( _t+1- _t )∈\-1,0,+1\, where positive values indicate that the agent succeeds in updating its internal belief about the ground-truth answer. 4.2 Injecting directional critiques into policy-gradient Margin-aware auxiliary objective. We inject the directional critique via an auxiliary objective that (i) acts locally at the critiqued steps and (i) induces a gradient that can be combined with standard policy-gradients without modifying the task reward. To this end, for a trajectory τ with labels ztt=0H−1\z_t\_t=0^H-1, define the positively and negatively critiqued index sets τ:=t:zt=+1P_τ:=\t:\ z_t=+1\, and τ:=t:zt=−1N_τ:=\t:\ z_t=-1\222Note that we do not discriminate ztUz_t^U and ztQz_t^Q for simplicity and the notation applies to both.. Whenever the counts |τ|>0|P_τ|>0 and |τ|>0|N_τ|>0, we define a intra-trajectory likelihood-margin objective ℒ^(ω;τ):=1|τ|∑t∈τlogπω,t−1|τ|∑t∈τlogπω,t. L(ω;τ):= 1|P_τ| _t _τ _ω,t- 1|N_τ| _t _τ _ω,t. (2) Here logπω,t _ω,t denotes the log-probability assigned by the agent to the decision taken at step t under parameters ω. Note that the “decision” may correspond to an action-selection (query) or a belief-update decision, and the construction here thus applies to both AS and BT channels. If |τ|=0|P_τ|=0 or |τ|=0|N_τ|=0, we set ℒ^=0 L=0 and do not apply the auxiliary term to the trajectory. Eq. 2 directly encourages the agent to increase the log-probability mass on positively critiqued decisions relative to negatively critiqued ones, without introducing intermediate rewards or training a separate discriminator for ztz_t. Crucially, Eq. 2 is additive over time and therefore naturally compatible with multi-turn credit assignment. Implied per-step coefficients. Notably, the auxiliary Eq. 2 has a gradient of the same form as standard policy gradients: ∇ωℒ^=∑t=0H−1ut∇ωlogπω,t;ut:=1|τ|if zt=+1,−1|τ|if zt=−1,0if zt=0. _ω\, L= _t=0^H-1u_t _ω _ω,t;\,\,\,u_t:= cases 1|P_τ|&if z_t=+1,\\[1.0pt] -1|N_τ|&if z_t=-1,\\[1.0pt] 0&if z_t=0. cases (3) In Eq. 3, the sign of utu_t matches the critique direction, so the auxiliary gradient pushes probability mass in the intended direction. Furthermore, ∑t=0H−1ut=0 _t=0^H-1u_t=0 exhibits a centering property, which means the auxiliary term induces a pure likelihood margin rather than a uniform likelihood shift. Minimal injection via advantage reweighting. Let (ω)J(ω) denote the standard policy-gradient RL algorithm (e.g., PPO), We consider the augmented surrogate ℒ^aug(ω):=(ω)+λτ[ℒ^(ω;τ)], L_aug(ω)\;:=\;J(ω)\;+\;λ\,E_τ\! [ L(ω;τ) ], (4) where λ>0λ>0 controls the strength of critique injection, and the expectation is taken over on-policy trajectories. The resulting gradient update can thus be written as ∇ωℒ^aug(ω)∝τ[∑t=0H−1(At+λut)∇ωlogπω,t]. _ω\, L_aug(ω)\; \;E_τ\! [ _t=0^H-1 (A_t+λ\,u_t )\, _ω _ω,t ]. (5) Eq. 5 shows that injecting the critiques requires only a minimal modification to the actor update: it suffices to apply an additive shaping to the advantage A^t⟵At+λut, A_t\ \ A_t\;+\;λ\,u_t, (6) while keeping the outcome reward and the underlying RL optimization machinery unchanged. Since the coefficients utu_t are derived from the likelihood-margin objective, the resulting update can be directly interpreted as reallocating policy-gradient magnitude from negatively critiqued steps to positively critiqued ones within the same trajectory, aligned with the directional critiques. (a) (b) (c) (d) Figure 3: Training dynamics of rewards, evaluated under the PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW– as-only and PPO with AReW– as+bt. Table 1: Main results (average outcome reward on test sets) across three domains and seven active reasoning tasks. Preference Estimation Medical Diagnosis Troubleshooting PE-GS=2 PE-GS=3 PE-FD=8 PE-FD=6 MediQ FloDial-Easy FloDial-Hard Direct Inference o4-mini 17.11 21.15 8.42 12.47 74.67 35.00 26.33 Qwen-2.5-7B-Inst. 15.00 10.00 2.67 4.20 39.00 24.33 13.33 Llama-3.1-8B-Inst. 18.00 12.33 3.14 5.70 35.25 24.33 17.33 PPO-trained (Qwen-2.5-7B-Inst.) Vanilla 24.00 18.33 30.52 32.03 50.50 37.33 21.33 AReW– AS only 46.00 ↑ 22.0 32.00 ↑ 13.7 39.62 ↑ 9.1 42.10 ↑ 10.1 57.25 ↑ 6.8 43.67 ↑ 6.3 36.00 ↑ 14.7 AReW– AS + BT 49.33 ↑ 25.3 80.33 ↑ 62.0 47.89 ↑ 17.4 44.47 ↑ 12.4 61.25 ↑ 10.8 41.00 ↑ 3.7 42.33 ↑ 21.0 PPO-trained (Llama-3.1-8B-Inst.) Vanilla 27.33 11.00 55.21 6.00 63.50 24.33 31.00 AReW– AS only 49.00 ↑ 21.7 73.00 ↑ 62.0 55.61 ↑ 0.4 56.91 ↑ 50.9 71.75 ↑ 8.3 41.67 ↑ 17.3 42.00 ↑ 11.0 AReW– AS + BT 54.67 ↑ 27.3 77.67 ↑ 66.7 61.28 ↑ 6.1 54.65 ↑ 48.7 70.75 ↑ 7.3 44.33 ↑ 20.0 49.00 ↑ 18.0 4.3 Theoretical and Practical Discussion In contrast to the theoretical characterization of SeL in Thm. 3.4, the directional critique breaks the negative confounding of limited AS and BT. Let ^Q(ω) T_Q(ω) and ^U(ω) T_U(ω) denote the corresponding critique-shaped updates induced by the likelihood-margin objective in Eq. 4, respectively. The following proposition demonstrates the effectiveness of AReW is determined by the “weighted accuracy” of ztz_t critiques. See the complete formulation in Appendix B.5. Proposition 4.1. Under the setting of Sec. 4.2 and Appendix B.5, denote the stepwise weight as wt(ω):=|ut||AtB(btB,qt)|∥∇ωlogπωQ(qt∣btB)∥2w_t(ω):=|u_t|\, |A_t^B(b_t^B,q_t) |\,\| _ω _ω^Q(q_t b_t^B)\|^2 and let W(ω):=[∑t=0H−1wt(ω)].W(ω):=E\! [ _t=0^H-1w_t(ω) ]. Then the critique quality is measured by the weighted accuracy AccQ(ω):=[∑t=0H−1wt(ω) 1zt=yt][∑t=0H−1wt(ω)]∈[0,1].Acc_Q(ω)\;:=\; E\! [ _t=0^H-1w_t(ω)\,1\z_t=y_t\ ]E\! [ _t=0^H-1w_t(ω) ]∈[0,1]. Moreover, the first-order improvement in AS informativeness induced by AReW satisfies (and the BT-side analogue) ℐth(^Q(ω))−ℐth(Q(ω))=ηW(ω)(2AccQ(ω)−1)+o(η).I_th\! ( T_Q(ω) )-I_th\! (T_Q(ω) )=η\,W(ω)\, (2\,Acc_Q(ω)-1 )+o(η). In particular, AReW does not rely on perfectly accurate critiques. The proposition has shown that AReW is effective whenever AccQ(ω)>12Acc_Q(ω)> 12. Moreover, we empirically exhibit that AReW is robust to the noise in critiques (see details in Robustness Analyses in Sec. 5.2). Practical discussion. In practice, our framework naturally maps to LLM agents with alternating action-selection (AS) and belief-tracking (BT) rounds during interaction. In each AS round, the agent issues a query intended to reduce uncertainty over candidate states s∈s . In the subsequent BT round, the agent is instructed to explicitly update its per-candidate confidence using the latest feedback, where the confidence assigned to the ground-truth candidate s⋆s induces Ψ^t _t. In practice, the required critiques are easy to obtain. For AS, query-level signals ztQz_t^Q can be directly inferred from user feedback. For BT, stepwise critique labels ztUz_t^U can be constructed by comparing changes in candidate confidence (particularly that of s⋆s ) upon receiving feedback. We defer concrete prompt templates and implementation details to Appendix C.3. 5 Experiments 5.1 Experimental Setup Datasets. We evaluate AReW across three interactive reasoning domains that share a common outcome-only supervision structure but differ substantially in semantics and feedback mechanisms: preference estimation, medical diagnosis, and troubleshooting. The first two domains correspond to PE-G, PE-F, and MediQ introduced in Sec. 2. To further assess generality, we additionally consider a troubleshooting domain based on the FloDial dataset (Raghu et al., 2021; Hu et al., 2024), which consists of multi-turn diagnostic dialogues for resolving user-reported issues. Across all three domains, reward is provided only at the end of interaction and depends solely on the quality of the final decision. Baselines. To evaluate the effectiveness of AReW, we compare it against the following baselines: 1) Direct Inference without Training, where we evaluate representative proprietary reasoning LLMs o4-mini; 2) PPO (Schulman et al., 2017), 3) GRPO (Shao et al., 2024), and 4) GSPO (Zheng et al., 2025). See more details in Appendix C.2. For each RL mechanism, we consider two variants of AReW: as-only and as+bt, which work on the AS side critique ztQz_t^Q and both AS and BT sides ztQ,ztUz_t^Q,z_t^U, respectively. Evaluation metrics. We report average reward as well as AS and BT capability proxies on the test datasets. PE-F is evaluated under a continuous reward defined by normalized similarity improvement, whereas all other datasets use binary rewards. Details of rewards and the proxies used to approximate AS and BT are provided in Appendix C.3. Implementation details. The main experiments of RL training are conducted on Qwen2.5-7B-Instruct (Yang et al., 2024) and LLaMA-3.1-8B-Instruct (Grattafiori et al., 2024). For the PE-G and PE-F tasks, the interactive feedback is rule-based; for the MediQ and FloDial datasets, we leverage Qwen2.5-14B-Instruct to simulate the “user” and provides the interactive feedback. See more details in Appendix C.3. (a) (b) (c) (d) Figure 4: Evaluations of AS and BT capabilities under PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW. (a) (b) (c) (d) Figure 5: (a): outcome-RL reduces sensitivity to interactive feedback while increasing belief consistency. (b)/(c): Evaluations of AS and BT capabilities under GRPO and GSPO (Qwen-2.5-7B-Instruct). (d): Training dynamics of rewards under different strength of AReW. 5.2 Experimental Results and Analyses We present experimental results to evaluate the effectiveness of AReW and to analyze its impact on learning dynamics in interactive reasoning tasks. We begin with overall performance comparisons, followed by analyses that examine how AReW affects reward optimization, AS and BT capabilities, and robustness analysis. Overall Performance. We first report the main results across the evaluated domains. Table 1 summarizes final-task performance. We further visualize dynamics of reward optimization and information-related behaviors. Specifically, Fig. 3 shows episode-level reward trajectories, while Fig. 4 reports proxy measurements of AS and BT capabilities introduced in Section 2. As shown in Table 1, AReW consistently outperform the vanilla PPO baseline, achieving notable improvements in 27 out of 28 evaluated settings. Among these, as+bt further largely outperform the as-only variants in 11 out of 14 cases. These results indicate the effectiveness of AReW on mitigating self-locking. Reward dynamics. Fig. 3 reveals three representative training behaviors. In some tasks (Fig. LABEL:fig:3.1.1), vanilla PPO fails to improve rewards throughout training, remaining trapped in SeL while AReW effectively breaks SeL and gained continual improvement. In others (Fig. LABEL:fig:3.1.2 and LABEL:fig:3.1.4), rewards can have some limited increase, but AReW achieves faster convergence and higher asymptotic performance. Notably, we also observe cases where reward curves appear comparable across methods, yet AReW yields superior AS and BT proxy scores and better final-task performance (See Table 1), corresponding to Obs. 3 discussed in Sec. 2.3. AS and BT behaviors. Fig. 4 further disentangles these effects. The as-only variant outperform the baseline in AS proxies in all cases, and the as+bt variant achieve additional gains in BT over the as-only. Interestingly, improving AS alone already leads to measurable improvements in BT, reflecting the intrinsic coupling between information acquisition and belief updates, illustrating that breaking SeL bring benefits for both AS and BT channels. Effectiveness across different RL algorithms. Beyond PPO, we additionally consider group-based RL variants, including GRPO and GSPO. While these methods sample multiple trajectories per step, we empirically observe that self-locking behaviors can still arise in interactive settings, suggesting that increased rollout multiplicity alone may not resolve the underlying coupling between AS and BT. As shown in Fig. LABEL:fig:4.2.2 and LABEL:fig:4.2.3, our AReW consistently improves final task performance, while simultaneously strengthening AS and BT proxies, analogous to those in PPO. These results indicate that the effectiveness of AReW proves effective across different RL mechanisms. Robustness Analysis. We evaluate the robustness of AReW by randomly flipping stepwise directional critiques with probability α. As shown in Table 2, increasing the perturbation level leads to a gradual reduction in final performance. Nevertheless, AReW consistently remains competitive with, and often outperforms, the vanilla baseline across a wide range of α. This trend is consistent with Proposition 4.1, which suggests that performance gains can be sustained as long as the weighted accuracy of critiques is not severely degraded. Even under strong perturbations (e.g., α=0.5α=0.5), AReW does not collapse. Overall, these results indicate that AReW is robust to critique noise and can tolerate imperfect directional signals in practice. Effect of the reweighting strength λuλ u. We examine the impact of the reweighting strength λuλ u on training dynamics. As illustrated in Fig. LABEL:fig:4.2.4, insufficient reweighting fails to provide enough signal to escape the self-locking regime, resulting in slow and limited improvement. Conversely, overly aggressive reweighting accelerates early optimization but often leads to unstable training and eventual performance collapse. We conjecture that large λuλ u amplifies high-variance advantage estimates and over-emphasizes a small subset of steps, making the policy update brittle and sensitive to noise. These observations suggest a non-trivial trade-off between exploration, stability, and convergence speed. We leave more adaptive or structure-aware reweighting strategies as an important direction for future work. Table 2: Final Performance under different strength of directional critique perturbation (controlled by α). Vanilla Perturbation ratio α 0 0.1 0.2 0.3 0.4 0.5 PE-GS=3 18.3 80.3 40.0 65.0 31.3 22.3 30.3 FloDial-Hard 21.3 36.0 30.3 29.0 27.6 30.6 23.3 6 Conclusion We study information self-locking (SeL) in long-horizon active reasoning and show that it arises from a structural failure of credit assignment with bidirectional coupling between action selection (AS) and belief tracking (BT). We provide both theoretical and empirical evidence that standard outcome-based RL can be trapped in SeL. We propose AReW, a critique-driven reweighting approach that selectively reallocates optimization signal along trajectories. Experiments demonstrate consistent gains, robustness to noisy critiques, effectiveness on multiple RL mechanisms, and improved training dynamics, AS and BT capabilities across multiple benchmarks. We believe this perspective opens up new directions for designing robust learning mechanism for interactive reasoning agents. References K. Badola, J. Simon, A. Hosseini, S. M. M. Carthy, T. Munkhdalai, A. Goyal, T. Kočiskỳ, S. Upadhyay, B. Fatemi, and M. Kazemi (2025) Multi-turn puzzles: evaluating interactive reasoning and strategic dialogue in llms. arXiv preprint arXiv:2508.10142. Cited by: Appendix A, §C.1, §2.1, §2.2. Y. Deng, L. Liao, L. Chen, H. Wang, W. Lei, and T. Chua (2023) Prompting and evaluating large language models for proactive dialogues: clarification, target-guided, and non-collaboration. arXiv preprint arXiv:2305.13626. Cited by: Appendix A. S. Dou, Y. Liu, H. Jia, L. Xiong, E. Zhou, W. Shen, J. Shan, C. Huang, X. Wang, X. Fan, et al. (2024) Stepcoder: improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391. Cited by: Appendix A. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. Z. Hu, C. Liu, X. Feng, Y. Zhao, S. Ng, A. T. Luu, J. He, P. W. W. Koh, and B. Hooi (2024) Uncertainty of thoughts: uncertainty-aware planning enhances information seeking in llms. Advances in Neural Information Processing Systems 37, p. 24181–24215. Cited by: Appendix A, §C.1, §5.1. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra (1998) Planning and acting in partially observable stochastic domains. Artificial intelligence 101 (1-2), p. 99–134. Cited by: §2.1. W. Kwan, X. Zeng, Y. Jiang, Y. Wang, L. Li, L. Shang, X. Jiang, Q. Liu, and K. Wong (2024) Mt-eval: a multi-turn capabilities evaluation benchmark for large language models. arXiv preprint arXiv:2401.16745. Cited by: Appendix A. P. Laban, H. Hayashi, Y. Zhou, and J. Neville (2025) Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120. Cited by: §1. S. Li, V. Balachandran, S. Feng, J. Ilgen, E. Pierson, P. W. W. Koh, and Y. Tsvetkov (2024) Mediq: question-asking llms and a benchmark for reliable interactive clinical reasoning. Advances in Neural Information Processing Systems 37, p. 28858–28888. Cited by: §C.1, §2.2. Y. Li, X. Shen, X. Yao, X. Ding, Y. Miao, R. Krishnan, and R. Padman (2025) Beyond single-turn: a survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717. Cited by: §1. Z. Liang, D. Yu, W. Yu, W. Yao, Z. Zhang, X. Zhang, and D. Yu (2024) Mathchat: benchmarking mathematical reasoning and instruction following in multi-turn interactions. arXiv preprint arXiv:2405.19444. Cited by: Appendix A. A. Plaat, M. van Duijn, N. van Stein, M. Preuss, P. van der Putten, and K. J. Batenburg (2025) Agentic large language models, a survey. arXiv preprint arXiv:2503.23037. Cited by: §1. D. Raghu, S. Agarwal, S. Joshi, et al. (2021) End-to-end learning of flowchart grounded task-oriented dialogs. arXiv preprint arXiv:2109.07263. Cited by: §C.1, §5.1. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2015) High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. Cited by: §C.2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §C.2, §5.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §C.2, §5.1. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §C.3.2. S. S. Srivastava and V. Aggarwal (2025) A technical survey of reinforcement learning techniques for large language models. arXiv preprint arXiv:2507.04136. Cited by: §1. Y. Wan, J. Wu, M. Abdulhai, L. Shani, and N. Jaques (2025) Enhancing personalized multi-turn dialogue with curiosity reward. arXiv preprint arXiv:2504.03206. Cited by: Appendix A. H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li (2025a) SPA-rl: reinforcing llm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732. Cited by: Appendix A. S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, F. Wu, G. Wang, and E. Hovy (2024) Reinforcement learning enhanced llms: a survey. arXiv preprint arXiv:2412.10400. Cited by: §1. Y. Wang, Z. Wei, X. Zhu, and Y. Meng (2025b) Beyond outcome reward: decoupling search and answering improves llm agents. arXiv preprint arXiv:2510.04695. Cited by: Appendix A, §1. S. Wu, M. Galley, B. Peng, H. Cheng, G. Li, Y. Dou, W. Cai, J. Zou, J. Leskovec, and J. Gao (2025) Collabllm: from passive responders to active collaborators. arXiv preprint arXiv:2502.00640. Cited by: §1. F. Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng, et al. (2025) Towards large reasoning models: a survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686. Cited by: §1. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. (2024) Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §5.1. H. Yu, Z. Qi, Y. Zhao, K. Nottingham, K. Xuan, B. P. Majumder, H. Zhu, P. P. Liang, and J. You (2025) Sotopia-rl: reward design for social intelligence. arXiv preprint arXiv:2508.03905. Cited by: Appendix A. Y. Yu, Z. Wang, W. Ma, Z. Guo, J. Zhan, S. Wang, C. Wu, Z. Guo, and M. Zhang (2024) Steptool: a step-grained reinforcement learning framework for tool learning in llms. Cited by: Appendix A. G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, et al. (2025a) The landscape of agentic reinforcement learning for llms: a survey. arXiv preprint arXiv:2509.02547. Cited by: §1. Z. Zhang, Z. Chen, M. Li, Z. Tu, and X. Li (2025b) RLVMR: reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents. arXiv preprint arXiv:2507.22844. Cited by: Appendix A. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §C.2, §5.1. Z. Zhou, X. Feng, Z. Zhu, J. Yao, S. Koyejo, and B. Han (2025) From passive to active reasoning: can large language models ask the right questions under incomplete information?. arXiv preprint arXiv:2506.08295. Cited by: Appendix A, §1, §2.1. D. Zou, Y. Chen, J. Wang, G. YANG, M. Li, Q. Da, J. Cheng, P. Li, and Y. Gong (2026) Reducing belief deviation in reinforcement learning for active reasoning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §C.1. Appendix A Related Work Active Reasoning requires LLMs to interact with external sources and actively acquire missing information to solve complex tasks. Prior work has improved LLMs’ ability to handle ambiguity and incompleteness through making clarification and information-seeking actions. For example, Proactive CoT (Deng et al., 2023) prompts LLMs to identify ambiguous problems and generate clarification questions, while UoT (Hu et al., 2024) quantifies the contribution of each question in reducing uncertainty. However, challenges remain when transitioning from LLMs’ single-turn success to multi-turn active reasoning (Kwan et al., 2024; Liang et al., 2024; Badola et al., 2025), even with several advanced strategies such as tree-based searching or post-training approaches, as highlighted in existing works (Zhou et al., 2025). There have been existing works which target RL on active reasoning (Zou et al., 2026). In our work, we identify a unique mechanism named information self-locking, which is sourced from a structural failure of credit assignment with bidirectional coupling between action selection (AS) and belief tracking (BT). This is consistent with empirical observations from real-world agentic use. For example, Wang et al. (2025b) uncovers the phenomenon specific to search agents: systematic deficiencies in search behaviors arise under outcome-only training and ultimately degrade final answer quality. Credit Assignment and Multi-turn RL. Credit assignment is crucial to long-horizon or multi-turn RL. Existing methods have extensively explored rule-based approaches (Yu et al., 2024; Dou et al., 2024; Zhang et al., 2025b) to shape intermediate rewards. Several recent works also proposed to measure the progress of stepwise actions toward overall task completion as intermediate rewards. Specifically, CURIO (Wan et al., 2025) constructs a potential function over an ideal belief state to assign intermediate rewards, assuming that the latent state space is finite and enumerable. Sotopia-RL (Yu et al., 2025) relies on reward labeling with proprietary LLMs. SPA-RL (Wang et al., 2025a) trains reward models for intermediate rewards by enforcing a summation constraint with respect to the final outcome reward. In our work, instead of working on complicated reward shaping or resorting to external models, we leverage easy-to-use binary directional critiques to make a minimal injection to policy gradient which is mathematically derived from a margin-aware auxiliary objective, in order to provide non-degenerate and stable learning signals to help agents escape from self-locking. Appendix B More Details on the Theory B.1 Notations & Problem Setup We consider active reasoning, where an LLM agent interacts with an external environment to acquire missing information and infer the underlying solution through multi-turn interaction. This can be modeled as a POMDP (,,,T,O,R,γ)(S,Q,O,T,O,R,γ), where S is the space of unobservable latent states, Q the action space, O the observation space, T(s′∣s,a)T(s s,a) the transition dynamics, O(o∣s,a)O(o s,a) the observation model, R the reward function, and γ the discount factor. In our setting, the latent state is fixed within each episode of horizon H, denoted by s⋆∈s . We assume deterministic feedback: for all (s,q)∈×(s,q) ×Q, the observation kernel O(⋅∣s,q)O(· s,q) is a point mass at o=(s,q)o= O(s,q); in particular, ot=(s⋆,qt)o_t= O(s ,q_t). Model Belief. We work with a belief-state abstraction where the agent is associated with an explicit model belief btM∈Δ()b_t^M∈ (S) which represents the agent’s internal understanding of the latent state and what information remains missing at each turn t∈0,…,Ht∈\0,…,H\. Note that this abstraction does not assume the agent internally stores btMb_t^M; rather, btMb_t^M is a well-defined object induced by the model parameters ω and the interaction history, which we use purely for theoretical analysis. We quantify truth-aligned confidence by the potential Ψ(b):=b(s⋆)∈[0,1] (b):=b(s )∈[0,1], which measures the model’s confidence mass assigned to the true latent state and serves as our working notion of belief quality. Model Trajectory. Agentic behaviour in active reasoning can be decomposed into two coupled processes: Action Selection (AS) and Belief Tracking (BT). A single agent with parameters ω induces: (i) a BT-channel belief update kernel πωU _ω^U that produces the next belief given the latest interaction, and (i) an AS-channel query policy πωQ _ω^Q that selects queries conditioned on the current belief. Concretely, define the BT-conditioning context c0c_0 as a fixed initial context (e.g., task instructions), and ct+1:=(btM,qt,ot)c_t+1:=(b_t^M,q_t,o_t) for t∈0,…,H−1t∈\0,…,H-1\. For each turn t, the resulted belief-MDP evolves via query generation qt∼πωQ(⋅∣btM)q_t _ω^Q(· b_t^M), observation ot=(s⋆,qt)o_t= O(s ,q_t), and belief update bt+1M∼πωU(⋅∣ct+1)b_t+1^M _ω^U(· c_t+1). We write the full trajectory τ∼πωτ _ω as τ=(b0M,q0,o0,b1M,q1,o1,…,bH−1M,qH−1,oH−1,bHM)τ=(b_0^M,q_0,o_0,\;b_1^M,q_1,o_1,\;…,\;b_H-1^M,q_H-1,o_H-1,\;b_H^M), where the trajectory likelihood factorizes as pω(τ)=πωU(b0M∣c0)∏t=0H−1πωQ(qt∣btM)∏t=0H−1πωU(bt+1M∣ct+1),wherect+1=(btM,qt,ot). p_ω(τ)= _ω^U(b_0^M c_0)\; _t=0^H-1 _ω^Q(q_t b_t^M)\; _t=0^H-1 _ω^U(b_t+1^M c_t+1),\,\,\,where\,\,\,c_t+1=(b_t^M,q_t,o_t). (7) Outcome reward. The environment returns an outcome reward R(τ)∈[0,1]R(τ)∈[0,1] (e.g., correctness of the inferred solution). We assume the expected reward is a (non-decreasing) function of the terminal potential: Assumption B.1. There exists a non-decreasing LRL_R-Lipschitz function f:[0,1]→[0,1]f:[0,1]→[0,1] s.t., [R(τ)|bHM]=f(Ψ(bHM))E\! [R(τ)\ |\ b_H^M ]=f\! ( (b_H^M) ). B.2 Two Capability Indices Oracle Bayesian belief. To decouple AS informativeness from the model’s BT mechanism, we introduce an oracle (Bayesian), theoretically optimal belief update process. Given deterministic feedback o=(s⋆,q)o= O(s ,q), define the Bayesian update operator (⋅,q,o) BayesUpd(·,q,o) by ((b,q,o))(s):=b(s) 1(s,q)=o∑s′∈b(s′) 1(s′,q)=o. ( BayesUpd(b,q,o) )(s)\;:=\; b(s)\,1\ O(s,q)=o\ _s b(s )\,1\ O(s ,q)=o\. Oracle-belief MDP. For analysis purposes only, we consider an oracle-belief MDP that evolves under Bayesian belief updates while sharing the same query policy as the agent. Fix a prior b0B∈Δ()b_0^B∈ (S) with b0B(s⋆)>0b_0^B(s )>0. For each turn t∈0,…,H−1t∈\0,…,H-1\, an oracle-belief trajectory evolves as query generation conditioned on oracle belief qt∼πωQ(⋅∣btB)q_t _ω^Q(· b_t^B), observation ot=(s⋆,qt)o_t= O(s ,q_t), and Bayesian belief update bt+1B=(btB,qt,ot)b_t+1^B= BayesUpd(b_t^B,q_t,o_t). This yields τB:=(b0B,q0,o0,b1B,q1,o1,…,bHB)τ^B:=(b_0^B,q_0,o_0,\;b_1^B,q_1,o_1,\;…,\;b_H^B). Notably, the oracle-belief MDP is introduced solely for analysis, aiming to isolate AS information supply, and we do not require the actual agent to access oracle belief btBb_t^B. We now quantify the informativeness of a query policy by the belief improvement it induces under the oracle-belief dynamics. Definition B.2. Consider a oracle-belief MDP trajectory τB=(b0B,q0,o0,…,bHB)τ^B=(b_0^B,q_0,o_0,…,b_H^B) induced by the query policy πωQ _ω^Q and Bayesian belief dynamics (⋅) BayesUpd(·). Define one-step oracle belief progress as ItB:=Ψ(bt+1B)−Ψ(btB)I_t^B:= (b_t+1^B)- (b_t^B). Define the theoretical AS informativeness of a query policy πωQ _ω^Q as the expected total improvement in oracle belief quality under the oracle-belief MDP, ℐth(ω):=τB∼πωB[∑t=0H−1ItB].I_th(ω)\;:=\;E_τ^B _ω^B [ _t=0^H-1I^B_t ]. We next characterize how much of the theoretical supplied information is actually absorbed by the agent model’s belief-tracking dynamics. Definition B.3. For an on-policy trajectory τ=(b0M,q0,o0,…,bHM)τ=(b_0^M,q_0,o_0,…,b_H^M) generated under πω _ω, define the one-step potential change at turn t as ΔΨtM:=Ψ(bt+1M)−Ψ(btM) _t^M:= (b_t+1^M)- (b_t^M). We decompose ΔΨtM _t^M into its one-sided components: the absorbed belief progress ItM,+:=(ΔΨtM)+I_t^M,+:=( _t^M)_+, and the self-destructive belief drift ItM,−:=(−ΔΨtM)+I_t^M,-:=(- _t^M)_+. Only the redeemable component contributes positively to belief improvement. The training-level belief-tracking (BT) index is defined as CBT(ω):=τ∼πω[∑t=0H−1ItM,+].C_BT(ω):=E_τ _ω\! [ _t=0^H-1I_t^M,+ ]. To relate AS informativeness to BT progress under learning dynamics, we impose the following mild regularity assumptions. Each assumption is accompanied by a brief intuition clarifying its role: Asmp. B.4 states that harmful belief drift is independent of the specific query chosen, and is instead driven by deficiencies in the agent’s BT mechanism. Asmp. B.5 upper-bounds the amount of belief progress that can be absorbed in a single step, and further ties such progress to the evidence revealed under the oracle belief dynamics. Asmp. B.6 enforces a Lipschitz-type stability of future belief quality with respect to the current potential, reflecting a conservative propagation of belief improvements under weak belief-tracking. Assumption B.4. For any turn t and any query q∈q , the conditional expectation of the self-destructive drift given the current belief is invariant to the query choice, i.e., ω[ItM,−∣btM,qt=q]=ω[ItM,−∣btM]E_ω\! [I_t^M,- b_t^M,q_t=q ]=E_ω\! [I_t^M,- b_t^M ] almost surely for all q∈q . Assumption B.5. For any turn t, there exists a constant Cmax≥1C_ ≥ 1 such that along on-policy rollouts, the conditional expectation satisfies ω[ItM,+∣btM,qt]≤CmaxCBT(ω)E_ω\! [I_t^M,+ b_t^M,q_t ]≤ C_ \,C_BT(ω) almost surely over the realized query qtq_t; and there exists a constant κU>0 _U>0 such that along on-policy rollouts, ω[ItM,+|btM,qt]≤κU⋅ItBE_ω\! [I_t^M,+\ |\ b_t^M,q_t ]\ ≤\ _U· I_t^B a.s. over the realized query qtq_t. Assumption B.6. For any 0<t<t′≤H0<t<t ≤ H, there exist a constant κc _c such that for and any realizable model beliefs b,b~∈Δ()b, b∈ (S) at turn t, |ω[Ψ(bt′M)|btM=b]−ω[Ψ(bt′M)|btM=b~]|≤κc|Ψ(b)−Ψ(b~)|. |\ E_ω\! [ (b_t ^M)\ |\ b_t^M=b ]-E_ω\! [ (b_t ^M)\ |\ b_t^M= b ]\ |\ ≤\ _c\, | (b)- ( b) |. Locking regime. We formalize the notion of self-locking via a two-dimensional low-information plus low-BT region: Definition B.7 (Locking regime). For given thresholds δ,ε>0δ, >0, we define the locking regime as the subset of parameter space characterized by simultaneously low theoretical action-selection informativeness and low belief-tracking capability: ℛδ,ε:=ω∈Ω:ℐth(ω)≤δ,CBT(ω)≤ε.R_δ, \;:=\; \ω∈ :\ I_th(ω)≤δ,\ \ C_BT(ω)≤ \. This region represents a two-dimensional low-information plus low-BT regime in which neither the query policy nor the belief-tracking dynamics can induce substantial positive progress. To analyze local training dynamics within the locking regime, we require a mild first-order regularity of the two indices. Assumption B.8 (Local first-order expandability of indices). Fix δ,ε>0δ, >0 and consider ω∈ℛδ,εω _δ, . There exist constants GI,GC<∞G_I,G_C<∞ such that ‖∇ωℐth(ω)‖≤GI\| _ωI_th(ω)\|≤ G_I and ‖∇ωCBT(ω)‖≤GC\| _ωC_BT(ω)\|≤ G_C. Moreover, for any ω′∈ℛδ,εω _δ, satisfying ‖ω′−ω‖=O(η)\|ω -ω\|=O(η), the following first-order expansions hold uniformly over ℛδ,εR_δ, : ℐth(ω′)=ℐth(ω)+⟨∇ωℐth(ω),ω′−ω⟩+o(η)I_th(ω )=I_th(ω)+ _ωI_th(ω),\ ω -ω +o(η), and CBT(ω′)=CBT(ω)+⟨∇ωCBT(ω),ω′−ω⟩+o(η)C_BT(ω )=C_BT(ω)+ _ωC_BT(ω),\ ω -ω +o(η), where o(η)/η→0o(η)/η→ 0 as η→0η→ 0 uniformly for ω∈ℛδ,εω _δ, . B.3 Projected Drifts and the 2D Locking Regime Policy-gradient decomposition of the outcome objective. We begin by expressing the outcome-based policy gradient in a form that admits a decomposition across the AS and BT channels. With the outcome objective J(ω):=τ∼πω[R(τ)]J(ω):=E_τ _ω[R(τ)], ∇ωJ(ω)=τ∼πω[R(τ)∇ωlogpω(τ)], _ωJ(ω)=E_τ _ω\! [R(τ)\, _ω p_ω(τ) ], and hence (via Eq. 7) ∇ωlogpω(τ)=∑t=0H∇ωlogπωU(btM∣ct)+∑t=0H−1∇ωlogπωQ(qt∣btM). _ω p_ω(τ)= _t=0^H _ω _ω^U(b_t^M c_t)+ _t=0^H-1 _ω _ω^Q(q_t b_t^M). We next impose a mild regularity assumption to control the policy-gradient magnitude. Assumption B.9. There exist finite constants GQ<∞G_Q<∞ and GU<∞G_U<∞ such that for all ω∈ℛδ,εω _δ, and t∈0,…,H−1t∈\0,…,H-1\, we have ∥∇ωlogπωQ(qt∣btM)∥≤GQ\| _ω _ω^Q(q_t b_t^M)\|≤ G_Q and ∥∇ωlogπωU(btM∣ct)∥≤GU\| _ω _ω^U(b_t^M c_t)\|≤ G_U almost surely under τ∼πωτ _ω. Channel-isolated Stage-wise advantages. As mentioned before, we have decomposed the agentic behavior into two coupled but conceptually distinct channels: AS and BT. To characterize how outcome-based training signals propagate through each channel, we introduce channel-isolated stage-wise value functions, in which the other channel is treated as part of the induced environment dynamics. For the BT channel, the belief state btMb_t^M itself is regarded as the decision variable. For t∈0,…,Ht∈\0,…,H\, define QtJ,U(c,b):=[R(τ)∣ct=c,btM=b],VtJ,U(c):=b∼πωU(⋅∣c)[QtJ,U(c,b)],Q^J,U_t(c,b):=E\! [R(τ) c_t=c,\ b_t^M=b ], V^J,U_t(c):=E_b _ω^U(· c)\! [Q^J,U_t(c,b) ], and the corresponding advantage AtJ,U(c,b):=QtJ,U(c,b)−VtJ,U(c).A^J,U_t(c,b):=Q^J,U_t(c,b)-V^J,U_t(c). Similarly, for the AS channel, the query qtq_t is treated as the decision variable. For t∈0,…,H−1t∈\0,…,H-1\, define QtJ,Q(b,q):=[R(τ)∣btM=b,qt=q],VtJ,Q(b):=q∼πωQ(⋅∣b)[QtJ,Q(b,q)],Q^J,Q_t(b,q):=E\! [R(τ) b_t^M=b,\ q_t=q ], V^J,Q_t(b):=E_q _ω^Q(· b)\! [Q^J,Q_t(b,q) ], with advantage AtJ,Q(b,q):=QtJ,Q(b,q)−VtJ,Q(b).A^J,Q_t(b,q):=Q^J,Q_t(b,q)-V^J,Q_t(b). Channel-isolated outcome update directions. Using the above definitions, the policy-gradient update directions induced by the outcome objective J can be written as gJ,Q(ω):=τ∼πω[∑t=0H−1∇ωlogπωQ(qt∣btM)AtJ,Q(btM,qt)]gJ,U(ω):=τ∼πω[∑t=0H∇ωlogπωU(btM∣ct)AtJ,U(ct,btM)].g_J,Q(ω):=E_τ _ω\! [ _t=0^H-1 _ω _ω^Q(q_t b_t^M)\,A^J,Q_t(b_t^M,q_t) ] g_J,U(ω):=E_τ _ω\! [ _t=0^H _ω _ω^U(b_t^M c_t)\,A^J,U_t(c_t,b_t^M) ]. Accordingly, define the query-channel projected update Q(ω):=ω+ηgJ,Q(ω),T_Q(ω)\ :=\ ω+η\,g_J,Q(ω), and the BT-channel projected update U(ω):=ω+ηgJ,U(ω).T_U(ω)\ :=\ ω+η\,g_J,U(ω). Note that these are virtual updates used for only mechanism analysis. B.4 2D One-Sided Self-Locking via Projected Drifts We begin by bounding the channel-isolated outcome advantages, which quantify the strength of the outcome-based learning signal available to each channel. Proposition B.10 (AS-channel outcome advantages). Assume Assumptions B.1, B.4, B.5, and B.6 hold, then for any on-policy executed query qtq_t at turn t, ω[|AtJ,Q(btM,qt)|]≤LRκcCmaxCBT(ω).E_ω\! [ |A^J,Q_t(b_t^M,q_t) | ]\ ≤\ L_R\, _c\,C_ \,C_BT(ω). Moreover, under Assumption B.9, ‖gJ,Q(ω)‖≤KQCBT(ω)\|g_J,Q(ω)\|\ ≤\ K_QC_BT(ω), where KQ:=HGQLRκcCmaxK_Q:=H\,G_Q\,L_R _c\,C_ \,. Proposition B.11 (BT-channel outcome advantages). Assume Assumptions B.1, B.4, B.5, and B.6 hold, then for any on-policy generated belief btMb_t^M at turn t, ω[|AtJ,U(ct,btM)|]≤ 2LR(CBT(ω)+κUℐth(ω)).E_ω\! [ |A_t^J,U(c_t,b_t^M) | ]\ ≤\ 2L_R (C_BT(ω)\ +\ _U\,I_th(ω) ). Moreover, under Assumption B.9, ‖gJ,U(ω)‖≤KU,Iℐth(ω)+KU,CCBT(ω)\|g_J,U(ω)\|\ ≤\ K_U,I\,I_th(ω)\ +\ K_U,C\,C_BT(ω), where KU,I:=2(H+1)GULRκUK_U,I:=2(H+1)\,G_U\,L_R _U and KU,C:=2(H+1)GULRK_U,C:=2(H+1)\,G_U\,L_R. We now translate the channel-wise gradient bounds into one-step projected drifts of the two capability indices within the locking regime. For convenience, we denote by ΔQ+ _Q^+ and ΔU+ _U^+ the positive parts of the one-step changes induced by the projected updates QT_Q and UT_U. Proposition B.12 (AS projected drift is controlled by BT level). Under Assumption B.8 and the conclusion of Proposition B.10, for all ω∈ℛδ,εω _δ, , ΔQ+ℐth(ω):=(ℐth(Q(ω))−ℐth(ω))+≤ηαCBT(ω)+o(η),α:=GIKQ. _Q^+I_th(ω)\ :=\ (I_th(T_Q(ω))-I_th(ω) )_+\ ≤\ η\,α\,C_BT(ω)\ +\ o(η), α:=G_IK_Q. Proposition B.13 (BT projected drift is controlled by AS and BT levels). Under Assumption B.8 and the conclusion of Proposition B.11, for all ω∈ℛδ,εω _δ, , ΔU+CBT(ω):=(CBT(U(ω))−CBT(ω))+≤η(βIℐth(ω)+βCCBT(ω))+o(η), _U^+C_BT(ω)\ :=\ (C_BT(T_U(ω))-C_BT(ω) )_+\ ≤\ η ( _I\,I_th(ω)\ +\ _C\,C_BT(ω) )\ +\ o(η), where βI:=GCKU,I,βC:=GCKU,C. _I:=G_C\,K_U,I, _C:=G_C\,K_U,C. We now combine the channel-wise projected drift bounds into a unified two-dimensional description of the local training dynamics within the locking regime. Theorem B.14 (2D one-sided self-locking). Fix δ,ε>0δ, >0 and consider ω∈ℛδ,εω _δ, . Under the conclusions of Propositions B.12 and B.13, the one-sided projected drifts satisfy the following componentwise inequality: (ΔQ+ℐth(ω)ΔU+CBT(ω))⪯η(0αβIβC)(ℐth(ω)CBT(ω))+o(η), pmatrix _Q^+I_th(ω)\\[2.0pt] _U^+C_BT(ω) pmatrix\ \ η pmatrix0&α\\ _I& _C pmatrix pmatrixI_th(ω)\\[2.0pt] C_BT(ω) pmatrix\ +\ o(η), where ⪯ denotes elementwise inequality. Remark B.15. Theorem B.14 controls ΔQ+ _Q^+ and ΔU+ _U^+, i.e., the positive parts of the projected drifts. Note that our theoretical result does not assert that ℐthI_th or CBTC_BT cannot decrease, oscillate, or be affected by algorithmic stabilizers (entropy bonuses, clipping, weight decay, etc.). Rather, it formalizes self-locking as the absence of a strong upward training signal inside the low-information and low-BT regime: even when improvements happen, the theorem shows they can only be of small magnitude, scaling linearly with the current levels ℐthI_th and CBTC_BT. Remark B.16. The AS-side drift bound is purely BT-limited: ΔQ+ℐth(ω)≲ηCBT(ω). _Q^+I_th(ω)\ \ η\,C_BT(ω). Thus, when BT is weak, the query policy receives only a weak positive signal to increase informativeness (consistent with the intuition that AS learning is masked by BT failures in Sec. 2.3). In contrast, the BT-side drift bound is non-dual: ΔU+CBT(ω)≲η(ℐth(ω)+CBT(ω)), _U^+C_BT(ω)\ \ η (I_th(ω)+C_BT(ω) ), where the additional self-term CBT(ω)C_BT(ω) captures a “self-improvement” channel: even under limited information supply, the BT mechanism can still realize some gains by better utilizing the same evidence. At the same time, the presence of βIℐth(ω) _I\,I_th(ω) formalizes the bottleneck: when AS information supply stays low, BT improvements cannot scale beyond the evidence-limited envelope, yielding the empirically observed pattern that BT may improve early but tends to plateau once AS remains uninformative, as mentioned in Sec. 2.3. Proposition B.17. Fix δ,ε>0δ, >0 and let ω0∈ℛδ,ε _0 _δ, . Let m:=maxα,βI+βCm:= \α,\ _I+ _C\, defining K:=⌊1ηmlog(ε+Cηℐth(ω0)+Cη)⌋+,K\ :=\ 1η m\, \! ( +CηI_th( _0)+Cη ) _+, any projected evolution consistent with the one-step drift bounds of Theorem B.14 cannot leave ℛδ,εR_δ, within the first K steps. B.5 Why AReW breaks Information Self-locking In this part, we characterize how and when AReW proposed in Sec. 4 breaks information self-locking effectively. For simplicity, we specialize to the AS (query-policy) side and consider only binary critiques zt∈+1,−1z_t∈\+1,-1\ on query steps. The analysis for the BT side is entirely analogous and omitted for brevity. Critique quality and oracle “good” queries. Recall that AS informativeness is measured by the theoretical index ℐth(ω)I_th(ω), defined as the expected total improvement in oracle belief quality under the oracle-belief MDP (Definition B.2). As discussed before, a query can be assessed independently of the LLM agent’s belief-tracking mechanism πωU _ω^U by its effect on the terminal oracle confidence Ψ(bHB) (b_H^B) under the oracle-belief dynamics. Specifically, we give the following definitions: Definition B.18 (Oracle-good Query). Under the setting of Appendix B.1, consider a oracle-belief MDP trajectory τB=(b0B,q0,o0,…,bHB)τ^B=(b_0^B,q_0,o_0,…,b_H^B) induced by the query policy πωQ _ω^Q and Bayesian belief dynamics (⋅) BayesUpd(·). Define the expected terminal oracle confidence following a query q at oracle belief b as mt(b,q):=[Ψ(bHB)|btB=b,qt=q].m_t(b,q)\;:=\;E\! [ (b_H^B)\ |\ b_t^B=b,\ q_t=q ]. A query is oracle-good at step t if it yields a higher terminal oracle confidence than the policy average at the same oracle belief, i.e., mt(btB,qt)>q′∼πωQ(⋅∣btB)[mt(btB,q′)].m_t(b_t^B,q_t)\;>\;E_q _ω^Q(· b_t^B)\! [m_t(b_t^B,q ) ]. By telescoping of oracle belief progress, ∑k=tH−1IkB=Ψ(bHB)−Ψ(btB) _k=t^H-1I_k^B= (b_H^B)- (b_t^B), the stepwise AS advantage with respect to ℐthI_th is given by AtB(btB,qt)=mt(btB,qt)−q′∼πωQ(⋅∣btB)[mt(btB,q′)],A_t^B(b_t^B,q_t)=m_t(b_t^B,q_t)-E_q _ω^Q(· b_t^B)\! [m_t(b_t^B,q ) ], which measures how much the chosen query improves the expected terminal oracle confidence relative to the policy’s average choice at the same oracle belief. We therefore define the oracle direction label yt:=sign(AtB(btB,qt))∈+1,−1,y_t:=sign\! (A_t^B(b_t^B,q_t) )∈\+1,-1\, indicating whether the step-t query contributes positively or negatively to ℐthI_th under the oracle-belief dynamics. Efficacy of AReW and weighted accuracy. The critique label zt∈+1,−1z_t∈\+1,-1\ can be viewed as an approximation of the oracle direction label yty_t. For critique injection to improve AS informativeness beyond outcome-only learning, it is necessary that the injected signal aligns, on average, with the true stepwise contributions to ℐthI_th. In particular, positive critique should be assigned more frequently to oracle-good queries than to oracle-bad ones. Moreover, different steps along the trajectory do not contribute equally to AS informativeness. This leads naturally to a notion of weighted accuracy of the critique labels with respect to the oracle direction labels. The following proposition shows that, under a mild regularity condition, the effectiveness of AReW in improving AS informativeness is fully determined by this weighted accuracy. Proposition B.19 (Weighted accuracy characterizes AReW improvement). Under the setting of Sec. 4.2 and Def. B.18, denote the step weight as wt(ω):=|ut||AtB(btB,qt)|∥∇ωlogπωQ(qt∣btB)∥2w_t(ω):=|u_t|\, |A_t^B(b_t^B,q_t) |\,\| _ω _ω^Q(q_t b_t^B)\|^2 and let W(ω):=[∑t=0H−1wt(ω)].W(ω):=E\! [ _t=0^H-1w_t(ω) ]. Then the critique quality is measured by the weighted accuracy AccQ(ω):=[∑t=0H−1wt(ω) 1zt=yt][∑t=0H−1wt(ω)]∈[0,1].Acc_Q(ω)\;:=\; E\! [ _t=0^H-1w_t(ω)\,1\z_t=y_t\ ]E\! [ _t=0^H-1w_t(ω) ]∈[0,1]. (8) Moreover, the first-order improvement in AS informativeness induced by AReW satisfies ℐth(^Q(ω))−ℐth(Q(ω))=ηW(ω)(2AccQ(ω)−1)+o(η).I_th\! ( T_Q(ω) )-I_th\! (T_Q(ω) )\;=\;η\,W(ω)\, (2\,Acc_Q(ω)-1 )\;+\;o(η). (9) In particular, AReW improves AS informativeness beyond the baseline update if and only if AccQ(ω)>12Acc_Q(ω)> 12. B.6 Proofs B.6.1 Proof of Proposition B.10 Proof. Fix a turn t∈0,…,H−1t∈\0,…,H-1\ and condition on the current on-policy model belief btM=b_t^M=b. For any query q∈q , define the one-step next belief bt+1M,(q)b_t+1^M,(q) as the belief obtained by executing qt=q_t=q, observing ot=(s⋆,q)o_t= O(s ,q) (deterministic), and sampling the BT update bt+1M,(q)∼πωU(⋅∣ct+1)b_t+1^M,(q) _ω^U(· c_t+1) with ct+1=(b,q,ot)c_t+1=(b,q,o_t). Define the corresponding one-step potential change ΔΨt(q):=Ψ(bt+1M,(q))−Ψ(b) _t^(q):= (b_t+1^M,(q))- (b), and its one-sided parts ItM,+(q):=(ΔΨt(q))+I_t^M,+(q):=( _t^(q))_+, ItM,−(q):=(−ΔΨt(q))+I_t^M,-(q):=(- _t^(q))_+. By the scalar identity x=(x)+−(−x)+x=(x)_+-(-x)_+ applied to x=ΔΨt(q)x= _t^(q), we have the decomposition ΔΨt(q)=ItM,+(q)−ItM,−(q), _t^(q)\;=\;I_t^M,+(q)\;-\;I_t^M,-(q), equivalently, Ψ(bt+1M,(q))=Ψ(b)+ItM,+(q)−ItM,−(q). (b_t+1^M,(q))\;=\; (b)\;+\;I_t^M,+(q)\;-\;I_t^M,-(q). Taking conditional expectations given btM=b_t^M=b and qt=q_t=q yields ω[Ψ(bt+1M,(q))∣btM=b,qt=q]=Ψ(b)+ω[ItM,+(q)∣btM=b,qt=q]−ω[ItM,−(q)∣btM=b,qt=q].E_ω\! [ (b_t+1^M,(q)) b_t^M=b,\ q_t=q ]= (b)+E_ω\! [I_t^M,+(q) b_t^M=b,\ q_t=q ]-E_ω\! [I_t^M,-(q) b_t^M=b,\ q_t=q ]. (10) Assumption B.4 states that, for any q, ω[ItM,−(q)∣btM=b,qt=q]=ω[ItM,−∣btM=b],E_ω\! [I_t^M,-(q) b_t^M=b,\ q_t=q ]=E_ω\! [I_t^M,- b_t^M=b ], which is independent of q. Therefore, subtracting equation 10 for two queries q,q′q,q gives ω[Ψ(bt+1M,(q))∣b,q]−ω[Ψ(bt+1M,(q′))∣b,q′]=ω[ItM,+(q)∣b,q]−ω[ItM,+(q′)∣b,q′].E_ω\! [ (b_t+1^M,(q)) b,q ]-E_ω\! [ (b_t+1^M,(q )) b,q ]=E_ω\! [I_t^M,+(q) b,q ]-E_ω\! [I_t^M,+(q ) b,q ]. (11) By Assumption B.5, along on-policy rollouts, ω[ItM,+∣btM=b,qt]≤CmaxCBT(ω).E_ω\! [I_t^M,+ b_t^M=b,\ q_t ]≤ C_ \,C_BT(ω). Thus, for any realized btM=b_t^M=b and any q in the support of πωQ(⋅∣b) _ω^Q(· b), ω[ItM,+(q)∣btM=b,qt=q]≤CmaxCBT(ω).E_ω\! [I_t^M,+(q) b_t^M=b,\ q_t=q ]≤ C_ \,C_BT(ω). (12) Combining Eq. 11 and 12 yields the bound on the range of the conditional mean next-step confidence: supqω[Ψ(bt+1M,(q))∣btM=b,qt=q]−infqω[Ψ(bt+1M,(q))∣btM=b,qt=q]≤CmaxCBT(ω), _q\ E_ω\! [ (b_t+1^M,(q)) b_t^M=b,\ q_t=q ]- _q\ E_ω\! [ (b_t+1^M,(q)) b_t^M=b,\ q_t=q ]\ ≤\ C_ \,C_BT(ω), (13) where q∈supp(πωQ(⋅∣b))q ( _ω^Q(· b)). Now define the conditional mean terminal confidence given the current belief at time t+1t+1: Gt+1,H(b′):=ω[Ψ(bHM)∣bt+1M=b′].G_t+1,H(b ):=E_ω\! [ (b_H^M) b_t+1^M=b ]. Assumption B.6 implies that Gt+1,HG_t+1,H is κc _c-Lipschitz with respect to Ψ(⋅) (·): for any realizable b′,b~′b , b , |Gt+1,H(b′)−Gt+1,H(b~′)|≤κc|Ψ(b′)−Ψ(b~′)|. |G_t+1,H(b )-G_t+1,H( b ) |\ ≤\ _c\, | (b )- ( b ) |. (14) Now define the query-conditioned mean terminal confidence at time t: mH(q):=ω[Ψ(bHM)∣btM=b,qt=q].m_H(q):=E_ω\! [ (b_H^M) b_t^M=b,\ q_t=q ]. By the tower property, mH(q)=ω[Gt+1,H(bt+1M,(q))|btM=b,qt=q].m_H(q)=E_ω\! [G_t+1,H(b_t+1^M,(q))\ |\ b_t^M=b,\ q_t=q ]. At this point, we use the fact that Gt+1,HG_t+1,H is κc _c-Lipschitz in Ψ and combine it with the cancellation structure already captured in equation 13: the induced range of mH(q)m_H(q) over q is bounded by applying equation 14 to the extremal conditional means, giving supqmH(q)−infqmH(q)≤κc(supqω[Ψ(bt+1M,(q))∣btM=b,qt=q]−infqω[Ψ(bt+1M,(q))∣btM=b,qt=q]). _qm_H(q)- _qm_H(q)\ ≤\ _c\, ( _qE_ω[ (b_t+1^M,(q)) b_t^M=b,\ q_t=q]- _qE_ω[ (b_t+1^M,(q)) b_t^M=b,\ q_t=q] ). (15) Combining equation 15 with equation 13 yields supqmH(q)−infqmH(q)≤κcCmaxCBT(ω). _qm_H(q)- _qm_H(q)\ ≤\ _c\,C_ \,C_BT(ω). (16) QtJ,Q(b,q)=f(mH(q)),Q^J,Q_t(b,q)=f(m_H(q)), where f is LRL_R-Lipschitz. Therefore, supqQtJ,Q(b,q)−infqQtJ,Q(b,q)≤LR(supqmH(q)−infqmH(q)). _qQ^J,Q_t(b,q)- _qQ^J,Q_t(b,q)≤ L_R\, ( _qm_H(q)- _qm_H(q) ). Plugging equation 16 gives supqQtJ,Q(b,q)−infqQtJ,Q(b,q)≤LRκcCmaxCBT(ω). _qQ^J,Q_t(b,q)- _qQ^J,Q_t(b,q)\ ≤\ L_R\, _c\,C_ \,C_BT(ω). (17) By definition, VtJ,Q(b)=q∼πωQ(⋅∣b)[QtJ,Q(b,q)]V^J,Q_t(b)=E_q _ω^Q(· b)[Q^J,Q_t(b,q)] is a convex combination of QtJ,Q(b,q)q\Q^J,Q_t(b,q)\_q. Hence for any executed qtq_t, |AtJ,Q(b,qt)|=|QtJ,Q(b,qt)−VtJ,Q(b)|≤supqQtJ,Q(b,q)−infqQtJ,Q(b,q).|A^J,Q_t(b,q_t)|= |Q^J,Q_t(b,q_t)-V^J,Q_t(b) |≤ _qQ^J,Q_t(b,q)- _qQ^J,Q_t(b,q). Combining with equation 17 yields |AtJ,Q(btM,qt)|≤LRκcCmaxCBT(ω).|A^J,Q_t(b_t^M,q_t)|≤ L_R\, _c\,C_ \,C_BT(ω). By definition, gJ,Q(ω)=τ∼πω[∑t=0H−1∇ωlogπωQ(qt∣btM)AtJ,Q(btM,qt)].g_J,Q(ω)=E_τ _ω\! [ _t=0^H-1 _ω _ω^Q(q_t b_t^M)\,A^J,Q_t(b_t^M,q_t) ]. Taking norms and applying Jensen and the triangle inequality, ∥gJ,Q(ω)∥≤∑t=0H−1[∥∇ωlogπωQ(qt∣btM)∥⋅|AtJ,Q(btM,qt)|].\|g_J,Q(ω)\|≤ _t=0^H-1E\! [ \| _ω _ω^Q(q_t b_t^M) \|· |A^J,Q_t(b_t^M,q_t) | ]. Using Assumption B.9 gives ‖∇ωlogπωQ(⋅)‖≤GQ\| _ω _ω^Q(·)\|≤ G_Q a.s., hence ‖gJ,Q(ω)‖≤GQ∑t=0H−1[|AtJ,Q|].\|g_J,Q(ω)\|≤ G_Q _t=0^H-1E [ |A^J,Q_t | ]. Applying Proposition B.10 to each t yields ‖gJ,Q(ω)‖≤GQ∑t=0H−1LRκcCmaxCBT(ω)≤HGQLRκcCmaxCBT(ω).\|g_J,Q(ω)\|≤ G_Q _t=0^H-1L_R\, _c\,C_ \,C_BT(ω)≤ H\,G_Q\,L_R _cC_ \,C_BT(ω). ∎ B.6.2 Proof of Proposition B.11 Proof. Fix a turn t. Condition on the BT context ctc_t and draw two independent samples B,B′∼i.i.d.πωU(⋅∣ct)B,B i.i.d. _ω^U(· c_t). Let Y:=QtJ,U(ct,B),Y′:=QtJ,U(ct,B′),Y¯:=[Y∣ct]=VtJ,U(ct).Y:=Q_t^J,U(c_t,B), Y :=Q_t^J,U(c_t,B ), Y:=E[Y c_t]=V_t^J,U(c_t). Then AtJ,U(ct,B)=Y−Y¯A_t^J,U(c_t,B)=Y- Y, and for any fixed realization Y=yY=y, Jensen implies [|y−Y′|∣y,ct]≥|y−[Y′∣ct]|=|y−Y¯|E[|y-Y | y,c_t]≥|y-E[Y c_t]|=|y- Y|. Taking expectation over Y yields [|Y−Y¯|∣ct]≤[|Y−Y′|∣ct].E [|Y- Y| c_t ]\ ≤\ E [|Y-Y | c_t ]. Therefore, ω[|AtJ,U(ct,btM)|]≤[|QtJ,U(ct,B)−QtJ,U(ct,B′)|].E_ω\! [ |A_t^J,U(c_t,b_t^M) | ]\ ≤\ E\! [ |Q_t^J,U(c_t,B)-Q_t^J,U(c_t,B ) | ]. (18) By Assumption B.1, [R(τ)∣bHM]=f(Ψ(bHM))E[R(τ) b_H^M]=f( (b_H^M)) for a non-decreasing LRL_R-Lipschitz f. Hence, for any (ct,b)(c_t,b), QtJ,U(ct,b)=[f(Ψ(bHM))|ct,btM=b].Q_t^J,U(c_t,b)=E\! [f( (b_H^M))\ |\ c_t,\ b_t^M=b ]. For any b,b′b,b , using |[X]−[Y]|≤[|X−Y|]|\,E[X]-E[Y]\,| [|X-Y|], Lipschitzness of f, and the elementary inequality |x−y|≤x+y|x-y|≤ x+y for x,y∈[0,1]x,y∈[0,1], we obtain |QtJ,U(ct,b)−QtJ,U(ct,b′)| |Q_t^J,U(c_t,b)-Q_t^J,U(c_t,b ) | ≤LR([Ψ(bHM)∣ct,btM=b]+[Ψ(bHM)∣ct,btM=b′]). ≤ L_R (E[ (b_H^M) c_t,b_t^M=b]+E[ (b_H^M) c_t,b_t^M=b ] ). (19) Along any continuation after time t, Ψ(bHM)=Ψ(btM)+∑k=tH−1(Ψ(bk+1M)−Ψ(bkM))≤Ψ(btM)+∑k=tH−1(Ψ(bk+1M)−Ψ(bkM))+=Ψ(btM)+∑k=tH−1IkM,+. (b_H^M)= (b_t^M)+ _k=t^H-1 ( (b_k+1^M)- (b_k^M) )\ ≤\ (b_t^M)+ _k=t^H-1( (b_k+1^M)- (b_k^M))_+= (b_t^M)+ _k=t^H-1I_k^M,+. Taking conditional expectation given (ct,btM=b)(c_t,b_t^M=b) yields [Ψ(bHM)∣ct,btM=b]≤Ψ(b)+[∑k=tH−1IkM,+|ct,btM=b].E[ (b_H^M) c_t,b_t^M=b]\ ≤\ (b)\ +\ E\! [ _k=t^H-1I_k^M,+\ |\ c_t,b_t^M=b ]. (20) By iterated expectations, [∑k=tH−1IkM,+|ct,btM=b] \! [ _k=t^H-1I_k^M,+\ |\ c_t,b_t^M=b ] =∑k=tH−1[[IkM,+∣hk+1]|ct,btM=b] = _k=t^H-1E\! [E[I_k^M,+ h_k+1]\ |\ c_t,b_t^M=b ] ≤κU∑k=tH−1[Ψ(bk+1B)−Ψ(bkB)|ct,btM=b]. ≤ _U _k=t^H-1E\! [ (b_k+1^B)- (b_k^B)\ |\ c_t,b_t^M=b ]. (21) At this point, the right-hand side is the remaining Bayesian truth-gain along the realized transcript. Since our main statement is in terms of ℐth(ω)I_th(ω) (Def. B.2), we use the following mild presentation-level observation: conditioning the same parametric query policy πωQ _ω^Q on a more accurate belief (oracle belief btBb_t^B) cannot decrease its expected Bayesian truth-gain, so the expected remaining transcript gain is upper bounded by the oracle-belief MDP gain. Concretely, we bound ω[∑k=tH−1(Ψ(bk+1B)−Ψ(bkB))]≤τB∼πωB[Ψ(bHB)−Ψ(b0B)]=ℐth(ω),E_ω\! [ _k=t^H-1 ( (b_k+1^B)- (b_k^B) ) ]\ ≤\ E_τ^B _ω^B [ (b_H^B)- (b_0^B) ]=I_th(ω), (22) and hence (by dropping conditioning and applying tower property) from equation 21 we obtain [∑k=tH−1IkM,+|ct,btM=b]≤κUℐth(ω).E\! [ _k=t^H-1I_k^M,+\ |\ c_t,b_t^M=b ]\ ≤\ _U\,I_th(ω). (23) Combining equation 18, equation 19, equation 20, and equation 23, and averaging over B,B′B,B , we obtain [|AtJ,U(ct,btM)|]≤ 2LR([Ψ(btM)]+κUℐth(ω)).E\! [ |A_t^J,U(c_t,b_t^M) | ]\ ≤\ 2L_R (E[ (b_t^M)]+ _U\,I_th(ω) ). It remains to control [Ψ(btM)]E[ (b_t^M)] by [Ψ(b0M)]+CBT(ω)E[ (b_0^M)]+C_BT(ω). Along any rollout, Ψ(btM)=Ψ(b0M)+∑k=0t−1(Ψ(bk+1M)−Ψ(bkM))≤Ψ(b0M)+∑k=0t−1(Ψ(bk+1M)−Ψ(bkM))+≤Ψ(b0M)+∑k=0H−1IkM,+. (b_t^M)= (b_0^M)+ _k=0^t-1 ( (b_k+1^M)- (b_k^M) )\ ≤\ (b_0^M)+ _k=0^t-1 ( (b_k+1^M)- (b_k^M) )_+\ ≤\ (b_0^M)+ _k=0^H-1I_k^M,+. Taking expectation yields [Ψ(btM)]≤[Ψ(b0M)]+CBT(ω)E[ (b_t^M)] [ (b_0^M)]+C_BT(ω), which proves the first inequality in the proposition. By definition, gJ,U(ω)=τ∼πω[∑t=0H∇ωlogπωU(btM∣ct)AtJ,U(ct,btM)].g_J,U(ω)=E_τ _ω\! [ _t=0^H _ω _ω^U(b_t^M c_t)\,A_t^J,U(c_t,b_t^M) ]. Taking norms, applying Jensen, and using Assumption B.9 (‖∇ωlogπωU‖≤GU\| _ω _ω^U\|≤ G_U a.s.) gives ∥gJ,U(ω)∥≤∑t=0H[∥∇ωlogπωU(btM∣ct)∥⋅|AtJ,U|]≤(H+1)GU⋅max0≤t≤H[|AtJ,U|].\|g_J,U(ω)\|≤ _t=0^HE [\| _ω _ω^U(b_t^M c_t)\|·|A_t^J,U| ]≤(H+1)G_U· _0≤ t≤ HE[|A_t^J,U|]. Applying the established bound on [|AtJ,U|]E[|A_t^J,U|] completes the proof. ∎ B.6.3 Proof of Proposition B.12 Proof. Fix ω∈ℛδ,εω _δ, . Recall Q(ω)=ω+ηgJ,Q(ω)T_Q(ω)=ω+η g_J,Q(ω). By Assumption B.8 (first-order expandability of ℐthI_th on ℛδ,εR_δ, ), ℐth(Q(ω))−ℐth(ω)=⟨∇ωℐth(ω),Q(ω)−ω⟩+o(η)=η⟨∇ωℐth(ω),gJ,Q(ω)⟩+o(η).I_th(T_Q(ω))-I_th(ω)= _ωI_th(ω),\ T_Q(ω)-ω +o(η)=η _ωI_th(ω),\ g_J,Q(ω) +o(η). Taking the positive part and using (x+y)+≤x++|y|(x+y)_+≤ x_++|y| yields ΔQ+ℐth(ω)=(η⟨∇ωℐth(ω),gJ,Q(ω)⟩+o(η))+≤η(⟨∇ωℐth(ω),gJ,Q(ω)⟩)++|o(η)|. _Q^+I_th(ω)= (η _ωI_th(ω),\ g_J,Q(ω) +o(η) )_+≤η ( _ωI_th(ω),\ g_J,Q(ω) )_++|o(η)|. Next, apply Cauchy–Schwarz and the gradient bound ‖∇ωℐth(ω)‖≤GI\| _ωI_th(ω)\|≤ G_I: (⟨∇ωℐth(ω),gJ,Q(ω)⟩)+≤|⟨∇ωℐth(ω),gJ,Q(ω)⟩|≤‖∇ωℐth(ω)‖⋅‖gJ,Q(ω)‖≤GI‖gJ,Q(ω)‖. ( _ωI_th(ω),\ g_J,Q(ω) )_+≤ | _ωI_th(ω),\ g_J,Q(ω) |≤\| _ωI_th(ω)\|·\|g_J,Q(ω)\|≤ G_I\,\|g_J,Q(ω)\|. Using the assumed norm control ‖gJ,Q(ω)‖≤KQCBT(ω)\|g_J,Q(ω)\|≤ K_QC_BT(ω) gives ΔQ+ℐth(ω)≤ηGIKQCBT(ω)+|o(η)|. _Q^+I_th(ω)≤η\,G_I\,K_Q\,C_BT(ω)+|o(η)|. Finally, absorb |o(η)||o(η)| into o(η)o(η) (since |o(η)|/η→0|o(η)|/η→ 0 uniformly on ℛδ,εR_δ, ) to obtain the claimed bound with α:=GIKQα:=G_IK_Q. ∎ B.6.4 Proof of Proposition B.13 Proof. Fix ω∈ℛδ,εω _δ, . Recall U(ω)=ω+ηgJ,U(ω)T_U(ω)=ω+η g_J,U(ω). By Assumption B.8 (first-order expandability of CBTC_BT on ℛδ,εR_δ, ), CBT(U(ω))−CBT(ω)=⟨∇ωCBT(ω),U(ω)−ω⟩+o(η)=η⟨∇ωCBT(ω),gJ,U(ω)⟩+o(η).C_BT(T_U(ω))-C_BT(ω)= _ωC_BT(ω),\ T_U(ω)-ω +o(η)=η _ωC_BT(ω),\ g_J,U(ω) +o(η). Taking positive parts and using (x+y)+≤x++|y|(x+y)_+≤ x_++|y| gives ΔU+CBT(ω)=(η⟨∇ωCBT(ω),gJ,U(ω)⟩+o(η))+≤η(⟨∇ωCBT(ω),gJ,U(ω)⟩)++|o(η)|. _U^+C_BT(ω)= (η _ωC_BT(ω),\ g_J,U(ω) +o(η) )_+≤η ( _ωC_BT(ω),\ g_J,U(ω) )_++|o(η)|. Apply Cauchy–Schwarz and the gradient bound ‖∇ωCBT(ω)‖≤GC\| _ωC_BT(ω)\|≤ G_C: (⟨∇ωCBT(ω),gJ,U(ω)⟩)+≤|⟨∇ωCBT(ω),gJ,U(ω)⟩|≤‖∇ωCBT(ω)‖⋅‖gJ,U(ω)‖≤GC‖gJ,U(ω)‖. ( _ωC_BT(ω),\ g_J,U(ω) )_+≤ | _ωC_BT(ω),\ g_J,U(ω) |≤\| _ωC_BT(ω)\|·\|g_J,U(ω)\|≤ G_C\,\|g_J,U(ω)\|. Using the assumed norm control on ℛδ,εR_δ, , ‖gJ,U(ω)‖≤KU,Iℐth(ω)+KU,CCBT(ω),\|g_J,U(ω)\|\ ≤\ K_U,I\,I_th(ω)\ +\ K_U,C\,C_BT(ω), we obtain ΔU+CBT(ω)≤ηGC(KU,Iℐth(ω)+KU,CCBT(ω))+|o(η)|. _U^+C_BT(ω)≤η\,G_C (K_U,I\,I_th(ω)+K_U,C\,C_BT(ω) )+|o(η)|. Absorbing |o(η)||o(η)| into o(η)o(η) yields the stated bound with βI:=GCKU,I _I:=G_CK_U,I and βC:=GCKU,C _C:=G_CK_U,C. ∎ B.6.5 Proof of Corollary B.17 Proof. Since the o(η)o(η) term in Theorem B.14 is uniform over ℛδ,εR_δ, , there exist η0>0 _0>0 and c0<∞c_0<∞ such that for all η∈(0,η0]η∈(0, _0] and all ω∈ℛδ,εω _δ, , the remainder satisfies the componentwise bound ‖o(η)‖∞≤c0η2\|o(η)\|_∞≤ c_0η^2. Set C:=c0/mC:=c_0/m. Define the capability vector (boldface) (ω):=(ℐth(ω)CBT(ω))∈ℝ+2,M:=(0αβIβC),:=(1,1)⊤.x(ω):= pmatrixI_th(ω)\\ C_BT(ω) pmatrix _+^2, M:= pmatrix0&α\\ _I& _C pmatrix, 1:=(1,1) . Inside ℛδ,εR_δ, , Theorem B.14 yields the componentwise bound Δ+(ω)⪯ηM(ω)+c0η2 1. ^+x(ω)\ \ η\,M\,x(ω)\ +\ c_0η^2\,1. Moreover, for any scalar z we have z≤z+z≤ z_+, hence any increment in x is bounded above by its positive part. Therefore, an upper envelope for the accumulation of positive gains is given by the deterministic recursion k+1⪯(I+ηM)k+c0η2 1,0:=(ω0).x_k+1\ \ (I+η M)\,x_k\ +\ c_0η^2\,1, _0:=x( _0). (24) Let yk:=‖k‖∞y_k:=\|x_k\|_∞. Taking ∥⋅∥∞\|·\|_∞ on equation 24 and using ‖I+ηM‖∞=1+ηm\|I+η M\|_∞=1+η m gives yk+1≤(1+ηm)yk+c0η2.y_k+1\ ≤\ (1+η m)\,y_k\ +\ c_0η^2. Unrolling this scalar recursion and using (1+ηm)k≤ekηm(1+η m)^k≤ e^kη m yields yk≤ekηm(y0+c0mη)−c0mη=ekηm(y0+Cη)−Cη.y_k\ ≤\ e^kη m (y_0+ c_0mη )\ -\ c_0mη\ =\ e^kη m (y_0+Cη )\ -\ Cη. By the initialization condition CBT(ω0)≤ℐth(ω0)C_BT( _0) _th( _0), we have y0=‖0‖∞≤ℐth(ω0)y_0=\|x_0\|_∞ _th( _0). Hence, if ekηm(ℐth(ω0)+Cη)−Cη≤u⋆,e^kη m (I_th( _0)+Cη )-Cη\ ≤\ u_ , then yk≤u⋆y_k≤ u_ , which implies k⪯(u⋆,u⋆)⊤⪯(δ,ε)⊤x_k (u_ ,u_ ) (δ, ) . Solving the inequality for k gives k≤1ηmlog(u⋆+Cηℐth(ω0)+Cη).k\ ≤\ 1η m \! ( u_ +CηI_th( _0)+Cη ). Choosing K=⌊1ηmlog(u⋆+Cηℐth(ω0)+Cη)⌋+K= 1η m\, \! ( u_ +CηI_th( _0)+Cη ) _+ ensures the above condition holds for all integers k∈0,1,…,Kk∈\0,1,…,K\, proving the claimed finite-horizon trapping (escape-time lower bound) under the projected drift envelope. ∎ B.7 Proof of Proposition B.19 Proof. By definition, Q(ω)−ω=ηgJ,Q(ω), _Q(ω)-ω=η\,g_J,Q(ω), (25) ^Q(ω)−ω=ηgJ,Q(ω)+ηλQgaux,Q(ω). T_Q(ω)-ω=η\,g_J,Q(ω)+η\, _Qg_aux,Q(ω). (26) By Assumption B.8, for ω′=Q(ω)∈ℛδ,εω =T_Q(ω) _δ, we have ℐth(Q(ω)) _th\! (T_Q(ω) ) =ℐth(ω)+⟨∇ωℐth(ω),Q(ω)−ω⟩+o(η), =I_th(ω)+ _ωI_th(ω),\ T_Q(ω)-ω +o(η), (27) and for ω′=^Q(ω)∈ℛδ,εω = T_Q(ω) _δ, we have ℐth(^Q(ω)) _th\! ( T_Q(ω) ) =ℐth(ω)+⟨∇ωℐth(ω),^Q(ω)−ω⟩+o(η). =I_th(ω)+ _ωI_th(ω),\ T_Q(ω)-ω +o(η). (28) Subtracting equation 27 from equation 28 yields ℐth(^Q(ω))−ℐth(Q(ω)) _th\! ( T_Q(ω) )-I_th\! (T_Q(ω) ) =⟨∇ωℐth(ω),^Q(ω)−Q(ω)⟩+o(η). = _ωI_th(ω),\ T_Q(ω)-T_Q(ω) +o(η). Using the update definitions, ^Q(ω)−Q(ω)=ηλQgaux,Q(ω) T_Q(ω)-T_Q(ω)=η\, _Q\,g_aux,Q(ω), so ℐth(^Q(ω))−ℐth(Q(ω)) _th\! ( T_Q(ω) )-I_th\! (T_Q(ω) ) =ηλQ⟨∇ωℐth(ω),gaux,Q(ω)⟩+o(η) =η\, _Q\, _ωI_th(ω),\ g_aux,Q(ω) +o(η) =:ηλQΓQ(ω)+o(η). =:η\, _Q\, _Q(ω)\ +o(η). (29) We expand ΓQ(ω) _Q(ω). On the oracle-belief MDP, ℐth(ω)=τB[∑t=0H−1ItB]I_th(ω)=E_τ^B[ _t=0^H-1I_t^B] is a finite-horizon policy objective with action qt∼πωQ(⋅∣btB)q_t _ω^Q(· b_t^B). Thus, the policy-gradient theorem yields ∇ωℐth(ω)=τB[∑t=0H−1AtB(btB,qt)st(ω)],st(ω)=∇ωlogπωQ(qt∣btB). _ωI_th(ω)=E_τ^B\! [ _t=0^H-1A_t^B(b_t^B,q_t)\,s_t(ω) ], s_t(ω)= _ω _ω^Q(q_t b_t^B). The likelihood-gap auxiliary objective in Section 4.2 induces an additive update component of the form gaux,Q(ω)=τB[∑t=0H−1ut(z)st(ω)],ut(z)=|ut|zt,|ut|≥0,zt∈+1,−1.g_aux,Q(ω)=E_τ^B\! [ _t=0^H-1u_t(z)\,s_t(ω) ], u_t(z)=|u_t|\,z_t,\ \ |u_t|≥ 0,\ \ z_t∈\+1,-1\. By bilinearity and exchanging expectation with finite sums, ΓQ(ω)∝[⟨∑tAtBst,∑ubusu⟩]=[∑t,uAtBbu⟨st,su⟩]. _Q(ω)\ \ E\! [ _tA_t^B\,s_t,\ _ub_u\,s_u ]=E\! [ _t,uA_t^B\,b_u\, s_t,s_u ]. For simplicity of analysis, we remove the cross-time terms, giving ΓQ(ω)=[∑t=0H−1AtB(btB,qt)ut(z)‖st(ω)‖2]. _Q(ω)=E\! [ _t=0^H-1A_t^B(b_t^B,q_t)\,u_t(z)\,\|s_t(ω)\|^2 ]. Write AtB=|AtB|ytA_t^B=|A_t^B|\,y_t with yt=sign(AtB)∈±1y_t=sign(A_t^B)∈\± 1\ and ut=|ut|ztu_t=|u_t|\,z_t with zt∈±1z_t∈\± 1\. Then each summand becomes AtBbt‖st‖2=|ut||AtB|‖st‖2(ztyt)=wt(ω)(ztyt),A_t^B\,b_t\,\|s_t\|^2=|u_t|\,|A_t^B|\,\|s_t\|^2\,(z_ty_t)=w_t(ω)\,(z_ty_t), where wt(ω)w_t(ω) is exactly the weight defined above. Hence ΓQ(ω)=[∑t=0H−1wt(ω)ztyt]. _Q(ω)=E\! [ _t=0^H-1w_t(ω)\,z_ty_t ]. Since ztyt=+1z_ty_t=+1 iff zt=ytz_t=y_t and ztyt=−1z_ty_t=-1 iff zt≠ytz_t≠ y_t, we have ztyt=2 1zt=yt−1z_ty_t=2\,1\z_t=y_t\-1, and therefore ΓQ(ω) _Q(ω) =[∑twt(ω)(2 1zt=yt−1)] =E\! [ _tw_t(ω)\, (2\,1\z_t=y_t\-1 ) ] =2[∑twt(ω) 1zt=yt]−[∑twt(ω)] =2\,E\! [ _tw_t(ω)\,1\z_t=y_t\ ]-E\! [ _tw_t(ω) ] =W(ω)(2AccQ(ω)−1), =W(ω)\, (2\,Acc_Q(ω)-1 ), which is equation 9. The final equivalence follows immediately when W(ω)>0W(ω)>0. ∎ Appendix C Setup Details C.1 Dataset Details and Prompt Templates In this section, we present more details for the datasets and tasks evaluated in this work. Preference Estimation – Gated (PE-G), adapted from Badola et al. (2025) and Zou et al. (2026). Adapted from Badola et al. (2025), Gated-PE is an interactive preference inference task under constrained information acquisition. The agent is given a finite set of items =x1,…,xNX=\x_1,…,x_N\, where each item xix_i is represented by a known attribute vector i∈ℝDa_i ^D. The user is characterized by an unknown latent preference vector ⋆∈[0,1]Dw ∈[0,1]^D. Through interaction, the agent maintains and iteratively refines an estimate t∈[0,1]Dw_t∈[0,1]^D of this latent preference. At each decision point, the agent actively selects a low-dimensional attribute subspace St⊆1,…,DS_t \1,…,D\ and an item comparison (xi,xj)∈×(x_i,x_j) ×X designed to elicit the user’s preference feedback restricted to StS_t. Based on the observed feedback, the agent updates its belief state. The objective is to accurately recover ⋆w under sparse, outcome-based supervision. MediQ, adapted from Li et al. (2024). Adapted from Li et al. (2024), MediQ is an interactive medical inference task that models hypothesis-level belief tracking under partial observability. The agent is provided with a clinical vignette and an associated medical question whose answer lies in a finite hypothesis set of size D. The agent maintains a belief state t∈[0,1]Dw_t∈[0,1]^D, where each dimension represents the current support for a candidate hypothesis. Through iterative interaction, the agent actively queries the LLM-simulated user for diagnostic information, receives structured feedback, and updates each hypothesis score accordingly. The learning objective is to progressively concentrate belief mass onto the correct hypothesis. FloDial, adapted from Raghu et al. (2021); Hu et al. (2024) consists of multi-turn diagnostic dialogues for resolving user-reported issues. It provides a scenario where a customer support technician interacts with customers to identify and resolve faults or issues within computer systems, electronic devices, machinery, or other complex systems. The agent simulates the customer support technician, which chat with the customer to further check the specific issues of device through multi-turn interactions. C.2 Baseline Details Here we introduce RL algorithms used in our experiments. Formally, given an actor model πθ _θ, the likelihood of a response y to a query x under the policy πθ _θ is modeled as πθ(y|x)=∏t=1|y|πθ(yt|x,y<t) _θ(y|x)= _t=1^|y| _θ(y_t|x,y_<t). Given a query-response pair (x,y)(x,y), a verifier r generates its reward r(x,y)∈[0,1]r(x,y)∈[0,1]. Proximal Policy Optimization (PPO) (Schulman et al., 2017) employs the following objective for policy optimization: PPO(θ)=x∼,y∼πθold(⋅|x)[1|y|∑t=1|y|min(wt(θ)A^t,clip(wt(θ),1−ε,1+ε)A^t)], _PPO(θ)=E_x ,\,y _ _old(·|x) [ 1|y| _t=1^|y| (w_t(θ) A_t,\,clip (w_t(θ),1- ,1+ ) A_t ) ], (30) where the importance ratio of the token yty_t is defined as wt(θ)=πθ(yt|x,y<t)πθold(yt|x,y<t)w_t(θ)= _θ(y_t|x,y_<t) _ _old(y_t|x,y_<t), the advantage A^t A_t of yty_t is typically computed via Generalized Advantage Estimation (GAE) (Schulman et al., 2015) with temporal-difference errors, and ε is the clipping range of importance ratios. Group Relative Policy Optimization (GRPO) (Shao et al., 2024) proposes computing the relative advantage of each response within a group of responses of the same query using the following objective (omitting the KL regularization term): GRPO(θ)=x,yii=1G[1G∑i=1G1|yi|∑t=1|yi|min(wi,t(θ)A^i,t,clip(wi,t(θ),1−ε,1+ε)A^i,t)], _GRPO(θ)=E_x,\,\y_i\_i=1^G [ 1G _i=1^G 1|y_i| _t=1^|y_i| (w_i,t(θ) A_i,t,\,clip (w_i,t(θ),1- ,1+ ) A_i,t ) ], (31) where yii=1G∼πθold(⋅|x)\y_i\_i=1^G _ _old(·|x) and G is the group size. The importance ratio wi,t(θ)w_i,t(θ) and advantage A^i,t A_i,t of token yi,ty_i,t are defined as: wi,t(θ)=πθ(yi,t|x,yi,<t)πθold(yi,t|x,yi,<t),A^i,t=r(x,yi)−mean(r(x,yi)i=1G)std(r(x,yi)i=1G), w_i,t(θ)= _θ(y_i,t|x,y_i,<t) _ _old(y_i,t|x,y_i,<t),\,\, A_i,t= r(x,y_i)-mean (\r(x,y_i)\_i=1^G )std (\r(x,y_i)\_i=1^G ), (32) respectively, where all the tokens in yiy_i share the same advantage. Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) extends GRPO by defining the importance ratio at the sequence level with length normalization, with sequence-level clipping, rewarding, and optimization. The objective is: GSPO(θ)=x,yii=1G[1G∑i=1Gmin(si(θ)A^i,clip(si(θ),1−ϵ,1+ϵ)A^i)], _GSPO(θ)=E_x,\y_i\_i=1^G [ 1G _i=1^G (s_i(θ) A_i,\ clip(s_i(θ),1-ε,1+ε) A_i ) ], (33) where si(θ)=(πθ(yi|x)πθold(yi|x))1/|yi|=exp(1|yi|∑t=1|yi|logπθ(yi,t|x,yi,<t)πθold(yi,t|x,yi,<t)).s_i(θ)= ( _θ(y_i|x) _ _old(y_i|x) )^1/|y_i|= \! ( 1|y_i| _t=1^|y_i| _θ(y_i,t|x,y_i,<t) _ _old(y_i,t|x,y_i,<t) ). C.3 Supplementary Implementation Details C.3.1 Additional Setting on Datasets Here we provide additional implementation details. The maximum number of interaction turns is set at 10 for PE-G, 12 for PE-F, 8 for MediQ, 10 for FloDial-Easy and FloDial-Hard. For RL training, we define task-specific rewards aligned with their evaluation metrics: for PE-G and PE-F, the reward is the similarity improvement compared to the initial default guess (all 0.5), where PE-G leverages binary reward ([improvement]>0.031[improvement]>0.03) and PE-F leverages the continuous improvement value (normalized to [0,1][0,1]). For MediQ and FloDial, the reward is binary, checking if the final decision made by the agents aligns with the ground-truth. All rewards are provided only at the terminal step of each trajectory, consistent with the outcome-based RL setting. The AS and BT proxies on PE and MediQ datasets can be seen in Sec. 2. For the FloDial dataset, the query is considered as uninformative if the user replies “unknown”, and the BT proxy is defined by whether the agent can increase the confidence of the ground truth when receiving informative feedback. Critique construction (AS vs. BT). Across datasets, we use two lightweight stepwise critiques: ztQz_t^Q (AS-channel) evaluates whether the query step at turn t yields informative feedback, and ztUz_t^U (BT-channel) evaluates whether the subsequent belief update is consistent with that feedback. Below we summarize the dataset-specific instantiations. PE-G and PE-F. AS critique. Let the action be a comparison between two items, inducing a signed attribute-difference vector mij∈ℝDm_ij ^D (restricted to the focused coordinates when applicable). We mark the query as informative if it exhibits a non-trivial trade-off across coordinates, i.e., ztQ=+1⟺∃k,ℓ∈[D]s.t.mij,kmij,ℓ<0,z_t^Q=+1 ∃ k, ∈[D]\ s.t.\ m_ij,k\,m_ij, <0, and set ztQ=−1z_t^Q=-1 for repeated or invalid queries or when mijm_ij is one-signed. BT critique. Conditioned on an informative query (ztQ=+1z_t^Q=+1), we compare the updated preference estimate vtv_t against the previous valid estimate vt−1v_t-1 using similarity to the ground-truth preference w⋆w : ztU=sign(sim(vt,w⋆)−sim(vt−1,w⋆)),z_t^U=sign\! (sim(v_t,w )-sim(v_t-1,w ) ), and set ztU=0z_t^U=0 when the query is uninformative or invalid. MediQ. AS critique. Let ftf_t denote the patient feedback to the query at turn t. We set ztQ=+1z_t^Q=+1 iff the feedback is informative (i.e., not an “Unknown / cannot answer” response) and the query is not repeated; otherwise ztQ=−1z_t^Q=-1. We additionally explore counterfactual pattern (exploration on other datasets as future work). When counterfactual queries are enabled, we intervene the belief from the last turn and obtain the counterfactual query. We then discourage semantic repetition by requiring the counterfactual query to be sufficiently different from the actual query (measured by token-level overlap), and mark the step as uninformative when this difference is small. BT critique. Let wt∈[0,1]4w_t∈[0,1]^4 be the belief over 44 hypotheses and s⋆s the ground-truth hypothesis. Define the margin mar(wt;s⋆):=wt(s⋆)−maxs≠s⋆wt(s).mar(w_t;s ):=w_t(s )- _s≠ s w_t(s). If the feedback is uninformative, we enforce invariance: ztU=+1z_t^U=+1 if mar(wt;s⋆)=mar(wt−1;s⋆)mar(w_t;s )=mar(w_t-1;s ) and ztU=−1z_t^U=-1 otherwise. If the feedback is informative, we employ a counterfactual-consistency check: We intervene the observation to “unknown” and obtain wtcfw_t^cf to be a counterfactual update from the same previous belief wt−1w_t-1 but from the “unknown” response; then ztU=+1⟺‖wtcf−wt−1‖2≤‖wt−wt−1‖2,z_t^U=+1 \|w_t^cf-w_t-1\|_2≤\|w_t-w_t-1\|_2, and ztU=−1z_t^U=-1 otherwise. FloDial. AS critique. User feedback takes the form Yes/No when the query matches a reference diagnostic item, and Unknown otherwise. We set ztQ=+1,if feedback is Yes,0,if feedback is No,−1,if feedback is Unknown (no match).z_t^Q= cases+1,&if feedback is Yes,\\ 0,&if feedback is No,\\ -1,&if feedback is Unknown (no match). cases BT critique. Let wt∈[0,1]Sw_t∈[0,1]^S be the belief over candidate faults and s⋆s the ground-truth fault. When the feedback is informative (Yes/No), we encourage increasing confidence on s⋆s : ztU=+1⟺wt(s⋆)>wt−1(s⋆),ztU=−1otherwise.z_t^U=+1\ \ w_t(s )>w_t-1(s ), z_t^U=-1\ otherwise. When the feedback is uninformative (Unknown), we use an invariance-style rule: we require the belief to remain unchanged (e.g., ztU=+1z_t^U=+1 iff wt=wt−1w_t=w_t-1). C.3.2 Training Configurations All expriments are trained on a single node with 8 B200 GPUs, based on the implementations of Verl (Sheng et al., 2025). All training tasks on PPO are conducted for 200 steps (GRPO and GSPO 100 steps) with the actor model optimized using a learning rate of 1.0×10−61.0× 10^-6. For distributed training, we adopt Fully Sharded Data Parallelism (FSDP), using BFloat16 precision throughout both training and evaluation. For efficient LLM rollouts, we adopt vLLM 333https://docs.vllm.ai/en/latest/ with a tensor parallel size of 1. The rollout sampling uses a temperature of 1.0 for all datasets. For the PPO baseline, we use Generalized Advantage Estimation (GAE) with parameters λ=1λ=1 and γ=1γ=1. The clip ratio ε are set to 0.2. For GRPO training, we sample 3 responses per prompt, and the rollout parameters with the clip ratio are consistent with the PPO setting. For the GSPO algorithm, the clip ratio εlow _low and εhigh _high are set to 0.0003 and 0.0004, respectively, while others keep consistent with GRPO training. Input Prompts for the MovieRec Preference Estimation Dataset You are an interactive preference estimation agent. The goal is to infer a user’s len_attributes-dimensional hidden preference vector on movies through multi-round interaction. ## Setup: - You are given len_seen movies, each with scores on len_attributes dimensions (indexed 1…len_attributes1… [rgb]0,0.08,0.45 [named]pgfstrokecolorrgb0,0.08,0.45 \len\_attributes\): seen_movie_sample - Maintain an estimate of the user’s preference vector: Guess: w1,w2,…,wlen_attributesw_1,w_2,…,w_ [rgb]0,0.08,0.45 [named]pgfstrokecolorrgb0,0.08,0.45 \len\_attributes\ - Initialization: Guess: 0.5,0.5,…,0.50.5,0.5,…,0.5 ## Interaction Protocol: Interaction alternates between two types of rounds. 1) Action Round (odd-numbered rounds: 1,3,5,…) In an Action Round, you must choose which information to query. You must output exactly: <interact> Focus: k1,k2,k3 Pair: p1,p2 </interact> Rules: - Focus must contain exactly three distinct integers in [1,len_attributes][1, [rgb]0,0.08,0.45 [named]pgfstrokecolorrgb0,0.08,0.45 \len\_attributes\]. Ensure that all dimensions receive opportunities to be disambiguated over the course of interaction. - Pair must contain exactly two distinct integers in [1,len_seen][1, [rgb]0,0.08,0.45 [named]pgfstrokecolorrgb0,0.08,0.45 \len\_seen\]. - Avoid uninformative pairs, such as dominance pairs where one movie is better on all focused dimensions k1,k2,k3k_1,k_2,k_3. After you output Focus and Pair, the user will provide feedback: - User Feedback: “Yes”, “No”, or “Equal”. “Yes” means p1p_1 is preferred when considering only dimensions k1,k2,k3k_1,k_2,k_3. 2) Update Round (even-numbered rounds: 2,4,6,…) In an Update Round, you must update the preference estimate based on the most recent Focus, Pair, and User Feedback. You must output exactly: <interact> Guess: w1,w2,…,wlen_attributes </interact> Rules: - Guess must be comma-separated numbers. - Use the feedback to adjust the relative importance of the focused dimensions in a way consistent with the observed preference. ## Reasoning: Before each <interact> block, you may briefly reason about what to do in a: <scratch>... </scratch> Round 1 is an Action Round. Figure 6: Prompt Template for MovieRec Preference Estimation. Input Prompts for the MediQ Dataset You are an interactive medical inference agent. Your goal is to iteratively maintain and refine a 4-dimensional state vector that tracks the relative support for four potential hypotheses associated with a given medical question, through multi-round interaction. ## Setup: - You are given: (i) a clinical vignette describing a patient scenario, and (i) a medical question associated with this scenario. - The question admits four potential hypotheses, labeled A, B, C, and D. - Maintain a state vector (wA,wB,wC,wD)(w_A,w_B,w_C,w_D), where each w∈[0,1]w∈[0,1], representing your current level of support for each hypothesis. - Initialization: Guess: 0.5,0.5,0.5,0.50.5,0.5,0.5,0.5 - In each Action round, you can issue only one query. The query must be a single, atomic question intended to reduce uncertainty among the four hypotheses; compound or multi-part questions are not allowed. ## Alternating Interaction Protocol: Interaction alternates between two types of rounds: 1) Action Round (odd-numbered rounds: 1,3,5,…) - You must output only a query in the exact format: <interact> Query: … </interact> - After you output a Query, the user will provide feedback. 2) Update Round (even-numbered rounds: 2,4,6,…) - You must update and output only the state vector (Guess) based on: (a) the most recent Query, and (b) the feedback returned for that Query. - Output in the exact format: <interact> Guess: wA,wB,wC,wD </interact> - Guess must be comma-separated numbers. ## State Update Rules (in Update rounds, after feedback): - Each dimension can be adjusted independently, depending on how the feedback affects that hypothesis. ## Query Policy (in Action rounds): - Ask atomic, clinically meaningful queries that help differentiate among the four hypotheses. - Avoid repeating previously asked queries. Before each <interact> block, briefly reason (in a few sentences) about what you should do in a: <scratch>... </scratch> block, following the protocol above. Let’s get started: Clinical Vignette: clinical_vignette Medical Question: medical_question Potential Hypotheses: potential_hypotheses Round 1 is an Action Round. Output your first Query. Figure 7: Prompt Template for MediQ. Input Prompts for the FloDial Dataset You are an interactive troubleshooting diagnosis agent. Your goal is to iteratively maintain and refine a state vector that tracks the relative plausibility of num_candidates candidate descriptions for a given problem, through multi-round interaction. ## Setup: - You are given: (i) a task description describing the user’s problem, and (i) a list of num_candidates candidate descriptions. - Each candidate is a possible explanation, diagnosis, or resolution suggestion related to the problem. - Exactly one of the candidate descriptions best matches the actual situation described by the user. - Your goal is to identify which candidate description is the most consistent with the situation, by asking diagnostic yes/no questions. - Maintain a state vector (w1,w2,…,wnum_candidates)(w_1,w_2,…,w_ [rgb]0,0.08,0.45 [named]pgfstrokecolorrgb0,0.08,0.45 \num\_candidates\), where each wi∈[0,1]w_i∈[0,1], representing your current level of support for candidate i. - Initialization: Guess: 0.5,0.5,…,0.50.5,0.5,…,0.5 (length num_candidates) - In each Action Round, you issue only one query. ## Alternating Interaction Protocol: Interaction alternates between two types of rounds: 1) Action Round (odd-numbered rounds: 1,3,5,…) - Output only a query in the exact format: <interact> Query: … </interact> - The query must be a single, atomic yes/no diagnostic question intended to reduce uncertainty among the num_candidates candidate descriptions. - Avoid repeating previously asked queries. - Do not directly ask whether a specific candidate description is correct or incorrect. - Compound or multi-part questions are not allowed. 2) Update Round (even-numbered rounds: 2,4,6,…) - Update and output only the state vector (Guess), based on the most recent query and the feedback returned for that query. - Output in the exact format: <interact> Guess: w1,w2,…,wnum_candidates </interact> - Each wiw_i must remain within [0,1][0,1]. - If the user replies “Unknown”, leave all weights unchanged. - Each dimension can be adjusted independently by 0 (not changed), +0.1+0.1, or −0.1-0.1, depending on how the feedback affects that candidate. Before each <interact> block, briefly reason (in a few sentences) in a <scratch>... </scratch> block about what to ask or how the feedback affects your belief. Let’s get started. Task Description: task_description Candidate Descriptions: candidate_descriptions Round 1 is an Action Round. Output your first query. Figure 8: Prompt Template for FloDial.