Paper deep dive
Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents
Yanze Jiang, Mingxuan Li, Yuhao Wang, Shengfang Zhai, Jiaheng Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/24/2026, 5:55:09 AM
Summary
The paper introduces Comparison-Only Tiny Advisor (COTA), a framework for constructive runtime intervention in Large Language Model (LLM) agents. COTA uses a tiny comparator model to judge whether sampled alternative actions lead to better continuations than the actor's current proposal, rather than generating corrections itself. This approach allows for effective intervention without retraining the actor or requiring a powerful expert solver, improving performance across WebShop, ALFWorld, and tau^3-Retail benchmarks.
Entities (12)
Relation Signals (11)
COTA → evaluatedon → tau^3-Retail
confidence 99% · We evaluate COTA on WebShop, ALFWorld, and tau^3-Retail
COTA → evaluatedon → WebShop
confidence 99% · We evaluate COTA on WebShop, ALFWorld, and tau^3-Retail
COTA → evaluatedon → ALFWorld
confidence 99% · We evaluate COTA on WebShop, ALFWorld, and tau^3-Retail
COTA → usesmodel → Qwen2.5-0.5B-Instruct
confidence 97% · the tiny comparator in COTA is a full-parameter fine-tuned Qwen2.5-0.5B-Instruct comparator.
COTA → appliesto → DeepSeek-v4-Flash
confidence 95% · We evaluate DeepSeek-V4-Flash... as LLM agent actors... COTA improves the actor in every evaluated setting
COTA → appliesto → Qwen3-8b
confidence 95% · We evaluate Qwen3-8B... as LLM agent actors... COTA improves the actor in every evaluated setting
COTA → appliesto → Qwen3.6-35B-A3B
confidence 95% · We evaluate Qwen3.6-35B-A3B... as LLM agent actors... COTA improves the actor in every evaluated setting
COTA → uses → tiny comparator
confidence 95% · In COTA, a tiny comparator judges whether sampled alternatives lead to better continuations than the actor's proposal
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Failure detection alone is insufficient. Effective intervention must also provide a useful direction for recovery. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention. In COTA, a tiny comparator judges whether sampled alternatives lead to better continuations than the actor's proposal, and repeated comparisons determine when intervention is warranted. We train the comparator using pairwise supervision constructed from same-prefix counterfactual branches. Preferred alternatives are returned as non-binding advice, leaving the original actor to replan. Across WebShop, ALFWorld, and tau^3-Retail with three actors, COTA improves all nine evaluation settings and outperforms the compared baselines. These results show that constructive runtime intervention can remain effective even when the auxiliary model has substantially weaker task-solving capability than the actor.
Tags
Links
- Source: https://arxiv.org/abs/2608.21027v1
- Canonical: https://arxiv.org/abs/2608.21027v1
Trouble viewing inline? Open PDF directly →
Full Text
76,016 characters extracted from source content.
Expand or collapse full text
Don’t Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents Yanze Jiang Thanks: Equal contribution. Mingxuan Li11footnotemark: 1 Yuhao Wang Shengfang Zhai Thanks: Corresponding authors. Jiaheng Zhang22footnotemark: 2 Affiliation: National University of Singapore Affiliation: yanzejiang, e1553202, wangyuhao@u.nus.edu Affiliation: shengfang.zhai, jhzhang@nus.edu.sg Abstract LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Failure detection alone is insufficient. Effective intervention must also provide a useful direction for recovery. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention. In COTA, a tiny comparator judges whether sampled alternatives lead to better continuations than the actor’s proposal, and repeated comparisons determine when intervention is warranted. We train the comparator using pairwise supervision constructed from same-prefix counterfactual branches. Preferred alternatives are returned as non-binding advice, leaving the original actor to re-plan. Across WebShop, ALFWorld, and τ3τ^3-Retail with three actors, COTA improves all nine evaluation settings and outperforms the compared baselines. These results show that constructive runtime intervention can remain effective even when the auxiliary model has substantially weaker task-solving capability than the actor. 1 Introduction Large language model (LLM) agents are increasingly used for long-horizon tasks that require sequential reasoning, tool use, and interaction with external environments (47; 30). In such settings, decisions are consequential: a locally poor action can alter the environment, consume a limited interaction budget, and make subsequent recovery substantially harder. Runtime intervention offers a complementary way to improve reliability without retraining or replacing the underlying actor, by monitoring an ongoing trajectory and redirecting execution when needed (36; 48; 15). Yet deciding when to intervene is itself nontrivial. Recent work shows that even accurate failure predictors can reduce end-to-end performance by disrupting trajectories that would otherwise succeed (36), and argues that effective runtime control should depend on whether an available intervention improves the downstream outcome, rather than on continuation risk alone (48). Thus, useful intervention requires not only detecting problematic decisions, but also providing a recovery signal that actually improves what the agent does next. Existing approaches obtain constructive interventions by placing substantial task-solving functionality in the intervention pathway. One option is to hand control to a stronger expert when the current trajectory appears problematic (48; 24; 29). Another is to keep the original actor in control while using a separate critic to inspect its proposal and generate task-specific corrective feedback (15). Both approaches introduce a redundancy: the actor is already responsible for solving the task, yet the intervention component must again possess sufficient task understanding to produce a useful correction. This redundancy has a practical cost: expert handoff requires another capable solver, while corrective critics become increasingly difficult to make lightweight (5; 24). This motivates the question we study in this work: Can constructive runtime intervention be achieved with a tiny auxiliary model that neither independently solves the task nor generates a correction? Doing so presents two challenges. First, constructive intervention must provide more than a binary warning: when the actor proposes a poor action, it should offer a useful direction for replanning (3; 12). Yet asking the tiny auxiliary model itself to discover that direction would reintroduce open-ended task solving (20; 6). Second, intervention should reflect the long-term consequence of the current decision rather than its surface plausibility (33; 26; 41). Learning accurate absolute values for arbitrary state–action pairs is particularly demanding in long-horizon agent tasks (2; 11; 25; 40), while comparing actions observed under different trajectories mixes the effect of the action with differences in history and continuation policy (23; 14; 35). Our key observation is that comparison provides the weaker primitive we need: an action need not be certified as globally good to determine that the actor’s current proposal is locally weak. If the proposed action is repeatedly outperformed by plausible executable alternative actions, then the proposed action is likely a poor local choice, even when none of those alternative actions is itself optimal. Based on this observation, we propose Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention. At each decision point, we sample a small set of executable alternatives, and a tiny comparator performs only one primitive: predicting which of two actions from the same state leads to the better continuation under the frozen actor. Repeated pairwise judgments estimate how frequently the actor’s proposal is dominated by its alternatives; when this candidate-relative domination is sufficiently strong, the winning alternatives are returned as non-binding advice and the actor replans before execution. To train the comparator, we use same-prefix counterfactual branching, inspired by 48. Starting from an identical environment state, we vary only the branch-point action and then return control to the same frozen actor. We use the resulting sibling trajectories to construct actor-conditioned pairwise supervision that directly matches the comparison required by the runtime gate. The comparator therefore neither generates corrective actions nor predicts absolute action values; planning, generation, and execution remain entirely with the original actor. We evaluate COTA on WebShop (45), ALFWorld (31), and τ3τ^3-Retail (46), using Qwen3-8B, Qwen3.6-35B-A3B, and DeepSeek-V4-Flash as LLM agent actors, yielding nine evaluation combinations. Despite using only a 0.5B comparator, COTA improves the actor in every evaluated setting and achieves the strongest overall performance among the compared intervention baselines. The gains persist for substantially stronger actors, showing that effective constructive intervention need not rely on another task solver: a tiny model trained only for local comparison can guide actors far beyond its own task-solving capacity while leaving replanning to the actor. Our contributions are threefold: • We formulate constructive runtime intervention and show that it can be reduced from open-ended correction to a candidate-relative comparison problem, allowing useful intervention without requiring the auxiliary model to independently solve the task. • We propose COTA, a comparison-only framework that aggregates pairwise judgments through a statistically interpretable Monte Carlo gate. We further construct reliable pairwise supervision from same-prefix counterfactual trajectories, directly matching the comparison required at runtime. • We evaluate COTA across three interactive environments and three actor families. A 0.5B comparator improves all nine actor–environment combinations, including substantially stronger Qwen3.6-35B-A3B and DeepSeek-V4-Flash actors, while introducing only modest online overhead. 2 Problem Formulation Agent trajectory. Following the standard formulation of interactive language agents (47) and sequential decision making under partial observability (17), we consider a frozen LLM agent π interacting with an environment to accomplish a task x. We use actor to refer to its frozen LLM policy π, which produces reasoning and environment actions. At step t, the agent observes oto_t, produces a reasoning trace or plan ptp_t, and proposes an environment action ata_t. If ata_t is executed, the environment returns the next observation ot+1o_t+1. We denote the trajectory prefix before executing ata_t as: τt=(o0,p0,a0,…,ot−1,pt−1,at−1,ot,pt), _t=(o_0,p_0,a_0,…,o_t-1,p_t-1,a_t-1,o_t,p_t), (1) and let st=(x,τt)s_t=(x, _t) denote the actor-visible decision state. Thus, the proposed action is sampled from at∼π(⋅∣st)a_t π(· s_t). Here sts_t refers to the information available to the agent at decision time, rather than necessarily the latent state of the environment. Runtime intervention. We consider intervention at the pre-execution stage: the agent has already proposed ata_t, but the action has not yet been sent to the environment. This form of intervention follows the general human-intervention and shielding paradigm, in which an external mechanism monitors a proposed action and may prevent or modify it before execution (28; 1). Let gt∈0,1g_t∈\0,1\ denote the intervention decision, where gt=0g_t=0 executes the original proposal and gt=1g_t=1 withholds it. We refer to this general operation as runtime intervention (37; 7; 38). At this level of abstraction, we leave unspecified how the system responds after withholding an action. To characterize the consequence of an action, we use the standard action-value notion from reinforcement learning (39; 32). For any action a available at sts_t, define its value under the frozen actor π as: Qπ(st,a)=[R(τ)∣st,;at=a,;πthereafter],Q^π(s_t,a)=E [R(τ) s_t,;a_t=a,;π\ thereafter ], (2) where R(τ)R(τ) is the return of the completed trajectory. Qπ(st,a)Q^π(s_t,a) measures the downstream consequence of taking a at the current decision point and then returning control to the same actor. Constructive runtime intervention. Standard runtime intervention determines whether the current proposal should be executed, but need not specify how the actor should recover when it is rejected. We call an intervention constructive when, upon withholding ata_t, it additionally provides an advice action atadva_t^adv to the same actor. The advice is non-binding: after receiving it, the actor replans and produces a new proposal at′a_t . We say the intervention is constructive when the advice offers a genuinely better local direction, i.e., Qπ(st,atadv)>Qπ(st,at).Q^π(s_t,a_t^adv)>Q^π(s_t,a_t). The desired end-to-end effect is that the actor absorbs this information and replans to a better action, i.e., Qπ(st,at′)>Qπ(st,at).Q^π(s_t,a_t )>Q^π(s_t,a_t). We call the auxiliary module that performs this advice-producing intervention an advisor. Our goal is to realize such constructive intervention while removing task-solving capability from the advisor. 3 Methodology Directly estimating whether an action is “good” through its absolute continuation value Qπ(st,at)Q^π(s_t,a_t) is unnecessarily difficult for constructive runtime intervention. It is sufficient to determine whether the actor’s current proposal ata_t is poor compared with plausible alternatives available at the same decision point. Let μ(⋅∣st)μ(· s_t) denote a reference distribution over executable candidate actions. We define the candidate-relative domination rate: ρμ(st,at)=PrA∼μ(⋅∣st)(Qπ(st,A)>Qπ(st,at)). _μ(s_t,a_t)= _A μ(· s_t) (Q^π(s_t,A)>Q^π(s_t,a_t) ). (3) This quantity has a simple rank interpretation. If ρμ(st,at)=0.1 _μ(s_t,a_t)=0.1, only 10%10\% of candidates sampled from μ outperform the proposal; if ρμ(st,at)=0.8 _μ(s_t,a_t)=0.8, the proposal is dominated by 80%80\% of the candidate distribution. Equivalently, 1−ρμ1- _μ gives the proposal’s quantile in the candidate-value distribution, up to ties. Importantly, μ need not be an expert policy or contain the globally optimal action. If ρμ _μ is small, the proposal already compares favorably with ordinary alternatives; conversely, a large ρμ _μ provides evidence that even ordinary alternatives frequently improve upon it. Moreover, the alternatives that defeat the proposal naturally serve as advice for replanning. Constructive runtime intervention can therefore be reduced to a much narrower primitive: given the same state, which of two actions leads to the better continuation under the same actor? 3.1 Comparison-Only Tiny Advisor Based on this statistical insight, we propose COTA, a runtime intervention framework in which the learned advisor performs only pairwise action comparison, while candidate generation, replanning, and environment interaction remain with the frozen actor. At step t, the actor first proposes at∼π(⋅∣st)a_t π(· s_t). We then obtain K executable alternatives from the reference candidate mechanism, i.e., t=A1,…,AKA_t=\A_1,…,A_K\, Ai∼μ(⋅∣st).A_i μ(· s_t). The framework is agnostic to how μ is constructed; we describe the candidate mechanisms used in our experiments in Section 4.2. A tiny comparator CθC_θ judges each candidate only relative to the actor’s proposal. Its target is: Cθ(st,aA,aB)≈[Qπ(st,aA)>Qπ(st,aB)].C_θ(s_t,a_A,a_B) [Q^π(s_t,a_A)>Q^π(s_t,a_B) ]. (4) Thus, for each AiA_i, we define: B~i=Cθ(st,Ai,at)∈0,1, B_i=C_θ(s_t,A_i,a_t)∈\0,1\, (5) where B~i=1 B_i=1 indicates that AiA_i is predicted to outperform the current proposal. COTA rejects ata_t when at least R of the K alternatives defeat it: ∑i=1KB~i≥R. _i=1^K B_i≥ R. (6) Otherwise, ata_t is executed unchanged. When the gate fires, the predicted winners are ranked using the same comparator, and the highest-ranked candidate atadva_t^adv is returned to the actor as non-binding advice. The actor then replans from the same decision state conditioned on this advice and produces a new proposal at′a_t , which is reviewed before execution. The learned component therefore performs neither action generation nor absolute value estimation. Its only task is the local comparison in Eq. 4; the stronger actor remains responsible for planning and execution. 3.2 Comparator Training Training CθC_θ requires supervision for the relative continuation quality of two actions taken from the same state. Returns from unrelated trajectories are unsuitable for this purpose because they conflate the effect of the action with differences in preceding histories and subsequent behavior. We therefore construct training data using same-prefix counterfactual branches, following the classical idea of Monte-Carlo rollout evaluation from a shared decision state (34; 4). At a sampled decision state sts_t, we restore the same environment state, execute a branch-point action a, and then return control to the same frozen actor π. Let Y(st,a)=R(τ)Y(s_t,a)=R(τ), τ∼P(⋅∣st,a,π)τ P(· s_t,a,π) denote the resulting branch return. By the definition of the actor-conditioned action value in Eq. 2: [Y(st,a)]=Qπ(st,a).E\! [Y(s_t,a) ]=Q^π(s_t,a). (7) Hence, Q^M(st,a)=1M∑m=1MYm(st,a) Q_M(s_t,a)= 1M _m=1^MY_m(s_t,a) is a Monte Carlo estimate of Qπ(st,a)Q^π(s_t,a) with M independent continuations. For sibling actions aAa_A and aBa_B, their empirical branch returns directly provide pairwise supervision: B^(st,aA,aB)=[Q^M(st,aA)>Q^M(st,aB)]. B(s_t,a_A,a_B)=I [ Q_M(s_t,a_A)> Q_M(s_t,a_B) ]. (8) We fine-tune CθC_θ to predict these pairwise labels from the actor-visible state and the two candidate actions. Because both branches share the same prefix and continuation actor, the comparison isolates the downstream consequence of changing the branch-point action. Details of label construction and comparator implementation are provided in Section 4.2. 3.3 Statistical Accuracy The winner-count rule in Eq. 6 can be interpreted as a Monte Carlo estimate of the candidate-relative domination rate in Eq. 3. If the true pairwise relation were observable, define Bi=[Qπ(st,Ai)>Qπ(st,at)].B_i=I [Q^π(s_t,A_i)>Q^π(s_t,a_t) ]. Then: ρ^K=1K∑i=1KBi,[ρ^K]=ρμ(st,at). ρ_K= 1K _i=1^KB_i, \! [ ρ_K ]= _μ(s_t,a_t). (9) Thus, requiring R winners corresponds to testing whether the proposal is dominated by at least an R/KR/K fraction of the candidate distribution. At deployment, we replace the oracle comparisons BiB_i with the learned predictions B~i B_i: ρ^θ,K=1K∑i=1KB~i,gθ=[ρ^θ,K≥RK]. ρ_θ,K= 1K _i=1^K B_i, g_θ=I [ ρ_θ,K≥ RK ]. (10) Let ϵθ(st,at)=PrA∼μ(⋅∣st)(Cθ(st,A,at)≠[Qπ(st,A)>Qπ(st,at)]) _θ(s_t,a_t)= _A μ(· s_t) (C_θ(s_t,A,a_t) [Q^π(s_t,A)>Q^π(s_t,a_t) ] ) denote the comparator’s pairwise error under the candidate distribution. The estimation error then separates naturally into comparator error and finite-candidate sampling error. Proposition 3.1 (Accuracy of candidate-relative estimation). For any fixed (st,at)(s_t,a_t), assuming that the K candidates are sampled either independently from μ(⋅∣st)μ(· s_t), or uniformly without replacement from a finite candidate pool whose empirical distribution defines μ(⋅∣st)μ(· s_t): |[ρ^θ,K]−ρμ(st,at)|≤ϵθ(st,at). |E\! [ ρ_θ,K ]- _μ(s_t,a_t) |≤ _θ(s_t,a_t). (11) Moreover, with probability at least 1−δ1-δ, we have: |ρ^θ,K−ρμ(st,at)|≤ϵθ(st,at)+log(2/δ)2K. | ρ_θ,K- _μ(s_t,a_t) |≤ _θ(s_t,a_t)+ (2/δ)2K. (12) The first term captures errors made by the learned comparator, while the second is the standard Monte Carlo error from using only K candidates. Consequently, when the true domination rate is sufficiently far from the threshold R/KR/K, the learned gate gθ=[ρ^θ,K≥R/K]g_θ=I\! [ ρ_θ,K≥ R/K ] makes the same intervention decision as the oracle candidate-relative gate [ρ(st,at0)≥R/K]I[ρ(s_t,a_t^0)≥ R/K] with probability at least 1−δ1-δ. We provide the detailed proof and analysis in Appendix A. When intervention occurs, the same comparisons that establish that ata_t is weak also identify alternatives predicted to have higher continuation value. COTA therefore obtains both components of constructive runtime intervention—when to interrupt and what direction to expose—from the same comparison primitive. 4 Experiments We evaluate whether COTA can realize constructive runtime intervention with a tiny comparator, improving substantially larger actors. Our experiments address three questions: (i) can a tiny comparison-only advisor consistently improve actors of different scales across diverse interactive environments; (i) how do the comparison objective and constructive intervention mechanism contribute to the overall performance; and (i) can the candidate mechanism introduce additional action diversity while preserving the actor’s freedom to replan? Complete prompts, data construction, hyperparameters, implementation details, and cost accounting are provided in Appendix B. 4.1 Setups Models. All agents in our experiments follow a ReAct-style interaction loop that alternates a textual rationale, one environment action, and the resulting observation. We evaluate Qwen3-8B 11 1 https://huggingface.co/Qwen/Qwen3-8B (42), Qwen3.6-35B-A3B 22 2 https://huggingface.co/Qwen/Qwen3.6-35B-A3B (27), and DeepSeek-V4-Flash (284B parameters) (9). Unless stated otherwise, the tiny comparator in COTA is a full-parameter fine-tuned Qwen2.5-0.5B-Instruct 33 3 https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct (43) comparator. The actor itself remains frozen in every condition. Benchmarks. We study three environments with complementary action interfaces. (1) WebShop requires multi-step product search and purchasing under natural-language constraints, and provides partial reward in [0,1][0,1] (45). We collect branch supervision from tasks 0–4999 and evaluate on the disjoint tasks 5000–5499. (2) ALFWorld evaluates compositional household tasks in a text-based embodied environment (31); we use its standard training split for data collection and the valid-unseen split for evaluation. (3) τ3τ^3-Retail tests policy-constrained customer-service dialogue with state-changing tools and a simulated user (46); we retain the benchmark’s official train/test partition. Splits are always made at the task level, so sibling branches from one task never cross train, validation, and test sets. Metrics. WebShop reports mean reward as its primary metric. ALFWorld and τ3τ^3-Retail report success rate. Qwen actors are evaluated once on each of 500 WebShop tasks and 134 ALFWorld valid-unseen games. Retail uses all 40 test tasks with three seeds, giving 120 episodes per system. To control closed-source model cost, DeepSeek-V4-Flash uses fixed subsets of the same held-out WebShop and ALFWorld tasks, shared by all compared methods, while retaining the full three-seed Retail protocol. Baselines. We organize baselines by how they spend inference-time computation. (1) Selection methods score sampled alternatives and execute the selected action; our main representative is AgentPRM-style absolute-Q scoring with forced top-1 control (8). (2) Test-time deliberation is represented by Self-Reflection, which gives the actor one conservative opportunity to inspect and revise its own proposal (30). (3) Runtime intervention is represented by Asym-AC, where a separate critic produces free-form feedback before the actor replans (15). Unlike forced selection, our COTA returns non-binding preferred actions; the original actor decides what to execute after replanning. Prompt templates and benchmark adaptations appear in Appendix B.7. 4.2 Implementation Details. Prefix-branch supervision. We first complete a base trajectory, then restore selected prefixes and vary only the first branch action before handing control back to the same continuation actor. Branches are one level deep and are labeled solely by downstream environment outcomes. The branch sampling policy, and offline generation cost are deferred to Appendix B.2 and Appendix B.10. Comparator implementation. Each training example consists of the actor-visible state and an ordered pair of actions (aA,aB)(a_A,a_B), with the target being exactly one of A, B, or T, indicating that aAa_A is preferred, aBa_B is preferred, or the two are indistinguishable, respectively. Each physical action pair is included in both A/B orders. After preprocessing, this yields 55k, 23k, and 9k supervision examples for WebShop, ALFWorld, and τ3τ^3-Retail, respectively. At deployment, COTA accepts a winner only when the predictions under both input orders are semantically consistent. The comparator never observes actor-private thoughts, branch outcomes, or future states. Details of the A/B/T construction are provided in Appendix A.3, with full optimization and model-selection details in Appendix B.4. Intervention budget. At each reviewed step, K is the maximum number of alternatives and R is the minimum number that must consistently beat the proposal before intervention. We use K=4,R=1K=4,R=1 on WebShop and ALFWorld and K=4,R=2K=4,R=2 on τ3τ^3-Retail. We select these operating points using end-to-end performance on held-out validation tasks, rather than comparator accuracy alone, and fix them for all test actors. The validation sweep is reported in Appendix B.5. Candidate-action distributions. We instantiate tA_t in three ways: (i) environment candidates sample legal actions exposed by the environment, (i) offline candidates retrieve grounded actions collected at analogous branch states, and (i) small-LM candidates are proposed online by a compact generator—Qwen2.5-0.5B-Instruct—and then canonicalized and validity-filtered. WebShop primarily uses environment actions; ALFWorld uses environment actions, optionally augmented by a small LM for stronger actors; and Retail combines small-LM proposals with grounded offline backfill. For environment-only candidates, we sample uniformly without replacement from the available executable-action pool. When multiple candidate sources are used, each candidate is sampled independently from a fixed mixture distribution over the corresponding sources. The comparator architecture and gate are unchanged across these sources. Construction details are deferred to Appendix B.6. 4.3 Main Results Table 1: Comparison with baselines. Performance is mean reward on WebShop and success rate on ALFWorld and τ3τ^3-Retail. Avg. T is cumulative episode time normalized by the paired original actor. Origin COTA Self-Reflection AgentPRM Asym-AC Actor Environment Perf. Avg. T Perf. Avg. T Perf. Avg. T Perf. Avg. T Perf. Avg. T Qwen3-8B WebShop 0.3960 1.000× 0.5630 1.413× 0.2610 2.674× 0.3153 0.454× 0.1785 5.177× ALFWorld 82.84% 1.000× 90.30% 1.138× 75.37% 1.145× 2.24% 1.462× 35.82% 2.323× τ3τ^3-Retail 37.50% 1.000× 45.00% 2.024× 32.50% 1.626× 4.17% 1.811× 43.33% 1.934× Qwen3.6-35B-A3B WebShop 0.5662 1.000× 0.6813 1.451× 0.5546 2.325× 0.4189 0.583× 0.3454 5.476× ALFWorld 85.07% 1.000× 94.03% 1.302× 79.85% 1.767× 8.96% 1.664× 71.64% 3.640× τ3τ^3-Retail 62.50% 1.000× 65.00% 1.779× 55.00% 1.894× 5.00% 2.886× 61.67% 1.463× DeepSeek-V4-Flash WebShop 0.6085 1.000× 0.6867 1.009× 0.4333 2.732× 0.5128 0.232× 0.2786 5.229× ALFWorld 90.00% 1.000× 95.00% 0.873× 82.50% 1.234× 10.00% 1.484× 75.00% 2.115× τ3τ^3-Retail 80.83% 1.000× 82.50% 1.472× 67.50% 2.352× 3.33% 1.810× 70.00% 1.308× Performance. Table 1 shows that COTA achieves the best performance in all nine actor–environment settings and consistently improves the corresponding original actor. For Qwen3-8B, it raises WebShop reward from 0.39600.3960 to 0.56300.5630, ALFWorld success from 82.84%82.84\% to 90.30%90.30\%, and τ3τ^3-Retail success from 37.50%37.50\% to 45.00%45.00\%. The gains persist for the substantially stronger Qwen3.6-35B-A3B and DeepSeek-V4-Flash actors. This is important because the same 0.5B comparator is not merely compensating for a weak actor: it remains useful even when the proposal is produced by a substantially stronger model. Our analysis further shows that even strong actors occasionally make clearly suboptimal local decisions, which COTA can identify by design. In contrast, Self-Reflection degrades all nine settings, consistent with prior findings that intrinsic self-correction can fail to escape the model’s own erroneous reasoning without an external feedback signal, and may even degrade an initially better solution (13; 18). Asym-AC remains below COTA across all settings, suggesting that shrinking the intervention model while retaining a task-solving correction objective is difficult: producing a useful correction still requires the critic to understand the current trajectory and determine how the task should proceed (21; 12). AgentPRM degrades most severely on ALFWorld and τ3τ^3-Retail, where sparse binary returns make absolute action-value estimation particularly challenging for a 0.5B model. Overall, these results demonstrate that COTA can provide effective constructive runtime intervention despite having substantially weaker task-solving capability than the actor, without any task-solving fine-tuning. Our central design is thus demonstrated: narrowing the learned task to local comparison enables effective constructive intervention while leaving task-level replanning to the stronger actor. Efficiency. COTA incurs modest online overhead, averaging 1.38×1.38× the end-to-end episode time of the original actor, with seven of nine settings below 1.5×1.5×. The larger overheads occur on τ3τ^3-Retail, where candidate generation additionally invokes a small LLM. Avg. T can occasionally fall below 1×1× because useful intervention may reduce the number of interaction steps required to complete the task. Detailed token-level and offline costs are reported in Appendix B.10. 4.4 Ablation Study We ablate two design choices in COTA: the learning objective, comparing absolute Qπ(s,a)Q^π(s,a) estimation with our same-prefix pairwise comparison, and the intervention mechanism, comparing forced action execution with constructive runtime intervention as defined in Section 2. This yields a 2×22× 2 ablation, which we evaluate with Qwen3-8B and Qwen3.6-35B-A3B across all three environments. Table 2: Ablation of learning objective and intervention mechanism. WebShop reports mean reward; ALFWorld and τ3τ^3-Retail report success rate (%). “Constructive” denotes constructive runtime intervention: the proposed action is withheld only when intervention is triggered, a preferred alternative is returned as advice, and the actor replans before execution. WebShop ALFWorld τ3τ^3-Retail Objective Intervention Qwen3 Qwen3.6 Qwen3 Qwen3.6 Qwen3 Qwen3.6 Absolute Q Forced 0.3153 0.4189 2.24 8.96 4.17 5.00 Absolute Q Constructive 0.5491 0.6447 57.46 63.43 16.67 17.50 Pairwise comparison Forced 0.3782 0.6041 51.49 50.75 16.67 37.50 Pairwise comparison Constructive 0.5630 0.6813 90.30 94.03 45.00 65.00 Table 2 shows that both factors are important. First, directly executing the auxiliary model’s preferred action is consistently inferior to constructive intervention. Under absolute-Q supervision, replacing forced execution with constructive intervention raises Qwen3-8B from 2.24%2.24\% to 57.46%57.46\% on ALFWorld and from 4.17%4.17\% to 16.67%16.67\% on τ3τ^3-Retail, with the same trend for Qwen3.6. The pattern remains under pairwise supervision: forcing the predicted winner is substantially worse than returning it as advice and allowing the actor to replan, particularly on the two long-horizon environments. These results support the formulation in Section 2: a useful advisor should guide the stronger actor rather than directly take over its next action. Constructive intervention alone, however, does not account for the full gain. Holding the intervention mechanism fixed, replacing absolute-Q estimation with same-prefix pairwise comparison improves ALFWorld from 57.46%57.46\% to 90.30%90.30\% for Qwen3-8B and from 63.43%63.43\% to 94.03%94.03\% for Qwen3.6; on τ3τ^3-Retail, the corresponding gains are 16.67%→45.00%16.67\%→ 45.00\% and 17.50%→65.00%17.50\%→ 65.00\%. The pairwise objective directly matches the decision required by our advisor—whether an available alternative is better than the actor’s current proposal—without requiring the tiny model to estimate an absolute long-horizon value. The best performance therefore requires both components of COTA: a narrow comparison objective and constructive runtime intervention that delegates final replanning to the actor. 4.5 Action Diversity A central role of the candidate mechanism in COTA is to introduce local alternatives that the actor is unlikely to expose by repeated sampling alone, and then pass this additional diversity to the actor through non-binding advice. We therefore examine two questions: whether the candidate mechanism indeed broadens the local action distribution, and whether this diversity is transferred to the actor without forcing its replanning behavior toward the candidate distribution. Figure 1: Actor behavior after COTA intervention. Successor proposals are categorized as Adopt if they match a recommended action, Keep if they repeat the original proposal, and Novel otherwise. Actors frequently adopt recommendations while retaining substantial probability mass on Keep and Novel. Repeated actor sampling reveals a highly concentrated action distribution. With eight high-temperature samples, the actor produces only 2.28 distinct executable actions on average, corresponding to 28.5% unique-action utilization; for 150 of 300 prefixes, all eight samples collapse to the same canonical action. In contrast, our candidate mechanism produces 3.71 distinct actions from only four candidates, corresponding to 92.7% utilization. Thus, simply sampling the actor more often provides limited additional support, whereas the candidate mechanism exposes substantially more diverse local alternatives for intervention. This additional diversity is not confined to COTA’s candidate set, but is transmitted to the actor through intervention. As shown in Figure 1, actors adopt recommended actions with substantial probability across environments, demonstrating that candidate alternatives can alter the actor’s subsequent behavior. At the same time, replanning does not collapse toward the candidate distribution. For DeepSeek-V4-Flash, only 30.4%, 32.9%, and 29.2% of replans adopt a recommendation on WebShop, ALFWorld, and τ3τ^3-Retail, respectively, while 49.6%–63.6% retain the actor’s original proposal. Qwen3.6-35B-A3B further illustrates that this behavior is context-dependent: it adopts 67.8% of recommendations on WebShop but only 19.0% on τ3τ^3-Retail, where 51.4% of replans keep the original proposal and 29.5% produce a novel action. Together, these results show that COTA uses the candidate mechanism to inject otherwise-missing local diversity into the actor’s decision process, while preserving the actor’s freedom to accept, reject, or revise the advice before execution. 5 Related Work Runtime Intervention. Runtime intervention has a long history in safe sequential decision making. 28 formalize human intervention in reinforcement learning and train a supervised blocker to imitate human intervention decisions. 1 introduce shielding, where a reactive safety layer monitors the learner’s proposed action and corrects it when execution would violate a temporal-logic specification. More recently, agent-specific guardrails extend this idea to LLM agents. ShieldAgent (7) verifies action trajectories against explicit safety policies, while AgentSpec (37) intercepts agent executions and enforces user-defined runtime constraints. ProbGuard (38) further moves from reactive rules to proactive monitoring by estimating the probability of reaching unsafe states and triggering intervention before violations occur. Beyond safety enforcement, 36 show that accurate failure prediction alone does not guarantee beneficial intervention because corrections may also disrupt otherwise successful trajectories. 48 argue that runtime oversight should estimate the downstream advantage of available interventions rather than continuation risk alone. In this paper, we study how to realize constructive runtime intervention with a tiny model that neither solves the task nor generates task-specific corrections. Weak Critics for Strong Actors. A growing line of work leverages weaker models in specialized auxiliary roles to assist substantially stronger actors. 19 train small verifiers against stronger provers, showing that weak verification can shape strong-model outputs toward improved checkability and legibility. 22 propose Weak-for-Strong Harnessing (W4S), training a 7B meta-agent to optimize workflows that better harness fixed stronger executors such as GPT-4o. 10 introduce RAG-Critic, where a specialized 3B error critic provides fine-grained feedback to guide error-specific planning and improve RAG systems with backbones up to 70B. 44 develop Critique-Guided Improvement (CGI), training a specialized 8B critic to provide actionable natural-language feedback; the learned critic substantially improves actors including Llama-3-70B under critique-guided inference. 15 propose an asymmetric actor–critic framework in which an open-source critic monitors and intervenes on a fixed, stronger proprietary actor at runtime, improving task success and reliability. 16 formulate weak-critic strong oversight, showing that a weak critic need only provide useful revision directions, rather than solve or judge the full task, to improve a frozen stronger model, and further distill this improvement through on-policy critique distillation. Our work pushes this direction further by narrowing the required capability primitive of the auxiliary model from task solving or full critique to a non-task-solving local-comparison capability, enabling effective assistance with substantially smaller auxiliary models. 6 Conclusion We studied whether constructive runtime intervention requires an auxiliary model capable of solving the task itself. Our results show that it need not. COTA uses a 0.5B comparator trained on same-prefix branches to identify better local alternatives, while leaving generation, replanning, and execution to the stronger actor. Across all evaluated settings, COTA achieves the best performance. Ablations further support both design choices: pairwise comparison and actor-mediated replanning. More broadly, our results suggest that effective runtime intervention can come from narrowing the auxiliary model’s role from solving to comparing. References Alshiekh et al. (2018) M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, p. 2669–2678. Cited by: §2, §5. Arjona-Medina et al. (2019) J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter RUDDER: return decomposition for delayed rewards. In Advances in Neural Information Processing Systems, Vol. 32, p. 13544–13555. Cited by: §1. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Cited by: §1. Bertsekas and Castanon (1999) D. P. Bertsekas and D. A. Castanon Rollout algorithms for stochastic scheduling problems. Journal of Heuristics 5 (1), p. 89–108. Cited by: §3.2. Chen et al. (2024a) L. Chen, M. Zaharia, and J. Zou FrugalGPT: how to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §1. Chen et al. (2024b) X. Chen, M. Lin, N. Schärli, and D. Zhou Teaching large language models to self-debug. In International Conference on Learning Representations, Cited by: §1. Chen et al. (2025) Z. Chen, M. Kang, and B. Li ShieldAgent: shielding agents via verifiable safety policy reasoning. In International Conference on Machine Learning, Cited by: §2, §5. Choudhury (2025) S. Choudhury Process reward models for llm agents: practical framework and directions. arXiv preprint arXiv:2502.10325. Cited by: §B.7, §4.1. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §4.1. Dong et al. (2025) G. Dong, J. Jin, X. Li, Y. Zhu, Z. Dou, and J. Wen RAG-critic: leveraging automated critic-guided agentic workflow for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3551–3578. External Links: Document Cited by: §5. Gehring et al. (2022) C. Gehring, M. Asai, R. Chitnis, T. Silver, L. P. Kaelbling, S. Sohrabi, and M. Katz Reinforcement learning for classical planning: viewing heuristics as dense reward generators. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 32, p. 588–596. External Links: Document Cited by: §1. Gou et al. (2023) Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen Critic: large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738. Cited by: §1, §4.3. Huang et al. (2024) J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, Cited by: §4.3. Jiang and Li (2016) N. Jiang and L. Li Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 48, p. 652–661. Cited by: §1. Jiang et al. (2026) S. Jiang, Z. Zhang, Y. Zhang, S. Yang, W. Xia, and S. Soatto Asymmetric actor-critic for multi-turn llm agents. arXiv preprint arXiv:2604.00304. Cited by: §1, §1, §4.1, §5. Jin et al. (2026) C. Jin, J. Li, R. Wu, E. Zhang, and D. N. Metaxas Weak critics make strong learners: on-policy critique distillation for scalable oversight. In 3rd AI for Math Workshop: Toward Self-Evolving Scientific Agents, Cited by: §5. Kaelbling et al. (1998) L. P. Kaelbling, M. L. Littman, and A. R. Cassandra Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1–2), p. 99–134. External Links: Document Cited by: §2. Kamoi et al. (2024) R. Kamoi, Y. Zhang, N. Zhang, J. Han, and R. Zhang When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics 12, p. 1417–1440. External Links: Document Cited by: §4.3. Kirchner et al. (2024) J. H. Kirchner, Y. Chen, H. Edwards, J. Leike, N. McAleese, and Y. Burda Prover-verifier games improve legibility of llm outputs. arXiv preprint arXiv:2407.13692. Cited by: §5. Lin et al. (2024) Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, and Y. Yang CriticBench: benchmarking llms for critique-correct reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, p. 1552–1587. External Links: Document Cited by: §1. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, p. 46534–46594. Cited by: §4.3. Nie et al. (2025) F. Nie, L. Feng, H. Ye, W. Liang, P. Lu, H. Yao, A. Alahi, and J. Zou Weak-for-strong: training weak meta-agent to harness strong executors. In Proceedings of the Second Conference on Language Modeling, Cited by: §5. Oberst and Sontag (2019) M. Oberst and D. Sontag Counterfactual off-policy evaluation with gumbel-max structural causal models. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 4881–4890. Cited by: §1. Ong et al. (2025) I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica RouteLLM: learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Parisi et al. (2022) S. Parisi, D. Tateo, M. Hensel, C. D’Eramo, J. Peters, and J. Pajarinen Long-term visitation value for deep exploration in sparse-reward reinforcement learning. Algorithms 15 (3), p. 81. External Links: Document Cited by: §1. Pignatelli et al. (2024) E. Pignatelli, J. Ferret, M. Geist, T. Mesnard, H. van Hasselt, and L. Toni A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research. Cited by: §1. Qwen Team (2026) Qwen Team Qwen3.6-35B-A3B: agentic coding power, now open to all. External Links: Link Cited by: §4.1. Saunders et al. (2018) W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans Trial without error: towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 2067–2069. Cited by: §2, §5. Shen et al. (2024) Z. Shen, H. Lang, B. Wang, Y. Kim, and D. Sontag Learning to decode collaboratively with multiple language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 12974–12990. External Links: Document Cited by: §1. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, p. 8634–8652. Cited by: §1, §4.1. Shridhar et al. (2020) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht Alfworld: aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Cited by: §1, §4.1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2 edition, MIT Press, Cambridge, MA. Cited by: §2. Sutton (1988) R. S. Sutton Learning to predict by the methods of temporal differences. Machine Learning 3 (1), p. 9–44. Cited by: §1. Tesauro and Galperin (1996) G. Tesauro and G. R. Galperin On-line policy improvement using monte-carlo search. In Advances in Neural Information Processing Systems, Vol. 9. Cited by: §3.2. Thomas et al. (2015) P. S. Thomas, G. Theocharous, and M. Ghavamzadeh High-confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29. External Links: Document Cited by: §1. Vasudev et al. (2026) R. Vasudev, M. Russak, D. Bikel, and W. Alshikh Accurate failure prediction in agents does not imply effective failure prevention. arXiv preprint arXiv:2602.03338. Cited by: §1, §5. Wang et al. (2026a) H. Wang, C. M. Poskitt, and J. Sun AgentSpec: customizable runtime enforcement for safe and reliable llm agents. In Proceedings of the IEEE/ACM International Conference on Software Engineering, External Links: Document Cited by: §2, §5. Wang et al. (2026b) H. Wang, C. M. Poskitt, J. Wei, and J. Sun ProbGuard: proactive runtime monitoring for LLM agent safety via probabilistic prediction. Note: Accepted at ASE 2026 External Links: 2508.00500 Cited by: §2, §5. Watkins and Dayan (1992) C. J. C. H. Watkins and P. Dayan Q-learning. Machine Learning 8 (3), p. 279–292. External Links: Document Cited by: §2. Xi et al. (2026) Z. Xi, C. Liao, G. Li, Y. Yang, W. Chen, Z. Zhang, B. Wang, S. Jin, Y. Zhou, J. Guan, W. Wu, T. Ji, T. Gui, Q. Zhang, and X. Huang AgentPRM: process reward models for llm agents via step-wise promise and progress. In Proceedings of the ACM Web Conference, Cited by: §1. Xiong et al. (2024) W. Xiong, Y. Song, X. Zhao, W. Wu, X. Wang, K. Wang, C. Li, W. Peng, and S. Li Watch every step! llm agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 1556–1572. External Links: Document Cited by: §1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §4.1. Yang et al. (2025b) R. Yang, F. Ye, J. Li, S. Yuan, Y. Zhang, Z. Tu, X. Li, and D. Yang The lighthouse of language: enhancing llm agents via critique-guided improvement. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Document Cited by: §5. Yao et al. (2022a) S. Yao, H. Chen, J. Yang, and K. Narasimhan Webshop: towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems 35, p. 20744–20757. Cited by: §1, §4.1. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-Bench: a benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. Cited by: §1, §4.1. Yao et al. (2022b) S. Yao, J. Zhao, D. Yu, I. Shafran, K. R. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, Cited by: §1, §2. Zhang et al. (2026) C. Zhang, Z. Wan, X. Yu, J. Wu, Q. Wen, P. Zhou, W. Zhao, and I. Tsang Calibration is not control: why llm-agent oversight needs intervention. arXiv preprint arXiv:2606.21399. Cited by: §1, §1, §1, §5. Appendix A Additional Analysis This section provides additional analysis for the methodology in Section 3. We first clarify the same-prefix branch construction used for comparator supervision, and then prove Proposition 3.1. We further discuss the abstaining comparison interface used in the implementation, and the role of candidate support. A.1 Same-Prefix Branch Fix a decision state sts_t, a branch-point action a, and the frozen continuation actor π. By construction, a same-prefix branch first restores the environment to sts_t, executes a, and then returns control to the same actor π. Recall that: Y(st,a)=R(τ),τ∼P(⋅∣st,a,π).Y(s_t,a)=R(τ), τ P(· s_t,a,π). (13) Therefore, [Y(st,a)] [Y(s_t,a)] =τ∼P(⋅∣st,a,π)[R(τ)] =E_τ P(· s_t,a,π)[R(τ)] =Qπ(st,a), =Q^π(s_t,a), (14) which gives Eq. 7 in the main text. For M independent continuations, Q^M(st,a)=1M∑m=1MYm(st,a), Q_M(s_t,a)= 1M _m=1^MY_m(s_t,a), (15) and linearity of expectation gives: [Q^M(st,a)]=Qπ(st,a).E[ Q_M(s_t,a)]=Q^π(s_t,a). (16) Whenever the return is integrable, Q^M(st,a) Q_M(s_t,a) also converges almost surely to Qπ(st,a)Q^π(s_t,a) as M→∞M→∞. For two sibling actions aAa_A and aBa_B, define their true and empirical continuation-value gaps as: d(st,aA,aB) d(s_t,a_A,a_B) =Qπ(st,aA)−Qπ(st,aB), =Q^π(s_t,a_A)-Q^π(s_t,a_B), (17) d^M(st,aA,aB) d_M(s_t,a_A,a_B) =Q^M(st,aA)−Q^M(st,aB). = Q_M(s_t,a_A)- Q_M(s_t,a_B). (18) Then we have: [d^M(st,aA,aB)]=d(st,aA,aB).E[ d_M(s_t,a_A,a_B)]=d(s_t,a_A,a_B). (19) Thus the branch-return difference isolates the downstream consequence of changing the branch-point action while keeping both the preceding state and the continuation actor fixed. This is the quantity thresholded to construct the empirical supervision in Eq. 8. It is useful to distinguish unbiased value estimation from finite-sample pairwise labeling. Although Q^M(st,a) Q_M(s_t,a) is unbiased for Qπ(st,a)Q^π(s_t,a), the thresholded label [d^M>0]I[ d_M>0] need not be an unbiased estimator of [d>0]I[d>0]. In particular, comparisons with small true value gaps are more susceptible to finite-rollout noise. For example, suppose branch returns are normalized to [0,1][0,1], and suppose d^M d_M is formed from M independent paired branch-return differences, each lying in [−1,1][-1,1]. If d>0d>0, Hoeffding’s inequality gives: Pr(d^M≤0)≤exp(−Md22). ( d_M≤ 0)≤ \! (- Md^22 ). (20) Hence increasing the number of branch continuations primarily improves the reliability of supervision for comparisons whose true continuation values are close. A.2 Proof of Proposition 3.1 Fix (st,at)(s_t,a_t) throughout this subsection. For a candidate A∼μ(⋅∣st)A μ(· s_t), define: B(A) B(A) =[Qπ(st,A)>Qπ(st,at)], =I [Q^π(s_t,A)>Q^π(s_t,a_t) ], (21) B~(A) B(A) =Cθ(st,A,at). =C_θ(s_t,A,a_t). (22) For compactness, write: ρ ρ =ρμ(st,at)=A∼μ[B(A)], = _μ(s_t,a_t)=E_A μ[B(A)], (23) qθ q_θ =A∼μ[B~(A)]. =E_A μ[ B(A)]. (24) The pairwise error defined in Section 3.3 can be written as: ϵθ=PrA∼μ(B~(A)≠B(A)). _θ= _A μ ( B(A)≠ B(A) ). (25) Because both B(A)B(A) and B~(A) B(A) are binary, we have: |qθ−ρ| |q_θ-ρ| =|[B~(A)−B(A)]| = |E[ B(A)-B(A)] | ≤[|B~(A)−B(A)|] [| B(A)-B(A)|] =ϵθ. = _θ. (26) This step does not require independence across candidates; it is purely a single-candidate comparison between the learned and oracle relations. Now suppose first that A1,…,AKA_1,…,A_K are independently sampled from μ(⋅∣st)μ(· s_t), and define: B~i=B~(Ai),ρ^θ,K=1K∑i=1KB~i. B_i= B(A_i), ρ_θ,K= 1K _i=1^K B_i. (27) By linearity of expectation, we have: [ρ^θ,K]=qθ.E[ ρ_θ,K]=q_θ. (28) Combining Eqs. equation 26 and equation 28 yields: |[ρ^θ,K]−ρ|≤ϵθ, |E[ ρ_θ,K]-ρ |≤ _θ, (29) which proves Eq. 11. For the finite-candidate component, for independent candidate sampling, the concentration result follows directly from Hoeffding’s inequality. The same bound also holds when the K candidates are sampled uniformly without replacement from a finite candidate population: sampling without replacement is at least as concentrated as sampling with replacement for bounded finite-population averages. Therefore, under either sampling scheme, we have: Pr(|ρ^θ,K−qθ|≥ξ)≤2exp(−2Kξ2). \! ( | ρ_θ,K-q_θ |≥ξ )≤ 2 (-2Kξ^2). (30) With probability at least 1−δ1-δ, |ρ^θ,K−qθ|≤log(2/δ)2K. | ρ_θ,K-q_θ |≤ (2/δ)2K. (31) Using the triangle inequality together with Eq. 26, we have: |ρ^θ,K−ρ| | ρ_θ,K-ρ | ≤|ρ^θ,K−qθ|+|qθ−ρ| ≤ | ρ_θ,K-q_θ |+|q_θ-ρ| ≤ϵθ+log(2/δ)2K. ≤ _θ+ (2/δ)2K. (32) This proves Eq. 12 under both sampling schemes. A.3 Comparator Training and Inference Section 3.1 defines the ideal binary comparison target as Cθ(st,aA,aB)≈[Qπ(st,aA)>Qπ(st,aB)]C_θ(s_t,a_A,a_B) [Q^π(s_t,a_A)>Q^π(s_t,a_B) ] in Eq. 4. Here we describe how this oracle target comparator is realized in our implementation. Training uses same-prefix branch estimates to supervise the relative ordering of two actions, with an additional tie class for indistinguishable outcomes. At inference time, the resulting ternary model outputs are mapped back to the binary decision CθC_θ required by the runtime intervention rule. Training supervision. As described in Section 3.2, for two sibling actions aAa_A and aBa_B taken from the same state, we estimate their actor-conditioned continuation values using same-prefix Monte Carlo branches Q^M(st,aA) Q_M(s_t,a_A) and Q^M(st,aB) Q_M(s_t,a_B). The binary relation introduced in Eq. 8 B^(st,aA,aB)=[Q^M(st,aA)>Q^M(st,aB)] B(s_t,a_A,a_B)=I [ Q_M(s_t,a_A)> Q_M(s_t,a_B) ] is therefore the empirical counterpart of the strict ordering in Eq. 4. Since Q^M(st,a) Q_M(s_t,a) is a Monte Carlo estimate of Qπ(st,a)Q^π(s_t,a), the empirical ordering approaches the corresponding ordering of QπQ^π as the number of branch continuations increases. Our implementation uses the same ordering signal, but does not force every pair into a binary preference. Instead, we train the model with three possible outputs: A,Q^M(st,aA)−Q^M(st,aB)>γe,B,Q^M(st,aB)−Q^M(st,aA)>γe,T,|Q^M(st,aA)−Q^M(st,aB)|≤γe. cases A,& Q_M(s_t,a_A)- Q_M(s_t,a_B)> _e,\\[5.69054pt] B,& Q_M(s_t,a_B)- Q_M(s_t,a_A)> _e,\\[5.69054pt] T,& | Q_M(s_t,a_A)- Q_M(s_t,a_B) |≤ _e. cases Thus, A and B represent the two directions of the same pairwise ordering used in Eq. 8, while T represents an empirical tie region in the supervision; at deployment, such a prediction contributes no decisive winner. When γe=0 _e=0, this supervision directly reflects the empirical strict ordering: unequal branch estimates produce a preference for the action with the larger estimated continuation value, while equal estimates are labeled T. For ALFWorld and τ3τ^3-Retail, branch outcomes are binary, so we use γe=0 _e=0: outcomes 11 versus 00 produce a decisive preference, whereas 11 versus 11 and 00 versus 00 produce a tie. For WebShop, whose returns provide graded numerical feedback, we use a positive margin γe=γ _e=γ. This deliberately treats small empirical return differences as ties rather than forcing the model to learn a preference from a weak distinction. The margin is therefore a conservative implementation-level surrogate for the strict target in Eq. 4: sufficiently separated action pairs are trained according to their empirical ordering, whereas pairs within the margin contribute no decisive preference. This construction connects directly to the target in the main method. Because Q^M(st,a)→Qπ(st,a) Q_M(s_t,a)→ Q^π(s_t,a) as the number of independent branch continuations increases, the training signal increasingly reflects the ordering of the true actor-conditioned continuation values. With γe=0 _e=0, it approaches the strict relation in Eq. 4 away from exact ties. With γe>0 _e>0, the implementation intentionally abstains on sufficiently small value differences rather than attempting to resolve them. Binary comparator at inference. Although the model is trained with the three outputs A,B,T\ A, B, T\, the runtime gate in Section 3.1 requires only the binary judgment in Eq. 4. We obtain this judgment by evaluating each action pair in both input orders. Specifically, we set: Cθ(st,aA,aB)=1,the model predicts A for (st,aA,aB),and predicts B for (st,aB,aA),0,otherwise.C_θ(s_t,a_A,a_B)= cases1,& array[]lthe model predicts A for (s_t,a_A,a_B),\\ and predicts B for (s_t,a_B,a_A), array\\[8.53581pt] 0,&otherwise. cases (33) Thus, aAa_A is counted as defeating aBa_B only when both input orders express the same semantic preference for aAa_A. A tie prediction, or a pair of predictions that is inconsistent under input reversal, produces Cθ=0C_θ=0. Eq. 33 is the concrete implementation of the binary CθC_θ used in the main method. In particular, the runtime prediction in Eq. 5 is exactly B~i=Cθ(st,Ai,at), B_i=C_θ(s_t,A_i,a_t), after applying the bidirectional check above. The winner count in Eq. 6 and the candidate-relative estimate in Eq. 10 therefore operate on the same binary comparator defined in Section 3.1; the ternary outputs are only an implementation detail used to obtain that decision conservatively. We study the empirical effect of ternary supervision and bidirectional inference in Appendix B.9. A.4 Gate Decision Consistency Proposition 3.1 implies that, with probability at least 1−δ1-δ: |ρ^θ,K−ρμ(st,at)|≤ϵθ(st,at)+log(2/δ)2K. | ρ_θ,K- _μ(s_t,a_t) |≤ _θ(s_t,a_t)+ (2/δ)2K. (34) Therefore, whenever the true domination rate is separated from the intervention threshold by more than this error: |ρμ(st,at)−RK|>ϵθ(st,at)+log(2/δ)2K, | _μ(s_t,a_t)- RK |> _θ(s_t,a_t)+ (2/δ)2K, (35) the learned estimate and the true domination rate lie on the same side of the threshold with probability at least 1−δ1-δ. Consequently, we have: [ρ^θ,K≥RK]=[ρμ(st,at)≥RK]I [ ρ_θ,K≥ RK ]=I [ _μ(s_t,a_t)≥ RK ] (36) with probability at least 1−δ1-δ. Thus, away from an uncertainty region around the intervention threshold, the learned gate agrees with the oracle candidate-relative gate. Lower comparator error narrows this region through ϵθ _θ, while increasing K reduces the finite-candidate sampling term at the standard K−1/2K^-1/2 rate. A.5 Candidate Distribution The candidate-relative formulation does not require the reference mechanism μ to be an expert policy. However, constructive advice is necessarily limited by candidate support: the framework cannot expose a genuinely better alternative if the sampled candidate set contains none. Under independent sampling from μ, the probability that at least one of K candidates truly improves on the current proposal is: Pr[∃i:Qπ(st,Ai)>Qπ(st,at)]=1−[1−ρμ(st,at)]K. \! [∃ i:Q^π(s_t,A_i)>Q^π(s_t,a_t) ]=1- [1- _μ(s_t,a_t) ]^K. (37) For the finite-pool without-replacement setting, suppose exactly W of the N available candidates have higher continuation value than the proposal. Then: ρN=WN. _N= WN. (38) The probability that a uniformly sampled subset of size K contains at least one true winner is: Pr(at least one true winner)=1−(N−WK)(NK), (at least one true winner)=1- N-WK NK, (39) where the numerator is interpreted as zero when K>N−WK>N-W. These expressions describe a support ceiling rather than a quality assumption on μ. A weak reference distribution can still provide a meaningful candidate-relative comparison, but it will supply genuinely useful alternatives less frequently. Conversely, broader or stronger candidate support increases the opportunities for constructive replanning without changing the comparator’s learning objective. Appendix B Experimental Details This appendix records the complete experimental contract behind Section 4. We first specify the evaluation slices and actor protocols, then describe branch collection, comparator training, candidate construction, baselines, prompts, and cost accounting. B.1 Evaluation Slices and Actor Protocols Table 3: Evaluation slices. A repeat denotes one complete environment episode with a distinct seeded rollout. DeepSeek uses cost-controlled fixed subsets for WebShop and ALFWorld. Environment Local tasks DeepSeek tasks Local repeats DeepSeek repeats Max steps WebShop 500 50 1 1 20 ALFWorld 134 20 1 2 30 τ3τ^3-Retail 40 40 3 3 100 WebShop. The branch-data pool uses tasks 0–4999. The held-out evaluation set is tasks 5000–5499, which is never used for branching, preprocessing, checkpoint selection, or hyperparameter sweeps. The environment returns a reward in [0,1][0,1] based on matched product attributes. We call reward 1 an exact success and separately report the fraction of reward-0 trajectories. ALFWorld. Data collection uses training games, while evaluation uses the 134 games in the standard valid-unseen split. A trajectory succeeds only when the environment’s terminal goal condition is satisfied. The actor sees the task, the executed action–observation prefix, the current observation, and the current admissible action set. τ3τ^3-Retail. We use the official Retail test split of 40 tasks. Each system is evaluated under three rollout seeds, and success is averaged over all 120 episodes. The environment includes a simulated user, policy text, and a database-backed tool interface; one actor turn may be followed by a tool result or a user reply. B.2 Prefix-Branch Collection We generate supervision in two stages. First, the frozen Qwen3-8B source actor completes a linear base trajectory without intervention. Second, after termination, we select decision points across the trajectory and restore each exact prefix. Where the environment does not support snapshots, restoration resets the task and replays all preceding actions. For a base trajectory with T actions, the default sampler chooses min(⌈0.5T⌉,10) ( 0.5T ,10) temporally stratified decision points. At each verified point, we keep the base action and obtain up to five distinct legal alternatives. The branching action differs across siblings; every branch thereafter returns to the same frozen continuation actor, low-temperature decoding, remaining step budget, and terminal rule. Branch trajectories are never branched recursively. This produces linear rather than exponential data growth while preserving the same-prefix counterfactual interpretation. B.3 Pair Construction Within each verified prefix group, we construct both base–branch and branch–branch pairs. WebShop orders actions by downstream partial reward, while ALFWorld and Retail use terminal success. Small outcome differences are treated as ties when they do not support a reliable preference. Ties are downsampled to approximately 20% of the training data so that abundant same-outcome siblings do not dominate learning. Every retained physical pair is direction-augmented into both (A,B)(A,B) and (B,A)(B,A) orders. All members of a task or interaction tree remain in one split. Table 4: Comparator data split details. Environment Physical pairs Train Validation Test WebShop 35,117 55,856 7,078 7,300 ALFWorld 14,548 22,678 3,222 3,196 τ3τ^3-Retail 5097 8,910 1284 1278 B.4 Comparator Interface We full-parameter fine-tune Qwen2.5-0.5B-Instruct in bf16. Observable state representation. WebShop uses a deterministic, plan-free execution summary together with the current page and executable click/search actions. ALFWorld uses the raw task plus executed Action/Observation prefix, current observation, and admissible commands. Retail uses a deterministic policy and interaction state with grounded entity, confirmation, tool, and recent-turn evidence. None of these representations includes unexecuted actor thoughts. Generic A/B/T comparator prompt. The benchmark renderer fills the state block below; the decision interface is otherwise shared. [System] You are an expert judge for an interactive agent. Given the task, observable progress, current state, and two candidate next actions, decide which action is more likely to lead to correct task completion when the same base agent continues. If one is clearly better, answer A or B. Otherwise answer T. [User] # Task and observable trajectory benchmark-specific state renderer # Candidate actions A: action A B: action B # Answer Reply with exactly one letter: A, B, or T. At inference time, a candidate is a stable winner only when the model selects the same semantic action after swapping A and B. Predictions of T, order-inconsistent outputs, malformed generations, and service errors fail open. This makes deployment more conservative than unconstrained single-order classification. B.5 Choosing K and R K controls the number of alternatives reviewed at a state and R controls how many stable winners are required to reject the proposal. We select K and R using end-to-end performance on held-out validation tasks and freeze the selected values before test evaluation. We use the same operating point across actor models within each environment. Table 5: Example WebShop validation sweep for the intervention operating point. Configuration K R Mean reward Success (%) COTA, cautious 4 4 0.4412 18.2 COTA, medium 8 2 0.5547 28.4 COTA, active 4 1 0.6354 34.2 The validation sweep selects K=4,R=1K=4,R=1 for WebShop and ALFWorld, and K=4,R=2K=4,R=2 for τ3τ^3-Retail. These values are then fixed for all test actors. B.6 Candidate-Action Sources Environment actions. When the environment exposes legal actions, we sample without replacement. In WebShop these are current-page clicks and search; in ALFWorld they are admissible text commands. Offline actions. Offline retrieval draws concrete actions observed in branch data under a compatible decision condition. Retrieved actions are rebound to currently observed entities and rejected if required identifiers or policy prerequisites are absent. Retail uses this source to recover useful tool patterns that are unlikely to appear in a small online sample. Small-LM actions. A small LLM receives the actor-visible state and proposes alternatives under several decision modes, such as information gathering, progress, recovery, and a distinct alternative. The generator never votes on its own actions. Table 6: Default candidate source by benchmark and actor family. Environment Actor condition Candidate source WebShop all actors environment ALFWorld Qwen3-8B environment ALFWorld stronger actors small LM + environment τ3τ^3-Retail all actors small LM + offline B.7 Baselines and Prompt Templates AgentPRM-style absolute-Q selection. Following the practical AgentPRM formulation (8), we train one scalar critic per benchmark. Terminal outcome is mapped to [−1,1][-1,1], discounted by γtγ^t with γ=0.95γ=0.95, averaged over duplicate state–action records, and mapped back to [0,1][0,1]. A Qwen2.5-0.5B sequence classifier is trained with soft binary cross-entropy. At deployment it scores [a0,a1,…,aK][a_0,a_1,…,a_K] independently and executes the stable argmax; exact ties retain a0a_0. Candidate rationales are omitted so that every action has the same observable interface. [System] You are a process reward model for an interactive agent. Estimate how likely the candidate next action is to lead to successful completion when the same base agent continues. [User] # Task and observable trajectory state renderer # Candidate action one concrete action Self-Reflection. The actor receives one conservative self-check before execution. It may keep its original proposal; malformed or illegal revisions fail open. [CONSERVATIVE SELF-CHECK] The preceding proposal has not been executed. Default to keeping it. Change it only for a definite legality, repetition, grounding, policy, or task- progress error and only when a clearly better executable action is available. When uncertain, repeat the original action. Output exactly one revised action. For tool use, the prompt additionally requires an unexecuted tool call to remain exactly one tool call and forbids claiming that the operation has already occurred. Asym-AC. A separate critic sees only the task, executed observable trajectory, current state/action affordances, and the actor proposal. It produces concise free-form feedback, after which the actor receives one revision turn. On Retail, the critic additionally sees the relevant domain policy and concrete tool arguments. [System] You are a zero-shot action-verification Critic Agent. Ground the review only in the supplied task, observable trajectory, current observation, and environment affordances. Identify concrete errors or confirm that the proposal is appropriate. Give concise feedback; do not invent a hidden candidate set. [User] Task: task Executed trajectory: prefix Current observation/affordances: state Actor proposal: proposal Review legality, grounding, prior progress, and likely task progress. COTA feedback. The comparator never writes free-form criticism. If at least R candidates win, the actor receives a short, structured message identifying its unexecuted proposal and listing preferred alternatives in ranked order. The actor is explicitly allowed to keep the original action or synthesize a third action. [ADVISOR FEEDBACK] The proposed action proposal may not be the best next step. Preferred alternatives: 1. winner 1; 2. winner 2. Reconsider the current state and output the action that should actually be executed. The recommendations are non-binding. B.8 Absolute-Q Diagnostics Table 7: Validation quality of the three Qwen2.5-0.5B absolute-Q critics. Environment Pairwise ranking acc. (%) Spearman BCE WebShop 69.13 0.618 0.527 ALFWorld 64.23 0.366 0.635 τ3τ^3-Retail 68.20 0.423 0.639 Table 8: Online intervention frequency (%) for the absolute-Q critic. Forced reports states where argmax replaces the actor proposal; selective reports gate rounds that request actor replanning. Qwen3-8B Qwen3.6 Environment Forced Selective Forced Selective WebShop 61.1 46.5 48.8 26.6 ALFWorld 83.6 50.5 87.0 56.5 τ3τ^3-Retail 77.7 38.7 66.0 32.9 The scalar critic can show reasonable held-out ranking accuracy while remaining poorly calibrated for online control. In ALFWorld it frequently scores several legal but subgoal-irrelevant moves above the proposal, causing forced selection to replace more than 80% of actor actions. Constructive intervention absorbs some of these mistakes, but direct pairwise supervision remains substantially more effective in the closed loop. B.9 A/B versus A/B/T Table 9: Comparator-target ablation on 100 held-out WebShop tasks. Both rows use a bidirectional gate and environment-only K=4,R=1K=4,R=1 candidates. Training target Pref. consistency Valid output End reward Zero reward A/B only 39.51% 86.16% 0.5031 37.0% A/B/T 57.70% 99.53% 0.5442 22.0% Explicit ties improve order consistency and make abstention available when the branch outcomes do not justify a winner. Because this comparison uses 100 independently sampled trajectories, we treat the online difference as descriptive rather than a standalone significance claim. B.10 Cost Offline cost includes base rollout generation, branch continuations, preprocessing, and comparator fine-tuning and is reported in Table 10. It is amortized across every actor and every future evaluation that reuses the comparator, and is therefore kept separate from the online Avg. T values in Table 1. Table 10: Offline cost accounting. The time unit is H200 GPU-hour. Environment Base rollout Branch rollout Comparator FT Total GPU-hours WebShop 1.48 78.65 1.23 81.36 ALFWorld 1.97 28.39 0.80 31.15 τ3τ^3-Retail 0.27 7.79 0.36 8.42 Table 11: Representative online efficiency on 500 matched Qwen3-8B WebShop tasks. Token counts include prompt and completion tokens; episode time is cumulative end-to-end time. System Actor tokens Advisor tokens Episode time Success (%) Actor only 28.420M – 1.212 h 18.2 0.5B COTA, env-only 36.050M 97.255M 1.712 h 27.8 The normalized time reported in the main table uses paired cumulative episode time. It captures both extra model calls and any trajectory shortening caused by better decisions.