Paper deep dive
ReDAct: Uncertainty-Aware Deferral for LLM Agents
Dzianis Piatrashyn, Nikita Kotelevskii, Kirill Grishchenkov, Nikita Glazkov, Ivan Nasonov, Ilya Makarov, Timothy Baldwin, Preslav Nakov, Roman Vashurin, Maxim Panov
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 2:00:05 AM
Summary
ReDAct (Reason-Defer-Act) is a framework for LLM agents in sequential environments that optimizes the trade-off between inference cost and performance. It uses a small, inexpensive model for default actions and defers to a large, reliable model when the small model's predictive uncertainty exceeds a calibrated threshold. Experiments on ALFWorld and MiniGrid show that deferring approximately 15% of decisions to a larger model matches the performance of using the large model exclusively while significantly reducing costs.
Entities (5)
Relation Signals (4)
ReDAct â evaluatedon â ALFWorld
confidence 100% · We evaluate our approach in text-based embodied environments such as ALFWorld and MiniGrid
ReDAct â evaluatedon â MiniGrid
confidence 100% · We evaluate our approach in text-based embodied environments such as ALFWorld and MiniGrid
ReDAct â extends â ReAct
confidence 95% · In our setting, we start with the ReAct framework (Yao et al., 2022b), which decouples reasoning and action selection into two separate LLM calls, and extend it with deferral capabilities.
ReDAct â utilizes â Perplexity
confidence 90% · the best results are achieved with Perplexity (PPL)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recently, LLM-based agents have become increasingly popular across many applications, including complex sequential decision-making problems. However, they inherit the tendency of LLMs to hallucinate, leading to incorrect decisions. In sequential settings, even a single mistake can irreversibly degrade the trajectory, making hallucinations an even bigger problem. Although larger LLMs hallucinate less, they incur a significantly higher per-token cost. In this paper, we address this tradeoff by proposing ReDAct (Reason-Defer-Act). In ReDAct, an agent is equipped with two LLMs: a small, cheap model used by default, and a large, more reliable but expensive model. When the predictive uncertainty of the small model exceeds a calibrated threshold, the decision is deferred to the large model. We evaluate our approach in text-based embodied environments such as ALFWorld and MiniGrid and show that deferring only about 15% of decisions to the large model can match the quality of using it exclusively, while significantly reducing inference costs.
Tags
Links
- Source: https://arxiv.org/abs/2604.07036v1
- Canonical: https://arxiv.org/abs/2604.07036v1
Trouble viewing inline? Open PDF directly â
Full Text
66,387 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. ReDAct: Uncertainty-Aware Deferral for LLM Agents Dzianis Piatrashyn 1 , Nikita Kotelevskii 1 , Kirill Grishchenkov 2 , Nikita Glazkov 3,4 , Ivan Nasonov 5 , Ilya Makarov 4,6 , Timothy Baldwin 1 , Preslav Nakov 1 , Roman Vashurin 1 , Maxim Panov 1 1 Mohamed bin Zayed University of Artificial Intelligence 2 Ocapital 3 National University of Science and Technology (NUST) MISIS 4 AXXX 5 Ivannikov Institute for System Programming of the Russian Academy of Sciences 6 Trusted AI Center, RAS Abstract Recently, LLM-based agents have become increasingly popular across many applications, including complex sequential decision-making prob- lems. However, they inherit the tendency of LLMs to hallucinate, leading to incorrect decisions. In sequential settings, even a single mistake can irreversibly degrade the trajectory, making hallucinations an even bigger problem. Although larger LLMs hallucinate less, they incur a significantly higher per-token cost. In this paper, we address this tradeoff by proposing ReDAct (Reason-Defer-Act). In ReDAct, an agent is equipped with two LLMs: a small, cheap model used by default, and a large, more reliable but expensive model. When the predictive uncertainty of the small model exceeds a calibrated threshold, the decision is deferred to the large model. We evaluate our approach in text-based embodied environments such as ALFWorld and MiniGrid and show that deferring only about 15% of deci- sions to the large model can match the quality of using it exclusively, while significantly reducing inference costs. 1 Introduction Large language models (LLMs; Brown et al., 2020) are widely used nowadays across a range of domains (Bommasani et al., 2021; Chowdhery et al., 2023). Among many directions in which LLMs are deployed, one of the most recent and rapidly emerging is their use as autonomous agents (Yao et al., 2022b; Schick et al., 2023). Here, the LLM autonomously plans and decides which actions to take at each step, given the observed context (Yao et al., 2022b; Brohan et al., 2023; Huang et al., 2023). A particularly interesting instantiation of this paradigm appears in embodied and interactive environments. Text-based embodied benchmarks, such as ALFWorld (Shridhar et al., 2021), and grid-world environments, such as MiniGrid (Chevalier-Boisvert et al., 2023), place LLM agents in partially observed settings. Here, the agentâs states change irreversibly, and errors in actions accumulate over the entire episode. A single wrong action, such as stepping onto a lava tile or picking up the wrong object, can immediately degrade the trajectory and ruin the entire episode. Unlike static text generation and other classical (e.g., QA) LLM setups, these environments require that agents make reliable decisions at every step, because errors compound rather than remain isolated (Huang et al., 2023; Wang et al., 2023). This motivates a mechanism that helps the agent recognize mistaken or unreliable actions. When such actions are detected, the agent can refuse to act and defer the decision-making to a more capable model. Classical selective prediction and learning-to-defer frameworks (Geif- man & El-Yaniv, 2019; Mozannar & Sontag, 2020) formalize this idea. In the LLM setting, the agent can abstain from making a decision when its small model is unsure and defer 1 arXiv:2604.07036v1 [cs.CL] 8 Apr 2026 Preprint. Under review. to a larger, more capable model. This formulation connects naturally to cost-aware model routing and cascading (Chen et al., 2024; Yue et al., 2023; Ong et al., 2025), where the goal is to use expensive large models only when they are likely to improve the outcome. However, existing routing methods treat model selection as a per-query problem, choosing a single model for the entire request. They do not consider the repeated setting of an agent that sequentially makes decisions, nor deferral within an interactive trajectory. Prior work on uncertainty quantification (UQ) has developed numerous methods that allow quantifying the reliability of LLMsâ predictions (Vashurin et al., 2025a). These include, inter alia, semantic entropy (Kuhn et al., 2023), information-theoretic approaches (Fomicheva et al., 2020; Malinin & Gales, 2021), verbalized confidence (Tian et al., 2023) and others. However, these works normally evaluate uncertainty in single-turn language generation. Much less is known about the usefulness of uncertainty scores from a small LLM for reliable deferral to a large model in sequential environments. In this paper, we study this problem. We propose ReDAct (Reason-Defer-Act), a framework in which an agent is equipped with two LLMs: a small, cheap model and a large, more reliable but expensive one. When the small modelâs uncertainty is high, the agent defers the action decision to the large model. Our contributions are as follows. âąWe propose ReDAct, a framework for LLM agents in sequential environments in which the base agent can defer individual action decisions from a small model to a large model based on predictive uncertainty (see Section 4). âąWe measure the performance of multiple state-of-the-art UQ methods and show that action-level information-theoretic UQ measures provide the most informative deferral signal (see Section 4.1). âąWe evaluate our approach on ALFWorld and MiniGrid datasets using frontier-size LLMs (70B to 480B) and demonstrate that deferring only about 15% of decisions to the larger model can match its full performance, at a fraction of the inference cost (see Section 5). 2 Related Work Our work is placed at the intersection of several topics, namely uncertainty quantification, prediction deferral, and the application of agents in embodied environments. Let us review these areas one by one. Uncertainty Quantification. The area of predictive uncertainty quantification has been developing rapidly in recent years, given the increased scope of problems in which pre- dictive models are applied (Kendall & Gal, 2017; Gal & Ghahramani, 2016; H Ì ullermeier & Waegeman, 2021; Kotelevskii et al., 2025). As these problems involve high-risk applications, such as autonomous driving (Bojarski et al., 2016) and medical diagnostics (Kompa et al., 2021), for reliable use in real scenarios, one needs to be careful with model predictions and detect unreliable ones. The problem of predictive uncertainty quantification has been extensively studied across different domains (Gawlikowski et al., 2023; Abdar et al., 2021), and interest in uncertainty for LLMs has grown particularly rapidly in recent years. There are several key groups of methods for quantifying uncertainty in LLM predictions. We will emphasize three of them. The first is an information-theoretic group, which works directly with the probabilities of generated tokens (Fomicheva et al., 2020; Malinin & Gales, 2021). As the methods in this class require token or sequence probabilities, the model must be white-box in this sense. Another group works with the semantics of generated outputs (Lin et al., 2024; Kuhn et al., 2023). These methods typically require multiple sequence generations from the LLM, grouping them into semantically close clusters using Natural Language Inference (NLI), and computing a measure of âvarianceâ, e.g., entropy, over them. Hence, these methods may be computationally more demanding. Recently, approaches that bridge these two classes of methods have emerged (Vashurin et al., 2025b). The third group we want to emphasize 2 Preprint. Under review. Action Uncertainty True False < State State Action Accept Accept Large Model Small Model Figure 1: Overview of the proposed ReDAct framework. The agent uses the small model to propose an action Ì a s t and estimates its uncertaintyu t . Ifu t exceeds the thresholdÏ, the step is deferred to the large model, which proposes Ì a l t as the final decision. includes the so-called verbalized methods. These methods prompt the LLM to output an explicit âI am not sureâ token and estimate its probability (Kadavath et al., 2022), or ask the model to output its confidence directly Tian et al. (2023). There is no established leader among these methods, as different combinations of models, datasets, and problems can yield different rankings. Prediction deferral. The option of abstaining from a prediction has been considered in the literature for a general statistical problem setting (Chow, 2003; Geifman & El-Yaniv, 2019). It was also considered in different domains, such as images (Geifman & El-Yaniv, 2019) and text (Malinin & Gales, 2021). Deferral to a more advanced model was studied in both the image domain (Kotelevskii et al., 2024) and the text domain (Mozannar & Sontag, 2020; Verma & Nalisnick, 2022). For LLMs, the switch from a small model to a larger, more capable one was considered in (Chen et al., 2024; Yue et al., 2023; Ong et al., 2025). In particular, Ong et al. (2025) trained a supervised router to select between models. However, these works studied only single-step setups, e.g., QA, without a sequential agentic problem formulation. A notable exception is (Ren et al., 2023), which defers the decision to the oracle in steps where the predictive uncertainty is high. Although important, this work considers only conformal prediction, which requires scoring each possible action using an LLM. This can be cost-prohibitive in settings with a large number of possible actions at each step. Moreover, it considers only deferral to a human oracle, while we keep the system fully autonomous. Agents in embodied systems. Several works consider agents operating in embodied en- vironments. ALFWorld (Shridhar et al., 2021) is a text-based counterpart of the ALFRED benchmark (Shridhar et al., 2020). It consists of long compositional household tasks with ir- reversible state changes. MiniGrid (Chevalier-Boisvert et al., 2023) consists of grid-world en- vironments that test navigation and exploration under partial observability. WebShop (Yao et al., 2022a) simulates an online shopping task that requires multi-step search, comparison, and purchase decisions. LLM-based agents have been applied to these environments with growing success (Yao et al., 2022b; Wang et al., 2023; Shinn et al., 2023). However, in these settings, uncertainty-aware deferral between small and large LLMs within an agent has not been considered. 3 Background We start this section by formalizing our setup and introducing useful notation. 3 Preprint. Under review. Problem Statement. We consider an agent placed in an environment that it can interact with. The agent is equipped with two LLMs: a small, cheap modelM small used by default, and a large, more reliable but expensive modelM large . The goal of the agent is to complete a task (e.g., find an item in a maze), described in natural language in the input prompt, and do so effectively (e.g., in as few steps as possible). Each such interaction with the environment forms an episode, which can end either with success or failure. The episode consists of discrete stepstâ1, 2,. . .,T. At each stept, the agent observes the current state of the environments t âS. Then, using the small model, the agent generates a proposal action Ì a s t , and estimates the uncertainty scoreu t associated with this proposal. If u t is below a selected thresholdÏ(see Section 4), the proposed action is accepted,a t : = Ì a s t . Otherwise, the proposed action is rejected, and the decision is deferred to the large model, which generates its own proposal Ì a l t , accepted unconditionally:a t : = Ì a l t . In both cases, the accepted actiona t is applied to the environment, transitioning the agent to a new states t+1 . A schematic illustration of our approach is provided in Figure 1. The episode ends when the task is completed (success) or when the agent reaches the maximum number of stepsT (failure). If cost were not a concern, a rational actor would use the large model at every step. In practice, however, a tradeoff between the two models is desirable. We propose balancing this tradeoff using uncertainty quantification, where the uncertainty score associated with a proposed action is compared to a threshold to decide when deferral is needed. Uncertainty Quantification. Let us denote byy = [y 1 ,y 2 ,. . .,y L ]a sequence, generated by an LLM for an input promptx = [x 1 ,x 2 ,. . .,x n ]. One can compute the probability of the output sequence y as: p(y| x,Ξ) = L â i=1 p(y i | x, y <i ,Ξ),(1) whereΞ are the model parameters, and y <i = (y 1 , . . . , y iâ1 ). If these token-level probabilities are available, one can compute information-theoretic measures of predictive uncertainty. In this work, we consider three measures that use the small modelâs output probabilities directly, specifically Sequence Probability (SP), Perplexity (PPL), and Mean Token Entropy (MTE). All three aggregate token-level probabilities into a single scalar uncertainty score for the generated sequence. Their formal definitions are given in Appendix A. These methods require a token probability distribution but, unlike methods based on semantic similarity, do not require multiple generations or external NLI models. Hence, these methods are computationally cheaper and well-suited for step-level deferral decisions in a sequential setting. As we will see in Section 5, they also demonstrate the most promising results. Note that in general, one can consider multimodal environments (e.g., visual- language ones). In this scenario, one may consider visual-based uncertainty quantification measures. Higher uncertainty scores result in lower confidence in the generated action. Whenu t >Ï, the small modelâs proposal Ì a s t is considered unreliable, and the step is deferred to the large model. ReAct Framework. In our setting, we start with the ReAct framework (Yao et al., 2022b), which decouples reasoning and action selection into two separate LLM calls, and extend it with deferral capabilities. At each step, the agent first generates a reasoning process that analyzes the current position, then uses this reasoning to select the next action. Separating reasoning and action allows us to incorporate deferral naturally. Specifically, we compute the uncertainty score at the action-selection stage, after the small modelâs reasoning is complete. If the uncertainty is high, the action call is deferred to the large model. The large model makes its own reasoning from scratch and returns the action Ì a l t , which the agent 4 Preprint. Under review. ActionReasoning UQ metricPRRROC-AUCPRRROC-AUC Mean Token Entropy0.4240.7100.1680.596 Perplexity0.3990.6840.1930.616 Sequence Probability 0.4170.7040.2790.682 Table 1: Performance comparison of uncertainty metrics estimated over action selection and reasoning trace. accepts. Using the prompt from Figure 5, we obtain the reasoning, which we then use as input for the action-selection prompt presented in Figure 6 (see Appendix E). 4 Method In this section, we describe the ReDAct framework in detail. The method involves two key choices. First, we need to determine the stage at which uncertainty is computed and which class of uncertainty measures to use. Second, we need to select the deferral threshold. We describe both below. 4.1 Action vs Reasoning-Level Uncertainty As we follow the ReAct framework, we can defer to the large model when the uncertainty in the reasoning stage or the action selection step is significant. To determine which stage provides a more reliable deferral signal, we conduct an auxiliary experiment on the ALF- World dataset. Specifically, we simulate 100 trajectories with the ReAct framework using the Qwen3-80B model, resulting in a total of 2411 steps. The environment does not provide ground-truth actions for intermediate steps. To recover correctness labels, we use GPT-5.2 as an LLM-as-a-judge (Zheng et al., 2023) to annotate each step as either correct (meaningful exploration or appropriate action selection) or incorrect (e.g., cyclic behavior or repeatedly visiting the same room). After this labelling procedure, we obtain 1555 steps labelled as correct and 856 as incorrect. We then compute UQ metrics for this labeled dataset and evaluate them via PRR and ROC-AUC. Results. We report the results for reasoning-level and action-level UQ in Table 1 (see more results in Appendix D). For UQ methods estimated at the action selection step, we observe high PRR and ROC-AUC values, especially for Perplexity, Sequence Probability, and Mean Token Entropy. Apart from superior performance, these three methods do not require repeated model inference, making them ideal for both UQ performance and inference costs. In contrast, UQ methods calculated at the reasoning step show poor discriminative power. This result is not surprising, as uncertainty quantification over free-form reasoning is known to be a difficult problem (Ni et al., 2025). Based on this finding, ReDAct focuses exclusively on action-level UQ metrics. 4.2 Threshold Selection A central component of ReDAct is the selection of the deferral thresholdÏ. Specifically, if the uncertainty at the action stage exceedsÏ, the decision is deferred to the large model. We calibrate the threshold using a set ofN cal calibration episodes (the same episodes for which we reported results in Table 1). The calibration procedure is as follows. We fix a target average number of large model calls per episode, denotedK. For each UQ measure (SP, MTE, PPL), every uncertainty value observed across all steps and episodes is treated as a candidate threshold. We then select the candidate that yields an average number of large model invocations per episode closest toKon the calibration set. Note that the number of large model calls is monotonically non-increasing in the threshold, so this selection is well- defined and can also be performed effectively via binary search. For the random baseline, 5 Preprint. Under review. we set the deferral probability uniformly at each step toK/ Ì L, where Ì Lis the average number of steps per episode on the calibration set. Limitations. Although we avoid using the expensive large model during calibration, this introduces a distribution shift in the uncertainty scores. Specifically, in the procedure described above, we select the threshold based solely on the distribution of uncertainty values produced by the small model. However, once the large model is invoked during evaluation, it changes the distribution of uncertainty scores at subsequent steps. This results in a slight discrepancy between the expected and actual number of large model requests. As our experimental results show, this effect is not severe in practice. Furthermore, it could be eliminated by performing calibration during a warm-up run with enabled deferral. The initial deferral threshold would be selected and continuously updated until the desired average number of deferrals is achieved. 5 Experiments In this section, we evaluate ReDAct on text-based interactive environments. We consider both task performance and the economic benefits of uncertainty-based deferral. Note that while our results are focused on text-based environments, our methodology is not limited to them and can be extended to other modalities, e.g., visual-textual. 5.1 Experimental Setup We start by describing in detail our experimental setup, including the datasets, models, metrics, and baselines used for evaluation. Datasets. In our experiments, we consider two text-based embodied environments, specifi- cally ALFWorld (Shridhar et al., 2021) and MiniGrid (Chevalier-Boisvert et al., 2023). âąFor ALFWorld, we considered 400 episodes, with the maximum number of steps per episode capped at 50. For more details see Appendix B.1. âąFor MiniGrid, we considered 200 episodes, with the maximum number of steps per episode capped at 50. Note that in the main body of the paper, we use a full-view setting for MiniGrid. The reason is that the majority of models we considered (including large models) failed to produce reasonable results in a partial-view setup. For more details and partial-view results, see Appendix B.2. Models and Prices. As described in Section 3, the agent is equipped with two LLMs. The first is a small, inexpensive model that can even be run locally, without paying high per-token usage fees. However, it may be more prone to hallucinations than its expensive counterpart. As small models, we consider Qwen3-80B, Llama3.3-70B, and Llama4-Maverick. The second is a large, more reliable model with much higher per-token usage costs. As large models, we consider GPT-5.2, Qwen3-235B, and Qwen3-480B. To infer these models (except for GPT-5.2), we use the Together AI service. 1 More details on the models we use are provided in Appendix B.3. We also provide a comparative table with input/output token prices for different models in Table 7. Metrics. As described in Section 3, each episode has a binary outcome: success (1) or failure (0). The episode is considered successful if the agent achieves the goal within 50 steps. Otherwise, it is considered a failure. Therefore, one natural measure of an agentâs performance is the success rate, computed as the ratio of successful episodes to the total 1 https://w.together.ai/ 6 Preprint. Under review. BaseSettingMTESPPPLRandom Qwen3-80B Base0.683± 0.025 + GPT-5.20.788± 0.0200.798± 0.0250.808± 0.0110.770± 0.026 + Qwen3-480B 0.743± 0.0200.763± 0.0140.790± 0.0230.760± 0.029 Llama3.3-70B Base 0.635± 0.018 + GPT-5.2 0.768± 0.0170.763± 0.0250.800± 0.0280.750± 0.025 + Qwen3-480B0.755± 0.0200.760± 0.0280.773± 0.0170.690± 0.026 GPT-5.2Base0.783± 0.019 Qwen3-480BBase 0.793± 0.024 Table 2: Success rate on ALFWorld with bootstrapped standard deviation. For the small model without deferral, all methods coincide. BaseSettingMTESPPPLRandom Qwen3-80B Base0.855± 0.024 + GPT-5.20.930± 0.0150.950± 0.0130.950± 0.0130.930± 0.019 + Qwen3-235B0.930± 0.0130.915± 0.0170.940± 0.0200.900± 0.012 Llama4 Base 0.220± 0.024 + GPT-5.2 0.370± 0.0500.325± 0.0300.420± 0.0230.380± 0.039 + Qwen3-235B0.330± 0.0210.355± 0.0470.370± 0.0310.295± 0.027 GPT-5.2Base0.990± 0.007 Qwen3-235BBase0.925± 0.023 Table 3: Success rate on MiniGrid full-view with bootstrapped standard deviation. For the small model without deferral, all methods coincide. number of episodes: Success Rate = #Successful episodes #Total episodes . Another important dimension is cost. As shown in Table 7, large models are significantly more expensive than small models. Therefore, we also report the overall inference expenses alongside downstream performance. When running only the large or small model without deferral, the cost reduces to that of the corresponding model alone. For ReDAct, the reported cost is the cumulative cost of the whole agent (small and large models). Ideally, one would like to achieve a success rate comparable to that of the large model alone, but at a much lower cost. ReDAct is designed to achieve this tradeoff. Baselines for deferral. As discussed in Section 4, we focus on action-level uncertainty measures for deferral. Among those, information-theoretic measures (MTE, SP, and PPL) showed the best discriminative performance and are therefore used as the main deferral indicator in ReDAct. We calibrate the deferral threshold onN cal =100 episodes with a target ofK =5 large model calls per episode on average. As an additional baseline, we consider random large model invocation (see Section 4). Below, we report results for all of these choices. 5.2 Results Tables 2 and 3 present the main results of applying ReDAct to ALFWorld and MiniGrid, respectively. Every combination of small model, large model, and dataset shows that deferring to the large model guided by the UQ signal from the small model is superior to random deferral. Almost any choice of UQ measure improves the small modelâs success rate, but the best results are achieved with Perplexity (PPL). Most notably on ALFWorld, PPL-based deferral achieves the full performance of the large model while invoking it for onlyâŒ15% of the total number of steps. 7 Preprint. Under review. 02k4k7k10k Number of large model calls 0.650 0.675 0.700 0.725 0.750 0.775 0.800 Success rate Qwen3-80B Qwen3-80B + GPT5.2 Qwen3-80B + Qwen3-480B Llama3.3-70B Llama3.3-70B + GPT5.2 Llama3.3-70B + Qwen3-480B GPT5.2 Qwen3-480B Small model Large: GPT5.2 Large: Qwen3-480B Pareto front Figure 2: Pareto front of success rate vs. large model calls on ALFWorld with perplexity- based deferral. ReDAct enables small models to approach large model performance while using only 15% of large model calls. Additionally, we show that UQ-guided deferral achieves Pareto-optimality both with respect to raw number of large model calls (Figures 2 and 10) and to the actual inference costs (Figures 11 and 12). This further strengthens our claim that cost-aware model routing is useful, and UQ guidance provides a valid signal for it. Distribution of calls. Figure 3 provides further evidence of the distinct nature of UQ-based deferral compared to the random approach. The figure shows that the relative frequency of large model calls per step for random deferral results in an almost perfectly uniform 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 PPL Large model calls: 1447 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 Random Large model calls: 1487 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 PPL Large model calls: 697 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 Random Large model calls: 892 Figure 3: Qwen3-80B + GPT-5.2. Large model invocation frequency by step in ALFWorld (top row) and MiniGrid (bottom row). At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 8 Preprint. Under review. BaseSettingMTESPPPLRandom Qwen3-80B Base9.22 + GPT-5.215.4316.3916.2514.15 + Qwen3-480B 16.316.1515.3913.84 Llama3.3-70B Base29.24 + GPT-5.2 31.5630.7230.7629.27 + Qwen3-480B33.0532.1833.2131.82 GPT-5.2Base45.21 Qwen3-480BBase 52.44 Table 4: Costs for inference per model in $ for ALFWorld. Total is $512 distribution, as expected. For UQ-guided deferral distinct behavior is observed between datasets. For ALFWorld (see Figure 3, top row) the distribution is heavily skewed to the right. This explains why UQ-based deferral makes inference slightly more expensive on ALFWorld than random baseline, as shown in Table 4. During unsuccessful episodes, the small model becomes more uncertain as it digs itself deeper into a sub-optimal trajectory, and the likelihood of its uncertainty exceeding the deferral threshold increases. Indeed, for sequential agentic tasks, later steps have strictly longer prompts as they contain the full history of previous steps. Given the same average number of large model calls for random and UQ-based approaches, this results in the average large model call having a higher count of input tokens, thus making it marginally more expensive. This intuition is further confirmed by Tables 14, 15, 16 and 17 (see Appendix F). The total count of input tokens is much higher for UQ-based setups, but the final cost is only slightly increased due to the low per-token cost of inputs. For MiniGrid (see Figure 3, bottom row), the distribution of large model calls has a distinct bimodal characteristic. This implies severe dependency of the complexity of the task on the initial position of the agent. An unfavorable start leads to an early deferral, producing the first mode. The second mode can be explained by the same argument, as the one used for ALFWorld. The distributions for remaining combinations of models can be found in Appendix F. We report additional experimental results in Appendix F, including number of large model calls, detailed token counts, and average number of steps per episode across all experimental setups. Cost-effectiveness. To demonstrate the cost-effectiveness of ReDAct, we report actual inference costs in USD incurred by each evaluation setup on the ALFWorld benchmark in Table 4. It is evident that using the large model only when it matters results in real cost reduction without losing performance. Compared to the cost of using the large model exclusively, PPL-based deferral increases inference costs only modestly relative to running the small model alone. 6 Conclusion In this paper, we propose ReDAct, an uncertainty-aware deferral framework for LLM agents operating in sequential environments. The agent is equipped with two LLMs and defers individual action decisions from the small model to the large model when the small modelâs uncertainty is high. We evaluated our approach on text-based embodied environments and demonstrated that deferring only about 15% of decisions can match, and sometimes surpass, the performance of the large model alone at a fraction of the cost. We also found that action-level uncertainty provides the most effective deferral signal, with information-theoretic measures such as Perplexity performing particularly well. We believe that adaptive switching between models within an agent is a promising direction for cost-efficient deployment, and that ReDAct offers a simple yet effective approach. 9 Preprint. Under review. 7 Limitations While ReDAct significantly improves the performance of small models on agentic tasks at a fraction of the cost of using large models exclusively, our study has certain limitations. First, ReDAct relies on information-theoretic UQ methods, which require access to token-level probabilities. In practice, however, most models of the size considered here expose logprobs through inference provider APIs. Second, we consider only models with more than 70B parameters. This choice is dictated by task complexity: smaller models perform poorly on agentic tasks without specialized fine-tuning and are thus outside the scope of this work. Acknowledgements Part of the computational resources for our experiments were provided by Together AI. 10 Preprint. Under review. References Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243â297, 2021. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. In NIPS 2016 Deep Learning Symposium, 2016. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, p. 287â318. PMLR, 2023. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, p. 1877â1901, 2020. Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & Miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems, 2023. C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41â46, 2003. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1â113, 2023. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive un- certainty quantification of free-form large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), p. 5050â5063. Association for Computational Linguistics, August 2024. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-Polygraph: Uncertainty estimation for language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 446â461, Singapore, 2023. Association for Computational Linguistics. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr Ì ed Ì eric Blain, Francisco Guzm Ì an, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539â555, 2020. Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, p. 1050â1059. PMLR, 2016. 11 Preprint. Under review. Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56:1513â1589, 2023. Yonatan Geifman and Ran El-Yaniv. SelectiveNet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, p. 2151â2159. PMLR, 2019. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, p. 1769â1782. PMLR, 2023. Eyke H Ì ullermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457â506, 2021. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, volume 30, 2017. Benjamin Kompa, Jasper Snoek, and Andrew L Beam. Second opinion needed: communi- cating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1):4, 2021. Nikita Kotelevskii, Samuel Horv Ì ath, Karthik Nandakumar, Martin Tak Ì ac, and Maxim Panov. Dirichlet-based uncertainty quantification for personalized federated learning with improved posterior networks. IJCAI 2024, 2024. Nikita Kotelevskii, Vladimir Kondratyev, Martin Tak Ì a Ë c, Eric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via Bayesian esti- mation. In The Thirteenth International Conference on Learning Representations, 2025. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic in- variances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2024. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021. Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, p. 7076â7087. PMLR, 2020. Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, et al. Efficient test-time scaling of multi-step reasoning by probing internal states of large language models. arXiv preprint arXiv:2511.06209, 2025. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Advances in Neural Information Processing Systems, volume 37, p. 134507â134533, 2024. 12 Preprint. Under review. Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference on Robot Learning, 2023. Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, p. 68539â68551, 2023. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, p. 8634â8652, 2023. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10740â10749, 2020. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interac- tive learning. In International Conference on Learning Representations, 2021. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 5433â5442, Singapore, December 2023. Association for Computational Linguistics. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with LM-Polygraph. Transactions of the Association for Computational Linguistics, 13:220â248, 2025a. Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shelmanov, and Maxim Panov. CoCoA: A minimum Bayes risk framework bridg- ing confidence and consistency for uncertainty quantification in LLMs. In Advances in Neural Information Processing Systems, 2025b. Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, p. 22184â22202. PMLR, 2022. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, p. 20744â20757, 2022a. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022b. Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representations, 2023. 13 Preprint. Under review. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and Chatbot arena. In Advances in Neural Information Processing Systems, volume 36, p. 46595â46623, 2023. 14 Preprint. Under review. A Measures of Uncertainty We are using the following UQ methods: âą Sequence Probability (SP): U SP =â L â i=1 log p(y i | x, y <i ,Ξ).(2) âą Perplexity (PPL): U PPL =â 1 L L â i=1 log p(y i | x, y <i ,Ξ).(3) âą Mean Token Entropy (MTE): U MTE = 1 L L â i=1 H(y i | x, y <i ,Ξ),(4) where H(y i | x, y <i ,Ξ) is the entropy of the token distribution p(y i | x, y <i ,Ξ). B Additional Details on Evaluation Setup B.1 ALFWorld For ALFWorld, we report results on AlfredTWEnv type with all available task types: Pick & Place, Examine in Light, Clean & Place, Heat & Place, Cool & Place, Pick Two & Place. The calibration subset of 100 episodes was randomly sampled from validseen split. B.2 Minigrid For MiniGrid, we adopt full-view mode of MiniGrid-DoorKey-8x8-v0 environment with random seed 42. This environment has a key that the agent must pick up to unlock a door and then get to the goal square to successfully finish the episode. Calibration is performed under the same settings, but with a different random seed (993) to ensure no overlap with the test set. For threshold calibration, we run 100 episodes using this seed. We conducted a preliminary investigation in the partial-view settings. However, both the large and small models performed poorly in this regime, so we decided to focus on the full-view mode in the main results. The results for the partial-view mode are presented in Table 5. ModelSuccess Rate Qwen3-80B0.630 Llama4-Maverick 0.105 Llama3.3-70B0.000 GPT-OSS-20B0.000 Mistral-8x-7B0.000 Mistral-Small 0.000 Qwen3.5-9B0.000 GPT-5.20.865 Qwen3-480B0.215 Qwen3-235B0.201 Table 5: Success rate on MiniGrid partial-view across small and large models. 15 Preprint. Under review. Figure 4: Prediction-Rejection Ratio (PRR) Curve. The curve illustrates the quality of the non-rejected predictions as a function of the rejection rate. Oracle represents the optimal rejection strategy, Random is a random rejection, and UE is rejection based on the evaluated uncertainty estimation method. B.3 Models We report the full model names for all models inferred via Together AI: small models - Qwen/Qwen3-Next-80B-A3B-Instruct, meta-llama/Llama-3.3-70B-Instruct-Turbo and meta- llama/Llama-4-Maverick-17B-128E-Instruct-FP8, large models - Qwen/Qwen3-235B-A22B- Instruct-2507-tput and Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8. C Metrics Prediction Rejection Ratio. To evaluate the discriminative quality of UQ methods, we use the Prediction Rejection Ratio (PRR), which measures the effectiveness of uncertainty scores for identifying high-quality predictions (Malinin & Gales, 2021). PRR operates by progressively rejecting predictions with uncertainty scores above a threshold and observing how the average quality of the remaining predictions changes. To compute PRR, one needs to estimate areas under two different curves. The first is the area between the Prediction Rejection (PR) curve for the evaluated uncertainty score and a random baseline, and the second is the area between the oracle (the ideal uncertainty score that perfectly aligns with quality) and the random baseline. Formally, PRR is defined as follows: PRR = AUC unc â AUC rnd AUC oracle â AUC rnd .(5) The higher PRR values indicate better ordering, induced by the uncertainty scores, and approach the performance of an oracle. To prevent cases where excessive rejection artificially inflates quality measures, we compute PRR only up to a rejection threshold of 50%. Figure 4 gives a visual representation of the PRR calculation, highlighting the relationship between the uncertainty threshold and the quality measure. D Action vs Reasoning-level Uncertainty Results We provide full results for the action-level UQ in Table 6. The considered UQ methods can be grouped into two main categories. âą Information-based: Sequence Probability, Mean Token Entropy and Perplexity. âą Sample diversity: Number of Semantic Sets (Lin et al., 2024), Sum of Eigenval- ues (Lin et al., 2024), Eccentricity (Lin et al., 2024), Lexical Similarity (Fomicheva et al., 2020), Sentence SAR (Duan et al., 2024), Degree Matrix (Lin et al., 2024), Semantic Density (Qiu & Miikkulainen, 2024), Predictive Entropy (Kuhn et al., 2023) and CoCoA (Vashurin et al., 2025b). 16 Preprint. Under review. UQ metricPRRROC-AUC Number Semantic Sets0.1440.545 Sum of Eigenvalue 0.2130.575 Eccentricity0.2180.578 Lexical Similarity 0.2190.579 Sentence SAR0.2600.603 Cocoa PPL 0.3000.628 Degree Matrix0.3340.668 Semantic Density 0.3430.671 Predictive Entropy0.3560.663 Cocoa SP0.3570.662 Cocoa MTE 0.3630.664 Semantic Entropy0.3650.665 Perplexity0.3990.684 Sequence Probability 0.4170.704 Mean Token Entropy0.4240.710 Table 6: Performance comparison of uncertainty metrics estimated over action selection. You are an AI agent solving a task in an interactive environment. TASK DESCRIPTION: DESCRIPTION ENVIRONMENT HISTORY: HISTORY AVAILABLE COMMANDS: AVAILABLE COMMANDS Think step by step about the current situation and consider what action to take next. Your thought process: Figure 5: Prompt to elicit reasoning trace in the ReDAct framework for ALFWorld. For implementation, we use the LM-Polygraph library (Fadeeva et al., 2023). To collect data, we sample 100 tasks from ALFWorld and simulate 100 trajectories with the ReAct framework using Qwen3-80B as the small model, resulting in a total of 2411 steps. Each step was labeled by GPT-5.2 as an LLM-as-a-judge, with 1555 steps marked correct and 856 incorrect. E Prompts For reproducibility purposes, we report all prompts used in our experiments: âą ALFWorld. The reasoning prompt is displayed in Figure 5 and the action selection prompt in Figure 6. âąMiniGrid. The reasoning prompt is shown in Figure 7, while the action selection prompt is provided in Figure 8. F Additional Experimental Results Number of calls to large model. Tables 12 and 13 report the number of calls to a large model for the ALFWorld and MiniGrid benchmarks, respectively. Approximate parity has been achieved between various setups, with the exception of Llama3.3-70B runs on ALFWorld, where the random baseline used slightly fewer large model calls. On the other hand, in MiniGrid experiments with Qwen3-80B as the small model, the random baseline used slightly more calls. We attribute this to the limitations of the calibration procedure, discussed in detail in Section 4.2. 17 Preprint. Under review. You are an AI agent solving a task in an interactive environment. TASK DESCRIPTION: DESCRIPTION ENVIRONMENT HISTORY: HISTORY YOUR CURRENT REASONING: THOUGHTS AVAILABLE COMMANDS: AVAILABLE COMMANDS OUTPUT RULES: - Output exactly ONE line. - That line must be EXACTLY one of the AVAILABLE COMMANDS. - Do NOT output reasoning, explanation, punctuation, or extra words. Now output your chosen action (one line only): Figure 6: Prompt to select an action in the ReDAct framework for ALFWorld. ModelInput Tokens PriceOutput Tokens Price Qwen3-80B0.151.50 Llama3.3-70B0.880.88 Llama4-Maverick0.270.85 GPT-5.21.7514.00 Qwen3-480B2.002.00 Table 7: Input and Output token price by model in USD per million tokens. Distributions of calls to large model.Figures 13a through 16b depict the distributions of the large model calls for the remainder of the experimental setups. The general character- istics follow the patterns discussed in Section 5, with a notable exception of experiments where Llama4-Maverick acts as the small model on the MiniGrid benchmark. Here, UQ- guided deferral is distributed closer to a uniform distribution than in other setups. The low base quality of this model on MiniGrid can explain this - the model is bad enough for the task to not know when to express confidence, which leads to a flatter distribution of calls. Number of steps to success. Tables 10 and 11 report average number of steps until completion for successful episodes for ALFWorld and MiniGrid, respectively. Given the magnitude of standard deviations, we conclude that deferral does not significantly impact the expected completion time for both benchmarks. Token usage.Tables 14 through 17 report detailed breakdown of token usage along with associated costs for all experiments on ALFWorld. Pareto-optimality with respect to cost. Figures 11 and 12 report Pareto-optimality of ReDAct with respect to success rate and inference cost in USD. Almost all ReDAct variants lie on the Pareto front, with the exception of Llama4-Maverick on MiniGrid, where ReDAct lies behind the front due to its low base performance on the benchmark. Still, ReDAct doubles the small modelâs success rate at only a modest cost increase. 18 Preprint. Under review. You are an AI agent solving a task in an interactive environment. TASK DESCRIPTION: DESCRIPTION ENVIRONMENT HISTORY: HISTORY AVAILABLE COMMANDS: AVAILABLE COMMANDS Think step by step, but keep your reasoning short, precise, and grounded only in the current observation and history. Follow this reasoning procedure strictly: 1. Summarize only the important facts from the current observation and history. 2. Identify the current subgoal: - If the key has not been picked up, the subgoal is to get key. - If the key has been picked up but the door is not open, the subgoal is to open door. - If the door is open, the subgoal is to go to goal. 3. Identify the exact position of the agent and the current target using the provided grid. - State the agent position and facing direction. - State the target position. - Compute the targetâs relative location from the agent: same cell, directly ahead, behind, left, right, or requiring movement. - Determine whether the target is immediately interactable: - pickup is valid only if the key is in the cell directly in front. - toggle is valid only if the door is in the cell directly in front. 4. Check whether the last few actions show useless repetition or left-right oscillation. 5. Choose the one action that makes the most direct progress toward the current subgoal. 6. Do not change subgoal unless the current one is completed or impossible. Reasoning rules: - Turning changes orientation only; it does not move the agent. - pickup and toggle work only when the target object is directly in front of the agent. - Do not alternate left and right repeatedly unless there is a clear new reason. - If you already know the door exists but do not have the key, prioritize getting the key instead of returning attention to the door. After forming your initial reasoning, perform a brief self-critique: Critique checklist: - Did I use correct coordinates from the grid? - If I chose pickup or toggle, is the target actually in the front cell? - Does this action make progress toward the current subgoal? - Am I repeating recent actions (e.g., left-right oscillation)? - Am I mistakenly focusing on the door before getting the key? If any issue is found, correct your reasoning and update the action choice. Keep the critique short and focused. Do not repeat the full reasoning. Keep the reasoning compact and action-oriented. Your thought process: Figure 7: Prompt to elicit reasoning trace in the ReDAct framework for MiniGrid. 19 Preprint. Under review. You are an AI agent solving a task in an interactive environment. TASK DESCRIPTION: DESCRIPTION ENVIRONMENT HISTORY: HISTORY YOUR CURRENT REASONING: THOUGHTS AVAILABLE COMMANDS: AVAILABLE COMMANDS OUTPUT RULES: - Output exactly ONE line. - That line must be EXACTLY one of the AVAILABLE COMMANDS. - Do NOT output reasoning, explanation, punctuation, or extra words. Now output your chosen action (one line only): Figure 8: Prompt to select an action in the ReDAct framework for MiniGrid. You are an expert evaluator of agent trajectories in ALFWorld. Your task: Evaluate whether each step in the trajectory is helpful toward completing the task. Output format: Return ONLY a single JSON object. Each key must be âstep iâ (with i starting from 1). Each value must be an object with: - âlabelâ: 0 or 1 - âreasonâ: a short 1 sentence explanation Example structure: âstep 1â:âlabelâ: 1, âreasonâ: âGoing to the fridge is a reasonable way to look for food items such as eggs.â , âstep 2â: âlabelâ: 0, âreasonâ: âThis repeats a previous inspection without adding new information.â Do NOT include any extra text or code fences outside the JSON. Return ONLY the raw JSON. Labeling rules: A step should be labeled 1 (good) if: - It is a reasonable attempt to find the needed object for the task. - It involves searching a plausible location (e.g., opening a cabinet, drawer, fridge, etc.). - It examines or moves toward objects likely relevant to the task or the next logical subgoal. - It is part of logical exploration, even if the final goal is not yet achieved. A step should be labeled 0 (bad) if: - It repeats an action with no new benefit (e.g., re-examining the same object without a clear reason). - It returns to a place already fully explored without purpose. - It manipulates objects clearly irrelevant to the task. - It represents looping or wandering behavior that does not help progress. Important: - Exploration steps SHOULD be labeled 1 if they could reasonably lead to the target object. - Do NOT mark all steps as 0 unless literally every step is clearly irrelevant to the task. Now evaluate the following trajectory. Return ONLY the JSON. Trajectory: TRAJECTORY Figure 9: Prompt for GPT-5.2 to label steps in AlfWorld trajectories. 20 Preprint. Under review. BaseSettingMTESPPPLRandom Qwen3-80B Base 0 + GPT-5.2 1331144714471487 + Qwen3-480B1187119311401141 Llama3.3-70B Base0 + GPT-5.21446134914261099 + Qwen3-480B1445138314811139 GPT-5.2Base9898 Qwen3-480BBase10425 Table 8: Number of large model calls on ALFWorld. For the small model without deferral, all methods coincide. BaseSettingMTESPPPLRandom Qwen3-80B Base0 + GPT-5.2765763697892 + Qwen3-235B870895803920 Llama4-Maverick Base 0 + GPT-5.2 914915900901 + Qwen3-235B 9651019942932 GPT-5.2Base3820 Qwen3-235BBase5047 Table 9: Number of large model calls on MiniGrid. For the small model without deferral, all methods coincide. 0.85 0.90 0.95 1.00 Qwen3-80B Qwen3-80B + GPT5.2 Qwen3-80B + Qwen3-235B GPT5.2 Qwen3-235B 01k2k4k6k Number of large model calls 0.2 0.3 0.4 Llama4 Llama4 + GPT5.2 Llama4 + Qwen3-235B Small model Large: GPT5.2 Large: Qwen3-235B Pareto front Success rate Figure 10: Pareto front of success rate vs. large model calls on MiniGrid with perplexity- based deferral. ReDAct enables small models to approach large model performance while using only 15% of large model calls. 21 Preprint. Under review. BaseSettingMTESPPPLRandom Qwen3-80B Base 19.58± 11.72 + GPT-5.2 20.31± 12.0420.87± 12.6121.49± 13.4120.36± 12.10 + Qwen3-480B21.39± 12.6120.88± 13.2421.32± 12.7819.96± 12.42 Llama3.3-70B Base 20.83± 13.27 + GPT-5.2 20.78± 12.6820.47± 12.0921.32± 12.7820.35± 12.43 + Qwen3-480B21.12± 12.3419.44± 11.9121.09± 12.5419.35± 11.45 GPT-5.2Base17.73± 12.31 Qwen3-480BBase19.79± 12.19 Table 10: Average number of steps before success with standard deviation for ALFWorld. BaseSettingMTESPPPLRandom Qwen3-80B Base26.47± 8.62 + GPT-5.224.12± 7.0925.02± 7.7224.08± 6.8125.09± 7.61 + Qwen3-235B25.65± 7.5525.34± 7.4925.64± 8.0625.38± 7.81 Llama4-Maverick Base33.41± 9.79 + GPT-5.233.53± 7.0333.83± 8.4032.40± 8.7633.93± 8.89 + Qwen3-235B33.17± 8.9932.44± 8.9932.53± 8.8033.49± 9.18 GPT-5.2Base18.79± 4.12 Qwen3-235BBase23.23± 6.46 Table 11: Average number of steps before success with standard deviation for MiniGrid. BaseSettingMTESPPPLRandom Qwen3-80B + GPT-5.23.328± 5.2703.618± 5.4533.618± 5.3693.717± 2.852 + Qwen3-480B2.967± 4.7472.982± 4.4062.850± 4.4052.853± 2.328 Llama3.3-70B + GPT-5.23.615± 3.9903.373± 3.9472.850± 4.4052.748± 2.195 + Qwen3-480B3.612± 4.3433.458± 4.6333.703± 4.6382.848± 2.346 Table 12: Mean number of large model calls with standard deviation per episode for ALFWorld. BaseSettingMTESPPPLRandom Qwen3-80B + GPT-5.23.825± 3.4493.815± 3.2113.485± 2.4214.460± 2.414 + Qwen3-235B4.350± 3.5944.475± 3.7034.015± 3.1264.600± 2.532 Llama4-Maverick + GPT-5.2 4.570± 2.6734.575± 2.7084.500± 2.7064.505± 2.121 + Qwen3-235B 4.825± 2.7865.095± 2.8984.710± 2.5314.660± 2.125 Table 13: Mean number of large model calls with standard deviation per episode for MiniGrid. MTESPPPLRandom Small Model Input Tokens25,638,66925,850,19626,166,77725,463,015 Small Model Output Tokens2,530,4462,522,1102,546,1532,579,383 Large Model Input Tokens 3,905,2714,372,5424,236,1473,209,024 Large Model Output Tokens68,35076,76578,51460,285 Large Model Calls1,3311,4471,4471,487 Small Model price $7.647.667.747.69 Large Model price $7.798.738.516.46 Total price $ 15.4316.3916.2514.15 Table 14: ALFWorld with Qwen3-80B + GPT-5.2. Token usage and cost comparison across different uncertainty metrics. 22 Preprint. Under review. MTESPPPLRandom Small Model Input Tokens27,982,52826,776,45725,846,24426,049,938 Small Model Output Tokens 2,986,4142,886,5702,828,2772,823,600 Large Model Input Tokens3,582,0413,660,3923,411,1462,652,560 Large Model Output Tokens 228,614239,339229,266196,881 Large Model Calls1,1871,1931,1401,141 Small Model price $8.688.358.128.14 Large Model price $ 7.627.807.285.70 Total price $ 16.3016.1515.4013.84 Table 15: ALFWorld with Qwen3-80B + Qwen3-480B. Token usage and cost comparison across different uncertainty metrics. MTESPPPLRandom Small Model Input Tokens25,724,00525,400,45624,894,95626,120,373 Small Model Output Tokens1,381,5911,382,2681,367,7651,458,582 Large Model Input Tokens 3,827,3393,554,0583,833,1182,480,882 Large Model Output Tokens72,13966,60567,57247,121 Large Model Calls1,4461,3491,4261,099 Small Model price $23.8523.5723.1124.27 Large Model price $7.717.157.655.00 Total price $31.5630.7230.7629.27 Table 16: ALFWorld with Llama3.3-70B + GPT-5.2. Token usage and cost comparison across different uncertainty metrics. MTESPPPLRandom Small Model Input Tokens26,235,70425,074,27425,778,04227,914,399 Small Model Output Tokens1,613,2011,543,9821,596,7701,679,948 Large Model Input Tokens3,994,8204,108,7104,278,4142,692,680 Large Model Output Tokens275,371271,067282,763197,577 Large Model Calls 1,4451,3831,4811,139 Small Model price $24.5123.4224.0926.04 Large Model price $8.548.769.125.78 Total price $33.0532.1833.2131.82 Table 17: ALFWorld with Llama3.3-70B + Qwen3-480B. Token usage and cost comparison across different uncertainty metrics. 23 Preprint. Under review. 1020304050 Cost 0.650 0.675 0.700 0.725 0.750 0.775 0.800 Success rate Qwen3-80B Qwen3-80B + GPT5.2 Qwen3-80B + Qwen3-480B GPT5.2 Qwen3-480B Small model Large: GPT5.2 Large: Qwen3-480B Pareto front (a) Qwen3-80B 3035404550 Cost 0.650 0.675 0.700 0.725 0.750 0.775 0.800 Success rate Llama3.3-70B Llama3.3-70B + GPT5.2 Llama3.3-70B + Qwen3-480B GPT5.2 Qwen3-480B Small model Large: GPT5.2 Large: Qwen3-480B Pareto front (b) Llama3.3-70B Figure 11: Pareto front of success rate vs. cost on ALFWorld with perplexity-based deferral. Each plot shows the trade-off between performance and computational cost. (a): Qwen3-80B small model. (b): Llama3.3-70B small model. 5101520253035 Cost 0.84 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Success rate Qwen3-80B Qwen3-80B + GPT5.2 Qwen3-80B + Qwen3-235B GPT5.2 Qwen3-235B Small model Large: GPT5.2 Large: Qwen3-235B Pareto front (a) Qwen3-80B 0.85 0.90 0.95 1.00 GPT5.2 Qwen3-235B 05101520253035 Cost 0.2 0.3 0.4 Llama4 Llama4 + GPT5.2 Llama4 + Qwen3-235B Small model Large: GPT5.2 Large: Qwen3-235B Pareto front Success rate (b) Llama4-Maverick Figure 12: Pareto front of success rate vs. cost on MiniGrid with perplexity-based deferral. Each plot shows the trade-off between performance and computational cost. (a): Qwen3-80B small model. (b): Llama4-Maverick small model. 24 Preprint. Under review. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 MTE Large model calls: 1331 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 SP Large model calls: 1447 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 PPL Large model calls: 1447 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 Random Large model calls: 1487 (a) Qwen3-80B + GPT-5.2. Large model invocation frequency by step in ALFWorld. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + Qwen3-480B MTE Large model calls: 1187 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + Qwen3-480B SP Large model calls: 1193 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + Qwen3-480B PPL Large model calls: 1140 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + Qwen3-480B Random Large model calls: 1141 (b) Qwen3-80B + Qwen3-480B. Large model invocation frequency by step in ALFWorld. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 25 Preprint. Under review. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama3.3-70B + GPT5.2 MTE Large model calls: 1446 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama3.3-70B + GPT5.2 SP Large model calls: 1349 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama3.3-70B + GPT5.2 PPL Large model calls: 1426 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama3.3-70B + GPT5.2 Random Large model calls: 1099 (a) Llama3.3-70B + GPT-5.2. Large model invocation frequency by step in ALFWorld. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama3.3-70B + Qwen3-480B MTE Large model calls: 1445 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama3.3-70B + Qwen3-480B SP Large model calls: 1383 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama3.3-70B + Qwen3-480B PPL Large model calls: 1481 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama3.3-70B + Qwen3-480B Random Large model calls: 1139 (b) Llama3.3-70B + Qwen3-480B. Large model invocation frequency by step in ALFWorld. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 26 Preprint. Under review. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 MTE Large model calls: 765 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 SP Large model calls: 763 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + GPT5.2 PPL Large model calls: 697 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + GPT5.2 Random Large model calls: 892 (a) Qwen3-80B + GPT-5.2. Large model invocation frequency by step in MiniGrid. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + Qwen3-235B MTE Large model calls: 870 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + Qwen3-235B SP Large model calls: 895 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Qwen3-80B + Qwen3-235B PPL Large model calls: 803 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Qwen3-80B + Qwen3-235B Random Large model calls: 920 (b) Qwen3-80B + Qwen3-480B. Large model invocation frequency by step in MiniGrid. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 27 Preprint. Under review. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama4 + GPT5.2 MTE Large model calls: 914 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama4 + GPT5.2 SP Large model calls: 915 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama4 + GPT5.2 PPL Large model calls: 900 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama4 + GPT5.2 Random Large model calls: 901 (a) Llama4-Maverick + GPT-5.2. Large model invocation frequency by step in MiniGrid. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama4 + Qwen3-235B MTE Large model calls: 965 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama4 + Qwen3-235B SP Large model calls: 1019 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Frequency Llama4 + Qwen3-235B PPL Large model calls: 942 01020304050 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Llama4 + Qwen3-235B Random Large model calls: 932 (b) Llama4-Maverick + Qwen3-480B. Large model invocation frequency by step in MiniGrid. At each step, the frequency is defined as the number of large model calls divided by the number of episodes that reached that step. 28