Paper deep dive
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics
Weiliang Chen, Song Liao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/13/2026, 3:11:01 AM
Summary
The paper introduces RATED, a reliability-aware AI agent tool retrieval framework that combines semantic matching with dynamic reliability adaptation via contextual bandits. It addresses the gap in existing methods that ignore tool availability during execution. The authors propose EcoAgent, a benchmark of 5,172 real-world tools, and demonstrate that RATED significantly improves Mean Reciprocal Rank (MRR) and reduces fault selection rates under API outage scenarios compared to baselines like ToolLLM.
Entities (10)
Relation Signals (10)
EcoAgent → contains → 5,172 real-world tools
confidence 98% · EcoAgent, a benchmark of 5,172 real-world tools
RATED → uses → DRAGON
confidence 95% · RATED employs a frozen dual-encoder... based on DRAGON... for semantic matching
RATED → uses → Contextual Bandit
confidence 95% · RATED employs... a lightweight contextual bandit module that re-ranks candidates based on real-time execution feedback.
RATED → outperforms → ToolLLM
confidence 92% · RATED achieves a 52.2% relative MRR improvement over the ToolLLM API retriever
RATED → reduces → fault selection rate
confidence 90% · reduces fault selection rate from 86.2% to 2.9% under severe API outage scenarios
Contextual Bandit → implements → LinUCB
confidence 88% · RATED adopts discounted LinUCB... with a separate linear model per tool.
EcoAgent → derivedfrom → LangChain
confidence 85% · We crawl tool catalogs from LangChain... to construct EcoAgent
EcoAgent → derivedfrom → CrewAI
confidence 85% · We crawl tool catalogs from... CrewAI... to construct EcoAgent
EcoAgent → derivedfrom → LlamaIndex
confidence 85% · We crawl tool catalogs from... LlamaIndex... to construct EcoAgent
EcoAgent → derivedfrom → n8n
confidence 85% · We crawl tool catalogs from... n8n... to construct EcoAgent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As AI agents increasingly rely on external tools, accurate retrieval is essential for reliable behavior. Existing methods optimize semantic relevance but largely ignore tool availability during execution. To address this gap, we propose RATED (Reliability-Aware Tool Retrieval via Execution Dynamics), a retrieval framework that combines robust semantic matching with dynamic reliability adaptation. RATED employs a frozen dual-encoder with contrastive fine-tuning for semantic matching, and a lightweight contextual bandit module that re-ranks candidates based on real-time execution feedback. This design enables rapid adaptation to runtime tool failures without expensive model retraining. We introduce EcoAgent, a benchmark of 5,172 real-world tools and over 210,000 query-tool pairs for offline retrieval and dynamic reliability evaluation. Experimental results demonstrate that RATED achieves a 52.2% relative MRR improvement over the ToolLLM API retriever on real-world queries, and reduces fault selection rate from 86.2% to 2.9% under severe API outage scenarios with a 30% failure rate, while incurring negligible latency overhead of less than 1 ms over encode-only inference. The code and dataset are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.13889v1
- Canonical: https://arxiv.org/abs/2610.13889v1
Trouble viewing inline? Open PDF directly →
Full Text
36,464 characters extracted from source content.
Expand or collapse full text
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics Conference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 07–11, 2026; Rome, ItalyProceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, ItalyDOI: 10.1145/3799682.3839887ISBN: 979-8-4007-2539-5/2026/11CCS: Information systems Retrieval models and rankingCCS: Computing methodologies Artificial intelligence Weiliang Chen Affiliation: Department of Computer Science, Texas Tech University, Lubbock, Texas, USA email: weilchen@ttu.edu and Song Liao Affiliation: Department of Computer Science, Texas Tech University, Lubbock, Texas, USA email: song.liao@ttu.edu © c Abstract. As AI agents increasingly rely on external tools, accurate retrieval is essential for reliable behavior. Existing methods optimize semantic relevance but largely ignore tool availability during execution. To address this gap, we propose RATED (Reliability-Aware Tool retrieval via Execution Dynamics), a retrieval framework that combines robust semantic matching with dynamic reliability adaptation. RATED employs a frozen dual-encoder with contrastive fine-tuning for semantic matching, and a lightweight contextual bandit module that re-ranks candidates based on real-time execution feedback. This design enables rapid adaptation to runtime tool failures without expensive model retraining. We introduce EcoAgent, a benchmark of 5,172 real-world tools and over 210,000 query–tool pairs for offline retrieval and dynamic reliability evaluation. Experimental results demonstrate that RATED achieves a 52.2% relative MRR improvement over the ToolLLM API retriever on real-world queries, and reduces fault selection rate from 86.2% to 2.9% under severe API outage scenarios with a 30% failure rate, while incurring negligible latency overhead of less than 1 ms over encode-only inference. Code and dataset are available at https://github.com/Elliotes/EcoAgent. Keywords: AI agents; Tool retrieval; Contextual bandits; Contrastive learning †c-license: by 1. Introduction Large Language Models (LLMs) have demonstrated impressive capabilities in natural language processing but are inherently unable to execute real-world actions (Li et al., 2025; Naveed et al., 2025). To overcome this limitation, recent systems integrate LLMs with external tools and APIs (Mialon et al., 2023; Schick et al., 2023), yielding autonomous AI agents that fulfill actionable requests, such as querying live databases or managing enterprise workflows (Dong et al., 2024). The fundamental bottleneck for these systems has shifted from text generation to accurately identifying and retrieving the correct entry from vast, growing tool repositories (Patil et al., 2023). Figure 1. Tool repository growth and reliability gap on the left, and skewed tag distribution on the right.Left panel shows tool repository growth timeline from February 2025 to January 2026, with total tools (blue line) and reliable tools (orange line). Right panel shows tag word cloud visualization. To address the tool retrieval challenge, existing research has focused on optimizing semantic relevance for Large Language Models (LLMs). Foundational works like ToolLLM (Qin et al., 2023) and ToolExpNet (Zhang et al., 2025) treat tool retrieval as a static ranking task. They use dense retrieval or experience networks to map queries to tool documentation based entirely on semantic relevance. While approaches such as TRFeedback (Xu et al., 2024) and ToolRL (Qian et al., 2025) incorporate execution feedback, they still aim to optimize how the LLM plans and invokes tools in a static environment. However, real AI agent tool ecosystems are far from static and highly unreliable. We crawl tool catalogs from LangChain (LangChain AI, 2023), CrewAI (CrewAI, 2024), LlamaIndex (LlamaIndex, 2024), and n8n (n8n, 2024) to reveal the severity of this issue. As Figure 1 shows, despite rapid growth, over 50% of these tools exhibit critical flaws like broken links and missing authentication. Many tools are functionally similar, so retrieval cannot rely on semantics alone to identify which ones will execute successfully. To overcome this challenge, the system must adapt dynamically during execution. When a tool fails at runtime, the system must promptly adjust candidate ordering between requests. ToolRL cannot achieve this prompt adjustment because it uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024) to update the LLM from batched trajectories and still targets the planner. Contextual bandits (Li et al., 2010) effectively address this problem. They condition each decision on the current query and update from each individual execution outcome. Moreover, they observe feedback only for the selected tool and make immediate adjustments without retraining the semantic retriever. In this paper, we propose RATED (Reliability-Aware Tool retrieval via Execution Dynamics), a retrieval framework that dynamically adapts to runtime tool failures without human intervention. First, we introduce EcoAgent, a benchmark of 5,172 real-world agent tools and over 210,000 query–tool pairs. Second, RATED employs a dual-encoder fine-tuned with a contrastive objective for semantic matching, together with a lightweight contextual bandit that re-ranks the top-K candidates using a prior based on metadata and execution feedback, enabling the system to avoid failing tools. Experiments show that RATED alone keeps fault selection below 3% and recovers at both failure rates. Under a 30% failure injection protocol, RATED reduces fault selection rate to 2.9%, compared with 86.2% for the ToolLLM API retriever, while adding only 0.172 ms latency per step over encode-only inference. 2. The EcoAgent Benchmark We construct EcoAgent, a benchmark comprising two complementary datasets. EcoAgent complements existing tool-use datasets with metadata reliability scores for every tool and with an evaluation protocol for injected execution failures. EcoAgent-Syn supports robust training with synthetic pairs, and EcoAgent-Real supports rigorous evaluation using real-world queries. We aggregate 5,172 real-world tool definitions from four open-source AI agent platforms, namely LangChain, CrewAI, LlamaIndex, and n8n. Metadata are normalized into a shared schema with functional descriptions, names, endpoints, and authentication fields, which supports retrieval text construction and reliability scoring. To operationalize tool quality, each tool receives an automated metadata reliability score between 0 and 1 from three weighted signals, namely name–description consistency, source credibility, and documentation quality, which Section 3.3 defines. The benchmark does not rely on manual per-tool ratings. Tools with scores of at least 0.8 are marked as reliable, identifying 2,940 tools, or 56.8% of the repository. The encoder is trained only on EcoAgent-Syn, while retrieval and dynamic experiment results are validated on the EcoAgent-Real test split, with hyperparameters selected on the validation split. We build EcoAgent-Syn with Gemma-3-1b (Gemini Team, 2025). For each tool, the model receives the tool name, functional description, and metadata as context and generates 10–20 task-oriented queries. Queries are restricted to 5–120 words, and near-duplicates with Jaccard similarity above 0.7 are removed to preserve intent diversity. Figure 2 shows the prompt template, which we also release with the dataset. To improve robustness against spurious correlations, each query is paired with hard negatives from SBERT (Reimers and Gurevych, 2019) retrieval and random negatives. EcoAgent-Syn links 69,501 unique synthetic queries to 5,172 ground-truth tools, with 70,122 positive pairs and 140,244 negative pairs. For evaluation under realistic language distributions, we build EcoAgent-Real from ToolBench (Qin et al., 2023). Because the original dataset lacks ground-truth mappings to our tool repository, we implement a semi-automated labeling pipeline. GTE-Large (Li et al., 2023) retrieves the top-5 candidates per query, discards pairs with cosine similarity below 0.7, and Gemma-3-1b verifies the remainder, retaining only pairs with functional match scores of at least 0.85. Human experts review a stratified sample of 200 queries and agree with the LLM verifier on 94% of cases. After filtering and deduplication, EcoAgent-Real contains 6,379 unique queries and 9,946 verified query–tool pairs. Prompt for Query Synthesis [Task] You are generating diverse user queries for an AI agent tool recommendation system. [Inputs] tool_name the tool name | tool_description its functional description | num_queries how many queries to write [Requirements] (1) Each query should be 5-120 words. (2) Vary the intent types: direct action requests, troubleshooting questions, how-to questions. (3) Use natural, conversational language. (4) DO NOT copy phrases verbatim from the tool description. (5) Make queries task-oriented and practical. [Output] One query per line, no numbering or bullets. A boxed prompt template with four labelled parts. The task line asks the model to generate diverse user queries for an AI agent tool recommendation system. The inputs are the tool name, the tool description, and the number of queries to write. Five numbered requirements ask for queries of 5 to 120 words, varied intent types, natural language, no verbatim copying from the tool description, and task-oriented phrasing. The output line asks for one query per line without numbering or bullets. Figure 2. Prompt template that generates the EcoAgent-Syn queries. Figure 3. RATED architecture with DRAGON retrieval, bandit re-ranking, and execution feedback.RATED architecture with DRAGON dual-encoder retrieval, bandit re-ranking, and execution feedback. 3. Methodology 3.1. Problem Setup We formulate agent tool retrieval as sequential decision making over a shared repository. Let =a1,…,aMU=\a_1,…,a_M\ denote the tool pool, where M=5,172M=5,172 in EcoAgent. At step t, the agent issues query qtq_t. The retriever returns a candidate set t⊂A_t with |t|=K|A_t|=K, and the policy selects one tool at∈ta_t _t for execution. The objective is to maximize the probability of successful execution. Unlike static retrieval with a fixed mapping f(q,a)→0,1f(q,a)→\0,1\, production agents must cope with tools that fail at runtime, where tool utility depends on semantic relevance Ssem(qt,a)S_sem(q_t,a) and time-varying reliability. After execution, only the chosen tool yields binary feedback yt∈0,1y_t∈\0,1\, yielding partial feedback that we address with contextual bandits (Abe et al., 2003). Figure 3 summarizes the RATED pipeline. 3.2. Semantic Perception Module The semantic perception module implements the first stage of RATED. Given qtq_t, it scores all tools in U by SsemS_sem and returns the top-K candidate set tA_t defined in Section 3.1. The semantic encoder is based on DRAGON (Lin et al., 2023), fine-tuned on EcoAgent-Syn and frozen at deployment. New tools can be added by encoding their documentation with fTf_T without retraining the encoders. 3.2.1. DRAGON Dual-Encoder Architecture DRAGON provides separate query encoder fQf_Q and tool encoder fTf_T, which suits settings where user queries and tool documentation follow different lexical patterns. For query q and tool a, we take CLS representations and L2-normalize them. The semantic relevance score is the dot product of the normalized vectors. (1) q=norm(fQ(q)),a=norm(fT(a))e_q=norm(f_Q(q)), _a=norm(f_T(a)) (2) Ssem(q,a)=q⊤aS_sem(q,a)=e_q e_a 3.2.2. Contrastive Fine-tuning on EcoAgent-Syn Although DRAGON is effective for open-domain retrieval, agent tool descriptions use specialized API terminology. We fine-tune fQf_Q and fTf_T on EcoAgent-Syn with an in-batch InfoNCE objective (van den Oord et al., 2019). For a batch of B positive pairs (qi,ai+)i=1B\(q_i,a_i^+)\_i=1^B, we optimize (3) ℒInfoNCE=−1B∑i=1Blogexp(qi⊤ai+/τ)∑j=1Bexp(qi⊤aj+/τ)L_InfoNCE=- 1B _i=1^B (e_q_i e_a_i^+/τ) _j=1^B (e_q_i e_a_j^+/τ) where τ is a temperature hyperparameter. Hard negatives from EcoAgent-Syn construction in Section 2 are used in training. In-batch negatives provide additional contrast during optimization. 3.3. Reliability-Aware Re-ranking The second stage re-ranks tA_t from Section 3.2 using operational signals while the semantic encoders remain frozen. At step t, the context is the query embedding t=qt∈ℝdx_t=e_q_t ^d, the policy selects at∈ta_t _t, and observes reward yt∈0,1y_t∈\0,1\ only after execution, where yt=1y_t=1 denotes success and yt=0y_t=0 denotes errors or timeouts. Tools with little execution history are initialized from catalog metadata. We define a static prior rstatic(a)∈[0,1]r_static(a)∈[0,1] from three signals used in benchmark construction in Section 2, namely name–description consistency rcons(a)r_cons(a), source credibility wsrc(a)w_src(a), and documentation quality rqual(a)r_qual(a). (4) rcons(a)=cos(fEnc(aname),fEnc(adesc))r_cons(a)= (f_Enc(a_name),f_Enc(a_desc)) (5) rstatic(a)=λ1rcons(a)+λ2wsrc(a)+λ3rqual(a)r_static(a)= _1r_cons(a)+ _2w_src(a)+ _3r_qual(a) where ∑iλi=1 _i _i=1. These signals implement the catalog reliability score in Section 2, and all three read catalog fields only, so no manual rating is needed. For rconsr_cons, fEncf_Enc is the SBERT encoder of Section 2, which embeds the tool name and its description, and we take the cosine of the two normalized vectors. For wsrcw_src, each platform carries a fixed weight, namely 1.0 for an official listing, 0.9 for LangChain, CrewAI, and LlamaIndex, 0.8 for n8n, and 0.6 otherwise. For rqualr_qual, we standardize the description length and the tag-string length against their medians and pass 0.6zdesc+0.4ztag0.6\,z_desc+0.4\,z_tag through a logistic function. The 0.8 threshold is used for benchmark statistics, while rstaticr_static initializes the bandit. Each arm bias is initialized as a←rstatic(a)⋅0b_a← r_static(a)·x_0, which covers the cold-start case, where a new or rarely used tool carries no execution history and is therefore scored from its catalog metadata. Algorithm 1 Reliability-aware adaptation via discounted LinUCB 0: query stream qtt=1T\q_t\_t=1^T, frozen encoders fQf_Q and fTf_T, discount γ, exploration α, fusion weights w1,w2,w3w_1,w_2,w_3 0: tool ata_t executed at each step, with (a,a)(A_a,b_a) updated online 1: a←norm(fT(a))e_a (f_T(a)) for all a∈a offline, reused at every step 2: a←dA_a _d and a←rstatic(a)0b_a← r_static(a)\,x_0 cold start from metadata 3: for t=1t=1 to T do 4: t←norm(fQ(qt))x_t (f_Q(q_t)) 5: t←A_t← top-K tools by Ssem(qt,a)=t⊤aS_sem(q_t,a)=x_t e_a 6: for each a∈ta _t do 7: ^a←a−1a θ_a _a^-1b_a 8: Sbandit←^a⊤t+αt⊤a−1tS_bandit← θ_a x_t+α x_t A_a^-1x_t 9: Sfinal(a)←w1Ssem+w2Sbandit+w3rstatic(a)S_final(a)← w_1S_sem+w_2S_bandit+w_3r_static(a) 10: end for 11: at←argmaxa∈tSfinal(a)a_t← _a _tS_final(a) 12: execute ata_t and observe yt∈0,1y_t∈\0,1\ 13: at←γat+tt⊤A_a_t← _a_t+x_tx_t kept diagonal, so scoring is O(d)O(d) 14: at←γat+yttb_a_t← _a_t+y_tx_t 15: end for At runtime, the system must adapt to dynamic failures such as API outages. Standard LinUCB (Li et al., 2010) weights all past observations equally and therefore adapts slowly when arm quality shifts over time. Alternatively, a global multi-armed bandit (Global MAB) (Auer et al., 2002) tracks shared or context-free arm statistics. It is cheaper to maintain but cannot model payoffs that depend on the current query context. To overcome these limitations, RATED adopts discounted LinUCB (Garivier and Moulines, 2011) with a separate linear model per tool. This choice addresses non-stationarity by down-weighting older rewards before each update, ensuring that recent failures change tool scores more quickly. Each tool a maintains a∈ℝd×dA_a ^d× d and a∈ℝdb_a ^d. The reliability score combines exploitation and exploration. (6) Sbandit(t,a)=^a⊤t+αt⊤a−1tS_bandit(x_t,a)= θ_a x_t+α x_t A_a^-1x_t where ^a=a−1a θ_a=A_a^-1b_a. After executing ata_t and observing yty_t, we update with discount γ∈(0,1)γ∈(0,1). (7) at _a_t ←γat+tt⊤ ← _a_t+x_tx_t (8) at _a_t ←γat+ytt ← _a_t+y_tx_t Discounting emphasizes recent feedback and encourages timely switches to alternatives. 3.4. Hybrid Fusion for Tool Ranking RATED fuses semantic, bandit, and metadata signals into one score over tA_t. For query qtq_t and candidate a∈ta _t, we define (9) Sfinal(qt,a)=w1Ssem(qt,a)+w2Sbandit(t,a)+w3rstatic(a)S_final(q_t,a)=w_1S_sem(q_t,a)+w_2S_bandit(x_t,a)+w_3r_static(a) where w1+w2+w3=1w_1+w_2+w_3=1. Weights are tuned on the validation split and fixed for testing in Section 4. At each step, the system retrieves top-K tools by SsemS_sem, ranks tA_t by SfinalS_final, executes the highest-scoring tool, and updates bandit parameters from yty_t. Algorithm 1 summarizes one step of this loop. 4. Experiments 4.1. Experimental Setup All experiments run on a single NVIDIA GeForce RTX 5070 GPU (12 GB VRAM). DRAGON is fine-tuned on EcoAgent-Syn for one epoch with batch size 8, learning rate 1×10−51× 10^-5, 1,000 warmup steps, τ=0.05τ=0.05, mixed precision, and maximum sequence length 256. At inference, the system retrieves top-K=10K=10 tools by SsemS_sem and re-ranks them with Eq. 9. The bandit uses α=0.1α=0.1, γ=0.95γ=0.95, and 2,000 warm-up pairs, with diagonal covariance updates for O(d)O(d) scoring at each step. Fusion weights tuned on the EcoAgent-Real validation split are (w1,w2,w3)=(0.75,0.15,0.10)(w_1,w_2,w_3)=(0.75,0.15,0.10). Static weights (λ1,λ2,λ3)( _1, _2, _3) in Eq. 5 are (0.5,0.3,0.2)(0.5,0.3,0.2), and biases are initialized with 0=/dx_0=1/ d. Table 1. Retrieval performance on EcoAgent-Syn and EcoAgent-Real. Method EcoAgent-Syn EcoAgent-Real MRR Recall@1 Recall@5 MRR Recall@1 Recall@5 TF-IDF 0.680 0.607 0.769 0.274 0.116 0.326 BM25 0.235 0.178 0.300 0.169 0.078 0.194 OpenAI Ada-002 0.790 0.736 0.857 0.670 0.430 0.750 BGE-M3 0.765 0.702 0.840 0.511 0.289 0.579 GTE-Large 0.772 0.713 0.842 0.515 0.294 0.581 E5-Large-v2 0.746 0.674 0.830 0.597 0.466 0.756 DRAGON (pretrained) 0.751 0.690 0.823 0.518 0.407 0.654 SPLADE-v3 0.754 0.689 0.834 0.626 0.491 0.789 BGE-Reranker 0.596 0.483 0.735 0.642 0.491 0.840 ToolLLM API Retriever 0.724 0.666 0.795 0.406 0.222 0.409 RATED (w/o Contrastive) 0.751 0.690 0.823 0.518 0.407 0.654 RATED (Full) 0.845 0.804 0.893 0.618 0.516 0.740 4.2. Evaluation Metrics Offline retrieval is measured with MRR, Recall@1, and Recall@5 on the EcoAgent-Real test split. EcoAgent-Syn uses the same metrics for in-domain comparison. Dynamic reliability is evaluated over T=10,000T=10,000 sequential selections. At step t, the policy selects ata_t for query qtq_t with ground truth at∗a^*_t. The indicator Status(a,t)∈0,1Status(a,t)∈\0,1\ denotes whether tool a is available under simulated failure injection. To simulate a persistent outage, the simulator ranks the N distinct ground-truth tools of the evaluation stream by frequency, skips the top 10%, and marks the next ρNρ N tools as failing, where ρ is the failure rate. The failing set is fixed before the run, and its tools stay unavailable from failure onset to the end. Metrics are averaged over the window after failure onset post=2,001,…,5,000T_post=\2,001,…,5,000\, which contains Tpost=3,000T_post=3,000 steps. (10) Post-Fail SR=1Tpost∑t∈post(at=at∗)⋅Status(at,t),Post-Fail SR= 1T_post _t _postI(a_t=a^*_t)·Status(a_t,t), (11) FSR=1Tpost∑t∈post(Status(at,t)=0),FSR= 1T_post _t _postI(Status(a_t,t)=0), where FSR is the fault selection rate. Adaptation steps (AS) measure recovery speed using (12) tadapt=mint:SRwindow(t)≥θ×SRpeak,t_adapt= \t:SR_window(t)≥θ×SR_peak\, computed from failure onset with a sliding window of W=200W=200 steps, peak success rate before failure SRpeakSR_peak, and θ=0.9θ=0.9 at 20% failure rate or θ=0.8θ=0.8 at 30%. Static rankers without online feedback never reach the recovery target and are marked Failed. 4.3. Baselines Retrieval baselines include lexical matchers TF-IDF (Spärck Jones, 1972) and BM25 (Robertson and Walker, 1994), dense encoders OpenAI Ada-002 (OpenAI, 2022), BGE-M3 (Chen et al., 2025), GTE-Large (Li et al., 2023), and E5-Large-v2 (Wang et al., 2024), and neural IR models DRAGON (Lin et al., 2023), which is not fine-tuned on EcoAgent-Syn, and SPLADE-v3 (Lassance et al., 2024). We also include a BGE-Reranker-v2-m3 (Chen et al., 2025) cross-encoder that re-ranks DRAGON top-50 candidates, and the ToolLLM API retriever (Qin et al., 2023). For dynamic evaluation, two reactive baselines share the DRAGON ranker: Health Check Blacklist permanently excludes any tool after one failed execution, and Retry (k=3k=3) tries up to three candidates within one step and keeps no memory across steps. A batch GRPO proxy similar to ToolRL (Shao et al., 2024) updates weights periodically rather than online. RATED uses DRAGON fine-tuned on EcoAgent-Syn as the backbone and is evaluated as RATED (Full), RATED (w/o Contrastive), RATED (w/o Feedback), and RATED (w/ Global MAB). Table 2. Dynamic reliability evaluation under injected tool failures. Method Failure Rate 20% Failure Rate 30% Post-Fail SR (%) ↑ FSR (%) ↓ AS ↓ Post-Fail SR (%) ↑ FSR (%) ↓ AS ↓ BM25 18.8 20.0 Failed 10.2 30.0 Failed ToolLLM API Retriever 17.4 79.6 Failed 10.5 86.2 Failed OpenAI Ada-002 19.4 80.4 Failed 11.1 88.8 Failed DRAGON (pretrained) 78.1 18.1 Failed 75.7 21.2 Failed SPLADE-v3 16.0 7.7 Failed 15.8 9.1 Failed BGE-Reranker 83.2 17.5 Failed 81.0 20.5 Failed Health Check Blacklist 82.6 1.4 2,620 80.2 1.9 Failed Retry (k=3k=3) 84.8 16.4 70 82.9 19.1 Failed ToolRL-like Batch GRPO 80.3 5.4 2,620 79.5 6.1 1 RATED (w/o Feedback) 81.1 16.5 Failed 78.7 19.4 Failed RATED (w/ Global MAB) 78.1 10.1 3,852 74.3 13.2 2,718 RATED (Full) 83.4 2.5 1,603 82.1 2.9 0 4.4. Main Results Table 1 reports offline retrieval on EcoAgent-Syn and the held-out EcoAgent-Real test split. RATED (Full) ranks first on Syn across all metrics, and RATED (w/o Contrastive) matches pretrained DRAGON on both benchmarks, which isolates the gain from contrastive fine-tuning. On EcoAgent-Real, RATED (Full) improves MRR by 52.2% over the ToolLLM API retriever but stays below Ada-002 and SPLADE-v3. High offline ranking accuracy on real queries therefore does not guarantee robust tool selection when tools fail at runtime. Table 2 reports Post-Fail SR, FSR, and adaptation steps under 20% and 30% failure injection. Static retrievers, including ToolLLM and Ada-002, frequently select unavailable tools and never recover within the evaluation horizon. DRAGON (pretrained) and BGE-Reranker reduce FSR but remain unable to adapt online. Relative to RATED (w/o Feedback), RATED (Full) lowers FSR to 2.5% and 2.9% while it maintains Post-Fail SR above 82%, which is a 7.3×7.3× reduction over DRAGON (pretrained) and a 30×30× reduction over ToolLLM. Batch GRPO updates similar to ToolRL retain higher FSR, and health check blacklisting attains low FSR but recovers slowly. RATED (w/ Global MAB) stays inferior to the full model. These results support decoupling semantic matching from reliability estimation updated online from execution feedback. 4.5. Hyperparameter Sensitivity Figure 4 reports Post-Fail SR and FSR under 20% failure injection for two grid sweeps over fusion and bandit hyperparameters. Panel (a) varies w1w_1 and w3w_3 with bandit settings fixed. Post-Fail SR rises with w1w_1 and peaks at w1=0.75w_1=0.75 with w3=0.10w_3=0.10, whereas FSR grows slowly over the same range and stays in the low single-digit range throughout. We therefore select w1=0.75w_1=0.75 and w3=0.10w_3=0.10 on the validation split, which trades about one point of FSR for the highest Post-Fail SR. Panel (b) varies γ and α with fusion weights fixed at the validated setting. Post-Fail SR is highest at γ=0.95γ=0.95, where the three α curves nearly coincide, and FSR again grows slowly with γ. We therefore set γ=0.95γ=0.95 and α=0.1α=0.1 on the validation split. Two-panel sensitivity analysis under 20 percent failure injection. Left panel plots fault selection rate versus semantic weight w1 for three static-prior weights w3. Right panel plots fault selection rate versus discount factor gamma for three exploration coefficients alpha. Vertical dotted lines mark default hyperparameters. Figure 4. Hyperparameter sensitivity at 20% failure injection.Two-panel sensitivity analysis under 20 percent failure injection. Left panel plots fault selection rate versus semantic weight w1 for three static-prior weights w3. Right panel plots fault selection rate versus discount factor gamma for three exploration coefficients alpha. Vertical dotted lines mark default hyperparameters. 4.6. Efficiency Analysis We profile end-to-end inference at batch size 1 on the same hardware as Section 4. Table 3 reports average latency, P95 latency, and incremental overhead relative to DRAGON encoding only. The encode-only baseline averages 3.223 ms. RATED (w/o Feedback) applies only the static reliability prior in Eq. 5 and adds 0.025 ms. Contextual bandit re-ranking incurs modest additional cost. RATED (w/ Global MAB) adds 0.108 ms, and RATED (Full) adds 0.172 ms, reaching 3.395 ms average and 4.391 ms P95 latency. These overheads are far below batch GRPO updates similar to ToolRL at 17.839 ms average, which require periodic policy synchronization. Therefore, RATED is fast enough for interactive agent loops and still reduces fault selection. Table 3. Inference latency at batch size 1 (ms). Method Avg. Latency P95 Latency Δ Overhead DRAGON encode only 3.223 4.269 – ToolRL-like Batch GRPO 17.839 28.921 +14.616 RATED (w/o Feedback) 3.248 4.274 +0.025 RATED (w/ Global MAB) 3.331 4.351 +0.108 RATED (Full) 3.395 4.391 +0.172 5. Conclusion We introduced RATED, a reliability-aware tool retrieval framework that combines DRAGON encoders fine-tuned with contrastive learning and contextual bandit re-ranking from execution feedback. On EcoAgent, RATED achieves strong in-domain retrieval and remains competitive on real-world queries. Under injected tool failures, it reduces fault selection to 2.5–2.9% FSR, 7.3×7.3× lower than static DRAGON, with only 0.172 ms added latency over encode-only inference. These results highlight the critical importance of incorporating execution feedback into retrieval systems for autonomous agents operating in dynamic environments, and demonstrate that RATED provides an efficient solution for reliable tool discovery in agent systems. Evaluating this cold-start path on newly added tools and validating RATED in live agent runtimes remain future work. Acknowledgements. This work is supported by the National Science Foundation (NSF) under award number 2623261. GenAI Usage Disclosure We use Gemma-3-1b to generate the synthetic queries of EcoAgent-Syn and to verify candidate query–tool pairs during EcoAgent-Real labeling, and we release the prompt templates with the dataset. We used OpenAI’s ChatGPT to improve the clarity and grammar of sentences during the writing process. References Abe et al. (2003) N. Abe, A. W. Biermann, and P. M. Long Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica 37 (4), p. 263–293. External Links: ISSN 0178-4617, Link, Document Cited by: §3.1. Auer et al. (2002) P. Auer, N. Cesa-Bianchi, and P. Fischer Finite-time analysis of the multiarmed bandit problem. Machine Learning 47, p. 235–256. External Links: Document Cited by: §3.3. Chen et al. (2025) J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu M3-embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. External Links: 2402.03216, Link Cited by: §4.3. CrewAI (2024) CrewAI CrewAI. Note: GitHub repository External Links: Link Cited by: §1. Dong et al. (2024) X. Dong, X. Zhang, W. Bu, D. Zhang, and F. Cao A survey of llm-based agents: theories, technologies, applications and suggestions. In 2024 3rd International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC), Vol. , Shenyang, China, p. 407–413. External Links: Document Cited by: §1. Garivier and Moulines (2011) A. Garivier and E. Moulines On upper-confidence bound policies for switching bandit problems. In Proceedings of the 22nd International Conference on Algorithmic Learning Theory, ALT’11, Berlin, Heidelberg, p. 174–188. External Links: ISBN 9783642244117 Cited by: §3.3. Gemini Team (2025) Gemini Team Gemini: a family of highly capable multimodal models. External Links: 2312.11805, Link Cited by: §2. LangChain AI (2023) LangChain AI LangChain. Note: GitHub repository External Links: Link Cited by: §1. Lassance et al. (2024) C. Lassance, H. Déjean, T. Formal, and S. Clinchant SPLADE-v3: new baselines for SPLADE. External Links: 2403.06789, Link Cited by: §4.3. Li et al. (2025) J. Li, Y. Gao, Y. Yang, Y. Bai, X. Zhou, Y. Li, H. Sun, Y. Liu, X. Si, Y. Ye, Y. Wu, Y. Lin, B. Xu, B. Ren, C. Feng, and H. Huang Fundamental capabilities and applications of large language models: a survey. ACM Comput. Surv. 58 (2). External Links: ISSN 0360-0300, Link, Document Cited by: §1. Li et al. (2010) L. Li, W. Chu, J. Langford, and R. E. Schapire A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, W ’10, New York, NY, USA, p. 661–670. External Links: Document Cited by: §1, §3.3. Li et al. (2023) Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang Towards general text embeddings with multi-stage contrastive learning. External Links: 2308.03281, Link Cited by: §2, §4.3. Lin et al. (2023) S. Lin, A. Asai, M. Li, B. Oguz, J. Lin, Y. Mehdad, W. Yih, and X. Chen How to train your DRAGON: diverse augmentation towards generalizable dense retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, p. 6385–6400. External Links: Link Cited by: §3.2, §4.3. LlamaIndex (2024) LlamaIndex LlamaIndex. Note: GitHub repository External Links: Link Cited by: §1. Mialon et al. (2023) G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, E. Grave, Y. LeCun, and T. Scialom Augmented language models: a survey. External Links: 2302.07842, Link Cited by: §1. n8n (2024) n8n n8n. Note: Workflow automation platform External Links: Link Cited by: §1. Naveed et al. (2025) H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian A comprehensive overview of large language models. ACM Trans. Intell. Syst. Technol. 16 (5). External Links: ISSN 2157-6904, Link, Document Cited by: §1. OpenAI (2022) OpenAI New and improved embedding model. Note: https://openai.com/index/new-and-improved-embedding-model/Accessed: 2026-01-23 Cited by: §4.3. Patil et al. (2023) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez Gorilla: large language model connected with massive apis. External Links: 2305.15334, Link Cited by: §1. Qian et al. (2025) C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tür, G. Tur, and H. Ji ToolRL: reward is all tool learning needs. External Links: 2504.13958, Link Cited by: §1. Qin et al. (2023) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world APIs. External Links: 2307.16789, Link Cited by: §1, §2, §4.3. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-bert: sentence embeddings using siamese bert-networks. External Links: 1908.10084, Link Cited by: §2. Robertson and Walker (1994) S. E. Robertson and S. Walker Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’94, Berlin, Heidelberg, p. 232–241. External Links: ISBN 038719889X Cited by: §4.3. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. External Links: 2302.04761, Link Cited by: §1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, §4.3. Spärck Jones (1972) K. Spärck Jones A statistical interpretation of term specificity and its application in retrieval. Journal of documentation 28 (1), p. 11–21. Cited by: §4.3. van den Oord et al. (2019) A. van den Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. External Links: 1807.03748, Link Cited by: §3.2.2. Wang et al. (2024) L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei Text embeddings by weakly-supervised contrastive pre-training. External Links: 2212.03533, Link Cited by: §4.3. Xu et al. (2024) Q. Xu, Y. Li, H. Xia, and W. Li Enhancing tool retrieval with iterative feedback from large language models. External Links: 2406.17465, Link Cited by: §1. Zhang et al. (2025) Z. Zhang, Z. Chen, H. Zhu, Z. Chen, N. Du, and X. Li ToolExpNet: optimizing multi-tool selection in LLMs with similarity and dependency-aware experience networks. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 15706–15722. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §1.