Paper deep dive
In-Context Reinforcement Learning for Tool Use in Large Language Models
Yaoqi Ye, Yiran Zhao, Keyu Duan, Zeyu Zheng, Kenji Kawaguchi, Cihang Xie, Michael Qizhe Shieh
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/13/2026, 12:39:48 AM
Summary
In-Context Reinforcement Learning (ICRL) is a novel, supervision-efficient framework for training Large Language Models (LLMs) to use external tools. Unlike traditional pipelines that require expensive supervised fine-tuning (SFT), ICRL uses few-shot prompting during reinforcement learning (RL) rollouts. The framework employs a curriculum that gradually reduces the number of in-context examples, transitioning the model from few-shot guidance to autonomous, zero-shot tool use. Experiments demonstrate that ICRL achieves state-of-the-art performance on reasoning and tool-use benchmarks, outperforming SFT-based methods in data efficiency and accuracy.
Entities (5)
Relation Signals (3)
ICRL → optimizes → LLM
confidence 95% · ICRL merges the efficiency of prompting with the adaptability of RL, offering a scalable, supervision-light alternative to traditional SFT+RL pipelines.
ICRL → trainedon → Natural Questions
confidence 95% · We use the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019) as the primary training corpus.
ICRL → utilizes → GRPO
confidence 95% · To ensure stability, we adopt GRPO (Shao et al., 2024) with loss masking to ignore non-trainable tool outputs.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2603.08068v1
- Canonical: https://arxiv.org/abs/2603.08068v1
Trouble viewing inline? Open PDF directly →
Full Text
42,480 characters extracted from source content.
Expand or collapse full text
In-Context Reinforcement Learning for Tool Use in Large Language Models Yaoqi Ye * 1 Yiran Zhao * 2 Keyu Duan 1 Zeyu Zheng 3 Kenji Kawaguchi 1 Cihang Xie 4 Michael Qizhe Shieh 1 Abstract While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limita- tions of their internal knowledge. A compelling approach to overcome this challenge is to aug- ment these models with external tools—such as Python interpreters for mathematical computa- tions or search engines for retrieving factual in- formation. However, enabling models to use these tools effectively remains a significant chal- lenge. Existing methods typically rely on cold- start pipelines that begin with supervised fine- tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is ex- pensive to annotate or synthesize. In this work, we proposeIn-ContextReinforcementLearning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompt- ing during the rollout stage of RL. Specifically, ICRLintroduces in-context examples within the rollout prompts to teach the model how to in- voke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero- shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use bench- marks. Results show thatICRLachieves state- of-the-art performance, demonstrating its effec- tiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines. 1 * Equal contribution 1 National University of Singapore 2 Salesforce AI Research 3 University of California Berkeley 4 University of California, Santa Cruz. Correspondence to: Yi- ran Zhao <zhaoyiran0924@gmail.com>, Michael Qizhe Shieh <michaelshieh@comp.nus.edu.sg>. Preprint. March 10, 2026. 1 Code is publicly available athttps://github.com/ applese233/ICRL 1. Introduction Recent advances in large language models (LLMs) (Guo et al., 2025; Yang et al., 2025; Seed et al., 2025; Team et al., 2025) have shown their effectiveness in addressing a wide range of complex tasks (Wang et al., 2024a; Hsiao et al., 2025; Shi et al., 2025; Qu et al., 2025). Nevertheless, a key limitation remains: these models rely on a fixed body of knowledge acquired during pretraining, which inherently restricts their ability to adapt to new or time-sensitive in- formation (Gao et al., 2023; Zhu et al., 2025; Wang et al., 2024b; Cheng et al., 2024; Matarazzo & Torlone, 2025). To mitigate this issue and enhance model flexibility, recent research has focused on enabling LLMs to interact with ex- ternal tools during inference. This includes generating and executing Python code for mathematical reasoning, lever- aging web search engines to access up-to-date and domain- specific content, and invoking dedicated helper models for specialized subtasks (Guo et al., 2024; Team et al., 2025; Li et al., 2025b; Jin et al., 2025a; Feng et al., 2025). The dominant training paradigms for LLMs either lever- age reinforcement learning (RL) with verifiable reward sig- nals (Guo et al., 2025; Jin et al., 2025a; Zhao et al., 2025), or adopt a cold-start strategy that begins with supervised fine-tuning (SFT) followed by an RL phase (Mei et al., 2025; Nguyen et al., 2025). Directly applying RL from scratch often yields poor performance, as the model lacks initial tool-use abilities and struggles with ineffective exploration. While incorporating an SFT stage can guide the model to- ward a more favorable initialization, it typically requires a large amount of high-quality labeled data, which is expen- sive to annotate or synthesize. In this work, we introduceIn-ContextReinforcement Learning (ICRL), a lightweight and supervision-efficient framework for training LLMs to perform tool-augmented reasoning. Unlike prior approaches that rely on SFT,ICRL teaches tool use directly through RL rollouts that are aug- mented with in-context demonstrations. Specifically, during RL training, we construct each rollout prompt by prepending a small number of few-shot examples that illustrate how to reason step-by-step, invoke tools in a structured format, and generate final answers. These demonstrations serve as soft supervision during exploration, guiding the model toward successful behavior without requiring labeled trajectories. 1 arXiv:2603.08068v1 [cs.AI] 9 Mar 2026 In-Context Reinforcement Learning for Tool Use in Large Language Models Figure 1.ICRLtraining workflow. The model is trained through a multi-stage curriculum that gradually reduces the number of in-context examples in the rollout template. At each stage, the LLM generates tool-augmented rollouts, receives rewards, and updates its policy via reinforcement learning, enabling a transition from imitation to autonomous tool use. Furthermore, as training progresses, we gradually reduce the number of demonstrations included in these rollout prompts, transitioning the model from few-shot to zero-shot settings. This progressive reduction forms a curriculum that encour- ages the model to internalize tool-use strategies and produce structured outputs autonomously, without relying on prompt- based scaffolding. We optimize the model using RL with a reward that balances task accuracy and format correctness. To ensure stability, we adopt GRPO (Shao et al., 2024) with loss masking to ignore non-trainable tool outputs. By em- bedding and gradually removing demonstrations from the RL rollouts,ICRLmerges the efficiency of prompting with the adaptability of RL, offering a scalable, supervision-light alternative to traditional SFT+RL pipelines. We conduct comprehensive experiments across a range of QA and reasoning benchmarks to evaluate the effective- ness ofICRL. Without relying on supervised fine-tuning or ground-truth tool traces,ICRLachieves state-of-the-art per- formance on challenging QA datasets, outperforming strong baselines such as ZeroSearch (Sun et al., 2025), Search- R1 (Jin et al., 2025a), and ParallelSearch (Zhao et al., 2025) by up to 8.9 on Qwen2.5-3B (Yang et al., 2024b) and 7.3 on Qwen2.5-7B in average exact match accuracy. The gains are especially pronounced on multi-hop reasoning tasks, where ICRLachieves double-digit improvements on datasets like TriviaQA (Joshi et al., 2017), 2Wiki (Ho et al., 2020), and Musique (Trivedi et al., 2022). Furthermore, in contrast to methods such as O 2 -Searcher (Mei et al., 2025) that require cold-start SFT to learn complex tool-use behavior,ICRL learns such capabilities directly through in-context examples during RL rollouts—demonstrating superior data efficiency. Beyond web QA, we also evaluateICRLon math reasoning tasks involving code execution as a tool. On the AIME2024 and AIME2025 benchmarks,ICRLmatches or exceeds the performance of ReTool (Feng et al., 2025), a strong SFT+RL baseline, despite using no supervised pretraining. These results highlightICRL’s ability to generalize to di- verse tool-augmented reasoning domains and its potential as a unified, scalable framework for training tool-using models without costly supervision. 2.In-Context Reinforcement Learning (ICRL) In this section, we formally introduce tool use in LLMs and describe how RL can be applied to train such behavior. We also present the overall workflow ofICRL, detailing its training templates, learning process, and reward design. 2.1. Tool Use in LLMs When LLMs encounter queries that exceed the scope of their internal knowledge, they must leverage external tools to obtain updated information or perform more complex reasoning. For example, search engines can provide access to recent knowledge, while Python interpreters can be used to execute structured reasoning procedures. Formally, given a queryqand an external toolT, the model generates a responsey = (y 1 ,y 2 ,...,y |y| ), where each token is conditioned not only on the query and previous tokens, but also on a history of prior interactions with the tool. This defines a conditional distribution of the form: π θ (y | q,T ) = |y| Y t=1 π θ (y t | y <t ,q,H t )(1) Here,π θ is the model parameterized byθ, andH t denotes the sequence of previous actions taken by the model and the corresponding observations returned by the tool up to stept. Specifically, the interaction between the model and the tool is structured as a sequence of actions. At each time step, the model may choose to (i) perform in- ternal reasoning, (i) issue a query to the external tool, or (i) return a final answer. These actions are embedded 2 In-Context Reinforcement Learning for Tool Use in Large Language Models in the generated text in a structured format, such as XML tags, which distinguish reasoning steps from tool invoca- tions and answers. For example, a reasoning step might be denoted as<think>...</think>, a search query as<search>...</search>, a retrieved information as<information>...</information>, and a final answer as <answer>...</answer>. Furthermore, the tool functions as a response mechanism. For example, a search engine can be modeled as a retrieval functionT : V ∗ → V ∗ , whereV ∗ is the space of textual sequences. Given a search queryq ′ , the tool returns an ob- servationo =T (q ′ ), such as the top-kdocuments retrieved from a corpus. This observation is appended to the model’s context and used in subsequent generation steps. 2.2. RL with Tool Use RL Objective Function. After formulating the tool- augmented reasoning in LLMs as a Markov Decision Pro- cess (MDP), we can define the corresponding reinforcement learning (RL) objective as follows: max π θ E q∼D,y∼π θ (·|q,T ) [r φ (q,y)] − βD KL [π θ (y | q,T )∥π ref (y | q,T )], (2) whereπ θ is the policy LLM,π ref is the reference LLM,r φ is the reward function andD KL is KL-divergence measure. Loss Masking. Unlike traditional RL, which optimizes solely over model-generated tokens, tool-augmented rea- soning introduces retrieved content into the rollout se- quence—tokens that are not produced by the model and therefore do not reflect its internal reasoning or decision- making process. To address this, we adopt a loss masking strategy tailored for RL with tool use, which excludes re- trieved content from the optimization. Specifically, only tokens generated by the language model contribute to the policy gradient, while retrieved spans are masked out and excluded from the loss computation. This targeted optimiza- tion ensures that learning remains focused on the model’s own behavior—such as tool usage, intermediate reasoning, and final answers—without being affected by fixed, untrain- able content from external sources. GRPO with Tool Use.We adopt GRPO (Shao et al., 2024) to trainπ θ on the RL datasetD =q 1 ,q 2 ,· ,q n . Specif- ically, forq ∈D, we use the old policy from previous step π θ old to sample a group ofNindividual responsesτ i . Then, the RL loss is defined as: L GRPO (θ) =E τ i ∼π θ old (q),q∼D RL 1 P N i=1 |τ i | N X i=1 |τ i | X t=1 CLIP(r i,t (θ),A i ,ε)− β·D KL [π θ ∥π ref ], (3) where A i = R(τ i )− mean(R(τ i )| τ i ∼ π θ old (τ),i = 1, 2,...,N) std(R(τ i )| τ i ∼ π θ old (τ),i = 1, 2,...,N) , (4) and r i,t (θ) = π θ (τ i,t |q,τ i,<t )/π θ old (τ i,t |q,τ i,<t ). 2.3. ICRL Training Process. Rather than training models from scratch using reinforcement learning, which often suffers from sparse rewards and inefficient exploration, or relying exclusively on few-shot prompting, which incurs substantial inference overhead, we introduceICRL, a framework that integrates the strengths of both approaches.ICRLlever- ages the sample efficiency and inductive bias of few-shot prompting while benefiting from the exploration capabilities of reinforcement learning. At the beginning of training, we incorporate a small number of tool-use demonstrations into the model’s rollout tem- plate. These examples guide the model toward effective tool-augmented reasoning via in-context learning, akin to few-shot prompting. The resulting policy is denoted as: π θ (y |P N ,q,T ) = |y| Y t=1 π θ (y t |P N ,y <t ,q,H t ),(5) whereP N represents the few-shot prompt consisting ofN demonstration examples. Table 1 shows a concrete example of rollout template. After training for several steps, the model begins to acquire tool-use capabilities with the guidance of the initial few-shot promptP N . Once sufficient learning progress is observed, we pause training and reduce the number of demonstration examples in the prompt. The updated policy conditioned on a reduced promptP N−1 is defined as: π θ (y |P N−1 ,q,T ) = |y| Y t=1 π θ (y t |P N−1 ,y <t ,q,H t ), (6) whereP N−1 denotes a prompt withN − 1demonstration examples. This process is repeated iteratively, progressively reducing the number of demonstrations, until no examples remain in the prompt. Reward Design.We design a composite reward function that combines the answer accuracy and format correctness to provide a richer learning signal: r φ (q,y) = α· reward acc + (1− α)· reward format ,(7) whereαis the hyperparameter to balance two rewards. Specifically, the accuracy-based reward is computed us- ing exact match (EM) between the model’s predicted an- swer and the ground truth. The reward is assigned as 3 In-Context Reinforcement Learning for Tool Use in Large Language Models Table 1. Few-shot rollout template in ICRL. Few-Shot Prompt Template Solve the following problem step by step. You must conduct reasoning inside<think>...</think>every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by<search>query</search> and it will return results between<information>...</information>. You can search as many times as you want. Finally, provide the answer inside <answer>...</answer>. Here are some examples: Example Problem: q demo Example Solution: <think>...</think> <search>...</search> <information>...</information> <think>...</think> <answer> a </answer> (repeated for N examples Now solve the following problem: Actual Problem: question Table 2. Format violation penalties for computing reward format . ViolationRationale No <answer> tagMust provide structured answer Unbalanced <answer> tagsProper XML structure required No <think> tagShould demonstrate reasoning Unbalanced <think> tagsProper XML structure required No <search> usageShould utilize available tool Empty answer contentAnswer must be substantive reward acc = 1if the prediction exactly matches the cor- rect answer, and 0 otherwise. Thereward format component evaluates the model’s adher- ence to the expected structured output format, specifically the correct use of XML tags. It is defined as: reward format = 1.0− X v∈V penalty(v),(8) whereVdenotes the set of format violations identified in the model’s response. The penalty functionpenalty(v)assigns a predefined cost to each violation, as specified in Table 2. With the proposed reward design, we optimize the policy using the RL objective defined in Equation 3. The complete training procedure for ICRL is outlined in Algorithm 1. 3. Experiment 3.1. Setup Backbone Models. We applyICRLto the Qwen2.5 model family (Yang et al., 2024a), focusing primarily on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, and further evaluating on Qwen2.5-14B-Instruct. We also extend our experiments to the Qwen3 series (Yang et al., 2025), particu- larly Qwen3-8B, which incorporates RL enhancements. All these instruction-tuned models are widely adopted for ques- tion answering and reasoning tasks. We choose the instruct variants over base models due to their strong instruction- following capabilities, which enable faster and more stable Algorithm 1 ICRL Input:Initial policyπ θ , reference modelπ ref , tool T, initial few-shot promptP N , dataset partitions D (N) ,D (N−1) ,...,D (0) , reward functionr φ (·), number of RL steps T Output: Trained model π θ 1: for k = N to 0 do 2: // Step 1: Construct prompt with k demonstrations 3: P k ← select k examples fromP N 4: D (k) ← RL training subset for current prompt level 5:for t = 1 to T do 6:for q ∈D (k) do 7:π θ old ← π θ 8: SampleNtrajectoriesτ 1 ,...,τ N ∼ π θ old (q,P k ,T ) 9:for each trajectory τ i do 10:Compute reward: r φ (q,τ i ) 11:Compute normalized advantage A i 12:Compute importance weights r i,t (θ) 13:end for 14:Update policy: π θ ← π θ −∇ θ L GRPO 15:end for 16:end for 17: end for convergence during RL training. For improved training effi- ciency, all models are loaded using bfloat16 precision. Baselines.We evaluate the effectiveness ofICRLby com- paring it against several state-of-the-art methods for training tool-augmented LLMs. These baselines fall into three main categories. Direct prompting methods include models that perform inference using direct inputs or prompting strate- gies such as Chain-of-Thought (CoT) reasoning (Wei et al., 2022). Retrieval-based methods leverage external infor- mation through techniques like Retrieval-Augmented Gen- eration (RAG), including standard RAG (Lewis et al., 2020), Interleaving Retrieval Chain-of-Thought (IRCoT) (Trivedi et al., 2023), and Search-o1 (Li et al., 2025a). Fine-tuning- 4 In-Context Reinforcement Learning for Tool Use in Large Language Models Table 3. Main Results ofICRL: Exact Match (EM) Accuracy (%) on various difficult QA datasets. The best performance is set bold. The second best performance is underlined. ModelMethod Difficult Question Answering Average TriviaQAHotpotQA2WikiMusiqueBamboogle Qwen2.5-3B Direct28.814.924.42.02.414.50 CoT3.22.12.10.20.01.52 IRCoT31.216.417.16.724.019.08 Search-o147.222.121.85.432.025.70 RAG54.425.522.64.78.023.04 SFT29.218.624.84.411.217.64 R1-instruct44.920.827.56.019.223.68 Reject Sampling48.824.023.35.921.024.60 Search-R154.532.431.910.326.431.10 ZeroSearch57.427.430.09.811.127.14 ICRL72.635.439.220.033.640.16+8.94 Qwen2.5-7B Direct40.818.325.03.112.019.84 CoT18.59.211.12.223.212.84 IRCoT47.813.314.97.222.421.12 Search-o144.318.717.65.829.623.20 RAG58.529.923.55.820.827.70 SFT35.421.725.96.611.220.16 R1-base53.924.227.38.329.628.66 R1-instruct53.723.729.27.229.328.62 Reject Sampling59.233.129.612.335.533.94 Search-R161.037.041.414.636.838.16 ZeroSearch65.234.635.218.427.836.24 ParallelSearch62.842.942.419.741.141.78 ICRL75.442.653.626.048.049.12+7.34 based methods involve approaches such as SFT (Chung et al., 2024), RL without search (R1) (Guo et al., 2025), and Rejection Sampling (Ahn et al., 2024). We also include recent RL methods that integrate search capabilities, such as Search-R1 (Jin et al., 2025a), ZeroSearch (Sun et al., 2025), O 2 -Searcher (Mei et al., 2025), and ParallelSearch (Zhao et al., 2025). These baselines provide a comprehensive comparison to validate the generality and advantages of our proposed ICRL framework. Training Datasets. We use the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019) as the primary training corpus. The dataset is loaded via FlashRAG (Jin et al., 2025b), which provides preprocessed question-answer pairs with gold-standard answers. NQ contains real user queries from Google Search, each paired with Wikipedia passages that include the correct answer. To support our proposed training method, we randomly sampled three questions from the web and used GPT-5.2 2 to generate few-shot examples formatted according to the rollout template shown in Table 1. To simulate real-world tool-use behavior, we integrate the Serper API 3 across all models to retrieve live results from the Google Search engine. For fairness, each query retrieves the top 3 documents required for search-based reasoning. 2 https://platform.openai.com/docs/models/ gpt-5.2 3 https://serper.dev/ Evaluation Benchmarks.We evaluateICRLand various baselines on several widely-used QA benchmarks, including TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), Musique (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). Since our models are trained on the Natural Questions (NQ) dataset, we exclude NQ from the evaluation to avoid data leakage. These bench- marks cover diverse domains and reasoning types, providing a comprehensive assessment of model performance. Further- more, to ensure evaluation efficiency, we randomly sample up to 500 questions from each dataset. The selected bench- marks include both in-domain general QA tasks (e.g., Triv- iaQA, HotpotQA) and out-of-domain multi-hop QA tasks (e.g., 2Wiki, Musique, and Bamboogle), allowing us to thor- oughly test the generalization and reasoning capabilities of different methods. Reward. The hyperparameterαin Equation 7 is set to 0.8. Furthermore, for the format violation penalties in Ta- ble 2, the weights are set to 0.5, 0.2, 0.15, 0.1, 0.1, and 0.2 respectively from top to bottom. Implementation Details. We implement our method us- ing the Volcano Engine Reinforcement Learning (VeRL) framework (Sheng et al., 2024). Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct serve as the backbone models, trained with a learning rate of 1e- 5 In-Context Reinforcement Learning for Tool Use in Large Language Models Table 4. Results: Exact Match (EM) Accuracy (%) on various QA datasets. O 2 -Searcher applies cold-start SFT before RL, while our method (ICRL) applies RL without SFT. ModelMethodSFT Difficult Question Answering Average TriviaQAHotpotQA2WikiMusiqueBamboogle Qwen2.5-3B O 2 -Searcher✓59.738.837.416.034.437.26 ICRL✗72.635.439.220.033.640.16 Table 5. Example of multi-turn search-augmented question answering. The model correctly identifies George Washington as the president who set the two-term precedent and retrieves his inauguration date through two search queries. QuestionWhen did the president who set the precedent of a two term limit enter office? Golden AnswerApril 30, 1789 Model Response<think>I need to know which president set this precedent. I also need to know when he entered office. Let me start by finding out which president it is. </think> <search> president who set two term limit precedent </search> <information>Doc 1(Title: “Twenty-second Amendment to the United States Constitution”) Until the amendment’s ratification, the president had not been subject to term limits, but both George Washington and Thomas Jefferson (the first and third ... Doc 2(Title: “Amendment 22 – Term Limits for the Presidency”) The decision for George Washington – the First President of the United States – to voluntarily step down from office after two terms, established an unofficial ... Doc 3(Title: “The first U.S. president set his own term limit”) Washington wrote ... </information> <think>It seems George Washington set the precedent. Now I need to find out when he entered office. </think> <search> when did George Washington enter office as president </search> <information> Doc 1(Title: “Presidency of George Washington”) Washington was inau- gurated as the first President of the United States on April 30, 1789... Doc 2(Title: “George Washington”) Washington was unanimously elected ... Doc 3(Title: “President Washington’s First Term (1789-1792)”) George Washington was inaugurated ... </information> <think>Based on the information, George Washington entered office on April 30, 1789. </think> <answer> April 30, 1789 </answer> 6. For each query, we sample 8 rollout trajectories with a temperature of 1.0 to compute the group-relative advan- tage. The maximum prompt length is set to 5000 tokens to accommodate few-shot demonstrations, and the maximum response length is capped at 2048 tokens, allowing up to 6 search turns per query. To regularize the policy, we ap- ply a KL penalty with a coefficient of 0.001. Training is conducted on 4 NVIDIA A100 GPUs (80GB each), using a batch size of 64. We adopt Fully Sharded Data Parallel (FSDP) training with gradient checkpointing to optimize memory usage. For retrieval, we use a BM25 retriever that returns the top-3 documents for each search query. 3.2. Main Results Table 3 presents the main results comparingICRLto other baselines across five popular QA benchmarks. From the results, we can observe that: ICRLachieves state-of-the-art performance across QA benchmarks. As shown in Table 3,ICRLsignifi- cantly outperforms all baselines on both Qwen2.5-3B and Qwen2.5-7B models across five challenging QA datasets. On Qwen2.5-3B,ICRLachieves an average exact match (EM) score of 40.16, surpassing the best competing method, Search-R1 (31.10), by +8.94. The improvements are es- pecially pronounced on multi-hop datasets such as 2Wiki (+7.3), Musique (+9.7), and Bamboogle (+7.2), demonstrat- ingICRL’s strength in handling complex reasoning and tool-use scenarios. Furthermore, on Qwen2.5-7B,ICRLachieves an average EM score of 49.12, outperforming the strongest baseline, ParallelSearch (41.78), by +7.34. It achieves the best re- sults on four out of five datasets, including TriviaQA (75.4), 2Wiki (53.6), Musique (26.0), and Bamboogle (48.0). These results show thatICRLscales effectively with model size and generalizes well across both in-domain and out-of- domain QA tasks. The consistent gains over baselines that rely on supervised fine-tuning or reward modeling—such as ZeroSearch, Search-R1, and Reject Sampling—highlight the effectiveness of our in-context reinforcement learning framework in learning tool-use behaviors without explicit accuracy-based rewards or supervision. ICRLachieves better performance without SFT or la- beled data. Table 4 highlights a key advantage ofICRL: it achieves superior performance without requiring any 6 In-Context Reinforcement Learning for Tool Use in Large Language Models TriviaQA HotpotQA 2Wiki MuSiQue Bamboogle 0 20 40 60 80 100 75.4 42.6 53.6 26 48 20.8 17.8 26.8 9 14.4 EM Accuracy (%) (a) EM Accuracy 3~2~0 3~2~1~0 1234567 0 20 40 60 80 100 Number of Turns Cumulative Finish (%) (b) Finish Percent 3~2~0 3~2~1~0 Figure 2. Comparison of Qwen-7B trained for three stages (3~2~0) vs four stages (3~2~1~0). (a) EM Accuracy across five QA datasets. (b) Cumulative finish percent vs number of search turns, aggregated across all datasets. Table 6. Results: Qwen2.5-14B models Exact Match (EM) Accuracy (%) on various QA datasets. The best performance is set bold. ModelMethod Difficult Question Answering Average TriviaQAHotpotQA2WikiMusiqueBamboogle Qwen2.5-14B Direct52.022.628.26.015.224.80 CoT56.424.625.89.040.031.16 ICRL75.043.261.825.653.651.84 SFT, in contrast to O 2 -Searcher, which applies a cold-start SFT phase before reinforcement learning. Despite using no labeled tool traces or task-specific supervision,ICRL achieves a higher average EM score of 40.16 compared to 37.26 from O 2 -Searcher. It outperforms O 2 -Searcher on four out of five datasets, including substantial gains on Trivi- aQA (+12.9) and Musique (+4.0). These results demonstrate thatICRLcan learn effective tool-use strategies purely from in-context examples and reinforcement signals, offering a scalable and data-efficient alternative to methods that rely on costly annotation and pretraining. 3.3. Concrete Examples Table 5 presents a complete reasoning example fromICRL- Qwen2.5-7B on a question from the Bamboogle dataset. The model is tasked with answering a compositional query: identifying the president who established the two-term precedent and determining when he entered office. It first issues a search query to identify the relevant figure, correctly concluding that George Washington set the precedent. It then formulates a follow-up query to retrieve his inaugura- tion date and successfully extracts the correct answer, April 30, 1789. This case illustratesICRL’s ability to decompose complex questions, retrieve relevant information across mul- tiple turns, and maintain coherent reasoning without explicit intermediate supervision. It demonstrates the effectiveness of our framework in learning structured tool-use behaviors through in-context reinforcement learning. 4. Further Analysis 4.1. Ablation analysis Ablation on curriculum design for rollout reduction. We conduct an ablation study comparing two curricula for reducing the number of examples used in the rollout process: a three-stage schedule (3~2~0) and a four-stage schedule (3~2~1~0). As shown in Figure 2 (a), the three-stage variant achieves substantially higher EM accuracy across all five QA datasets. For instance, on TriviaQA and 2Wiki, the three-stage model reaches 75.4 and 53.6, compared to 20.8 and 26.8 with the four-stage version. Figure 2 (b) shows that the four-stage curriculum leads to faster decisions, with over 80% of queries finishing within two search turns. However, this comes at the cost of answer quality. These results suggest that aggressively reducing rollout length too early (via the intermediate stage with one example) encourages premature stopping and weakens multi-turn reasoning. In contrast, the simpler 3~2~0 curricu- lum maintains stronger performance by allowing the model to explore longer reasoning paths during training. Model Scaling and 14B Performance. To evaluate the scalability ofICRLacross larger models, we applied our 7 In-Context Reinforcement Learning for Tool Use in Large Language Models (a) Response Length(b) Reward(c) Number of Valid Search Figure 3. Training dynamics comparison across different few-shot settings (3-shot, 2-shot, 0-shot) for Qwen-7B model. method to Qwen2.5-14B-Instruct and report results in Ta- ble 6.ICRLsignificantly outperforms both direct prompting and CoT methods across all five QA datasets. In particu- lar, it achieves 75.0 EM on TriviaQA and 61.8 on 2Wiki, yielding a strong average EM score of 51.84—surpassing CoT by +20.7 and direct prompting by +27.0. These re- sults demonstrate thatICRLcontinues to scale effectively to larger model sizes and benefits from increased capacity, without requiring additional supervision or annotation. 4.2. Training process To understand howICRLevolves during training, we ana- lyze the learning curves across the 3-shot, 2-shot, and 0-shot curriculum stages using the Qwen2.5-7B model. In the early stages with demonstrations (3-shot and 2-shot), the model produces relatively stable and well-structured responses, as reflected in the consistent response lengths. As training progresses into the 0-shot stage, the response length initially drops due to the removal of in-context examples but gradu- ally increases again, indicating that the model is learning to independently compose longer and more structured outputs. Although the reward remains relatively steady throughout training and is based only on sparse signals—output format validity and final answer accuracy—the model still learns to use tools more effectively over time. This is most clearly reflected in the increasing number of valid tool calls during the 0-shot phase. The rise in valid tool usage indicates that ICRLsuccessfully encourages the model to internalize tool- use behavior, even without dense or step-level supervision. 4.3. Generalize Apart from conducting our method in web search tool- calling domain, we also evaluate our method by training the models’ abilities on code-writing and calling tools to run the python code to help them solve complex math problems. We Table 7. Results: Accuracy (%) on Math QA datasets. ModelMethodSFT Math QA AIME2024AIME2025 Qwen3-8B ReTool✓67.049.3 ICRL✗64.151.7 compare our results with ReTool, which is a SFT-RL train- ing framework that training models to learn code-writing and tool-calling on running code. ReTool (Feng et al., 2025) achieves the state-of-the-art performance on code- augmented long-form reasoning for solving math problems, but it also needs a lot of annotated data to apply cold-start SFT for models to learn the tool-calling format and under- stand the whole reasoning process. However, our method ICRLdoesn’t need to SFT the model first, and models can still reasoning properly from the In-Context Learning with the examples we provides in prompt. From Table 7, al- through out method underperforms ReTool on AIME2024 by 2.9%, it can achieve better result on AIME2025 with +2.4% accuracy. Which means our method still works for help models to learn other tool-calling operations and is more data-efficient than other methods that need cold-start SFT with thousands of annotated data. 5. Conclusion We introduceICRL, a simple yet powerful framework for training LLMs to use tools via in-context reinforcement learning, without requiring SFT or labeled tool traces. By incorporating few-shot demonstrations directly into the RL rollout prompts and gradually phasing them out,ICRLen- ables models to transition from imitation to autonomous tool use through reward-driven learning. Our method achieves strong performance across a range of QA and reasoning benchmarks, outperforming existing approaches that rely 8 In-Context Reinforcement Learning for Tool Use in Large Language Models on supervised data or frozen tool-use policies.ICRLalso generalizes across domains, including web search and code execution, demonstrating its flexibility and effectiveness. These results highlightICRLas a scalable and data-efficient alternative to traditional SFT+RL pipelines for enabling tool-augmented language models. Impact Statements This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reasoning: Pro- gresses and challenges. arXiv preprint arXiv:2402.00157, 2024. Cheng, J., Marone, M., Weller, O., Lawrie, D., Khashabi, D., and Van Durme, B. Dated data: Tracing knowl- edge cutoffs in large language models. arXiv preprint arXiv:2403.12958, 2024. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. Feng, J., Huang, S., Qu, X., Zhang, G., Qin, Y., Zhong, B., Jiang, C., Chi, J., and Zhong, W. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536, 2025. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1), 2023. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In IJCAI, 2024. Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing a multi-hop qa dataset for compre- hensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020. Hsiao, V., Fine-Morris, M., Roberts, M., Smith, L. N., and Hiatt, L. M. A critical assessment of LLMs for solving multi-step problems: Preliminary results. In AAAI 2025 Workshop LM4Plan, 2025. URLhttps: //openreview.net/forum?id=kFrqoVtMIy. Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025a. Jin, J., Zhu, Y., Dou, Z., Dong, G., Yang, X., Zhang, C., Zhao, T., Yang, Z., and Wen, J.-R. Flashrag: A mod- ular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, p. 737–740, 2025b. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kel- cey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information pro- cessing systems, 33:9459–9474, 2020. Li, X., Dong, G., Jin, J., Zhang, Y., Zhou, Y., Zhu, Y., Zhang, P., and Dou, Z. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025a. Li, X., Jin, J., Dong, G., Qian, H., Zhu, Y., Wu, Y., Wen, J.- R., and Dou, Z. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025b. Matarazzo, A. and Torlone, R. A survey on large language models with some insights on their capabilities and limi- tations. arXiv preprint arXiv:2501.04040, 2025. Mei, J., Hu, T., Fu, D., Wen, L., Yang, X., Wu, R., Cai, P., Cai, X., Gao, X., Yang, Y., et al.O 2 - searcher: A searching-based agent model for open- domain open-ended question answering. arXiv preprint arXiv:2505.16582, 2025. 9 In-Context Reinforcement Learning for Tool Use in Large Language Models Nguyen, X.-P., Pandit, S., Reddy, R. G., Xu, A., Savarese, S., Xiong, C., and Joty, S. Sfr-deepresearch: Towards effective reinforcement learning for autonomously rea- soning single agents. arXiv preprint arXiv:2509.06283, 2025. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the composi- tionality gap in language models. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, p. 5687–5711, 2023. Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., and Wen, J.-r. Tool learning with large lan- guage models: a survey. Frontiers of Computer Sci- ence, 19(8), January 2025.ISSN 2095-2236.doi: 10.1007/s11704-024-40678-2. URLhttp://dx.doi. org/10.1007/s11704-024-40678-2. Seed, B., Chen, J., Fan, T., Liu, X., Liu, L., Lin, Z., Wang, M., Wang, C., Wei, X., Xu, W., et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. Hybridflow: A flexi- ble and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., and Ren, Z. Tool learning in the wild: Empowering language models as automatic tool agents, 2025. URLhttps://arxiv.org/abs/ 2405.16533. Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y., Jiang, Y., Xie, P., Zhang, Y., Huang, F., and Zhou, J. Zerosearch: In- centivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588, 2025. Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop ques- tion composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Pro- ceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 10014–10037, 2023. Wang, C., Deng, Y., Lyu, Z., Zeng, L., He, J., Yan, S., and An, B. Q*: Improving multi-step reasoning for llms with deliberative planning, 2024a. URLhttps: //arxiv.org/abs/2406.14283. Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., and Li, J. Knowledge editing for large language models: A survey. ACM Computing Surveys, 57(3):1–37, 2024b. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2024a. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024b. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhut- dinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2369–2380, 2018. Zhao, S., Yu, T., Xu, A., Singh, J., Shukla, A., and Akkiraju, R. Parallelsearch: Train your llms to decompose query and search sub-queries in parallel with reinforcement learning. arXiv preprint arXiv:2508.09303, 2025. Zhu, Z., Liao, Y., Chen, Z., Wang, Y., Guan, Y., Wang, Y., and Wang, Y. Evolvebench: A comprehensive benchmark for assessing temporal awareness in llms on evolving knowledge. In Proceedings of the 63rd Annual Meeting 10 In-Context Reinforcement Learning for Tool Use in Large Language Models of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16173–16188, 2025. 11