Paper deep dive

Looped Language Models Improve Compositional Tool Calling

Andrei Cristian Popescu, Haitz Sáez de Ocáriz Borde, Pietro Liò

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 67

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/20/2026, 3:57:33 AM

Summary

This paper investigates the efficacy of looped language models (LLMs) for compositional tool calling, a setting requiring coordination of multiple API calls, state maintenance, and dependency handling. The authors evaluate native (Ouro) and retrofitted (Llama, OLMo) looped models against non-looped baselines on benchmarks including BFCL, NESTful, and API-Bank. Results indicate that recurrent computation significantly improves performance on compositional and dependency-aware tasks (multi-step, parallel, nested calls), with accuracy generally increasing with recurrent depth. Adaptive inference is shown to offer a favorable compute-performance trade-off by allocating computation only when necessary.

Entities (12)

API-Bank · benchmark · 98%BFCL · benchmark · 98%Looped Language Models · model-architecture · 98%NESTFUL · benchmark · 98%Compositional Tool Calling · task · 95%Llama 3.2-1B · model · 95%OLMo-2-1B · model · 95%Ouro 1.4B · model · 95%Ouro-2.6B · model · 95%Adaptive Inference · method · 90%Qwen3-1.7B · model · 90%Recurrent Depth · hyperparameter · 90%

Relation Signals (10)

BFCL → evaluates → Compositional Tool Calling

confidence 95% · BFCL v3 is our primary evaluation benchmark... BFCL evaluates both single-call function selection and independent multi-call generation.

NESTFUL → evaluates → Compositional Tool Calling

confidence 95% · NESTful evaluates hierarchical tool use through nested API workflows... emphasizes long-range dependencies and multi-step execution

Looped Language Models → improves → Compositional Tool Calling

confidence 95% · Our results suggest that looped language models are a promising architecture for agentic systems that require reliable planning, coordination, and execution of compositional tool use workflows.

Ouro 1.4B → isa → Looped Language Models

confidence 95% · We evaluate native and retrofitted looped language models... Ouro-1.4B SFT

Ouro-2.6B → isa → Looped Language Models

confidence 95% · We evaluate native and retrofitted looped language models... Ouro-2.6B SFT

API-Bank → evaluates → Compositional Tool Calling

confidence 90% · API-Bank... complements the compositional reasoning benchmarks by measuring API grounding and invocation accuracy.

Llama 3.2-1B → isretrofittedwith → Looped Language Models

confidence 90% · Retrofitting recurrence into Llama-3.2-1B and OLMo-2-1B likewise improves several compositional categories

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Looped language models have shown promising results on reasoning benchmarks, yet their potential for agentic tool use remains largely unexplored. We study this question in compositional tool-calling settings, where models must coordinate multiple API calls, maintain intermediate state, and preserve dependencies across tool interactions. We evaluate native and retrofitted looped language models on API-Bank, BFCL, and NESTful, comparing looped and non-looped models trained under matched supervised fine-tuning recipes and varying recurrent depth at inference time. In controlled experiments, recurrent computation generally benefits compositional and dependency-aware tool use, while providing smaller and more model-dependent gains on isolated API invocation. Accuracy on multi-step tool use generally increases with recurrent depth; adaptive inference, however, achieves a more favorable compute-performance trade-off by allocating additional computation only when needed. Our results suggest that looped language models are a promising architecture for agentic systems that require reliable planning, coordination, and execution of compositional tool use workflows.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,696 characters extracted from source content.

Expand or collapse full text

Looped Language Models Improve Compositional Tool Calling Andrei Cristian PopescuHaitz Sáez de Ocáriz BordePietro Liò acp96@cam.ac.uk hs788@cam.ac.uk pl219@cam.ac.uk Department of Computer Science and Technology University of Cambridge, United Kingdom Abstract Looped language models have shown promising results on reasoning benchmarks, yet their potential for agentic tool use remains largely unexplored. We study this question in compositional tool-calling settings, where models must coordinate multiple API calls, maintain intermediate state, and preserve dependencies across tool interactions. We evaluate native and retrofitted looped language models on API-Bank, BFCL, and NESTful, comparing looped and non-looped models trained under matched supervised fine-tuning recipes and varying recurrent depth at inference time. In controlled experiments, recurrent computation generally benefits compositional and dependency-aware tool use, while providing smaller and more model-dependent gains on isolated API invocation. Accuracy on multi- step tool use generally increases with recurrent depth; adaptive inference, however, achieves a more favorable compute-performance trade-off by allocating additional computation only when needed. Our results suggest that looped language models are a promising architecture for agentic systems that require reliable planning, coordination, and execution of compositional tool use workflows. Simple Multiple Parallel Parallel Multiple API-Bank 255075100 Ouro-2.6B SFT Ouro-1.4B SFT Qwen3-1.7B SFT Qwen3-4B SFT Llama-3.2-3B SFT (a) Native looped and baseline models. Simple Multiple Parallel Parallel Multiple API-Bank 15304560 OLMo-2-1B native SFT OLMo-2-1B looped SFT (d=8) Llama-3.2-1B native SFT Llama-3.2-1B looped SFT (d=8) (b) Retrofitted and native backbones. Figure 1: Looped computation primarily improves compositional tool use. (a) Under matched supervised fine-tuning conditions, Ouro models show their largest relative advantages on multi-call BFCL categories, while differences on the predominantly single-call API-Bank benchmark are less consistent. (b) Retrofitting recurrence into Llama-3.2-1B and OLMo-2-1B likewise improves several compositional categories relative to their non-recurrent counterparts. Category labels are color-coded: compositional tasks are shown in orange, while non-compositional tasks are shown in black. arXiv:2608.18171v1 [cs.AI] 17 Aug 2026 1 Introduction Pretrained language models are increasingly used as decision-making components in agentic systems, selecting and invoking external tools to complete user-directed tasks [Karpas et al., 2022, Yao et al., 2023, Schick et al., 2023, Shen et al., 2023]. Tool use provides a controlled setting for studying how pretrained representations are translated into actions: while simple requests may require only a single function call, more complex tasks require models to compose multiple calls, maintain intermediate state, and coordinate sequential or parallel dependencies. Reliable tool use therefore requires more than producing an individually plausible invocation; models must construct and preserve structured action sequences across multiple decisions. Looped language models provide a natural mechanism for strengthening this form of structured decision making. Instead of relying on a single forward pass, they repeatedly refine latent representa- tions before generating each token, increasing test-time computation without increasing parameter count. Although this iterative computation has shown promising results on reasoning benchmarks, its benefits for agentic behavior remain underexplored. Our main contributions are: 1. We assess whether iterative latent computation improves compositional tool use in a con- trolled setting requiring structured, multi-step decision making. 2. Our evaluation covers native and retrofitted looped models on API-Bank, BFCL, and NESTful, with matched fine-tuning comparisons and shared-backbone Llama and OLMo retrofits. 3. The largest benefits appear on multi-call and dependency-aware tasks, suggesting that recurrent computation is particularly useful when action selection must preserve structure across multiple decisions. 4.Increasing recurrent depth generally improves compositional tool use, while adaptive infer- ence achieves a better compute-performance trade-off by allocating fewer iterations when additional refinement is unnecessary. 2 Related Work We review two lines of work most closely related to our study: recurrent-depth architectures for latent computation, and methods for planning and composition in tool-using language models. Recurrent Depth and Latent ComputationLooped Transformers increase effective model depth by repeatedly applying a shared Transformer block, decoupling test-time computation from param- eter count. Early recurrent and parameter-sharing architectures include the Universal Transformer [Dehghani et al., 2019], Deep Equilibrium Models [Bai et al., 2019], and ALBERT [Lan et al., 2020]. More recent work introduced looped Transformers as a method for latent reasoning, where recurrent iterations refine hidden representations rather than extending the context with explicit reasoning tokens [Saunshi et al., 2025, Geiping et al., 2025, Zhu et al., 2025, Jeddi et al., 2026, Hao et al., 2025, Zelikman et al., 2024, Huang et al., 2025]. Looped models have demonstrated strong performance on algorithmic reasoning, length generalization, adaptive computation, and latent test-time scaling [Giannou et al., 2023, Anil et al., 2022, Yang et al., 2024, Kohli et al., 2026, Popescu et al., 2026]. We test whether these capabilities transfer to tool-use settings. Planning and Composition in Tool-Using Models Tool-augmented language models extend autoregressive generation by invoking external functions for retrieval, computation, and interaction with software environments. Early work demonstrated that language models can interleave reasoning with tool execution [Yao et al., 2023], learn API usage from self-supervised annotations [Schick et al., 2023], or route requests to specialized neural and symbolic modules [Karpas et al., 2022]. More recent approaches improve compositional tool use through explicit planning, including decomposing tasks into executable workflows [Shen et al., 2023, Lu et al., 2023], retrieving relevant APIs from large tool collections [Qin et al., 2023, Patil et al., 2023], and constructing dependency-aware execution graphs that schedule sequential and parallel function calls [Kim et al., 2024]. Unlike these approaches, which primarily enhance tool use through external planning, retrieval, or execution strategies, our 2 work investigates whether additional latent iterative computation in looped Transformers benefits compositional tool calling. Benchmarks such as API-Bank [Li et al., 2023], BFCL [Patil et al., 2025], and NESTful [Basu et al., 2025] are used to evaluate increasingly complex tool-use capabilities, ranging from single-function invocation to parallel, sequential, and nested API compositions. 3 Background We first introduce the recurrent computation used by looped Transformers, and then formalize compositional tool calling as structured prediction over sets of tool invocations and their dependencies. Looped TransformersIn contrast to classical Transformers that stack distinct Transformer layers, looped Transformers repeatedly apply a shared block of Transformer layers over multiple recurrent iterations. Each iteration refines the latent representation, increasing inference-time computation while keeping the parameter count fixed. Leth (0) denote the initial hidden representation of an input sequence. At recurrent iterationt, the shared Transformer block updates the latent state according toh (t) = F θ h (t−1) , whereF θ denotes the shared recurrent block. A shared output head then produces a prediction after every recurrent iteration,π (t) θ (y | x) = g θ h (t) ,producing a sequence of refined predictionsπ (1) θ ,...,π (T) θ , whereTis the maximum number of recurrent iterations. Looped Transformers naturally enable adaptive computation. Rather than executing a fixed number of recurrent iterations for every token, looped Transformers can dynamically allocate computation by selecting an exit iteration using either a learned halting policy or post-hoc criteria [Banino et al., 2021, Zhu et al., 2025, Popescu et al., 2026]. During inference, easier predictions can terminate after fewer recurrent iterations, improving the cost–accuracy trade-off. Compositional Tool Use Formalism LetT = τ 1 ,...,τ N denote the tools available to a lan- guage model, where each toolτ i is specified by a function name, a natural-language description, and an argument schema. Given a user requestx, the language model predicts a distribution over structured tool callsp θ (c k | x),wherec k = (f k ,a k ), f k ∈Tis the selected function anda k denotes its instantiated arguments. Executingc k returns an observationo k , which may be incorporated into subsequent model predictions. A tool use solution can be represented as a directed acyclic graph G x = (C x ,E x ),whereC x = c 1 ,...,c K is the set of tool calls and(c i ,c j ) ∈ E x indicates that callc j depends on the result of callc i . A single-call task has|C x | = 1. Independent calls have no dependencies and may be executed in parallel, whereas sequential calls are constrained by an ordering. In a dependent or nested sequence, the observation returned by an earlier call is used to instantiate an argument of a later call, such thata j = φ j (x,o i 1 ,...,o i m ),for predecessor callsc i 1 ,...,c i m . More complex workflows may combine parallel branches with sequential dependencies. We refer to the construction of multi-call solutions as compositional tool calling. Under this representation, inde- pendent calls haveE x =∅and dependent workflows haveE x ̸=∅. Thus,C x specifies the required tool calls, whileE x specifies the output-to-input dependencies between them. Using this formulation, existing benchmarks emphasize different components of the tool-calling process. API-Bank primarily evaluates individual tool callsc k = (f k ,a k ), focusing on correct tool selection, argument grounding, and API invocation. BFCL evaluates both single-call function selection and independent multi-call generation. Its Simple and Multiple categories require|C x | = 1, whereas its Parallel and Parallel- Multiple categories require|C x | > 1, withE x =∅throughout. Finally, NESTful emphasizes the dependency structureE x , where outputs returned by earlier calls are consumed as arguments of later calls, forming hierarchical execution chains. These benchmarks evaluate node-level call prediction, structured multi-call composition, and dependency-aware execution. 4 Experimental Setup This section describes the models, training procedure, evaluation benchmarks, and inference protocols used in our experiments. We compare looped and non-looped models under matched supervised fine-tuning recipes, and evaluate both fixed-depth and adaptive looped inference to study the role of further computation during tool use. 3 4.1 Model Families and Variants We evaluate looped language models in two complementary settings. First, we compare the looped Ouro-1.4B and Ouro-2.6B models against standard Transformer baselines from the Qwen3 and Llama families at similar parameter scales. We fine-tune the corresponding base checkpoints using the same dataset and optimization settings, and separately evaluate released instruction-tuned Qwen and Llama checkpoints up to 8B parameters as stronger public reference systems. The larger instruction- tuned checkpoints also provide reference points whose single-pass inference cost is closer to that of multiple loop iterations. Second, we evaluate retrofitted looped models based on OLMo-2-1B and Llama-3.2-1B. For each model, we compare the looped checkpoint against its corresponding non-looped parent after supervised fine-tuning with the same data and optimization recipe. These comparisons more directly isolate the effect of introducing looped computation while preserving the underlying pretraining family. Architectural details for Ouro and the retrofitted models are provided in Appendix A. Supervised Fine-TuningAll controlled models are supervised fine-tuned on the Hermes function calling dataset. Training examples are formatted using ChatML with the Hermes JSON tool-calling protocol. More details about the dataset can be found in Appendix B. All experiments use a maximum sequence length of 4,096 tokens, a 90/10 train-validation split, two training epochs, AdamW with learning rate2× 10 −5 , cosine learning-rate decay with 3% warmup, gradient accumulation of 2, per-device batch size of 2, LoRA rank 32, and bf16 precision. The experiments were run on an NVIDIA A100 80GB GPU. The Ouro models are fine-tuned using the original training objective, L(θ,φ) = T X t=1 p φ (t| x)L (t) θ (x) + β KL(p φ (·| x)∥π(·)), whereβ = 0.1,L (t) θ denotes the next-token cross-entropy at loop iterationt,p φ (t| x)is the learned distribution over loop iterations, andπis the uniform prior over loop depths. The retrofitted looped models do not learn an exit distribution. Instead, the number of loopsris sampled independently for each training batch using the Poisson-lognormal depth distribution proposed in the original paper McLeish et al. [2025]. The model is supervised at the final readout after the sampled number of iterations: L(θ) =E x E r∼D PLN " M−1 X ℓ=1 − logp (r) θ (x ℓ+1 | x 1:ℓ ) + β KL p θ 0 (·| x 1:ℓ ) p (r) θ (·| x 1:ℓ ) !# . Here,β = 0.1,p (r) θ is the next-token distribution of the adapted model afterrloop iterations andp θ 0 is the corresponding distribution of the frozen pre-adaptation model. Unlike Ouro, which learns a per-token distribution over loop iterations, the retrofitted model is trained using randomly sampled recurrence depths and always computes the supervised loss from the final recurrent readout. 4.2 Evaluation Benchmarks We evaluate tool use performance using three benchmarks covering different aspects of function calling: BFCL v3, NESTful, and API-Bank. These benchmarks aim to assess isolated API invo- cation, compositional multi-tool reasoning, and hierarchical tool execution. Additional details and representative examples for all three benchmarks are provided in Appendix D. BFCL v3. BFCL v3 [Patil et al., 2025] is our primary evaluation benchmark. We evaluate using the official BFCL v3 single-turn protocol and report AST accuracy on the non-live benchmark split. Following the benchmark protocol, we report results separately for the Simple, Multiple, Parallel, and Parallel-Multiple categories. Simple tasks require one tool call. Multiple tasks also require a single call, but the model must select the correct function from several candidate tool definitions. Parallel tasks require multiple independent invocations of a single function. Parallel- Multiple tasks combine function selection with multiple independent invocations across several candidate functions. Thus, in BFCL, Multiple refers to the number of candidate tools rather than the number of generated calls. Only the Parallel and Parallel-Multiple categories require multiple calls. The evaluator standardizes tool calls before comparison, making the evaluation robust to formatting differences while requiring semantic equivalence of the predicted calls. We report the official AST 4 accuracy, which scores a prediction as correct if its standardized abstract syntax tree matches the reference tool call. Representative examples for each category are provided in Appendix D.1. NESTful. NESTful [Basu et al., 2025] evaluates hierarchical tool use through nested API work- flows, where later calls depend on outputs produced by earlier ones. Unlike BFCL, the benchmark emphasizes long-range dependencies and multi-step execution rather than isolated function prediction. Following the benchmark protocol, we report Win Rate as the primary execution-based metric, together with function, parameter, partial-sequence, and full-sequence scores. A representative nested execution example is provided in Appendix D.2. API-Bank. API-Bank [Li et al., 2023] evaluates tool use across a diverse collection of real- world APIs spanning multiple domains. In contrast to BFCL and NESTful, most tasks involve a single API invocation, putting more emphasis on selecting the appropriate API and generating correct arguments than on coordinating multiple interacting tool calls. Hence, it complements the compositional reasoning benchmarks by measuring API grounding and invocation accuracy. We report three complementary metrics. Call Correctness applies the released API-specific functional checks, Exact requires an exact match of the API name and complete argument dictionary after parsing and conservative normalization, and Parse is the fraction of outputs that can be parsed as valid API calls. A representative API-Bank instance is provided in Appendix D.3. 4.3 Analyzing the Effect of Iterative Computation We evaluate the effect of successive recurrent iterations using both fixed-depth and adaptive inference protocols. These experiments are performed on the BFCL v3 evaluation split described above and a 500-example subset of NESTful. The NESTful subset is selected by taking examples evenly across the benchmark’s official ordering before any experiments are run, and is held constant across all models and inference settings. Fixed-Depth Inference. To evaluate the effect of additional looped computation, we perform fixed-depth inference by executing a predetermined number of loop iterations for every generated token. For the Ouro models, we evaluate depths 1 to 4, corresponding to the model’s maximum recurrent depth. For the retrofitted recurrent Llama-3.2-1B and OLMo-2-1B models, we evaluate 1, 2, 4, and 8 loop iterations. Modifying only the inference depth while keeping model parameters fixed isolates the contribution of recurrence independently of training. Adaptive Inference. In addition to fixed-depth inference, we evaluate adaptive looped compu- tation using Ouro’s pretrained exit gate. During generation, each token exits the inference loop once the cumulative exit probability exceeds a confidence thresholdq. We evaluate thresholds q ∈ 0.1, 0.3, 0.5, 0.7, 0.8and report both task performance and the average selected loop depth, computed as the mean number of loop iterations used across all generated tokens. 5 Results In this section, we start by evaluating whether looped models improve tool use over non-looped baselines. We then investigate whether these improvements arise from iterative computation itself through controlled inference-time depth ablations and whether recurrent computation can be allocated adaptively to improve the compute–performance trade-off. Looped language models improve tool use.Tables 1, 2, and 3 report results on BFCL, NESTful, and API-Bank. The controlled comparisons use identical datasets, optimization schedules, and LoRA configurations for looped and non-looped models. We evaluate both native Ouro models against similarly sized Transformer baselines and retrofitted recurrent variants of Llama-3.2-1B and OLMo-2-1B against their non-recurrent parents. Publicly released instruction-tuned Qwen and Llama checkpoints are included as reference systems. In both experimental settings, looped models perform better on compositional tool-calling tasks, whereas gains on isolated API invocation are smaller and model-dependent. On BFCL, the smallest differences occur on Simple tasks, where both looped and non-looped SFT models already perform 5 Table 1: BFCL semantic AST correctness (%; higher is better). Overall is the aggregate score across all task categories. The lower block compares looped and non-looped variants of each backbone, fine-tuned using identical data and optimization settings; looped variants use a fixed inference depth of 8 loop iterations. Base checkpoints omitted from the lower block score near zero throughout. ModelTrainingSimpleMultipleParallelPar.-Mult.Overall Ouro-1.4BBase65.367.035.044.555.4 SFT91.890.067.555.579.3 Ouro-2.6BBase80.575.52.01.047.9 SFT92.388.083.076.586.4 Qwen3-1.7BBase1.50.00.00.00.6 SFT67.057.03.09.540.7 Instruct91.891.583.581.087.9 Qwen3-4BBase72.057.51.00.540.6 SFT94.387.02.55.556.7 Instruct93.092.587.588.590.9 Qwen3-8BInstruct95.596.091.589.593.6 Llama-3.2-1BInstruct18.22.54.04.59.5 Llama-3.2-3BSFT88.286.069.060.578.4 Instruct33.837.00.00.020.9 Llama-3.1-8BInstruct46.541.00.00.026.8 OLMo-2-1BSFT59.250.014.512.539.1 Looped SFT55.058.026.514.541.8 Llama-3.2-1BSFT29.828.514.05.021.4 Looped SFT43.540.531.06.032.9 Table 2: NESTful official evaluation (higher is better). Win Rate is the primary metric. ModelTrainingFunction F1Parameter F1PartialFullWin Rate Ouro-1.4BBase0.9050.5390.1490.0910.110 SFT0.8990.5660.2190.1310.191 Ouro-2.6BBase0.9200.5950.2070.1280.190 SFT0.9220.6800.2950.2040.371 Qwen3-1.7BBase0.0000.0000.0000.0000.000 SFT0.0000.0000.0000.0000.000 Instruct0.9240.5550.2020.1090.134 Qwen3-4BBase0.0000.0000.0000.0000.000 SFT0.4580.3090.1560.0010.063 Instruct0.9710.7030.2850.1960.292 Qwen3-8BInstruct0.9790.7740.3290.2460.345 Llama-3.2-3BBase0.9110.5610.2080.1550.146 SFT0.9110.4950.1750.0930.095 Instruct0.9290.4190.1600.0330.060 Llama-3.1-8BInstruct0.6570.3420.1370.0300.073 strongly. Larger gains appear on the compositional categories, particularly Parallel and Parallel- Multiple. NESTful shows a similar pattern, where the Ouro models improve on the full evaluation, while the fixed-depth subset indicates that additional recurrence can also benefit the retrofitted models on hierarchical workflows. API-Bank shows much smaller differences, consistent with its greater emphasis on individual API selection and argument generation. Despite their relatively small parameter counts, the Ouro models remain competitive with several released Qwen and Llama instruction checkpoints. 6 Table 3: API-Bank evaluation (%; higher is better). (a) Controlled comparison between looped and non-looped models trained using identical supervised fine-tuning recipes. The lower block pairs each retrofitted backbone with its non-looped counterpart; looped variants are evaluated at a fixed inference depth of 8 loop iterations. (b) Comparison against publicly released instruction-tuned checkpoints. Call Correctness measures functional API-call correctness, Exact requires an exact API-name-and-argument match, and Parse reports valid call generation. (a) Controlled models BaseSFT ModelCallExactParseCallExactParse Ouro-1.4B73.067.695.175.170.297.2 Ouro-2.6B79.276.999.279.977.199.7 Qwen3-1.7B5.15.113.961.457.893.6 Qwen3-4B71.271.299.276.673.899.7 Llama-3.2-1B1.71.310.816.314.155.3 Llama-3.2-3B0.30.30.368.564.599.7 OLMo-2-1B1.90.535.737.133.299.5 OLMo-2-1B (Loop)0.10.01.534.030.390.7 Llama-3.2-1B1.71.310.816.314.155.3 Llama-3.2-1B (Loop)0.10.01.517.916.243.4 (b) Released instruct checkpoints ModelCallExactParse Ouro-1.4B (SFT)75.170.297.2 Ouro-2.6B (SFT)79.977.199.7 Qwen3-1.7B79.474.0100.0 Qwen3-4B78.775.3100.0 Qwen3-8B79.976.6100.0 Llama-3.2-1B5.15.113.9 Llama-3.2-3B74.670.798.5 Llama-3.1-8B78.874.8100.0 1234 0 25 50 75 100 Semantic AST accuracy (%) Ouro-1.4B 1234 0 25 50 75 100 Ouro-2.6B 1248 0 25 50 75 100 Llama-3.2-1B 1248 0 25 50 75 100 OLMo-2-1B Fixed recurrent depth SimpleMultipleParallelParallel-Multiple Figure 2: BFCL semantic AST accuracy by task category as fixed recurrent depth increases. Each curve reports accuracy on Simple, Multiple, Parallel, and Parallel-Multiple tasks. Tool-use performance increases with recurrent depth.The results in the previous section demon- strate that looped language models outperform their non-looped counterparts, but they do not dis- tinguish improvements arising from the loops themselves from those due to training. To isolate the contribution of looped computation, we vary the number of inference loop iterations while keeping the model fixed. Figure 2 shows that BFCL accuracy generally rises with recurrent depth, with the largest gains on compositional categories. For Ouro-1.4B and the retrofitted Llama model, Simple tasks saturate earlier, whereas the tool-selection-heavy Multiple category and the multi-call Parallel and Parallel-Multiple categories continue to benefit from further computation before plateauing. OLMo-2-1B shows a similar pattern on multi-call tasks. Ouro-2.6B instead reaches near-saturation after three iterations in all categories, suggesting that the larger model requires fewer refinement steps. NESTful exhibits the same depth effect: Win Rate increases as the number of recurrent iterations grows (Figure 3). Since later function calls explicitly depend on outputs from previous calls, the improvement with depth indicates that recurrence benefits dependent tool interactions. Across both Ouro models and the retrofitted Llama model, the effect persists when only inference depth is varied, separating it from differences introduced during training. Adaptive looped computation increases tool-calling efficiency. The previous subsection shows that additional inference loop iterations improve tool use performance but eventually saturate, with the saturation point varying across models and tasks. This suggests that a fixed inference depth is 7 1234 Fixed recurrent depth 0 10 20 30 40 Win Rate (%) Ouro-1.4B Ouro-2.6B (a) Native recurrent Ouro models. 1248 Fixed recurrent depth 0 1 2 3 4 5 6 Win Rate (%) Llama-3.2-1B OLMo-2-1B (b) Retrofitted recurrent Llama model. Figure 3: NESTful Win Rate as fixed recurrent depth increases. Native recurrent Ouro models benefit from additional recurrent computation, whereas the retrofitted recurrent Llama baseline remains substantially weaker. 1234 Mean recurrent depth per generated token 0 25 50 75 100 Semantic AST accuracy (%) BFCL 1234 Mean recurrent depth per generated token 0 10 20 30 40 Win Rate (%) NESTful Fixed Adaptive (a) Ouro-1.4B. 1234 Mean recurrent depth per generated token 0 25 50 75 100 Semantic AST accuracy (%) BFCL 1234 Mean recurrent depth per generated token 0 10 20 30 40 Win Rate (%) NESTful Fixed Adaptive (b) Ouro-2.6B. Figure 4: Adaptive recurrent computation on BFCL and NESTful. The x-axis reports the mean recurrent depth per generated token. Adaptive stopping improves the compute-performance frontier by allocating additional recurrent iterations only when beneficial. On BFCL, adaptive stopping matches or slightly exceeds the best fixed-depth operating point while executing fewer recurrent iterations on average. On NESTful, Ouro-2.6B reaches the same Win Rate as fixed depth 4 while using fewer recurrent iterations per generated token. suboptimal. A key advantage of looped language models is that they naturally support adaptive computation, allowing the model to dynamically determine how much iterative refinement each prediction requires. Next, we evaluate adaptive looped computation, where the model exits the inference loop on a per-token basis using the Ouro adaptive exit gate. Figure 4 compares adaptive and fixed-depth inference on BFCL and NESTful. Adaptive inference yields a better performance–compute trade-off for both Ouro models. It recovers most of the gains from deeper recurrence while using fewer loop iterations per generated token, and in several settings matches or exceeds the best fixed-depth configuration at lower average cost. Allocating recurrent computation by token difficulty thus yields a better compute–performance trade-off than applying a uniform depth. Iterative computation refines tool invocations.Figure 5 shows a representative NESTful example. At shallow recurrent depths, the model omits the dependent call or produces invalid function and variable references. By depth 3, it recovers the complete tool sequence and the correct output-to-input dependency, after which the prediction remains unchanged. This example suggests that recurrence can correct semantic errors in call structure and dependency binding, rather than merely output formatting. Additional examples appear in Appendix C. 8 Figure 5: Iterative refinement across recurrent depths. Ouro-1.4B on a two-step NESTful composition task. Calls are shown asfunction(argument=value) -> answer. At depth 1 the model emits a single call to a function absent from the tool catalogue and omits the dependent call; at depth 2 it recovers the two-call structure but the first function is again absent from the catalogue and the variable reference is invalid. Depths 3 and 4 match the gold sequence, including the output-to- input reference $var1.output_0$. 6 Conclusion Our empirical evaluation shows that looped computation is particularly beneficial when tool calling requires composition, dependency tracking, or coordination across multiple calls. On BFCL and NESTful, native Ouro models and retrofitted Llama and OLMo models generally outperform com- parable non-looped baselines on structured tool use tasks. Notably, although post-hoc recurrence improves several compositional tasks, retrofitted models remain substantially weaker than natively recurrent models on deeply nested workflows, suggesting that the effectiveness of recurrent refinement may depend on how representations are shaped during pretraining. By contrast, gains are smaller and more model-dependent on API-Bank, where most examples involve isolated API invocation. Fixed-depth experiments further show that multi-step tool use performance generally increases with recurrent depth, and qualitative analysis indicates that successive iterations can incrementally refine function selection, call structure, and intermediate dependencies. Adaptive stopping recovers most of the gains from deeper recurrent inference using fewer loop iterations on average, yielding a better compute-performance trade-off. The results presented in this paper suggest that looped language models are a promising foundation for agentic systems that must dynamically allocate computation while planning, coordinating, and executing compositional tool use workflows. 7 Limitations Our conclusions are drawn from a deliberately controlled setting. For the Ouro models in particular, no non-looped counterpart trained under identical pretraining conditions is publicly available, so we approximate a matched comparison by evaluating against both the Qwen3 and Llama families under the same fine-tuning recipe. The retrofit experiments, which share a backbone with their non-recurrent parents, isolate the architectural change more directly. Because additional recurrent iterations increase per-token compute, we also include larger instruction-tuned checkpoints as reference points, with inference costs that approximately upper-bound those of the looped models. Finally, all three benchmarks are static, single-turn evaluations. Extending this analysis to live, multi-turn settings such as the BFCL live and multi-turn categories, where the model must recover from failed executions across an episode, is left to future work. 9 References Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models, 2022. URL https://arxiv.org/abs/2207.04901. Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models, 2019. URLhttps: //arxiv.org/abs/1909.01377. Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder, 2021. URL https://arxiv.org/abs/2107.05407. Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, and Pavan Kapanipathi. Nestful: A benchmark for evaluating llms on nested sequences of api calls, 2025. URL https://arxiv.org/abs/2409.03797. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers, 2019. URL https://arxiv.org/abs/1807.03819. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach, 2025. URL https://arxiv.org/abs/2502.05171. Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers, 2023. URLhttps://arxiv. org/abs/2301.13196. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2025. URLhttps: //arxiv.org/abs/2412.06769. Wei Huang, Yizhe Xiong, Xin Ye, Zhijie Deng, Hui Chen, Zijia Lin, and Guiguang Ding. Fast quiet- star: Thinking without thought tokens, 2025. URL https://arxiv.org/abs/2505.17746. interstellarninja and Teknium.Hermes-function-calling-dataset-v1, 2026.URLhttps:// huggingface.co/NousResearch/hermes-function-calling-v1. Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. Loopformer: Elastic-depth looped transform- ers for latent reasoning via shortcut modulation, 2026. URLhttps://arxiv.org/abs/2602. 11451. Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhlgay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz, Amnon Shashua, and Moshe Tenenholtz. Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning, 2022. URL https://arxiv.org/abs/2205.00445. Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An llm compiler for parallel function calling, 2024. URLhttps://arxiv.org/ abs/2312.04511. Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, and Yuekun Yao. Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers, 2026. URLhttps://arxiv.org/abs/2604. 07822. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2020. URL https://arxiv.org/abs/1909.11942. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms, 2023. URL https://arxiv.org/abs/2304.08244. 10 Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models, 2023. URL https://arxiv.org/abs/2304.09842. Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, and Micah Goldblum. Teaching pretrained language models to think deeper with retrofitted recurrence, 2025. URLhttps: //arxiv.org/abs/2511.07384. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL https://arxiv.org/abs/2305.15334. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste- Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 48371–48392. PMLR, 13–19 Jul 2025. URLhttps://proceedings. mlr.press/v267/patil25a.html. Andrei Cristian Popescu, Haitz Sáez de Ocáriz Borde, and Pietro Liò. Adaptive depth in looped transformers: Diagnosing learned halting gates and trajectory readouts, 2026. URLhttps: //arxiv.org/abs/2607.20519. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL https://arxiv.org/abs/2307.16789. Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers, 2025. URLhttps://arxiv.org/abs/ 2502.17416. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL https://arxiv.org/abs/2302.04761. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URLhttps://arxiv.org/ abs/2303.17580. Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report, 2024. URL https://arxiv.org/abs/2408.11857. Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms, 2024. URL https://arxiv.org/abs/2311.12424. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URLhttps://arxiv.org/ abs/2210.03629. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URLhttps: //arxiv.org/abs/2403.09629. Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models, 2025. URL https://arxiv.org/abs/2510.25741. 11 A Looped Models Architectural Details This section summarizes the architectures and training procedures of the looped language models evaluated throughout this work. We first describe the native recurrent Ouro models and then the retrofitted recurrent models, focusing on the architectural and training differences relevant to our evaluation. A.1 Ouro Ouro [Zhu et al., 2025] is a family of pretrained looped language models that performs iterative latent computation by repeatedly applying a shared stack ofLdecoder-only Transformer layers. The recurrent block consists of standard Transformer layers with multi-head self-attention, Rotary Position Embeddings (RoPE), SwiGLU feed-forward networks, and sandwich RMSNorm. Rather than stacking independently parameterized layers, the same recurrent block is applied for up to T max = 4recurrent iterations, allowing additional test-time computation without increasing the number of model parameters. The released checkpoints are Ouro-1.4B, containing a 24-layer recurrent stack, and Ouro-2.6B, obtained by doubling the recurrent stack to 48 layers and continuing pretraining. Both models use a hidden size of 2048 and are pretrained on approximately 7.7T tokens. At each recurrent iterationt, Ouro predicts both the next-token distribution and a conditional halting probabilityλ (t) (x) = σ Linear φ (h (t) ) ,which induces an exit distribution over recurrent depths p φ (t| x) = λ (t) (x) Q j<t 1− λ (j) (x) ,with the remaining probability mass assigned to the final recurrent iteration. This exit distribution is used both during training to weight the prediction losses across recurrent depths and during inference to determine the adaptive computation depth. Training proceeds in two stages. Stage I jointly optimizes the language model and halting gate using L = T max X t=1 p φ (t| x)L (t) − βH(p φ (·| x)), whereL (t) is the language-model loss after recurrent iterationt. The entropy regularization term is equivalent to a KL penalty towards a uniform prior over exit depths, encouraging the model to utilize multiple recurrent iterations before specializing the halting policy. In Stage I, the language-model backbone is frozen and only the halting gate is optimized. Rather than supervising language-model predictions, the gate is trained from the marginal utility of ex- ecuting one additional recurrent iteration. Given the detached improvement in prediction loss I (t) i = max 0,L (t−1) i,stop −L (t) i,stop ,the improvement score is converted into a soft exit targety (t) i , and the gate is optimized using the binary cross-entropy objective L gate =− T max X t=1 h y (t) logλ (t) + (1− y (t) ) log 1− λ (t) i . This second stage improves the calibration of the adaptive exit policy while leaving the pretrained recurrent representations unchanged. Throughout this work, we evaluate the released Ouro-1.4B and Ouro-2.6B checkpoints. We consider both fixed-depth inference, where every token executes a predetermined number of recurrent iterations, and adaptive inference using the pretrained halting gate. A.2 Retrofitted Recurrent Models In addition to the native recurrent Ouro architecture, we evaluate the retrofitted recurrent language models of McLeish et al. [2025], which convert pretrained decoder-only Transformers into looped models while largely preserving their pretrained parameters. The original Transformer is partitioned into three components: a non-recurrent prelude, a shared recurrent block, and a non-recurrent coda. The recurrent block consists of a contiguous subset of Transformer layers whose parameters are shared across recurrent iterations. During inference, the prelude is executed once, the recurrent block is repeatedly applied for a configurable number of iterations, and the coda is executed once before producing the next-token prediction. This retrofit increases inference-time computation 12 through repeated latent refinement while maintaining the initialization and capabilities of the original pretrained model. In contrast to Ouro, the retrofitted model does not learn an adaptive halting policy. Instead, the number of recurrent iterationsris sampled independently for each training batch from a Poisson-lognormal depth distribution,λ ∼ LogNormal(μ,σ 2 ), r ∼ Poisson(λ),whereμandσcontrol the expected recurrence depth. The model is then supervised only after the sampled number of recurrent iterations using the objective L(θ) =E x E r∼D PLN " M−1 X ℓ=1 − logp (r) θ (x ℓ+1 | x 1:ℓ ) + βD KL p θ 0 (·| x 1:ℓ )∥ p (r) θ (·| x 1:ℓ ) # , wherep (r) θ denotes the next-token distribution afterrrecurrent iterations andp θ 0 is the corresponding distribution of the frozen pre-adaptation model. The KL regularization preserves the behavior of the original pretrained model while adapting it to recurrent computation. Unlike Ouro, which jointly optimizes predictions across all recurrent iterations using a learned exit distribution, the retrofitted model computes the language-model loss only from the final recurrent prediction corresponding to the sampled recurrence depth. Throughout this work, we evaluate released retrofitted recurrent checkpoints initialized from two pretrained model families: OLMo-2-0425-1B and Llama-3.2-1B. Since the model does not include a learned halting mechanism, we evaluate it only under fixed-depth inference by varying the number of recurrent iterations executed during generation. B Hermes Function-Calling Dataset All supervised fine-tuning experiments described in this work use the Hermes Function-Calling V1 dataset introduced by Nous Research [interstellarninja and Teknium, 2026, Teknium et al., 2024]. The dataset is a synthetic instruction-following corpus designed for training language models to produce structured function calls and JSON outputs from natural-language requests. It combines single-function and multi-function tool-calling conversations together with structured extraction, JSON-mode, and agentic interaction examples, all formatted according to the Hermes Function- Calling standard. The training examples include user requests, tool definitions expressed through JSON schemas, assistant-generated tool calls, tool responses, and final assistant replies, allowing models to learn both API selection and argument generation in multi-turn settings. The released dataset also incorporates updated function calling data derived from the Glaive function-calling corpus and additional synthetic tool use examples created by Nous Research. Throughout this work, all controlled supervised fine-tuning experiments use the same Hermes Function-Calling V1 training split and ChatML formatting described in Section 4. C Additional Qualitative Examples This section provides further qualitative examples of how predictions change with recurrent depth, complementing Figure 5. All examples are drawn from the NESTful subset described in Section 4.3, using fixed-depth inference. In each figure, the tool catalogue is the full candidate set supplied to the model for that instance, and calls are shown in the compact formfunction(argument=value) → label. The examples here illustrate the error types we observe at low recurrent depth: functions absent from the supplied catalogue, argument names drawn from other tool specifications, incorrect call ordering, spurious additional calls, and omitted dependent calls. Figure 6 illustrates several failure modes being corrected over recurrent refinement. The initial prediction is malformed, while the second iteration produces a syntactically valid but semantically incorrect sequence that introduces an unsupported function. Subsequent recurrent steps eliminate both the ordering error and the hallucinated tool invocation, converging to the gold execution trace without further modification. The remaining examples isolate narrower changes. Figure 7 shows a case where function selection is already correct at depth 1 and only the argument names differ from the specification. The model suppliesinput_str, which is a parameter of a different tool in the same catalogue, and repeats it as 13 USER QUERY Calculate the power of two for the highest power of two that divides 100. AVAILABLE TOOLS (gold functions in bold) highest_power_of_two, traverse_diagonal, calculate_power_of_two, non_whitespace_substrings, compute_degree, generate_boxes, recursive_function, random_between, ratio_saliency, parse_list_of_numbers, count_words_from_sentences, create_or_return_directory, append_and_return PREDICTED CALL SEQUENCE Gold1.highest_power_of_two(x=100) -> $var1 2.calculate_power_of_two(num=$var1.output_0$) -> $var2 Depth 1 incorrect emits tool schemas in place of arguments; nested tags; no parseable call <tool_call> <tool_call>"name": "highest_power_of_two", "arguments": "x": "description": "A positive integer.", "type": "integer", "output_parameters": "output_0": "description": "", "type": "integer"</tool_call> </tool_call> [truncated] Depth 2 incorrect order reversed; extra call absent from catalogue; variable reference invalid 1.calculate_power_of_two(num=100) -> $var_1 2.highest_power_of_two(x=100) -> $var_2 3.multiply(arg_0=2, arg_1=$var_2.result$) -> $var_3 Depth 3 correct order corrected; extra call removed; dependency correct 1.highest_power_of_two(x=100) -> $var_1 2.calculate_power_of_two(num=$var_1.output_0$) -> $var_2 Depth 4 correct same sequence as depth 3; label naming differs ($var1 vs $var_1) 1.highest_power_of_two(x=100) -> $var1 2.calculate_power_of_two(num=$var1.output_0$) -> $var2 Figure 6: Ordering and spurious-call correction. At depth 1 the output is malformed, emitting tool specifications rather than call arguments (excerpt shown truncated). At depth 2 both gold functions appear but in reversed order, followed by a call tomultiply, which is absent from the tool catalogue, and an invalid variable reference. Depths 3 and 4 match the gold sequence, differing from each other only in label naming. U S E R Q U E R Y Reverse the alphanumeric characters in the string "Hello, World!". Then, check if the parentheses in the reversed string are balanced. A V A I L A B L E T O O L S ( g o l d f u n c t i o n s i n b o l d ) get_status, vectorized_variance, find_matching_indices, non_whitespace_substrings, capitalize_sentence, serialize_employees_to_json, recursive_binary_search, ratio_saliency, extract_decimal_from_string, reverse_alphanumeric, check_parens, probability_of_decay, make_dict, compute_degree, parse_list_of_numbers, traverse_diagonal, count_words_from_sentences, to_ascii_digit, regexp_match PREDICTED CALL SEQUENCE Gold1.reverse_alphanumeric(s="Hello, World!") -> $var1 2.check_parens(string=$var1.output_0$) -> $var2 Depth 1 incorrect correct function; argument name from another tool spec, duplicated; call missing 1.reverse_alphanumeric(input_str="Hello, World!", input_str_2="Hello, World!") -> $var1 2.(no call emitted) Depth 2 / 3 / 4 correct arguments corrected; dependent call recovered; identical at all three depths 1.reverse_alphanumeric(s="Hello, World!") -> $var1 2.check_parens(string=$var1.output_0$) -> $var2 Figure 7: Argument correction. At depth 1 the model selects the correct first function but names its argumentinput_str, a parameter of a different tool in the same catalogue, duplicates it as input_str_2, and omits the dependent call. Depths 2 through 4 match the gold sequence and are identical to one another. a second argument. The correction at depth 2 therefore affects argument grounding and the presence of the dependent call, rather than the choice of function. Figure 8 shows a case in which the prediction changes between two consecutive incorrect depths. Depths 1 and 2 name different functions, neither of which appears in the supplied catalogue, and both omit the second call. The two correct depths differ only in the order of the keys within the second call’s argument dictionary, which the evaluator treats as equivalent. Finally, Figure 9 separates the recovery of a dependency from any change in function selection. The first call is already identical to the gold call at depth 2, and the only difference between depths 2 and 3 is the addition of the second call together with the reference$var1.output_0$. Depth 1 in this example produces an object with no function name at all, which the parser does not resolve to a call. Across these examples, the changes observed with additional recurrent depth involve which functions are named, which arguments they receive, how many calls are emitted, and how intermediate outputs are referenced. We describe only observable differences between predictions and the gold sequence, and we do not claim that these figures are representative of the error distribution over the full evaluation set. 14 USER QUERY Given a list of integers [1, 2, 3, 4, 5], sort it using the selection sort algorithm and then partition the sorted list into two parts. The first part should have a length of 3. AVAILABLE TOOLS (gold functions in bold) compute_degree, parse_list_of_numbers, my_divmod, extract_log_record, build_list_from_dicts, sum_integers, find_first_even_number, sort_dictionary_by_value, validate_image_data, minutes_difference, filter_product_labels, ratio_saliency, traverse_diagonal, selection_sort_non_local, non_whitespace_substrings, partition_list, count_words_from_sentences, first_odd PREDICTED CALL SEQUENCE Gold1.selection_sort_non_local(lst=[1, 2, 3, 4, 5]) -> $var1 2.partition_list(lst=$var1.output_0$, k=3) -> $var2 Depth 1 incorrect function absent from catalogue; argument name from another tool spec; call missing 1.pivot_1(input_list=[1, 2, 3, 4, 5], index=2) -> $var1 2.(no call emitted) Depth 2 incorrect different function, also absent from catalogue; dependent call still missing 1.split(input_list=[1, 2, 3, 4, 5]) -> $var1 2.(no call emitted) Depth 3 correct both gold functions selected; dependent call recovered with correct reference 1.selection_sort_non_local(lst=[1, 2, 3, 4, 5]) -> $var1 2.partition_list(k=3, lst=$var1.output_0$) -> $var2 Depth 4 correct same calls as depth 3; argument key order differs 1.selection_sort_non_local(lst=[1, 2, 3, 4, 5]) -> $var1 2.partition_list(lst=$var1.output_0$, k=3) -> $var2 Figure 8: Two distinct incorrect attempts before recovery. Depths 1 and 2 each emit a single call to a function absent from the tool catalogue and omit the required dependent call. At depth 1, the model also uses an argument name belonging to a different tool in the catalogue. Depths 3 and 4 select both gold functions and correctly pass $var1.output_0$, differing only in argument-key order. U S E R Q U E R Y Remove any leading or trailing commas, colons, or spaces from the string " :,hello,world,: ", then check if the resulting string has an attribute named "length" and concatenate its value to the string "The length of the string is: ". A V A I L A B L E T O O L S ( g o l d f u n c t i o n s i n b o l d ) find_result, count_words_from_sentences, non_whitespace_substrings, decode_num, dict_length, compute_degree, get_functions_in_module, remove_leading_trailing_characters, remove_trailing_slash, check_and_concatenate, parse_list_of_numbers, encode_numpy_array, ratio_saliency, traverse_diagonal, count_matches, last_element, pair_vars_with_rotated_list PREDICTED CALL SEQUENCE Gold1.remove_leading_trailing_characters(input_string=" :,hello,world,: ") -> $var1 2.check_and_concatenate(obj=$var1.output_0$, attribute_name="length", string="The length of the string is: ") -> $var2 Depth 1 incorrect no function name emitted; a computed literal is returned instead; parser yields no calls <tool_call> "result": "The length of the string is: " + "10" </tool_call> Depth 2 incorrect first call matches gold exactly; dependent call missing 1.remove_leading_trailing_characters(input_string=" :,hello,world,: ") -> $var1 2.(no call emitted) Depth 3 / 4 correct dependent call added with correct reference; first call unchanged from depth 2 1.remove_leading_trailing_characters(input_string=" :,hello,world,: ") -> $var1 2.check_and_concatenate(attribute_name="length", obj=$var1.output_0$, string="The length of the string is: ") -> $var2 Figure 9: Isolated recovery of a missing dependent call. Depth 1 emits an object with no function name, returning a computed literal, so the parser yields no calls. Depth 2 emits a first call that matches the gold sequence exactly but stops there. Depths 3 and 4 leave that first call unchanged and add the dependent call with the reference$var1.output_0$, isolating the recovery of the dependency from any change in function selection. D Tool-Calling Benchmark Examples This section provides representative examples from the tool-calling benchmarks evaluated in this work and relates their task structures to the formalism introduced in Section 3. We represent a tool use solution asG x = (C x ,E x ), whereC x is the set of required function calls andE x contains output- to-input dependencies between calls. The benchmarks emphasize complementary aspects of this structure: API-Bank focuses on evaluating the grounding of individual calls, BFCL evaluates function selection and the construction of independent multi-call sets, and NESTful evaluates dependent call graphs in which intermediate outputs are consumed by subsequent calls. Table 4 summarizes these differences before we present representative examples from each benchmark. D.1 Berkeley Function Calling Leaderboard The Berkeley Function Calling Leaderboard (BFCL) evaluates whether a language model can select and instantiate functions from natural-language requests [Patil et al., 2025]. We evaluate the non- live single-turn categories: Simple, Multiple, Parallel, and Parallel-Multiple. These categories differ in the number of candidate tools supplied to the model and in the number of calls that must be 15 Table 4: Structural comparison of the evaluated tool-calling benchmarks under the solution representa- tionG x = (C x ,E x ). API-Bank emphasizes individual-call grounding, BFCL evaluates increasingly complex independent call sets, and NESTful introduces explicit output-to-input dependencies. BenchmarkCandidate toolsCall structureDependenciesPrimary challenge API-Bankone or moretypically |C x | = 1E x =∅Tool and argument grounding BFCL Simple|T x | = 1|C x | = 1E x =∅Argument grounding BFCL Multiple|T x |> 1|C x | = 1E x =∅Function selection BFCL Parallelone or more|C x |> 1E x =∅Independent call generation BFCL Parallel-Multiple|T x |> 1|C x |> 1E x =∅Selection and call composition NESTfulmultiple|C x |> 1typically |E x |> 0Dependency-aware execution generated. They do not require the output of one call to be consumed by another. Such output-to-input dependencies are instead studied by the NESTful dataset. Using the notation introduced in Section 3, letT x ⊆ Tdenote the candidate tools supplied with requestx, and letC x = c 1 ,...,c K , c k = (f k ,a k ),denote the reference call set. For the BFCL single-turn categories considered here, the calls within an example are independent, and hence the dependency graph satisfiesE x =∅. The categories differ mainly in|T x |and|C x |. The following examples are reproduced in shortened form from the executable counterparts of the corresponding BFCL categories. Tool descriptions are shortened for readability. Simple.Simple examples supply a single tool and require one call:|T x | = 1, |C x | = 1, E x =∅. For example, the request “A biased die produces a six with probability0.6. If it is rolled20times, what is the probability of obtaining exactly five sixes?” is accompanied by the toolcalc_binomial_probability(n, k, p),and has the reference call C x =calc_binomial_probability(n=20, k=5, p=0.6).This category tests argument ex- traction and schema-conformant call generation. Multiple. Multiple examples supply between two and four candidate tools but require only one call:|T x | > 1, |C x | = 1, E x =∅.For example, a request asks for the probability of obtaining exactly 5 sixes in20fair-die rolls.The candidate set contains bothget_weather_data(coordinates) and calc_binomial_probability(n, k, p).The correct call isC x = calc_binomial_probability(n=20, k=5, p=1/6).Thus, “Multiple” refers to the presence of multiple candidate function definitions, rather than to the generation of multiple calls. The category tests function selection in addition to argument grounding. Parallel.Parallel examples require multiple independent invocations of a supplied function:|C x | > 1, E x =∅. For example, BFCL includes the request “Play songs from Taylor Swift and Maroon 5 for 20 minutes and 15 minutes, respectively, on Spotify.” with the tool spotify.play(artist, duration). The reference call set is C x = spotify.play(artist=Taylor Swift, duration=20), spotify.play(artist=Maroon 5, duration=15) . The calls do not consume one another’s outputs and may therefore be executed concurrently. The model must identify the correct number of calls and align each entity with its corresponding argu- ments. Parallel-Multiple. Parallel-Multiple combines multiple candidate functions with the genera- tion of multiple independent calls:|T x | > 1, |C x | > 1, E x =∅.For example, a re- quest asks both for the current weather in Ottawa and for the probability of obtaining 5 wins in ten independent attempts with success probability0.5.The candidate tools are 16 get_weather_data(coordinates) and calc_binomial_probability(n, k, p),and the ref- erence call set is C x = get_weather_data( coordinates=[45.4215,-75.6972]), calc_binomial_probability( n=10, k=5, p=0.5) . This category jointly tests function selection, call-set construction, and argument assignment across different tools. D.2 NESTful NESTful evaluates nested sequences of executable API calls in which the output of one function is passed as an argument to a subsequent function [Basu et al., 2025]. Each instance contains a user request, a catalog of available tools, a gold sequence of calls with arguments, and the final answer obtained by executing that sequence. The released evaluation set contains 1861 instances drawn from mathematical reasoning and coding domains, together with executable implementations of the corresponding functions. Using the notation of Section 3, a NESTful solution is represented asG x = (C x ,E x ),where C x = c 1 ,...,c K contains the required function calls and(c i ,c j ) ∈ E x whenever the output ofc i is used to instantiate an argument ofc j . Unlike the BFCL single-turn categories considered in this work, NESTful therefore evaluates both call-set construction and explicit output-to-input dependencies. Its call structure is generally a directed acyclic graph rather than merely a collection of independent calls. NESTful assigns a unique label to the output of every call. If a callc i is assigned the label$var_i, a later argument can refer to one of its output fields using notation such as $var_i.result$.This makes the dependency relation explicit in the serialized reference sequence. For example, consider the request “Find the average of all the numbers between 6 and 34 that are divisible by 5.” The corresponding solution consists of the call set C x =c 1 ,c 2 ,c 3 ,c 4 , where c 1 = add(6, 4), c 2 = subtract(34, 4), c 3 = add(out(c 1 ), out(c 2 )), c 4 = divide(out(c 3 ), 2). ThecorrespondingdependencygraphhasC x = c 1 ,c 2 ,c 3 ,c 4 andE x = (c 1 ,c 3 ), (c 2 ,c 3 ), (c 3 ,c 4 ).Callsc 1 andc 2 are independent and may be evaluated in paral- lel. Callc 3 depends on both of their outputs, whilec 4 depends on the output ofc 3 . The longest dependency path therefore has two edges: c 1 → c 3 → c 4 or c 2 → c 3 → c 4 . Executing the complete sequence produces the gold answer 20. This example illustrates the distinction between node composition and dependency composition. The model must first select the correct calls and instantiate their constant arguments, as in ordinary multi- call generation. It must additionally assign outputs to variables, select the correct output fields, and bind those variables to the arguments of subsequent calls. Errors can therefore arise from selecting an incorrect function, constructing an incorrect argument, omitting a call, producing the wrong ordering, or referencing the wrong intermediate variable. NESTful reports complementary metrics at different levels of the predicted solution. Function and parameter scores measure the correctness of individual calls, partial sequence matching measures how much of the gold call sequence is recovered, and full sequence matching requires the complete sequence of functions and arguments to match. The benchmark additionally executes the predicted calls and reports a win rate based on whether the resulting answer is correct. These metrics distinguish locally plausible calls from complete, dependency-preserving solutions. D.3 API-Bank API-Bank evaluates tool-augmented language models through conversational API usage [Li et al., 2023]. Each instance consists of a user request, a collection of available APIs, an expected API 17 invocation, the API response, and the final assistant response. Compared to BFCL and NESTful, API-Bank evaluates the grounding of individual tool calls rather than the construction of multi-call workflows. An API-Bank example is represented asG x = (C x ,E x ),where|C x | = 1, E x =∅. The task is therefore to select the correct function and instantiate its arguments from the user request before integrating the returned observation into the final response. For example, consider the dialogue “Can you calculate(5 + 6)× 3for me?” The available API isCalculator(formula),whereformulais a string containing an arithmetic expression. The corresponding reference call isC x =Calculator(formula = ’(5+6)*3’).Since the instance requires a single API invocation, its dependency relation is E x =∅. The example tests whether the model can identify the required API, translate the natural-language request into the expected argument representation, and produce a correctly formatted invocation. More generally, API-Bank evaluates API selection and argument generation within conversational contexts, which may contain information accumulated across multiple dialogue turns. Unlike BFCL, which explicitly evaluates the construction of independent multi-call sets, and NESTful, which evaluates output-to-input dependencies between calls, the API-Bank setting considered in our evaluation emphasizes the grounding and invocation of individual APIs. 18

Instant research discovery