Paper deep dive
MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand
Shouang Wei, Houcheng Min, Xinpeng Dong, Xin Lin, Sen Cui, Bo Jiang, Zhongxiang Dai, Kun Kuang, Guandong Xu, Fei Wu, Min Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/9/2026, 12:05:06 AM
Summary
MetaForge is a self-evolving multimodal agent framework that dynamically retrieves, adapts, and forges tools on demand to overcome the limitations of static tool inventories. It operates through a closed loop of Decide, Retrieve, Adapt, and Forge stages, trained via multi-turn GRPO with a composite reward function. The framework manages a dynamic tool pool containing executable, model, and forged tools, and demonstrates superior accuracy, efficiency, and generalization across 12 multimodal benchmarks.
Entities (20)
Relation Signals (18)
MetaForge → evaluatedon → ChartQA
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... ChartQA
MetaForge → evaluatedon → DocVQA
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... DocVQA
MetaForge → evaluatedon → TallyQA
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... TallyQA
MetaForge → evaluatedon → OCRVQA
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... OCRVQA
MetaForge → evaluatedon → CLEVR-M
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... CLEVR-M
MetaForge → evaluatedon → ScienceQA
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... ScienceQA
MetaForge → evaluatedon → MathVista
confidence 95% · Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. ... MathVista
MetaForge → implements → Decide
confidence 95% · MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: statically predefined tool inventories fail to generalize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise-induced errors. We propose MetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge-retrieve-adapt-forge-recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, retrieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learning, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 benchmarks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and generalization, validating a paradigm shift from static tool inventories to on-demand self-evolution.
Tags
Links
- Source: https://arxiv.org/abs/2606.01801v1
- Canonical: https://arxiv.org/abs/2606.01801v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
171,956 characters extracted from source content.
Expand or collapse full text
MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand Shouang Wei 1,* , Houcheng Min 1,* , Xinpeng Dong 2 , Xin Lin 1 , Sen Cui 3 , Bo Jiang 1 , Zhongxiang Dai 4 , Kun Kuang 2 , Guandong Xu 5 , Fei Wu 2 , Min Zhang 1,†,‡ 1 East China Normal University 2 Zhejiang University 3 Tsinghua University 4 The Chinese University of Hong Kong, Shenzhen 5 The Education University of Hong Kong 52285901019@stu.ecnu.edu.cn, mzhang@cs.ecnu.edu.cn Abstract Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: stat- ically predefined tool inventories fail to gener- alize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise- induced errors. We proposeMetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic be- havior into four coupled stages: Decide (judg- ing whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge–retrieve–adapt–forge–recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, re- trieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learn- ing, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 bench- marks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and general- ization, validating a paradigm shift from static tool inventories to on-demand self-evolution. 1 Introduction Multimodal agents have made significant progress in recent years in visual understanding and complex reasoning tasks (Jian et al., 2025; Ning et al., 2025; Liao et al., 2025). However, many real-world sce- narios still exceed their built-in capabilities, such as fine-grained text recognition and complex visual analysis. To bridge these capability gaps, existing studies have integrated external tools, such as OCR and image-processing modules, into multimodal agents to enhance their task-solving abilities. * Equal contribution.†Corresponding author.‡Project leader. What color is the umbrella? What is the price of the blue mug? What is the difference between the highest and lowest bar? Generate HTML for this webpage. Thinking: Calling OpenCV Answer: Yellow Answer: Yellow Thinking: Calling OCR Thinking: Calling OpenCV Answer: $19.99 Answer: $9.99 Thinking: Calling OpenCV Thinking: Calling OpenCV Answer: 27 Answer: parameter error Thinking: Calling OCR Thinking: forge tool Answer: Correct HTML Answer: Wrong HTML Thinking: Calling forged tool Thinking: Calling Python Code Thinking: Calling OCR Tool Result: Tool Result: MetaForge Existing Method Tool Result: Existing Method MetaForge 12,35,27,8 27 Existing Method Bulid...... (a) Decide (b) Retrieve (d) Forge Existing Method Static Tool Pool: OpenCV OCR Python Code Thinking: No Tool Needed Tool Result: $19.99 $9.99... $19.99 yellow (c) Adapt Tool Result: Tool Result: Thinking: Calling OpenCV Tool Result: 12,35,27,8 Tool Result: parameter error MetaForge MetaForge Tool Result: Tool Result: forged tool Tool Result: HTML Figure 1: Comparison examples between existing tool- augmented methods and our method. Existing work on tool-augmented multimodal agents mainly falls into two categories. One type uses a single tool intensively to obtain fine-grained visual evidence and external knowledge through cropping, zooming, or multimodal retrieval (Fan et al.; Su et al., 2025b; Zheng et al., 2025; Narayan et al., 2025; Wu et al., 2025b). The other type builds multi-tool orchestration frameworks that in- tegrate code interpreters, specialized vision models, and search engines into a reasoning loop (Zhang et al.; Zhao et al.; Lu et al.). Although these methods improve their perfor- mance on specific tasks, they still face two core lim- itations when they use tools for complex reasoning tasks. First, statically predefined tool inventories fail to generalize to unseen scenarios. Second, in- discriminate tool invocation incurs redundant cost and noise-induced errors. As Fig. 1 shows, these limitations lead to four failure modes. In (a), the model cannot accurately judge whether it needs 1 arXiv:2606.01801v1 [cs.MA] 1 Jun 2026 a tool. It therefore makes redundant calls in sce- narios that allow direct answers. In (b), the model retrieves a tool, but the selected tool does not match the task requirement. This mismatch causes exe- cution failure. In (c), the model identifies relevant tools, but it cannot adapt the call parameters to the current task context. This causes parameter errors. In (d), the task falls outside the coverage of the stat- ically predefined tool inventory. The model lacks an on-demand tool forging mechanism, so it cannot close the capability gap and ultimately fails. Over- all, these cases show that existing methods lack a unified mechanism for deciding whether to invoke tools, adapting tool calls, and expanding the toolset when capability gaps arise. To this end, we proposeMetaForge, a self- evolving multimodal agent that retrieves, adapts, and forges tools on demand.MetaForge models tool use as a closed-loop capabil- ity orchestration process consisting of De- cide–Retrieve–Adapt–Forge: the agent first de- termines whether an external tool is needed; if so, it retrieves suitable tools from a dynamic tool li- brary and adapts the invocation according to the task context. When existing tools cannot satisfy the requirement, the system forges new skills online and, after validation, registers them back into the tool library for future reuse. Through this mech- anism,MetaForgecan continuously expand its capabilities across different tasks and tools, rather than being limited to a predefined set of tools. To train the model to learn such an adaptive or- chestration strategy, we adopt multi-turn GRPO training and design a composite reward function that jointly optimizes answer correctness, tool re- trieval, and parameter adaptation, invocation neces- sity, skill forging and reuse, and format parsabil- ity. In particular, we introduce cost-related rewards to reduce unnecessary tool calls, thereby improv- ing inference efficiency while maintaining perfor- mance. In addition, to address the potential ex- pansion of the tool library caused by online skill accumulation, we design a capacity-constrained dy- namic tool-library management mechanism. The main contributions are as follows: • We proposeMetaForge, a framework for static tool users into on-demand, self-evolving capability orchestrators. With an online skill- forging mechanism and dynamic tool manage- ment, the agent can identify capability, gener- ate new tools, and reuse them efficiently. •We propose a GRPO training paradigm for multi-turn tool interaction, using a compos- ite reward to jointly optimize tool invocation, adaptation, forging, reuse, and efficiency. •We conduct systematic experiments on mul- tiple multimodal benchmarks under both IID and OOD tool settings, demonstrating the ad- vantages ofMetaForgein terms of accu- racy, efficiency, and generalization. 2 Related Work 2.1 Multimodal Tool-Augmented Reasoning Multimodal tool-augmented reasoning mainly fol- lows two lines. The first line focuses on deep use of one or a few visual tools, using cropping, zooming, or retrieval to obtain fine-grained visual evidence and external knowledge (Fan et al.; Su et al., 2025b; Zheng et al., 2025; Narayan et al., 2025; Wu et al., 2025b). The second line builds tool-augmented or multi-tool orchestration frameworks that integrate code execution, search, vision models, and special- ized perception tools into the reasoning loop. Some methods further train models to improve tool invo- cation (Zhang et al.; Zhao et al., 2026; Lu et al.; Song et al., 2026; Wu et al., 2025c; Yang et al., 2025; Hong et al., 2025; Su et al., 2025a; Ding et al., 2026; Wei et al., 2026). These methods im- prove model performance on specific tasks, but they usually operate within statically predefined tool inventories and fixed invocation patterns in practice. As a result, they may struggle to decide whether tool use is warranted, retrieve a suitable tool, or adapt the invocation to the current task context. Our work instead learns adaptive orches- tration, allowing the model to decide whether to answer directly, reuse existing tools, adapt tool calls, or forge new ones on demand when needed. 2.2 Dynamic Skill Synthesis and Evolution To overcome the capability ceiling of statically predefined tool inventories, prior work explores reusable skill synthesis and evolution. Offline skill synthesis generates reusable skills from task de- scriptions before deployment (Qian et al., 2023; Yuan et al.; Cai et al.; Wölflein et al., 2025), while experience-based skill evolution uses interaction histories to retain, revise, or remove skills (Wang et al.; Qiu et al., 2025; Huang et al., 2025; Sun et al., 2025). Recent work also extracts structured 2 Multimodal Tasks Caption General OCR Table Document Geospatial Query Image <creating_skill> <tool_calling><tool_response> <tool_calling> <tool_response> <thinking><thinking> Executable Tools Model Tools Grounding DINO OpenCV Qwen-VLMinerUPaddle OCR EasyOCR SAM Qwen- Image-Edit Dynamic Tool Pool R e w a r d Retrieve/Adapt Reward Decide Reward Forge Reward GRPO Policy Update <answer> Forged Tool Validation and Registration Capacity-Constrained Tool Pool Management Update Router 1 Router 2 Existing Tools Inadequate Existing Tools Adequate Multimodal Agent Format Reward Forged Tools SKILL.md Scripts Multi Turn ... Figure 2: Overview of MetaForge, a self-evolving multimodal agent that retrieves, adapts, and forges tools on demand. At each reasoning turn, Router 1 triggers<creating_skill>when existing tools are inadequate, and the forged tool is validated and registered into the Dynamic Tool Pool; Router 2 issues<tool_calling>when existing tools are adequate. The Dynamic Tool Pool contains Executable Tools, Model Tools, and Forged Tools. After multi-turn interaction, MetaForge outputs the final<answer>, while Retrieve/Adapt, Decide, Forge, and Format rewards supervise GRPO policy updates. skills from historical multi-path trajectories for re- trieval and adaptation during inference (Jiang et al., 2026). These approaches provide useful reusable knowledge, but often depend on predefined task descriptions or accumulated historical trajectories, and do not address cases where a current multi- modal task exposes a capability gap in the avail- able tool pool. In contrast, our online tool forging mechanism identifies such gaps during interaction, generates executable skills through planning and implementation, validates them, and registers them back into the tool pool for future reuse. 3 Methodology 3.1 Problem Formulation We propose MetaForge, a self-evolving multimodal agent for adaptive tool orchestration. Givenx = [I,T ], whereI ∈Iis visual input andTthe textual prompt, MetaForge outputsˆythrough multi-turn interaction with a dynamic tool pool. MetaForge divides the tool pool into three com- plementary types: •Executable tools: deterministic OCR, code in- terpretation, and symbolic computation; • Model tools: specialized vision models for ob- ject detection and visual understanding; • Forged tools: reusable programs that the agent creates on demand when existing tools cannot satisfy the current task. In implementation, each forged tool is a skill with a Markdown specifica- tion, a JSON schema, and executable scripts. All tools share a unified interfacef= (name, desc, schema, type), e.g.: f ex = Qwen3-VL-32B |z name , “reasoning . . . ” |z desc , prompt : str, img_idx : int |z schema , model tool |z type . At turnt, the policy selects an action given the historyh t = x,a 1 ,o 1 ,...,a t−1 ,o t−1 and the tool poolT t : a t ∼ π θ (a| h t ,T t ),(1) wherea t has three forms: invoking a tool from T t , forging a task-specific tool, or outputting the final answer. Each action yields an observationo t appended to history, andT t expands when validated forged tools are added. The process ends when the policy answers or reaches N turns. 3 3.2 Online Tool Forging Static tool sets cannot cover all requirements of diverse multimodal tasks. We introduce an online tool forging mechanism that enables the policy to expand the tool pool during interaction. When no existing tool can handle a task, the policy triggers acreate_skillaction with a natural language requirementd. A tool-forging modelG φ then pro- duces a forged tool through a two-stage process. 3.2.1 Two-Stage Tool Forging Planning stage.G φ takes the requirementdand produces a structured plan: z = G plan φ (d),(2) wherezspecifies the forged tool name, description, parameter interface, and required scripts. Implementation stage.G φ generates code for each planned script conditioned on d and z: c i = G code φ (d,z,i), i = 1,...,N z ,(3) where N z is the number of scripts. Each forged tool is a triplets = (m s ,p s ,e s ): a Markdown descriptionm s recording name, func- tion, scenarios, and calling conventions; a JSON schemap s defining parameters, types, and con- straints; and a script sete s containing Python scripts with parameterized inputs and outputs. 3.2.2Forged Tool Validation and Registration Each candidate forged tool must pass a two-part validation gate before registration: v(ˆs;d) = v exec (ˆs)· v sem (ˆs,d),(4) wherev exec ∈ 0, 1checks whether all scripts compile and execute without errors, andv sem ∈ 0, 1 uses a judge model to verify that the forged tool’s behavior matchesd. Only candidates who pass both checks join the tool pool: T t+1 =T t ∪Q ∗ t ,(5) whereQ ∗ t denotes validated forged tools at t. 3.3 Composite Reward Training via Multi-Turn GRPO Online tool forging addresses missing tools, but the policy must also learn when to answer directly, when to retrieve an existing tool, how to adapt the tool call to the task, and when to forge or reuse a forged tool. We optimize the policy with Group Relative Policy Optimization (GRPO) and design a composite reward with four components: Re- trieve/Adapt, Decide, Forge, and Format rewards. 3.3.1 Reward Function Design For each inputx, we sample a group of rollouts G(x) = τ 1 ,...,τ M from the current policy, where eachτ i is a multi-turn trajectory. The re- ward for rollout τ decomposes as: R(τ ) = r ra (τ ) + r dec (τ ) + r forge (τ ) + r fmt (τ ). (6) Each component targets a distinct aspect of desir- able agent behavior. retrieve/adapt reward. The retrieve/adapt reward is a binary signal for whether the answer matches ground truth: r ra (τ ) = ( 1, if ˆy = y, 0, otherwise. (7) This reward optimizes the result of retrieving use- ful tools, adapting their calls to the task context, forging tools when needed, or answering directly when no tool is necessary. Decide reward. The decide reward provides an efficiency ranking within correct rollouts. It en- courages the policy to decide when tool use is nec- essary and avoid redundant calls. LetG + (x) = τ ∈ G(x) | ˆy τ = y denote the correct rollouts. For τ ∈G + (x), the normalized latency is: ˆ ℓ(τ ) = ℓ(τ )− L + max L + min − L + max , L + max > L + min , 0,otherwise, (8) whereℓ(τ )denotes the total tool-call latency ofτ, andL + min = L + min (x) ,L + max = L + max (x)are the minimum and maximum latencies inG + (x). The Decide reward is: r dec (τ ) = ( ˆ ℓ(τ ), τ ∈G + (x), 0,otherwise. (9) By normalizing within the correct group, the De- cide reward is mapped to the range[0, 1], maintain- ing a consistent range with other reward compo- nents while assigning higher scores to faster, cor- rectly completed trajectories. It is zero for incorrect rollouts, ensuring the policy is never incentivized to sacrifice correctness for speed. Forge reward. The forge reward encourages the policy to forge and reuse forged tools, but only within correct trajectories, preventing exploitation through trivial tool generation. LetA τ denote all 4 tool actions in trajectory τ : r forge (τ ) = 1[Correct(τ )] · 1[Forge(τ )] + 1[ReuseForged(τ )] /2, (10) whereCorrect(τ )indicates thatτproduces the correct final answer. TheForgeterm indicates suc- cessful online tool forging, requiring the policy to invokecreate_skilland produce a forged tool that passes validation (§3.2.2): Forge(τ ) =1 ∃c∈A τ : Create(c)∧Valid(c) . (11) TheReuseForgedterm indicates successful invo- cation of previously registered forged tools: ReuseForged(τ ) = 1 ∃c∈A τ : ForgedTool(c)∧NoError(c) , (12) whereForgedTool(c)indicates thatcinvokes a registered forged tool, andNoError(c)indicates error-free execution. Together, these terms incen- tivize a forge-and-reuse lifecycle: forging tools when existing tools are insufficient, and reusing them in later tasks to avoid redundant construction. Format reward. Each response must follow a structured format (e.g., tool-call syntax and answer delimiters). The format reward enforces this: r fmt (τ ) = ( 0.2,format-compliant, −0.2, otherwise. (13) Format-compliant trajectories receive a positive reward, while violations incur a mild penalty, en- couraging the policy to maintain a consistent output structure throughout the interaction. 3.3.2 Policy Optimization We adopt GRPO to optimize the policy. For input x, the policyπ θ old samplesMrollouts and com- putes their rewardsR(τ i ). Rewards are standard- ized within the group to obtain advantages: A(τ i ) = R(τ i )− mean M j=1 R(τ j ) std M j=1 R(τ j ) + ε .(14) The policy is updated by maximizing a clipped sur- rogate with KL regularization. Letρ i = π θ /π θ old denote the importance ratio and ̄ρ i = clip(ρ i , 1− ε, 1+ε) the clipped ratio: J GRPO (θ) =E x∼D τ i ∼π θ old 1 M M X i=1 min ρ i A(τ i ), ̄ρ i A(τ i ) − βD KL (π θ ∥π ref ) . (15) 3.4 Capacity-Constrained Tool Pool Management As forged tools accumulate through online forg- ing (§3.2), the dynamic tool pool grows. Since all tool descriptions are injected into the model’s context, an excessively large pool occupies the lim- ited context window and dilutes attention to critical information. Meanwhile, overlapping forged tools waste the exploration budget during training. To address this, we manage the forged-tool portion of the pool through two mechanisms: semantic deduplication and usage-based retention, keeping it compact and non-redundant. LetP n denote the forged-tool pool before the n-th training step, andKthe maximum capacity. Each training step processes a batch of samples, each explored through multiple rollout trajectories with multi-turn interactions. 3.4.1 Forged Tool Merging During training stepn, trajectories across samples may independently forge tools for similar needs, producing a validated forged-tool setQ ∗ n with po- tential functional overlaps. We perform semantic deduplication to elimi- nate this redundancy: a judge model extracts each forged tool’s semantic summary, groups forged tools with overlapping functionality (the number of forged tools in each group is the group size), evalu- ates members within each group for completeness and generality, and retains the highest-quality one: ̃ Q n = Merge(Q ∗ n ),(16) where ̃ Q n contains the highest-quality forged tool from each functional group. 3.4.2 Usage-Based Retention For each forged toolsin the pool, we track how many times it is invoked during each interaction turn and maintain a cumulative usage countu n (s) across training steps. The deduplicated forged tools join the current pool to formU n = P n ∪ ̃ Q n . If|U n | ≤ K, we 5 Models (↓) / Datasets (→) IID DatasetsOOD DatasetsALL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. AVG. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M AVG. AVG. w/ IID Tools GPT-5.496.8572.9580.2577.3557.7584.0078.1990.9684.0397.9572.0087.4536.4078.13 78.16 Claude-Sonnet-4-698.1873.8082.2088.2481.4083.7184.5992.1292.2297.4972.1887.9093.1589.18 86.88 Gemini-3.1-Pro-Preview96.9572.6581.4881.9868.6088.5081.6991.6888.7797.9269.5387.0337.1578.68 80.19 Qwen2.5-VL-7B93.4368.2271.8370.2445.3199.1074.6974.9548.9471.4757.0473.7364.4065.09 69.89 Qwen3-VL-8B89.8070.2077.4061.7070.6595.7877.5982.1684.5851.1560.7078.0363.5070.02 73.80 Qwen3-VL-32B94.7072.7582.7085.1569.1099.1483.9286.7658.3283.1571.9087.5069.3576.16 80.04 InternVL3.5-14B87.6071.1866.4669.9859.2072.2771.1280.8367.0844.0035.0769.0666.8860.49 65.80 InternVL3.5-38B91.8073.5072.9070.2086.7578.3578.9285.7674.9549.1528.7085.5676.7566.81 72.86 VTool-R1-7B93.1073.4579.3066.7577.8594.5780.8486.8052.2362.3562.3079.1269.0068.63 74.74 R1-Onevision-7B83.2069.2076.1061.6054.1087.0071.8776.8848.9349.7052.4081.6546.7559.39 65.63 PyVision-Image-7B-RL69.9541.5037.2547.4537.4548.0046.9347.2838.8925.1533.2031.7350.7537.83 42.38 DeepEyesV2-7B89.9070.2558.7072.7582.60 50.7170.8285.1627.0428.3066.8062.8671.1056.88 63.85 PixelReasoner-RL-v1-7B92.3071.9077.2567.1574.6593.3579.4381.4483.4570.3063.1079.1263.7073.52 76.48 AdaReasoner-7B89.4066.2573.2068.7067.7591.7176.1782.1685.3966.5555.5074.7146.3568.44 72.31 TikArt-8B96.7073.0582.4083.0563.5598.7882.9286.4491.0964.9570.3084.0866.9577.30 80.11 ZwZ-8B96.8073.6581.5080.3567.6098.9283.1485.5272.6356.2570.5086.0162.5572.24 77.69 MetaForge97.9080.5286.5187.3677.9899.6288.3290.4989.6098.3478.0189.6881.7087.9788.14 w/ OOD Tools GPT-5.496.5572.8079.4577.7059.1083.8578.2490.4483.6797.8273.0087.2035.9578.01 78.13 Claude-Sonnet-4-698.0872.0582.55 87.9981.3083.5084.2592.1292.8197.4974.5086.1693.8589.49 86.87 Gemini-3.1-Pro-Preview96.1372.7081.2581.8070.2587.5781.6291.6088.6597.7069.6586.8239.2578.95 80.28 Qwen2.5-VL-7B93.6367.4678.1870.6650.0198.6676.4375.4151.9573.1556.8874.4266.6066.40 71.42 Qwen3-VL-8B90.2071.1576.9565.3572.8595.7178.7082.0484.6851.7058.8077.9363.2069.73 74.21 Qwen3-VL-32B95.1574.40 78.6583.4077.3598.7884.6286.1659.7483.0570.0088.2079.2577.73 81.18 InternVL3.5-14B88.2063.8559.1059.2653.9070.6465.8374.4463.4739.5029.9065.3468.9556.93 61.38 InternVL3.5-38B92.1073.4071.7570.9086.6576.2178.5085.8075.6536.6529.7084.0880.5565.41 71.95 VTool-R1-7B93.2573.1077.9069.4078.2592.1480.6786.4053.8862.1063.5081.0170.8569.62 75.15 R1-Onevision-7B83.9069.1075.7561.3053.3587.2171.7777.0850.0649.3552.7080.6148.6559.74 65.76 PyVision-Image-7B-RL77.0053.3052.7555.8549.8068.0059.4569.2459.9427.3045.5041.5948.2548.64 54.04 DeepEyesV2-7B93.0573.1056.7574.6084.0570.2175.2986.6029.5647.0064.2063.5574.5560.91 68.10 PixelReasoner-RL-v1-7B91.8573.5578.4566.1071.8093.7879.2680.2482.9075.7561.3080.5665.6574.40 76.83 AdaReasoner-7B92.3066.8071.2571.1069.6081.8575.4881.0486.5671.6058.1075.4549.1070.31 72.90 TikArt-8B94.7074.3079.3081.1570.6098.4283.0881.3691.2569.3570.3083.8368.7077.47 80.27 ZwZ-8B95.6073.9081.7080.8575.5098.9284.4185.2075.6477.3569.3085.8776.3578.29 81.35 MetaForge97.2281.1786.0087.6789.1499.5890.1390.1888.1598.7279.7889.7380.4087.8388.98 Table 1: Accuracy (%) across 12 multimodal benchmarks under IID and OOD tool settings. Results are grouped into IID and OOD datasets with per-group and overall averages. Bold andunderlinedenote the best and second-best per tool setting. All reported results are averaged over three independent runs. setP n+1 = U n . When the capacity is exceeded, we apply a usage-based retention policy that prior- itizes two types of forged tools: well-established ones that the policy has repeatedly relied on, and newly forged ones addressing frequent needs. This allows the pool to preserve tools with either strong historical utility or high current demand. Specif- ically, we first retain existing forged tools whose cumulative usage count exceeds an upper-quartile thresholdδ n = Q 3 + 1.5 IQR, computed over u n (s)| s∈P n , forming the high-priority set: H n =s∈P n | u n (s)≥ δ n .(17) The remaining capacity is filled with newly forged tools from ̃ Q n , ranked in descending order by group size from the merging stage. A larger group size indicates more training samples independently requested similar functionality, reflecting higher demand and thus higher retention priority. 4 Experiments 4.1 Experimental Setup Dataset and tool set splits. To evaluate task and tool generalization, we split datasets and tools into IID and OOD settings. The detailed dataset, tool, and split compositions appear in Appendices A.1. Metrics.We use accuracy (ACC) as the primary metric. Multiple-choice answers are evaluated by exact option match, and open-ended answers are judged by GPT-5 (Singh et al., 2025). We also re- port tool usage behavior and inference efficiency. Tool Usage Entropy (TUE) measures the concen- tration of tool selection: TUE =− X i:w i >0 w i P j w j log 2 w i P j w j ,(18) wherew i is the number of calls to tooliduring evaluation, including forged tools, and lower TUE 6 indicates more concentrated selection. Tool SR is the ratio of successful tool calls to all tool calls. Forged Tool Success Rate (FSR) is the ratio of forged tools that pass validation and enter the tool pool to all forging attempts. AET, RLA, and ITC measure interaction turns, end-to-end latency, and generated tokens. Implementationdetails. Weimplement MetaForge on VerlTool (Jiang et al., 2025) and train Qwen3-VL-8B-Instruct (Bai et al., 2025a) with GRPO on 8 H200 GPUs.Each input usesM =8rollouts and at mostT =10turns. Appendix A.3 provides sampling, optimization, and reward configuration. Baselines. We compare 16 baselines in three groups. The first includes closed-source models: GPT-5.4 (OpenAI, 2026), Claude Sonnet 4.6 (An- thropic, 2026), and Gemini 3.1 Pro (DeepMind, 2026). The second includes open-source mul- timodal models from Qwen2.5-VL (Bai et al., 2025b), Qwen3-VL (Bai et al., 2025a), and In- ternVL3.5 (Wang et al., 2025b) families across scales. The third contains tool-augmented mul- timodal models closest to our setting, including VTool-R1-7B (Wu et al., 2025c), R1-Onevision- 7B (Yang et al., 2025), PyVision-Image-7B- RL (Zhao et al., 2026), DeepEyesV2-7B (Hong et al., 2025), PixelReasoner-RL-vl-7B (Su et al., 2025a), AdaReasoner-7B (Song et al., 2026), TikArt-8B (Ding et al., 2026), and ZwZ-8B (Wei et al., 2026). Appendix A.2 gives descriptions. 4.2 Performance Comparison Adaptive tool orchestration improves overall ac- curacy. Table 1 shows that MetaForge achieves the best accuracy across 12 benchmarks under both IID and OOD tool settings. The gains hold across IID and OOD datasets, suggesting that the model does not merely learn dataset-specific tool routines. We attribute this robustness to online tool forging, multi-turn GRPO with composite rewards, and tool pool management, which supply missing tools, im- prove call validity, and reduce calls. Fixed tool-use policies lack adaptive invocation. Several tool-augmented baselines underperform their base models. R1-Onevision-7B, PyVision- Image-7B-RL, and DeepEyesV2-7B are all built on Qwen2.5-VL-7B, yet achieve lower overall ac- curacy. This suggests that policies trained with a fixed tool set can produce erroneous calls when the available tools change, offsetting the benefits of Model Tool ForgingInference Efficiency Att. Reg. Err. Dup. FSR ACC AET RLAITC w/ IID Tools Qwen2.5-VL-7B14566681145.5 69.89 1.07 2.31176.41 Qwen3-VL-8B34526314.7 73.80 1.46 3.75239.06 VTool-R1-7B11027691424.5 74.74 1.39 6.44266.05 R1-Onevision-7B952255.6 65.63 1.01 0.14302.71 PyVision-Image-7B-RL651945129.2 42.38 5.02 75.50 1148.74 DeepEyesV2-7B1286 269 9358220.9 63.85 3.01 10.11 537.88 PixelReasoner-RL-v1-7B492519551.0 76.48 1.55 10.22 312.36 AdaReasoner-7B16675642745.2 72.31 2.84 18.09 329.56 TikArt-8B291 130 1332844.7 80.11 2.22 11.77 298.99 ZwZ-8B609 138 3799222.7 77.69 1.65 12.79 223.31 MetaForge190123511664.788.142.2810.99280.87 w/ OOD Tools Qwen2.5-VL-7B230 131831657.0 71.42 1.07 0.97174.28 Qwen3-VL-8B28522117.9 74.21 1.40 2.17272.68 VTool-R1-7B862457527.9 75.15 1.41 2.27262.26 R1-Onevision-7B633050.0 65.76 1.01 0.01304.99 PyVision-Image-7B-RL321517046.9 54.04 5.83 2.50 1087.73 DeepEyesV2-7B404 114 2603028.2 68.10 2.60 2.44642.37 PixelReasoner-RL-v1-7B6733221249.3 76.83 1.58 1.82306.81 AdaReasoner-7B200501321825.0 72.90 2.52 4.39312.68 TikArt-8B292 161 1003155.1 80.27 2.56 7.70383.35 ZwZ-8B442814263.6 81.351.45 0.90236.67 MetaForge12110191.788.982.304.97336.65 Table 2: Tool forging and inference efficiency. Att., Reg., Err., and Dup. denote attempts, registrations, gen- eration errors, and removed duplicates. tool augmentation. In contrast, MetaForge’s uni- fied tool interface and online tool forging allow the model to invoke available tools adaptively and forge tools when existing tools are insufficient. Tool complementarity matters. Closed-source models remain stable under IID/OOD tools, likely because their intrinsic strengths reduce reliance on external tools. By contrast, open-source multi- modal models are more sensitive to tool changes: QwenVL benefits more from OOD tools, whereas InternVL benefits more from IID tools. Because both settings use the same online tool forging mech- anism, the main differences lie in executable tools and model tools. For executable tools, IID tools em- phasize perception functions (OCR, grounding, seg- mentation), whereas OOD tools emphasize reason- ing functions (symbolic computation, code execu- tion, chart parsing), suggesting that tools help more when they match model gaps. For model tools, QwenVL can call InternVL/Step3 under OOD tools, while InternVL can call QwenVL under IID tools, indicating that cross-family multimodal mod- els provide complementary gains. These results show that tool augmentation depends not only on tool functionality, but also on complementarity be- tween tools and the calling model. 4.3 Tool Usage Behavior Analysis We analyze tool usage behavior with TUE, Tool SR, and task accuracy (Task ACC). Figure 3 shows 7 that methods with more dispersed selection and lower call success usually obtain lower task accu- racy. Base models and existing tool-augmented methods often show this pattern because fixed tool sets generalize poorly to unseen tools. When tools change, they tend to make mistakes or issue invalid calls. Without tool-forging training, their forged tools are also more likely to be non-executable or misaligned with task needs. These errors reduce Tool SR and disrupt subsequent reasoning, thereby lowering task accuracy. In contrast, MetaForge maintains a more concen- trated selection, higher call success, and better task performance under both IID and OOD tool settings. This advantage stems from improved call quality and enhanced generalization. The Retrieve/Adapt, Format, and Decide rewards encourage useful tool use, valid formatting, and concise interaction. The unified tool interface helps the model understand and invoke unseen tools, while the Forge reward encourages forging and reusing forged tools when existing tools are insufficient. Together, these de- signs improve selection precision and the reliability of forged-tool construction. Qwen2.5-VL-7B Qwen3-VL-8B VTool-R1-7B R1-Onevision-7B PyVision-Image-7B-RL DeepEyesV2-7B PixelReasoner-RL-v1-7B AdaReasoner-7B TikArt-8B ZwZ-8B MetaForge 0 1 2 3 4 5 6 7 Tool Use Entropy 3.03 2.64 3.06 3.93 3.62 3.57 2.87 3.58 2.72 3.03 1.30 3.22 3.72 3.71 4.00 3.68 4.02 2.49 4.22 3.45 3.01 1.12 Base ModelsTool-Augmented Models 0 20 40 60 80 100 120 Task ACC / Tool Success Rate (%) 98.24%96.59% 88.98%88.14% TUE (IID)TUE (OOD)Task ACC (IID)Task ACC (OOD)Tool SR (IID)Tool SR (OOD) Figure 3: Comparison of TUE, Tool SR, and Task ACC across methods. 4.4 Tool Forging Behavior Analysis All methods use the same inference prompt and therefore have the same opportunity to forge tools. Table 2 shows that base models and existing tool- augmented methods obtain low FSR, indicating that prompting alone is insufficient for reliable tool forging. Without dedicated training, these methods struggle to decide when a new forged tool is needed and may create redundant ones. Even when they trigger forging, low generation quality can cause validation failure. In contrast, MetaForge achieves the highest FSR under both tool settings. This improvement comes from the combined effect of the Forge reward, two- stage tool forging, and decide reward. The Forge reward favors forged tools that help solve the task, two-stage forging improves alignment with task needs, and the Decide reward suppresses unneces- sary creation. Together, these components improve forged-tool quality and registration success. 4.5 Inference Efficiency Analysis Table 2 shows three efficiency-accuracy patterns among existing methods. Some methods, such as R1-Onevision-7B, have AET close to 1 and low latency, but bypass tools and gain little from aug- mentation. Others, such as PyVision-Image-7B- RL, incur many turns and high token costs without accuracy gains, indicating redundant interaction. A third group, such as TikArt-8B, has efficiency comparable to MetaForge but shows an accuracy gap. Under similar overhead, precise tool selection and invocation determine final performance. In contrast, MetaForge achieves the highest ac- curacy with efficient inference under IID/OOD set- tings. AET stays around 2.3 turns, showing stabil- ity. Its RLA (IID: 10.99s, OOD: 4.97s) and ITC (IID: 280.87, OOD: 336.65) remain relatively low among tool-augmented models. This efficiency comes from the Decide reward, which penalizes redundancy. Among correct trajectories, those with fewer calls and lower latency receive higher re- wards, guiding efficiency during training. 5 Conclusion We propose MetaForge, a self-evolving multimodal agent for adaptive tool orchestration. MetaForge addresses the two limitations of tool-augmented multimodal agents, namely statically predefined tool inventories and indiscriminate tool invocation, by learning a closed Decide-Retrieve-Adapt-Forge loop. Combined with multi-turn GRPO, a com- posite reward that balances answer correctness, tool retrieval, parameter adaptation, invocation ne- cessity, and forged-skill reusability, and capacity- constrained tool pool management, MetaForge learns when to answer, when to reuse existing tools, when to adapt tool calls, and when to forge new skills on demand. Experiments on 12 benchmarks against 16 baselines show consistent gains in ac- curacy, efficiency, and generalization under both IID and OOD dataset/tool settings. Further anal- ysis confirms more reliable tool selection, higher tool-call and skill-registration success, and effec- tive reuse of forged skills. Overall, MetaForge shifts multimodal agents from static tool invento- ries toward on-demand self-evolution. 8 Limitations Although MetaForge achieves strong results in adaptive tool orchestration and on-demand toolset evolution, several limitations remain. • In our implementation, each forged tool is implemented as an executable skill, consist- ing of a Markdown specification, an inter- face schema, and Python scripts. This form is mainly suited for programmatic task-level operations, such as code processing, chart analysis, and numerical computation. There- fore, Forge does not imply that the model can solve all multimodal tasks. Tasks requir- ing additional models or external resources, such as video synthesis, may remain beyond the current skill design. Future work can ex- plore stronger tool-forging models, richer skill forms, and more reliable validation mecha- nisms to broaden the supported task range. •Our evaluation covers 12 benchmarks but re- mains centered on image-text tasks. We have not evaluated settings involving audio, video, temporal reasoning, or interactive environ- ments. While the results show strong gen- eralization across the evaluated tasks, whether the same gains transfer to richer multimodal settings remains unclear. Future work can ex- tend the decide-retrieve-adapt-forge loop to richer inputs and study whether online skill forging remains effective for temporal and audio-visual reasoning. • MetaForge uses the decide reward and capacity-constrained tool pool management to reduce redundant calls and tool library growth, but the efficiency objective focuses on latency. In deployment, tool calls also introduce mon- etary cost and context usage. Future work can incorporate these factors into tool invocation, balancing monetary cost, context usage, and response time while preserving accuracy. References Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019. Tallyqa: Answering complex counting ques- tions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084. Anthropic. 2026.Introducing claude sonnet 4.6.https://w.anthropic.com/news/ claude-sonnet-4-6. Accessed: 2026-02-17. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Tanvir Bhathal and Asanshay Gupta. 2025. Websight: A vision-first architecture for robust web agents. arXiv preprint arXiv:2508.16987. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 4291–4301. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. In The Twelfth International Conference on Learning Representations. Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoub- hik Debnath, Ronghang Hu, Didac Suris, Chai- tanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, and 1 others. 2025. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719. Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler- Lussier, and Ningchuan Xiao. 2022. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, and 1 others. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711. Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. Hitab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1094–1110. Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, and 1 others. 2025. Paddleocr-vl: Boosting multilingual document pars- ing via a 0.9 b ultra-compact vision-language model. arXiv preprint arXiv:2510.14528. 9 Google DeepMind. 2026.Gemini 3.1 pro. https://deepmind.google/models/ model-cards/gemini-3-1-pro/.Ac- cessed: 2026-02-19. Hao Ding, Zhichuan Yang, Weijie Ge, Ziqin Gao, Chaoyi Lu, and Lei Zhao. 2026. Tikart: Aperture- guided observation for fine-grained visual reason- ing via reinforcement learning.arXiv preprint arXiv:2602.14482. Tianjiao Du, Junming Cao, Qinyue Wu, Wei Li, Beijun Shen, and Yuting Chen. 2019. Cocoqa: Question answering for coding conventions over knowledge graphs. In 2019 34th IEEE/ACM International Con- ference on Automated Software Engineering (ASE), pages 1086–1089. IEEE. Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching- Chen Kuo, Yuting Zheng, Xinze Guan, and Xin Eric Wang. Grit: Teaching mllms to think with images. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913. Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. 2025. Deepeyesv2: Toward agentic multimodal model. arXiv preprint arXiv:2511.05271. Xu Huang, Weiwen Liu, Xingshan Zeng, Yuefeng Huang, Xinlong Hao, Yuxian Wang, Yirong Zeng, Chuhan Wu, Yasheng Wang, Ruiming Tang, and 1 others. 2025. Toolace-dev: Self-improving tool learn- ing via decomposition and evolution. arXiv preprint arXiv:2505.07512. Pu Jian, Donglei Yu, Wen Yang, Shuo Ren, and Jiajun Zhang. 2025. Teaching vision-language models to ask: Resolving ambiguity in visual questions. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3619–3638. Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, and 1 others. 2025. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055. Guanyu Jiang, Zhaochen Su, Xiaoye Qu, and Yi R Fung. 2026. Xskill: Continual learning from experi- ence and skills in multimodal agents. arXiv preprint arXiv:2603.12056. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min- joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Euro- pean conference on computer vision, pages 235–251. Springer. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Ha- jishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern recognition, pages 4999–5007. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clini- cally generated visual questions and answers about radiology images. Scientific data, 5(1):180251. Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. 2025. Do- clayllm: An efficient multi-modal extension of large language models for text-rich document understand- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4038–4049. AD Lindström and S Abraham. 2022. Clevr-math: A dataset for compositional language, visual and math- ematical reasoning. In Proceedings of the 16th In- ternational Workshop on Neural-Symbolic Learning and Reasoning, volume 3212, pages 155–170. Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Man- dar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023a. Deplot: One-shot visual language rea- soning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10381–10399. Fangyu Liu, Guy Emerson, and Nigel Collier. 2023b. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635–651. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jian- wei Yang, Hang Su, and 1 others. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38–55. Springer. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic frame- work with extensible tools for complex reasoning. In ICLR 2025 Workshop on Foundation Models in the Wild. U-V Marti and Horst Bunke. 2002. The iam-database: an english sentence database for offline handwrit- ing recognition. International journal on document analysis and recognition, 5(1):39–46. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark 10 for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pages 2263– 2279. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Ena- mul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 14662–14684. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthe- nis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, pages 1697–1706. Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter con- ference on applications of computer vision, pages 2200–2209. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE. Kartik Narayan, Yang Xu, Tian Cao, Kavya Nerella, Vishal M Patel, Navid Shiee, Peter Grasch, Chao Jia, Yinfei Yang, and Zhe Gan. 2025. Deepmmsearch- r1: Empowering multimodal llms in multimodal web search. arXiv preprint arXiv:2510.12801. Maizhen Ning, Zihao Zhou, Qiufeng Wang, Xiaowei Huang, and Kaizhu Huang. 2025. Gns: Solving plane geometry problems by neural-symbolic rea- soning with multi-modal llms. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pages 24957–24965. Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, and 1 others. 2025. Mineru2. 5: A decoupled vision-language model for efficient high-resolution document parsing. arXiv preprint arXiv:2509.22186. OpenAI.2026.IntroducingGPT-5.4. https://openai.com/index/ introducing-gpt-5-4/. Accessed: 2026-03- 05. Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In European conference on computer vision, pages 647– 664. Springer. Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6922–6939. Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Ji- acheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, and 1 others. 2025. Alita: Generalist agent enabling scalable agentic reasoning with mini- mal predefinition and maximal self-evolution. arXiv preprint arXiv:2505.20286. Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answer- ing using world knowledge. In European conference on computer vision, pages 146–162. Springer. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 8317–8326. Mingyang Song, Haoyu Sun, Jiawei Gu, Linjie Li, Luxin Xu, Ranjay Krishna, and Yu Cheng. 2026. Adareasoner: Dynamic tool orchestration for iterative visual reasoning. arXiv preprint arXiv:2601.18631. Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. 2025a. Pixel reasoner: Incentiviz- ing pixel space reasoning via curiosity-driven rein- forcement learning. In The Thirty-ninth Annual Con- ference on Neural Information Processing Systems. Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, and 1 others. 2025b. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617. Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, and Jiaqi Wang. 2025. Seagent: Self-evolving computer use agent with au- tonomous learning from experience. arXiv preprint arXiv:2508.04700. Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. Visualmrc: Machine reading comprehension on docu- ment images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13878– 13888. Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, and 1 others. 2025a. 11 Step-3 is large yet affordable: Model-system co- design for cost-effective decoding. arXiv preprint arXiv:2507.19427. Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2words: Au- tomatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pages 498– 510. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. Voyager: An open-ended embod- ied agent with large language models. Transactions on Machine Learning Research. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025b. In- ternvl3. 5: Advancing open-source multimodal mod- els in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Lai Wei, Liangbo He, Jun Lan, Lingzhong Dong, Yu- tong Cai, Siyuan Li, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, and 1 others. 2026. Zoom- ing without zooming: Region-to-image distilla- tion for fine-grained multimodal perception. arXiv preprint arXiv:2602.11858. Georg Wölflein, Dyke Ferber, Daniel Truhn, Ognjen Arandjelovic, and Jakob Nikolas Kather. 2025. Llm agents making agent tools. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 26092–26130. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, and 1 others. 2025a. Qwen-image technical report.arXiv preprint arXiv:2508.02324. Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. 2025b. Mmsearch- r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670. Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengx- iang Zhai, and Klara Nahrstedt. 2025c. Vtool-r1: Vlms learn to think with images via reinforcement learning on multimodal tool use. arXiv preprint arXiv:2505.19255. Zhengzhuo Xu, Bowen Qu, Yiyan Qi, SiNan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of diversely aligned expert connector for chart understanding. In The Thirteenth International Conference on Learning Representations. Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, and 1 others. 2025. R1- onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 2376–2385. Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. Craft: Customizing llms by creating and retrieving from specialized toolsets. In The Twelfth International Conference on Learning Representations. YiFan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, and 1 others. Thyme: Think beyond images. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code. Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, and Chen Wei. 2026. Pyvision-rl: Forging open agentic vision models via rl. arXiv preprint arXiv:2602.20739. Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei. Pyvision: Agentic vision with dynamic tooling. In First Work- shop on Multi-Turn Interactions in Large Language Models. Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023. Robut: A systematic study of table qa robustness against human-annotated ad- versarial perturbations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6064– 6081. Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. 2025. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277–3287. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei- Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004. 12 A Experimental Details A.1 Dataset and Tool Set Splits To evaluate task and tool generalization, we split both datasets and tools into In-Domain (IID) and Out-of-Domain (OOD) settings. • Dataset Split. The training set contains 9,500 samples from 19 datasets across six categories: document understanding, general VQA, OCR, table understanding, image captioning, and geospatial reasoning.Table 3 summarizes datasets, categories, and sample counts. For evaluation, IID datasets share these categories but come from different sources.They in- clude DocVQA (Mathew et al., 2021), Tal- lyQA (Acharya et al., 2019), OCRVQA (Mishra et al., 2019), RoBUT-WikiSQL (Zhao et al., 2023), Localized Narratives (Pont-Tuset et al., 2020), and MapQA (Chang et al., 2022). OOD datasets cover six task categories unseen during training: chart, diagram, code, math, science, and spatial math. They include ChartQA (Masry et al., 2022), AI2D (Kembhavi et al., 2016), WebSight (Bhathal and Gupta, 2025), Math- Vista (Lu et al., 2023), ScienceQA (Saikh et al., 2022), and CLEVR-Math (Lindström and Abra- ham, 2022). We evaluate MetaForge on 12 benchmarks, with 24,005 test samples. Ta- ble 4 reports benchmarks, splits, sample counts, and data sources. We construct test sets in three ways. For ChartQA, AI2D, MathVista, and ScienceQA, we use full official test sets. For DocVQA, TallyQA, OCRVQA, RoBUT- WikiSQL, MapQA, Localized Narratives, and CLEVR-Math, we sample subsets to reduce evaluation cost. WebSight does not provide an official test split, so we sample from its training split. Because our training set excludes Web- Sight and code-generation tasks, this evaluation does not overlap with our training data. •Tool Set Split.The IID tool setting uses the 10 tools available during training, while the OOD setting replaces them with 8 unseen tools.Both settings keep the shared meta- toolscreate_skillandrun_skill, so all agents can forge and execute reusable tools when existing tools are insufficient. Table 5 lists all tools used in the IID and OOD tool settings. DatasetCategorySamples InfographicVQA (Mathew et al., 2022)Document500 AOKVQA (Schwenk et al., 2022)General VQA500 COCOQA (Du et al., 2019)General VQA500 VQAv2 (Goyal et al., 2017)General VQA500 Visual7W (Zhu et al., 2016)General VQA500 TQA (Kembhavi et al., 2017)General VQA500 VQARAD (Lau et al., 2018)General VQA500 VSR (Liu et al., 2023b)Image Captioning500 MapQA (Chang et al., 2022)Geospatial500 STVQA (Biten et al., 2019)OCR500 TextVQA (Singh et al., 2019)OCR500 VisualMRC (Tanaka et al., 2021)OCR500 IAM (Marti and Bunke, 2002)OCR500 Screen2Words (Wang et al., 2021)OCR500 TATQA (Zhu et al., 2021)Table500 FinQA (Chen et al., 2021)Table500 HiTab (Cheng et al., 2022)Table500 RoBUT-WTQ (Zhao et al., 2023)Table500 RoBUT-SQA (Zhao et al., 2023)Table500 Total6 categories9,500 Table 3: Composition of the training set across datasets and task categories. DatasetSplitSamplesSource DocVQA (Mathew et al., 2021)IID2,000Subset TallyQA (Acharya et al., 2019)IID2,000Subset OCRVQA (Mishra et al., 2019)IID2,000Subset RoBUT-WikiSQL (Zhao et al., 2023)IID2,000Subset MapQA (Chang et al., 2022)IID2,000Subset Localized Narratives (Pont-Tuset et al., 2020)IID1,400Subset ChartQA (Masry et al., 2022)OOD2,500Full AI2D (Kembhavi et al., 2016)OOD3,088Full WebSight (Bhathal and Gupta, 2025)OOD2,000 Train-split MathVista (Lu et al., 2023)OOD1,000Full ScienceQA (Saikh et al., 2022)OOD2,017Full CLEVR-Math (Lindström and Abraham, 2022) OOD2,000Subset Total24,005 Table 4: Test set composition. “Full” denotes the com- plete official test set; “Subset” denotes a random subset of the official test set; “Train-split” denotes a sample drawn from the training set due to the absence of an official test split. A.2 Baseline Details Table 6 summarizes the 16 baselines used in our experiments. A.3 Hyperparameters Configuration Table 7 summarizes the training configuration. B Extended Experimental Analysis B.1 Forged Tool Pool Evolution Analysis To analyze tool forging during evaluation, we track how the forged-tool pool grows as each method is evaluated on samples from the 12 datasets, ordered 1 https://github.com/JaidedAI/EasyOCR 2 https://github.com/opencv/opencv 3 https://w.wolframalpha.com/ 13 Setting ToolTypeFunction IID Tools IIDQwen3-VL-8B (Bai et al., 2025a)Model ToolGeneral visual understanding and VQA. IIDQwen3-VL-32B (Bai et al., 2025a)Model Tool Stronger multimodal reasoning for complex visual analysis. IIDQwen3-VL-235B-A22B (Bai et al., 2025a) Model ToolLarge-scale visual reasoning and answer verification. IIDGroundingDINO (Liu et al., 2024)Model ToolOpen-vocabulary object localization. IIDSAM3 (Carion et al., 2025)Model ToolObject and region segmentation. IIDPaddleOCR (Cui et al., 2025)Model ToolText detection and recognition in images. IIDEasyOCR 1 Model ToolLightweight OCR for scene text. IIDMinerU2.5 (Niu et al., 2025)Model ToolLayout analysis and structured document extraction. IIDQwen-Image-Edit (Wu et al., 2025a)Model ToolInstruction-based image editing. IIDOpenCV 2 Executable ToolClassical image operations such as crop, resize, thresh- olding, and edge detection. OOD Tools OODInternVL3.5-38B (Wang et al., 2025b)Model ToolLarge multimodal model from a different architecture family. OODInternVL3.5-14B (Wang et al., 2025b)Model ToolEfficient multimodal model from a different architec- ture family. OODStep3 (Wang et al., 2025a)Model ToolMultimodal reasoning over image-text inputs. OODWolframAlpha 3 Executable Tool Math, science, unit conversion, and symbolic calcula- tion. OODUniChart (Masry et al., 2023)Model ToolChart-specific visual question answering. OODDeplot (Liu et al., 2023a)Model ToolChart-to-table conversion. OODChartMoE (Xu et al.)Model ToolChart understanding and reconstruction. OODPython Code (Jiang et al., 2025)Executable Tool Sandboxed numerical computation and data process- ing. Table 5: IID and OOD tools used in our experiments. IID tools are available during training, while OOD tools are unseen during training. GroupModelDescription Closed-sourceGPT-5.4 (OpenAI, 2026)Proprietary multimodal model from OpenAI. Closed-sourceClaude Sonnet 4.6 (Anthropic, 2026)Proprietary multimodal model from Anthropic. Closed-sourceGemini 3.1 Pro (DeepMind, 2026)Proprietary multimodal model from Google. Open-source multimodal modelQwen2.5-VL-7B-Instruct (Bai et al., 2025b)Open-source QwenVL model used as a 7B-scale baseline. Open-source multimodal modelQwen3-VL-8B-Instruct (Bai et al., 2025a)Base policy family and main 8B-scale multimodal baseline. Open-source multimodal modelQwen3-VL-32B-Instruct (Bai et al., 2025a)Larger Qwen3-VL baseline for scale comparison. Open-source multimodal modelQwen3-VL-235B-A22B-Instruct (Bai et al., 2025a) Large-scale Qwen3-VL baseline. Open-source multimodal modelInternVL3.5-14B-Instruct (Wang et al., 2025b)InternVL-family baseline with a different architecture. Open-source multimodal modelInternVL3.5-38B-Instruct (Wang et al., 2025b)Larger InternVL-family baseline. Tool-augmented multimodal model VTool-R1-7B (Wu et al., 2025c)Reinforcement learning method for multimodal chain-of-thought and tool use. Tool-augmented multimodal model R1-Onevision-7B (Yang et al., 2025)Converts visual inputs into structured text for cross-modal reasoning. Tool-augmented multimodal model PyVision-Image-7B-RL (Zhao et al., 2026)Learns multi-turn tool invocation with cumulative tool rewards. Tool-augmented multimodal model DeepEyesV2-7B (Hong et al., 2025)Uses cold-start training and reinforcement learning for tool usage be- havior. Tool-augmented multimodal model PixelReasoner-RL-vl-7B (Su et al., 2025a)Learns pixel-level zooming and frame selection through reinforcement learning. Tool-augmented multimodal model AdaReasoner-7B (Song et al., 2026)Trains tool usage as general reasoning behavior with Tool-GRPO. Tool-augmented multimodal model TikArt-8B (Ding et al., 2026)Interleaves language reasoning with zooming and segmentation opera- tions. Tool-augmented multimodal model ZwZ-8B (Wei et al., 2026)Distills inference-time zooming behaviors into single-forward-pass primitives. Table 6: Baseline models used for comparison. by dataset. As Figure 4 shows, existing baselines exhibit two distinct forging patterns. One group continuously forges tools in large numbers, regard- less of whether matching tools already exist for the current task. The other group barely forges any tools, even in OOD dataset regions where clear tool gaps exist. Both patterns reveal the same de- ficiency: these models fail to identify tool gaps 14 ItemValue FrameworkVerlTool (Jiang et al., 2025) Base policyQwen3-VL-8B-Instruct (Bai et al., 2025a) Tool-forging modelQwen3-VL-235B-A22B-Instruct (Bai et al., 2025a) RL algorithmGRPO Python version3.10 PyTorch version2.6.0 Transformers version4.51.3 vLLM version0.8.4 Hardware8× NVIDIA H200 GPUs, 141 GB each PrecisionBF16 mixed precision Rollouts per input M8 Sampling temperature / top-p1.0 / 1.0 Clipping ratio ε0.2 KL coefficient β0.002 Learning rate3e-6 Batch size128 Epochs / update steps3 / 223 Training timeapproximately 80 hours Maximum prompt length24,576 tokens Maximum response length24,576 tokens Maximum action length8,192 tokens Maximum observation length16,384 tokens Maximum interaction turns T10 Forged-tool pool capacity K50 LLM judgeGPT-5 (Singh et al., 2025) Table 7: Training hyperparameters configuration. between task requirements and the available tools, and thus cannot adapt their forging behavior. In contrast, MetaForge adapts tool forging to the gap between task requirements and existing tools. Under IID tools, it forges almost no tools on the first six datasets and then increases sharply on OOD datasets such as WebSight and MathVista before plateauing. Under OOD tools, broader tool cover- age leaves fewer gaps, so the model forges almost no tools. This behavior comes from the reward design: when existing tools suffice, tool forging adds latency without improving accuracy. When tools fall short, tool forging becomes necessary for the correct answer. Through competition among sampled trajectories, the model learns to forge tools when needed. B.2 Reward Function Analysis The Retrieve/Adapt reward is the core objective and activates the auxiliary rewards, so we ablate only the three auxiliary rewards. As shown in Table 8, removing any auxiliary reward reduces accuracy, confirming that each component is use- ful. Removing the format reward causes the largest drop across nearly all datasets. This reward en- forces valid tool-call syntax; without it, the model more often produces malformed calls that cannot be parsed or executed, blocking tool feedback for later decisions. Removing the Forge reward mainly hurts OOD datasets, with a large drop on tasks such as WebSight. These cases often contain clear tool gaps, showing that the Forge reward is important for encouraging on-demand tool forging. Remov- ing the Decide reward causes the smallest drop, but the decline appears under both IID and OOD tool settings. This suggests that Decide supervision en- courages concise calling paths and reduces redun- dant interaction rounds, which lowers the chance of intermediate errors. B.3 Backbone Generalization Analysis We train MetaForge on three backbones: Qwen2.5- VL-7B, Qwen3-VL-8B, and InternVL3.5-14B, covering architectures (QwenVL and InternVL) and scales (7B/8B/14B). As shown in Table 9, MetaForge brings gains across all backbones, in- dicating that it does not depend on specific archi- tecture, scale, or pretraining strategy. Weaker back- bones benefit more. InternVL3.5-14B has the low- est baseline ALL AVG (65.80/61.38), but improves to 85.54/84.69 after training, yielding the largest gain. The stronger Qwen3-VL-8B reaches the best overall performance of 88.14/88.98. These results demonstrate that adaptive tool call and online tool forging are backbone-agnostic. They compensate for the inherent limitations of different backbone models through tool interaction and forging. B.4 Tool Usage Entropy Analysis To compare tool selection, we report call distribu- tions and per-dataset TUE over 12 datasets (Fig- ures 5 and 6). Distributions are grouped into Model tools, Executable tools, and Forged tools. Existing methods show three patterns. First, some methods over-invoke tools. PyVision-Image- 7B-RL and DeepEyesV2-7B make thousands to tens of thousands of calls per dataset, with TUE around 3.5–4.0. This suggests that they compensate for weak selection by trying many tools rather than choosing a concise combination. Second, some methods underuse external tools. R1-Onevision- 7B issues only single-digit calls on most datasets, indicating limited ability or tendency to use the available tool pool. Third, some methods are bi- ased toward tool categories. TikArt-8B and ZwZ- 8B reach low TUE on some IID datasets (0.1–0.5), but their TUE rises to 2.6–3.2 on WebSight and MathVista, with overall TUE of 2.7–3.5. This in- dicates partial task differentiation, but not a stable orchestration strategy across tasks. In contrast, MetaForge shows task-adaptive tool orchestration under both IID and OOD tool settings. Under IID tools (Figure 5), calls on 11 datasets are dominated by Model tools, suggesting that the pol- 15 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 50 100 150 200 250 300 Cumulative Registered Skills IID DatasetsOOD Datasets (a) w/ IID Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 25 50 75 100 125 150 175 200 Cumulative Registered Skills IID DatasetsOOD Datasets (b) w/ OOD Tools Qwen2.5-VL-7B Qwen3-VL-8B VTool-R1-7B R1-Onevision-7B PyVision-Image-7B-RL DeepEyesV2-7B PixelReasoner-RL-v1-7B AdaReasoner-7B TikArt-8B ZwZ-8B Ours Figure 4: Cumulative registered forged tools over the evaluation process under IID and OOD tool settings. Reward ComponentsIID DatasetsOOD DatasetsALL r ra r forge r fmt r dec DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. AVG. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M AVG. AVG. w/ IID Tools Qwen3-VL-8B (Bai et al., 2025a)89.8070.2077.4061.7070.6595.7877.5982.1684.5851.1560.7078.0363.5070.02 73.80 ✓97.9080.5286.5187.3677.9899.6288.3290.4989.6098.3478.0189.6881.7087.9788.14 ✓✗✓96.2574.2579.4080.3068.8898.7182.9785.8887.8273.4570.2085.9182.0080.88 81.92 ✓✗✓92.3068.5575.7078.7564.8596.2879.4185.9285.6289.9568.2083.5379.5582.13 80.77 ✓✗96.0073.6583.05 83.8076.3598.5785.2487.6489.2890.8072.5086.5678.0584.1484.69 w/ OOD Tools Qwen3-VL-8B (Bai et al., 2025a)90.2071.1576.9565.3572.8595.7178.7082.0484.6851.7058.8077.9363.2069.73 74.21 ✓97.2281.1786.0087.6789.1499.5890.1390.1888.1598.7279.7889.7380.4087.8388.98 ✓✗✓95.8574.8078.7078.4075.0099.2183.6685.7287.5075.0071.0085.3781.0080.93 82.30 ✓✗✓92.2065.8071.8576.9565.3097.5078.2784.6487.0791.2567.8085.2778.6582.45 80.36 ✓✗94.4073.9583.6083.7578.2598.2185.3685.4889.5090.6571.7086.1278.5583.6784.51 Table 8: Ablation study of reward components under IID and OOD tool settings.✓and✗indicate the inclusion and removal of a reward component, respectively. We compare the full model with all four rewards against three variants, each removing one of r forge , r fmt , and r dec . Models (↓) / Datasets (→) IID DatasetsOOD DatasetsALL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. AVG. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M AVG. AVG. w/ IID Tools Qwen2.5-VL-7B (Bai et al., 2025b)93.4368.2271.8370.2445.3199.1074.6974.9548.9471.4757.0473.7364.4065.09 69.89 MetaForge95.8075.8582.7582.3587.1599.2187.1984.3687.2496.2064.3083.8378.1582.3584.77 Qwen3-VL-8B (Bai et al., 2025a)89.8070.2077.4061.7070.6595.7877.5982.1684.5851.1560.7078.0363.5070.02 73.80 MetaForge97.9080.5286.5187.3677.9899.6288.3290.4989.6098.3478.0189.6881.7087.9788.14 InternVL3.5-14B (Wang et al., 2025b)87.6071.1866.4669.9859.2072.2771.1280.8367.0844.0035.0769.0666.8860.49 65.80 MetaForge95.9072.9083.2583.7078.6099.0085.5687.3287.7596.6078.2085.0778.2085.5285.54 w/ OOD Tools Qwen2.5-VL-7B (Bai et al., 2025b)93.6367.4678.1870.6650.0198.6676.4375.4151.9573.1556.8874.4266.6066.40 71.42 MetaForge95.4576.7583.1082.9087.0099.3587.4384.5689.2496.0064.1084.3879.4082.9585.19 Qwen3-VL-8B (Bai et al., 2025a)90.2071.1576.9565.3572.8595.7178.7082.0484.6851.7058.8077.9363.2069.73 74.21 MetaForge97.2281.1786.0087.6789.1499.5890.1390.1888.1598.7279.7889.7380.4087.8388.98 InternVL3.5-14B (Wang et al., 2025b)88.2063.8559.1059.2653.9070.6465.8374.4463.4739.5029.9065.3468.9556.93 61.38 MetaForge94.3573.7581.6580.8078.5098.5784.6085.5286.3294.8579.5083.9378.5584.7884.69 Table 9: Generalization across backbone multimodal models. MetaForge is applied to three backbones spanning two architecture families (QwenVL, InternVL) and different parameter scales (7B/8B/14B). icy recognizes their broad coverage in this setting. WebSight is the exception: forged-tool calls dom- inate because the existing tools lack the required website-generation function, so the model forges task-specific tools. Under OOD tools (Figure 6), the pattern changes. Executable tools increase on ChartQA, WebSight, and MathVista, showing that the policy can use newly available executable tools when they better match the task. Forged-tool calls nearly disappear, indicating that the expanded OOD tool set already covers most needs. Claude- Sonnet-4-6 also obtains relatively low TUE (IID 1.81, OOD 1.63), but it remains Model-dominant in both settings, suggesting focused selection without the same adaptive shift across tool settings. These results align with MetaForge’s design. The unified tool interface places Model tools, Ex- ecutable tools, and Forged tools in one action space. Multi-turn GRPO optimizes the policy to retrieve, adapt, forge, and reuse tools according to task needs. Online tool forging expands the tool pool when existing tools are insufficient. Capacity- constrained tool pool management deduplicates overlapping forged tools and retains high-value 16 ones. Together, these mechanisms lead to a com- pact and task-sensitive tool selection strategy. B.5Analysis of Tool Invocation Success Rates To thoroughly assess invocation robustness across different task scenarios and tool configurations, we report per-dataset tool call success rates for each model and conduct a comparative analysis under both the IID and OOD tool settings. Under the IID tool setting, per-dataset tool call success rates reveal significant stratification among models. Ex- isting baselines span a wide range of overall suc- cess rates, from below 50% (e.g., R1-Onevision-7B at 47.6%) to around 90%. Most baselines exhibit large cross-dataset fluctuations, with some drop- ping to 0% on specific datasets. Under the OOD tool setting, most baselines ex- hibit performance shifts between the two tool sets: some degrade substantially (e.g., VTool-R1-7B drops from 72.1% to 58.5%), while others improve (e.g., Qwen2.5-VL-7B rises from 46.5% to 66.2%). This sensitivity to tool set composition exposes the lack of robust tool generalization in existing meth- ods. MetaForge never encounters the OOD tools during training, yet its success rate improves from 96.6% to 98.2%. All 12 datasets exceed 84.6%, and 9 of them surpass 98%. This result indicates that MetaForge learns general-purpose tool orches- tration rather than memorizing specific tool config- urations. The unified tool interface enables flexible dispatch of unfamiliar tools based on functional descriptions. Furthermore, online tool forging en- ables the model to forge task-specific tools on de- mand, effectively compensating for the limitations of the predefined tool set. Together, these two com- ponents ensure consistently high tool call success rates under arbitrary tool configurations. B.6 Error Pattern Analysis To investigate the root causes of invocation failures across methods, we broadly categorize the 11 error types into three groups: (1) Hallucination errors, includingunknown_skill(invoking a non- existent forged tool),missing_entrypoint (invoking a non-existent script within a forged tool, where the model fabricates script names without consulting the tool description), and invalid_entrypoint(specifying a non- compliant script entry point, as only.pyand.sh formats are supported), reflecting generally poor awareness of the available tool pool; (2) Parame- ter errors, includinginvalid_parameters/ invalid_arguments(passing invalid values), missing_parameters(omitting required pa- rameters),andmissing_image_index/ invalid_image_index(missing or out-of- range image indices), reflecting a limited under- standing of tool interface specifications; (3) Execu- tion errors, includingruntime_error(inter- nal script execution failure, typically due to miss- ing internal arguments),request_failed(ex- ternal service exceptions such as network instabil- ity), andtimeout(subprocess execution time- out), which typically arise mainly from environ- mental factors or forged-tool code quality rather than model behavior defects. As shown in the figure, the majority of exist- ing methods concentrate their failures on halluci- nation and parameter errors. PyVision-Image-7B- RL produces over 41K failed calls under the IID setting, withinvalid_parametersalone ac- counting for 19,785 instances, indicating a severe lack of understanding of tool parameter formats. DeepEyesV2-7B accumulates over 21K failures, withmissing_entrypointreaching 7,610, revealing frequent hallucinated invocations of non- existent scripts within forged tools. R1-Onevision- 7B exhibits the most severe dysfunction, issuing only 231 calls in total with a success rate of merely 47.62%. Of its failed calls,unknown_skill alone accounts for 59 instances, suggesting that the model lacks the basic ability to recognize and select from the available tool pool. Under the OOD set- ting,unknown_skillerrors increase substan- tially for most methods (VTool rises from 472 to 1,096; PixelReasoner rises from 607 to 1,424), in- dicating that methods trained on fixed tool sets lack generalization to unseen tools. MetaForge achieves the highest overall suc- cess rates of 96.59% (IID) and 98.24% (OOD) among all methods. The errors consist primar- ily ofruntime_error(515 instances, con- centrated on WebSight forged-tool code execu- tion anomalies) andinvalid_image_index (357/411 instances, concentrated on ScienceQA image index handling), both of which are execution- class errors attributable to environmental fac- tors rather than deficiencies in the model’s tool understanding.More importantly, hal- lucination errors (missing_entrypoint+ unknown_skill) total only 14 and 9 under IID and OOD, respectively, validating the general- ized invocation behavior endowed by the unified tool interface and Forge reward. Notably, Claude- 17 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 200 400 Tool Call Count Qwen2.5-VL-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 6000 Qwen3-VL-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 VTool-R1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 TUE 2.96 2.16 3.16 2.91 1.72 2.04 2.00 2.74 3.24 2.28 3.13 2.39 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 1.78 1.70 2.76 1.54 2.16 0.34 1.12 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 2.77 2.00 2.89 2.87 1.79 2.74 1.65 2.44 3.38 2.47 3.02 2.25 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 20 40 60 Tool Call Count R1-Onevision-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 5000 10000 15000 PyVision-Image-7B-RL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 5000 10000 15000 DeepEyesV2-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 TUE 2.65 2.70 2.86 3.16 2.95 2.87 1.38 2.99 1.50 3.08 2.60 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 2.48 2.81 3.11 3.48 2.78 3.41 2.70 3.07 3.91 3.56 3.81 3.35 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 3.77 3.86 2.90 3.80 2.65 2.40 3.30 3.96 2.02 3.40 3.41 3.93 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 Tool Call Count PixelReasoner-RL-v1-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 6000 AdaReasoner-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2500 5000 7500 TikArt-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 TUE 2.66 2.27 3.41 2.60 1.92 2.64 2.07 2.42 3.65 2.30 3.14 2.29 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 2.47 3.11 1.96 2.87 2.93 3.28 2.14 3.01 2.83 3.42 2.88 3.87 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 1.39 0.52 1.51 2.38 0.53 1.34 3.22 1.67 0.73 2.34 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 5000 10000 Tool Call Count ZwZ-8B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 Claude-Sonnet-4-6 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 MetaForge DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 TUE 1.60 0.30 1.66 2.38 1.59 2.66 0.66 0.53 2.11 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1 2 1.87 1.72 2.35 2.24 1.42 0.58 1.58 1.88 1.43 1.79 1.93 1.14 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 0.57 0.49 0.50 3.55 TUE Tool Usage & TUE IID Tools Figure 5: Tool call distribution and per-dataset Tool Usage Entropy (TUE) for each model under the IID tool setting. 18 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 100 200 300 Tool Call Count Qwen2.5-VL-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 6000 Qwen3-VL-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 VTool-R1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 TUE 3.08 2.74 2.67 3.29 2.78 1.71 2.67 2.07 2.93 2.42 2.97 2.75 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 0.80 0.56 1.20 2.09 0.70 0.89 2.35 0.80 3.45 2.78 2.19 0.95 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 2.81 2.70 3.72 3.58 3.20 2.98 3.79 3.39 3.10 2.98 3.49 2.57 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 20 40 Tool Call Count R1-Onevision-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 5000 10000 15000 PyVision-Image-7B-RL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 5000 10000 DeepEyesV2-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 TUE 1.37 2.53 2.30 2.04 3.32 2.98 2.86 3.48 2.25 1.30 2.50 2.06 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 3.23 3.22 2.94 2.99 3.37 2.99 3.24 3.24 3.08 3.24 3.16 3.04 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 3.03 2.71 3.42 3.22 1.91 3.43 3.10 3.64 2.30 3.29 3.22 3.46 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 Tool Call Count PixelReasoner-RL-v1-7B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 AdaReasoner-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 6000 TikArt-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 4 TUE 2.59 2.14 3.24 2.06 2.12 2.61 2.28 2.05 2.85 2.01 2.46 1.86 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0.0 2.5 5.0 1.78 3.14 2.82 3.19 3.39 2.84 3.56 3.34 3.27 3.90 3.47 4.93 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 2.27 2.32 2.33 1.51 2.38 1.73 3.03 3.02 2.63 2.61 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 Tool Call Count ZwZ-8B Model Tools Executable Tools Forged Tools DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1000 2000 Claude-Sonnet-4-6 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2000 4000 MetaForge DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 TUE 0.61 0.33 1.65 1.98 0.93 0.70 2.39 0.59 2.33 2.70 2.55 0.73 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 2 1.59 1.39 1.63 1.98 1.38 1.13 2.48 1.30 0.72 2.05 1.36 0.66 TUE DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M 0 1 2 0.30 0.66 1.74 0.33 1.10 2.05 0.95 TUE Tool Usage & TUE OOD Tools Figure 6: Tool call distribution and per-dataset Tool Usage Entropy (TUE) for each model under the OOD tool setting. 19 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 29.1 51.8 52.5 31.6 59.3 44.6 49.4 38.5 35.6 44.6 54.0 54.0 46.5 Qwen2.5-VL-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 100.0100.0 98.2 94.4 100.0100.0 97.9 94.5 71.7 97.5 93.8 86.7 82.6 Qwen3-VL-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 50.9 87.3 47.0 55.8 92.3 81.4 94.9 79.8 50.7 72.9 74.2 86.8 72.1 VTool-R1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 25.0 50.0 38.9 25.0 33.3 71.2 42.9 23.7 0.00.0 54.2 86.7 47.6 R1-Onevision-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 48.4 53.1 53.7 39.9 57.0 67.8 58.3 53.0 59.9 58.2 56.4 59.6 55.4 PyVision-Image-7B-RL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 56.4 59.6 42.9 56.9 32.8 85.4 60.0 63.8 31.3 72.1 55.3 59.0 53.3 DeepEyesV2-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 63.6 85.2 45.2 48.9 87.8 83.4 84.8 79.4 43.7 84.1 72.6 89.0 76.2 PixelReasoner-RL-v1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 57.8 85.7 86.7 88.5 80.1 83.9 90.5 67.1 94.1 77.2 70.6 80.9 79.5 AdaReasoner-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 98.2 99.9 99.5 97.2 98.4 100.0 98.7 98.1 79.7 97.7 82.2 88.3 91.4 TikArt-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 96.4 100.0100.0 81.1 100.0100.0 93.0 98.1 64.4 100.0 96.0 80.1 76.2 ZwZ-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 100.0 98.4 93.3 96.2 99.1 100.0 99.6 94.8 43.8 97.2 96.3 99.5 88.3 Claude-Sonnet-4-6 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 99.9 100.0 99.9 99.8 97.7 100.0100.0 99.1 89.0 98.2 87.0 100.0 96.6 MetaForge Tool Call Success Rate per Dataset IID Tools Figure 7: Per-dataset tool call success rates for each model under the IID tool setting. Existing baselines exhibit large cross-dataset variance, with some dropping to near 0% on specific datasets. MetaForge achieves at least 87% on all 12 datasets and exhibits the smallest cross-dataset variance. 20 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 72.1 75.7 44.6 50.0 66.7 70.8 80.7 68.6 59.2 79.2 73.4 65.0 66.2 Qwen2.5-VL-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 97.5 99.8 90.9 89.5 99.5 100.0 97.5 99.1 80.2 92.9 90.0 98.7 86.3 Qwen3-VL-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 65.9 72.4 52.0 51.6 64.5 58.8 55.8 57.6 54.3 57.4 54.0 71.6 58.5 VTool-R1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 20.0 56.5 9.1 81.8 26.7 64.6 44.4 38.7 33.3 100.0 37.5 72.2 52.0 R1-Onevision-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 71.8 75.8 69.7 73.9 65.7 74.0 69.4 68.2 75.5 75.1 73.4 76.4 72.0 PyVision-Image-7B-RL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 72.3 80.2 66.2 63.2 81.8 78.2 75.2 68.8 28.3 68.0 52.3 75.0 59.5 DeepEyesV2-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 69.6 79.8 50.3 50.7 72.4 60.5 84.0 77.4 43.9 75.6 63.7 81.4 71.3 PixelReasoner-RL-v1-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 81.0 80.3 88.1 85.9 89.2 88.3 86.8 79.6 83.6 76.2 87.9 79.0 83.7 AdaReasoner-7B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 99.7 99.8 87.6 79.1 93.2 96.8 97.9 97.0 74.7 94.5 86.1 89.8 88.8 TikArt-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 Success Rate (%) 99.4 99.7 82.1 93.8 98.3 99.7 98.3 99.1 83.1 90.0 80.9 100.0 89.5 ZwZ-8B DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 85.2 87.1 80.5 87.2 82.5 96.4 68.0 83.1 94.0 78.1 84.8 95.6 84.5 Claude-Sonnet-4-6 DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M Overall 0 20 40 60 80 100 120 100.0 99.7 99.1 99.8 99.7 100.0 99.8 98.7 99.9 98.3 84.6 100.0 98.2 MetaForge Tool Call Success Rate per Dataset OOD Tools Figure 8: Per-dataset tool call success rates for each model under the OOD tool setting. Most baselines show notable performance shifts between the two tool sets, whereas MetaForge improves from an overall 96.6% (IID) to 98.2% (OOD), with all datasets exceeding 84.6%. 21 Sonnet-4-6, serving as the closed-source upper bound, concentrates its errors almost exclusively onrequest_failed(915 under IID, 1,293 un- der OOD), which stem from external API service instability rather than any deficiency in tool un- derstanding. This further corroborates that once a model acquires robust tool understanding, the dominant failure mode shifts from model-intrinsic deficits (e.g., hallucinated invocations and param- eter misspecification) to external execution condi- tions beyond the model’s control. B.7 Joint Sensitivity Analysis of Learning Rate and Format Reward The actor learning rate and the format reward mag- nitude are two key hyperparameters that jointly govern training dynamics. The learning rate con- trols the step size of policy gradient updates, while the format reward magnitude determines how strongly the model is penalized or rewarded for syntactic compliance in tool calls. We con- duct a grid search over learning rateslr ∈ 1e-6, 2e-6, 3e-6, 4e-6, 5e-6and format reward magnitudesα ∈ 0.1, 0.2, 0.3, 0.4, 0.5, yield- ing 25 configurations evaluated under both IID and OOD tool settings. As shown in Table 10, the combination of lr=3e-6andα=0.2achieves the highest overall accuracy under both tool settings (IID: 88.14%, OOD: 88.98%). Several observations emerge from the grid. First, for any fixed format reward magni- tude,lr=3e-6consistently yields the best or near- best accuracy, confirming it as the optimal learn- ing rate across a range of format reward strengths. Second, for any fixed learning rate,α=0.2con- sistently outperforms other magnitudes, indicating that a moderate format penalty provides the best balance between syntactic compliance and behav- ioral flexibility. Third, the two hyperparameters exhibit a mild interaction effect: at lower learning rates (e.g.,1e-6), performance is relatively insen- sitive toαbecause the small policy updates limit the influence of any individual reward signal; at higher learning rates (e.g.,5e-6), the choice ofα becomes more consequential, as large policy up- dates amplify the impact of the format reward, and an overly strong penalty (α≥ 0.4) combined with aggressive optimization leads to compounded per- formance degradation. B.8 Case Study To more intuitively demonstrate the advantages of MetaForge, we select six representative cases from the validation set and compare it against three types of baselines: the base model (Qwen3-VL-8B), a tool-augmented model (TikArt-8B), and a closed- source model (Claude-Sonnet-4-6). These cases are organized into three categories: (1) Type A: only MetaForge answers correctly while all three baselines fail, demonstrating superior accuracy through adaptive tool orchestration; (2) Type B: all models answer correctly but MetaForge requires significantly fewer interaction turns, demonstrating its ability to achieve an optimal trade-off between accuracy and inference cost through the Decide reward; and (3) Type C: MetaForge forges task- specific tools when existing tools are insufficient and succeeds, while baselines either fail to forge tools or create ineffective ones, demonstrating its ability to extend the tool pool on demand. Type A: MetaForge Achieves Higher Accuracy via Adaptive Tool Orchestration. Case A1 is a ChartQA sample evaluated under the OOD dataset and OOD tool setting, requiring extraction of a nu- merical difference from a bar chart (ground truth: 52). MetaForge completes the task in 4 turns by orchestrating three tools: it first calls Chart- Moe to extract the PET bottle collection rates, rec- ognizes that the returned country names do not match the chart, then switches to the suitable tool InternVL3.5-14B-Instruct for re-extraction, and finally invokespython_codeto compute the difference. This trajectory illustrates a key be- havior acquired through multi-turn GRPO train- ing: the ability to detect tool failure from obser- vations and adaptively switch to suitable tools. In contrast, Qwen3-VL-8B calls UniChart, then python_code, but computes from incorrect val- ues. TikArt-8B attempts to solve the problem using onlypython_codewithout chart extrac- tion. Claude-Sonnet-4-6 submits an answer directly without invoking tools, producing an incorrect re- sult. Case A2 is a MathVista sample evaluated un- der the OOD dataset and OOD tool setting, re- quiring computation of the average between the longest light-blue bar and the shortest gray bar in a chart (ground truth: 273). MetaForge decomposes the task into two stages: first calling UniChart to extract bar values (415 and 131), then invoking python_codeto compute the mean, yielding 22 Qwen2.5-VL-7B Qwen3-VL-8B VTool-R1-7B R1-Onevision-7B PyVision-Image-7B-RL DeepEyesV2-7B PixelReasoner-RL-v1-7B AdaReasoner-7B TikArt-8B ZwZ-8B Ours Claude-Sonnet-4-6 Qwen2.5-VL-7B Qwen3-VL-8B VTool-R1-7B R1-Onevision-7B PyVision-Image-7B-RL DeepEyesV2-7B PixelReasoner-RL-v1-7B AdaReasoner-7B TikArt-8B ZwZ-8B Ours Claude-Sonnet-4-6 0 10000 20000 30000 40000 Number of Failed Calls IID ToolsOOD Tools Error Pattern Distribution Across Models runtime_error missing_entrypoint unknown_skill request_failed invalid_parameters missing_parameters invalid_image_index missing_image_index invalid_arguments timeout invalid_entrypoint Figure 9: Distribution of error types across models under IID (left) and OOD (right) tool settings. Errors are categorized into Hallucination (e.g., invoking non-existent tools), Parameter (e.g., missing or invalid arguments), and Execution (e.g., runtime exceptions) errors. MetaForge produces the fewest total errors, with residual failures concentrated in execution-class errors attributable to environmental factors rather than tool-understanding deficien- cies. the correct answer. In contrast, Qwen3-VL-8B attempts to answer without tool usage and misiden- tifies the bar values, producing an incorrect result; TikArt-8B usespython_codefor computation but operates on incorrect values, also getting the wrong answer; and Claude-Sonnet-4-6 answers di- rectly without tool invocation, producing an incor- rect result. These cases demonstrate that multi-turn GRPO training enables MetaForge to learn adap- tive tool selection that recovers from tool failures, yielding higher task accuracy. Type B: Optimizing Accuracy-Efficiency Trade- off via Decide Reward. Case B1 is a Math- Vista sample evaluated under the OOD dataset and OOD tool setting, requiring identification of the accuracy difference between the most and least accurate algorithms in a bar chart (ground truth: 80). MetaForge solves it in just 2 turns by di- rectly calling InternVL3.5-14B-Instruct, which ac- curately extracts all four bar values in a single in- vocation, enabling immediate answer computation. In comparison, Qwen3-VL-8B requires 4 turns, sequentially trying UniChart, Deplot, and Chart- Moe before obtaining usable values; TikArt-8B requires 5 turns, calling UniChart twice, then De- plot andpython_code; Claude-Sonnet-4-6 re- quires 3 turns, first calling ChartMoe and then cross-validating with InternVL3.5-14B-Instruct be- fore computing the answer. Case B2 is a CLEVR-Math sample evaluated under the OOD dataset and OOD tool setting, ask- ing how many cylinders remain after subtracting cyan ones (ground truth: 2). MetaForge com- pletes the task in 2 turns with a single call to InternVL3.5-14B-Instruct, which counts the cylin- ders and identifies the cyan one in one step. Qwen3- VL-8B requires 3 turns, calling InternVL3.5-14B- Instruct twice with redundant queries. Claude- Sonnet-4-6 requires 3 turns, first calling Qwen3- VL-8B-Instruct, which returns an imprecise de- scription, then switching to InternVL3.5-14B- Instruct for a targeted count. TikArt-8B exhibits inefficient behavior, consuming 10 turns: it cre- ates and runs four forged tools implemented as skills: image-scene-summarizer, visual-content- summarizer, small-brown-metallic-object-tracker, and small-gray-metallic-object-tracker, each trig- gering a multi-step workflow of reading tool doc- umentation, preparing arguments, and executing scripts, before finally falling back to InternVL3.5- 14B-Instruct. These cases demonstrate that the De- cide reward within the GRPO objective incentivizes the model to converge on the most informative tool in fewer steps, resulting in a Pareto-superior trade- off between accuracy and efficiency compared to baselines that rely on exhaustive enumeration or redundant invocations. 23 lr α IID DatasetsOOD DatasetsALL DocVQA TallyQA OCRVQA RoBUT MapQA Loc.Narr. AVG. ChartQA AI2D WebSight MathVista ScienceQA CLEVR-M AVG. AVG. w/ IID Tools 1e-6 0.194.8175.5981.8682.1371.6197.6983.9585.8185.3589.5870.9284.9075.0381.93 82.94 0.295.4376.8482.7383.1473.4298.1784.9686.7286.3791.2472.4386.1476.8583.29 84.12 0.395.2376.1982.1482.6272.6697.9184.4686.1085.9290.2471.5385.4775.7582.50 83.48 0.494.4275.2481.5881.4671.7997.6683.6985.3585.0789.3170.2484.9474.4881.57 82.63 0.594.2573.7580.2580.6570.0897.4082.7383.8683.6086.7267.2883.5773.8579.81 81.27 2e-6 0.196.4477.7584.5485.4376.9098.8986.6688.0487.7295.7475.1987.4380.6985.80 86.23 0.297.1279.3485.6386.2878.4299.2887.6889.5788.9397.4876.8488.9282.3487.35 87.51 0.396.9978.9685.4386.0578.1399.2187.4689.3588.7796.9576.3988.6781.9787.02 87.24 0.496.6278.2784.6985.1677.4498.9186.8588.5487.8595.8974.9487.8980.8485.99 86.42 0.595.6077.5083.5684.1474.2698.5585.6087.2686.8193.3173.1287.8979.6784.68 85.14 3e-6 0.197.2378.8485.4485.4375.8099.2987.0189.0688.5095.7076.3188.6680.2786.42 86.71 0.297.9080.5286.5187.3677.9899.6288.3290.4989.6098.3478.0189.6881.7087.9788.14 0.397.62 80.0986.0886.8877.5499.4587.9490.1189.2897.5577.3789.4281.1387.4887.71 0.497.3579.2085.5685.9976.7299.2487.3489.2988.6395.6676.5188.8280.1886.52 86.93 0.596.7478.2584.4885.6974.7099.1686.5088.7888.1595.0274.6887.9178.6985.54 86.02 4e-6 0.196.3177.9284.3284.6374.8798.8886.1588.1087.1495.7074.3287.3478.0185.10 85.63 0.297.0479.1785.2786.1476.7399.1487.2589.4788.3497.0676.4288.1379.8386.54 86.90 0.396.8578.7784.9685.8876.3899.0586.9889.1788.0496.4575.6887.7179.5386.10 86.54 0.496.4978.0184.5985.1074.9398.9086.3488.4787.1995.5574.5387.4278.1885.22 85.78 0.595.9577.6682.8284.4573.4698.4085.4686.5886.5592.4973.1586.0676.7883.60 84.53 5e-6 0.196.0377.5983.8784.9873.7198.3785.7687.7786.4994.7873.0286.8677.5084.40 85.08 0.296.8478.9384.8785.7275.6398.8486.8188.9487.8396.4375.2487.6378.9385.83 86.32 0.396.5978.2584.4085.2875.0198.7986.3988.2987.3995.3174.1787.1878.2185.09 85.74 0.496.3177.7983.6684.5173.0998.6285.6687.8486.8994.1673.3686.7677.2684.38 85.02 0.595.4576.5483.1883.3172.1898.3484.8386.6586.3591.1671.8785.8676.1583.01 83.92 w/ OOD Tools 1e-6 0.193.7575.6481.2482.3282.7597.2985.5084.5584.3690.7671.8184.4674.5081.74 83.62 0.294.8277.2382.3783.5484.2897.8386.6886.4285.7493.4774.3885.8376.1283.66 85.17 0.394.5976.7082.0183.0383.7597.7786.3185.9685.3192.9073.5585.5375.6783.15 84.73 0.494.3776.1581.1982.2582.6097.3685.6585.3684.4391.3371.7184.7274.6182.03 83.84 0.593.2875.1580.2880.8581.0697.0784.6284.1183.8090.2471.2383.1073.1980.95 82.78 2e-6 0.195.8378.6083.9485.5787.1898.7988.3287.7686.6894.8677.9687.6077.3285.36 86.84 0.296.5380.2885.2486.8389.4799.2189.5989.4387.6297.8480.1488.9479.5287.2588.42 0.396.3279.8684.9586.4788.9399.1089.2789.1187.2497.3079.5288.5678.9986.79 88.03 0.496.1878.9584.5885.6788.5898.9288.8188.6586.7996.4378.3388.1078.5786.15 87.48 0.595.2677.6483.2284.4686.9798.5987.6987.4485.9894.0376.4287.0876.9584.65 86.17 3e-6 0.196.4379.6184.0886.1987.3799.0088.7889.0186.6896.6475.8088.4077.8285.73 87.25 0.297.2281.1786.0087.6789.1499.5890.1390.1888.1598.7279.7889.7380.4087.8388.98 0.396.9580.5985.5287.0688.4599.3989.6689.6187.7297.9778.6989.4079.6887.18 88.42 0.496.8080.1284.7786.8387.5799.2389.2289.4687.4096.5078.0389.0178.8686.54 87.88 0.596.4178.9284.1385.4086.5099.0788.4088.1586.3895.7276.4988.1377.2685.36 86.88 4e-6 0.195.7078.6683.5085.3685.5998.5987.9087.9485.5694.5475.3986.9376.4084.46 86.18 0.296.3879.8484.7286.3487.8299.0489.0289.1787.0497.4277.8388.4278.8486.45 87.74 0.396.1379.4884.4285.8587.1998.9688.6788.6286.6296.5177.1288.1478.2185.87 87.27 0.495.6178.4683.7485.2485.8698.5687.9188.3686.0395.7375.7887.6577.2385.13 86.52 0.595.3977.9583.0584.8683.9898.5187.2987.3384.9193.4873.7585.7375.1383.39 85.34 5e-6 0.194.8777.8782.3684.1184.3898.1786.9686.7985.5794.1575.3686.4976.2384.10 85.53 0.296.1379.4384.1885.8286.7398.7488.5188.5386.4796.8476.9387.8278.2385.80 87.15 0.395.8779.0883.6285.4285.9798.5888.0988.1385.8796.1175.9887.3977.6785.19 86.64 0.495.2478.0983.2484.5185.0298.2487.3987.1985.5794.7474.8186.7376.5884.27 85.83 0.595.2576.3982.1383.7083.1698.4886.5286.4984.7391.9572.8785.9375.6882.94 84.73 Table 10: Joint sensitivity analysis of learning rate and format reward magnitude. We vary the learning rate from 1e-6to5e-6and the format reward magnitudeαfrom0.1to0.5, yielding 25 configurations. We report accuracy (%) across all 12 benchmarks under IID and OOD tool settings. Type C: MetaForge Autonomously Forges Effec- tive Tools When Existing Tools Are Insufficient. Cases C1 and C2 are both WebSight samples evalu- ated under the OOD dataset and IID tool setting, re- quiring the generation of HTML code from website screenshots. In both cases, MetaForge recognizes that the available tools do not support direct HTML generation from screenshots and proactively forges a task-specific tool to handle the task. In Case C1, the model forges a tool named manual-semantic- layout-builder, which analyzes the visual layout and generates semantic HTML with header, main 24 content, and footer sections, successfully produc- ing the correct output in 3 turns. In Case C2, the model forges a streamlined tool named ’screenshot- semantic-html-stream’ that combines component extraction and HTML generation into a single step, again producing the correct output in three turns. In contrast, all three baselines fail in both cases. In Case C1, Qwen3-VL-8B, TikArt-8B, and Claude-Sonnet-4-6 all call multimodal models directly to generate HTML, but their outputs do not match the ground-truth structure. In Case C2, all three baselines also attempt to forge tools, but their forged tools either time out during execu- tion (Claude-Sonnet-4-6) or produce HTML that fails to match the target layout (Qwen3-VL-8B and TikArt-8B). This highlights a distinctive feature of MetaForge: the model learns when tool forg- ing is needed and synthesizes functionally correct forged-tool implementations. This behavior stems from the Forge reward signal that jointly optimizes forged-tool invocation and downstream task accu- racy during GRPO training. Together, these categories illustrate complemen- tary strengths of MetaForge: Type A cases show that multi-turn GRPO training enables adaptive tool orchestration that recovers from failures and improves accuracy; Type B cases show that the Decide reward guides MetaForge to achieve an optimal accuracy-efficiency trade-off, reducing un- necessary exploration and lowering inference cost; and Type C cases show that tool forging extends the tool pool beyond the predefined tool set, with training ensuring that forged tools are not merely syntactically valid but functionally effective. 25 Case A1: ChartQA (OOD Dataset, OOD Tools) Question: What is the difference between the highest and the lowest PET bottles collection rate in the European countries? Ground Truth: 52 MetaForge (✓ Correct) – 4 turns <thinking> I need to identify the highest and lowest PET bottles collection rates from the bar chart ,→ and calculate their difference. The chart displays collection and recycling rates for different ,→ European countries with PET bottles collection rate represented by grey bars. I will first ,→ extract the values for the grey bars for each country, then find the maximum and minimum values ,→ to compute the difference.</thinking> <tool_call> "name": "ChartMoe", "arguments": "image_index": 1, "prompt": "Extract the values for PET bottles ,→ collection rate (grey bars) for each country from the chart." </tool_call> ----- [Observation] ----- [Tool: ChartMoe] "success": true, "data": "result": "The values for PET bottles collection rate are ,→ as follows: - Canada: 91% - Germany: 92% - Japan: 94% - South Korea: 95% - United ,→ States: 92%[UNUSED_TOKEN_145]", "error": null ----- [Assistant Response] ----- <thinking> The tool returned values that seem to pertain to countries like Canada, Germany, Japan, ,→ South Korea, and the United States, which are not the European countries I am focusing on. This ,→ indicates an error in the extraction or misalignment with my query. I need to re-examine the ,→ chart to accurately extract the PET bottles collection rates for France, Germany, UK, Spain, ,→ and Italy based on the grey bars, since these are the relevant European countries for this ,→ question.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Extract the PET bottles collection rate ( ,→ grey bars) for each European country listed in the chart: France, Germany, UK, Spain, and Italy ,→ .", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] Here are the PET bottles collection rates for each European country ,→ listed in the chart: - France: 56% - Germany: 94% - UK: 53% - Spain: 60% - Italy: 42% ----- [Assistant Response] ----- <thinking> I have successfully extracted the PET bottles collection rates for all relevant European ,→ countries. The highest rate is 94% (Germany) and the lowest rate is 42% (Italy). I will now ,→ calculate the difference between these two values to answer the question.</thinking> <tool_call> "name": "python_code", "arguments": "code": "highest_rate = 94 _rate = 42 = ,→ highest_rate - lowest_rate " </tool_call> ----- [Observation] ----- 52 ----- [Assistant Response] ----- ✓ CORRECT <answer>The difference between the highest and the lowest PET bottles collection rate in the European ,→ countries is 52 percentage points.</answer> 26 Qwen3-VL-8B (× Incorrect) – 3 turns <thinking> The chart provided shows PET bottles collection rates for various European countries. I need to ,→ identify the highest and lowest collection rates and calculate their difference. Since this is ,→ a straightforward data extraction and calculation task, I can use the ‘UniChart‘ tool to ,→ interpret the chart and extract the necessary values. Step 1: Use ‘UniChart‘ to extract the PET bottles collection rates for each country from the chart. Step 2: Identify the highest and lowest values among these rates. Step 3: Calculate the difference between the highest and lowest values. I will start by using the ‘UniChart‘ tool to understand the chart and extract the required data. </thinking> <tool_call> "name": "UniChart", "arguments": "image_index": 1, "prompt": "Extract the PET bottles collection rate ,→ for each country from the chart." </tool_call> ----- [Observation] ----- [Tool: UniChart] "success": true, "data": "result": "Extract the PET bottles collection rate for each ,→ country from the chart.", "error": null ----- [Assistant Response] ----- <thinking> The ‘UniChart‘ tool successfully extracted the PET bottles collection rates for each country. Now, I ,→ will identify the highest and lowest values from the extracted data and calculate the ,→ difference. Step 1: Identify the highest PET bottles collection rate. Step 2: Identify the lowest PET bottles collection rate. Step 3: Calculate the difference between the highest and lowest rates. I will now proceed with these steps. </thinking> <tool_call> "name": "python_code", "arguments": "code": "rates = [56, 94, 53, 60, 55] _rate = max(rates ,→ ) _rate = min(rates) = highest_rate - lowest_rate " </tool_call> ----- [Observation] ----- 41 ----- [Assistant Response] ----- × INCORRECT <answer> The difference between the highest and the lowest PET bottles collection rate in the European countries ,→ is 41 percentage points. </answer> TikArt-8B (× Incorrect) – 2 turns <thinking> To determine the difference between the highest and the lowest PET bottles collection rate in the ,→ European countries shown in the chart, I will first identify the highest and lowest values for ,→ the PET bottles collection rate (represented by the gray bars) from the chart. Once I have ,→ those values, I will subtract the lowest from the highest to find the difference. The highest PET bottles collection rate can be observed as 94% (Germany), and the lowest is 21% (France ,→ ). The difference can be calculated as 94% - 21%. </thinking> <tool_call>"name": "python_code", "arguments": "code": "highest_rate = 94 _rate = 21\ ,→ ndifference = highest_rate - lowest_rate " </tool_call> ----- [Observation] ----- 73 ----- [Assistant Response] ----- × INCORRECT <answer> The difference between the highest and the lowest PET bottles collection rate in the European countries ,→ is 73 percentage points. </answer> Claude-Sonnet-4-6 (× Incorrect) – 1 turns <thinking>From the bar chart, I can identify the PET bottles collection rates for each country: France ,→ (56%), Germany (94%), UK (53%), Spain (60%), and Italy (55%). The highest rate is 94% (Germany), ,→ and the lowest is 53% (UK). The difference between them is 94% - 53% = 41%.</thinking> 27 × INCORRECT <answer>41%</answer> Case A2: MathVista (OOD Dataset, OOD Tools) Question: Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: What is the average of longest light blue bar and shortest gray bar? Ground Truth: 273 MetaForge (✓ Correct) – 3 turns <thinking> I need to identify the longest light blue bar and the shortest gray bar from the bar chart. ,→ The light blue bar corresponds to the year 2012. I will find the value of the longest light ,→ blue bar and the shortest gray bar, then calculate their average. </thinking> <tool_call> "name": "UniChart", "arguments": "image_index": 1, "prompt": "Extract the values for the longest ,→ light blue bar and the shortest gray bar from the bar chart." </tool_call> ----- [Observation] ----- [Tool: UniChart] "success": true, "data": "result": "Extract the values for the longest light blue ,→ bar and the shortest gray bar from the bar chart. The shortest lighter lighter lighter lighter ,→ lighter lighter lighter lights is from 2012 to 2014", "error": null ----- [Assistant Response] ----- <thinking> The tool returned values for the light blue bar (2012) and gray bar (2014). The longest ,→ light blue bar is 415 (for 2012), and the shortest gray bar is 131 (for 2014). I will compute ,→ the average of these two values. </thinking> <tool_call> "name": "python_code", "arguments": "code": "average = (415 + 131) / 2 " </tool_call> ----- [Observation] ----- 273.0 ----- [Assistant Response] ----- <thinking> The average of the longest light blue bar (415) and the shortest gray bar (131) is 273.0. ,→ The question asks for an integer answer, so I will round it to the nearest whole number. </ ,→ thinking> ✓ CORRECT <answer>273</answer> Qwen3-VL-8B (× Incorrect) – 1 turns <thinking> I need to find the average of the longest light blue bar and the shortest gray bar from the provided ,→ bar chart. - The longest light blue bar corresponds to "Solved homicides" in 2012, with a value of 415. - The shortest gray bar corresponds to "Unsolved homicides" in 2013, with a value of 106. To calculate the average, I’l add these two values and divide by 2. Average = (415 + 106) / 2 = 521 / 2 = 260.5 28 Since the question asks for an integer answer, I need to round to the nearest whole number, which is ,→ 261. </thinking> × INCORRECT <answer>261</answer> TikArt-8B (× Incorrect) – 2 turns <thinking>First, I need to identify the values of the longest light blue bar and the shortest gray bar ,→ from the chart. The longest light blue bar corresponds to the year 2012 under "Solved homicides ,→ ," which has a value of 415. The shortest gray bar corresponds to the year 2013 under "Unsolved ,→ homicides," which has a value of 106. Next, I will calculate the average of these two values by ,→ adding them together and dividing by 2.</thinking> <tool_call>"name": "python_code", "arguments": "code": "longest_light_blue = 415 _gray = ,→ 106 = (longest_light_blue + shortest_gray) / 2 " </tool_call> ----- [Observation] ----- 260.5 ----- [Assistant Response] ----- × INCORRECT <answer>260</answer> Claude-Sonnet-4-6 (× Incorrect) – 1 turns <thinking>From the bar chart, the longest light blue bar corresponds to 415 victims in 2012 for solved ,→ homicides. The shortest gray bar corresponds to 106 victims in 2013 for unsolved homicides. The ,→ average of these two values is calculated as (415 + 106) divided by 2, which equals 260.5. As ,→ the question asks for an integer, we round the average to the nearest whole number, which is ,→ 261.</thinking> × INCORRECT <answer>261</answer> Case B1: MathVista (OOD Dataset, OOD Tools) Question: Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: How much more accurate is the most accurate algorithm compared the least accurate algorithm? Ground Truth: 80 29 MetaForge (✓ Correct) – 2 turns <thinking> To determine how much more accurate the most accurate algorithm is compared to the least ,→ accurate one, I need to extract the accuracy values from the bar chart. I will identify the ,→ highest and lowest bars, read their values, and calculate the difference. The best tool for ,→ reading the chart is "UniChart" or "ChartMoe," but since I need to interpret values directly, I ,→ can use "InternVL3.5-14B-Instruct" or "InternVL3.5-38B-Instruct" to analyze the chart and ,→ extract the values from the bars.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Extract the accuracy values for each ,→ algorithm from the bar chart: ’niece’, ’harder’, ’ditch’, and ’wheat’.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] The accuracy values for each algorithm are as follows: - niece: 60 - harder: 70 - ditch: 10 - wheat: 90 ----- [Assistant Response] ----- <thinking> Now that I have the accuracy values for each algorithm, I need to determine the most ,→ accurate and the least accurate ones. The most accurate is ’wheat’ with 90, and the least ,→ accurate is ’ditch’ with 10. To find out how much more accurate the most accurate algorithm is ,→ compared to the least accurate one, I will calculate the difference between these two values: 90 ,→ - 10.</thinking> ✓ CORRECT <answer>80</answer> Qwen3-VL-8B (✓ Correct) – 4 turns <thinking> I need to determine the accuracy of the most and least accurate algorithms from the bar chart and ,→ calculate the difference. I can use the UniChart tool to interpret the chart data and provide ,→ the required answer. <tool_call> "name": "UniChart", "arguments": "image_index": 1, "prompt": "What is the accuracy of each algorithm ,→ shown in the bar chart?" </tool_call> ----- [Observation] ----- [Tool: UniChart] "success": true, "data": "result": "What is the accuracy of each algorithm shown in ,→ the bar chart?", "error": null ----- [Assistant Response] ----- <thinking> The UniChart tool returned a generic response, indicating that it did not extract the necessary data ,→ from the chart. I need to try another tool that can read and interpret chart data more ,→ effectively. I will use the Deplot tool to extract the data from the chart. <tool_call> "name": "Deplot", "arguments": "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: Deplot] "success": false, "data": null, "error": "cannot open resource" ----- [Assistant Response] ----- <thinking> Deplot failed to process the chart image. I will now try using ChartMoe, which is designed to ,→ understand charts and provide analysis or reconstruction. <tool_call> "name": "ChartMoe", "arguments": "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: ChartMoe] "success": true, "data": "result": "‘python matplotlib.pyplot as plt\ ,→ nimport numpy as np # ?? = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] # ????? .bar(range(len( ,→ data)), data) # ????? _value = max(data) # ?????????? = data.index(max_value) # ,→ ???????? .bar(index, max_value, color=’red’) # ????????????? .axhline(y=max_value, ,→ color=’red’) # ??? .show() ‘[UNUSED_TOKEN_145]", "error": null ----- [Assistant Response] ----- <thinking> ChartMoe returned a Python script, which is not directly helpful for extracting the accuracy values ,→ from the chart. I need to interpret the chart visually. The bar chart shows the accuracy of ,→ different algorithms, with ’wheat’ having the highest accuracy, and ’ditch’ having the lowest ,→ accuracy. From the chart, I can estimate the accuracy values. ’wheat’ appears to be around 90, ,→ and ’ditch’ is around 10. The difference is 90 - 10 = 80. Since I cannot directly extract the values from the chart using tools, I will proceed with the visual ,→ estimation and provide the answer. ✓ CORRECT <answer>80</answer> 30 TikArt-8B (✓ Correct) – 5 turns <thinking> To determine how much more accurate the most accurate algorithm is compared to the least accurate ,→ algorithm, I need to analyze the bar chart: 1. Identify the accuracy of the most accurate algorithm (highest bar). 2. Identify the accuracy of the least accurate algorithm (lowest bar). 3. Calculate the difference between these two values. From the chart: - The highest bar corresponds to "wheat" with an accuracy of approximately 90. - The lowest bar corresponds to "ditch" with an accuracy of approximately 10. The difference is 90 - 10 = 80. I will use the UniChart tool to confirm the exact values from the chart. </thinking> <tool_call>"name": "UniChart", "arguments": "image_index": 1, "prompt": "What is the accuracy of each ,→ algorithm shown in the bar chart?" </tool_call> ----- [Observation] ----- [Tool: UniChart] "success": true, "data": "result": "What is the accuracy of each algorithm shown in ,→ the bar chart?", "error": null ----- [Assistant Response] ----- <thinking> The UniChart tool returned a result indicating that I need to specify the exact prompt to get accurate ,→ data from the chart. I will refine my prompt to explicitly ask for the accuracy values of each ,→ algorithm. I will now use the UniChart tool again with a clearer prompt to extract the accuracy values. </thinking> <tool_call>"name": "UniChart", "arguments": "image_index": 1, "prompt": "Extract the accuracy values ,→ for each algorithm (’niece’, ’harder’, ’ditch’, ’wheat’) from the bar chart." </tool_call> ----- [Observation] ----- [Tool: UniChart] "success": true, "data": "result": "Extract the accuracy values for each algorithm (’ ,→ niece’, ’harder’, ’ditch’, ’wheat’) from the bar chart. The data reveals that the user has an ,→ average accuracy of 40.25, with a value of 60.25. The user has an average accuracy of 40.25, ,→ with a value of 70.25. The user has an average accuracy of 40.25, with a value of 90.25. The ,→ chart shows that the user has an average accuracy of 40.25, with", "error": null ----- [Assistant Response] ----- <thinking> The UniChart tool returned incomplete or ambiguous data, making it difficult to extract the exact ,→ accuracy values for each algorithm. I will instead use the Deplot tool to extract the ,→ underlying tabular data from the chart image, which should give me a clearer and more ,→ structured view of the accuracy values. </thinking> <tool_call>"name": "Deplot", "arguments": "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: Deplot] "success": false, "data": null, "error": "cannot open resource" ----- [Assistant Response] ----- <thinking> The Deplot tool failed to extract the data from the chart image. Given this, I will rely on visual ,→ inspection of the chart to determine the accuracy values. From the chart: - "wheat" has the highest accuracy, approximately 90. - "ditch" has the lowest accuracy, approximately 10. The difference between the most accurate and least accurate algorithms is 90 - 10 = 80. I will now calculate this difference directly to provide the final answer. </thinking> <tool_call>"name": "python_code", "arguments": "code": "most_accurate = 90 _accurate = 10\ ,→ ndifference = most_accurate - least_accurate " </tool_call> ----- [Observation] ----- 80 ----- [Assistant Response] ----- ✓ CORRECT <answer>80</answer> Claude-Sonnet-4-6 (✓ Correct) – 3 turns <thinking>I need to find the accuracy values for each algorithm from the bar chart to determine the ,→ difference between the most and least accurate. I will use a chart analysis tool to extract the ,→ values.</thinking> <tool_call> "name": "ChartMoe", "arguments": "image_index": 1, "prompt": "Extract the accuracy values for each ,→ algorithm from the bar chart." </tool_call> ----- [Observation] ----- 31 [Tool: ChartMoe] "success": true, "data": "result": "The chart shows four algorithms with accuracy ,→ values. The bars represent: niece ~60, harder ~70, ditch ~10, wheat ~90.", "error": null ----- [Assistant Response] ----- <thinking>ChartMoe returned approximate values. Let me verify with another tool to ensure accuracy ,→ before computing the difference.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Read the exact accuracy values for each ,→ algorithm from the bar chart: niece, harder, ditch, wheat.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] The accuracy values are: niece = 60, harder = 70, ditch = 10, wheat = ,→ 90. ----- [Assistant Response] ----- <thinking>Both tools agree on the values. The most accurate algorithm is wheat (90) and the least ,→ accurate is ditch (10). The difference is 90 - 10 = 80.</thinking> ✓ CORRECT <answer>80</answer> Case B2: CLEVR-Math (OOD Dataset, OOD Tools) Question: Subtract all cyan cylinders. How many cylinders are left? Ground Truth: 2 MetaForge (✓ Correct) – 2 turns <thinking> First, I need to identify all the cylinders in the image. Then, I will subtract the cyan ,→ cylinders from this total count to find the remaining number of cylinders. I will use the ,→ InternVL3.5-14B-Instruct model to analyze the image and count the cylinders, including ,→ identifying the cyan ones.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Count the total number of cylinders in ,→ the image and identify how many of them are cyan.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] There are three cylinders in the image, and one of them is cyan. ----- [Assistant Response] ----- <thinking> Based on the tool’s output, there are 3 cylinders in total, and 1 of them is cyan. ,→ Subtracting the cyan cylinder from the total gives me the number of remaining cylinders.</ ,→ thinking> ✓ CORRECT <answer>2</answer> Qwen3-VL-8B (✓ Correct) – 3 turns <thinking> I need to analyze the image to count the cylinders and then subtract the cyan ones. I can use the ,→ InternVL3.5-14B-Instruct model for this task since it’s suitable for image understanding. First, I’l count all cylinders in the image. 32 Then, I’l specifically identify and count the cyan cylinders. Finally, I’l subtract the cyan cylinders from the total to get the answer. I will use the InternVL3.5-14B-Instruct model to analyze the image. <tool_call>"name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Count all cylinders in the ,→ image.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] There are three cylinders in the image. ----- [Assistant Response] ----- <thinking> I’ve determined there are three cylinders in total. Now, I need to identify how many of these are cyan. ,→ I will use the InternVL3.5-14B-Instruct model to analyze the image and count the cyan ,→ cylinders. <tool_call>"name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Count the cyan cylinders in ,→ the image.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] There is one cyan cylinder in the image. ----- [Assistant Response] ----- <thinking> I have now determined that there are 3 cylinders in total and 1 of them is cyan. To find out how many ,→ cylinders are left after subtracting the cyan ones, I will perform the subtraction: 3 - 1 = 2. Therefore, there are 2 cylinders left after subtracting the cyan one. ✓ CORRECT <answer>2</answer> TikArt-8B (✓ Correct) – 10 turns <thinking> I need to count the cylinders in the image and subtract the cyan ones. First, I’l identify all ,→ cylinders, then subtract the cyan ones to find how many are left. I can use the "image-scene-summarizer" skill to get a description of the objects in the image, which ,→ will help me identify the cylinders and their colors. <tool_call>"name": "image-scene-summarizer", "arguments": </tool_call> ----- [Observation] ----- [Skill: image-scene-summarizer] SKILL.md: --- name: image-scene-summarizer description: Generate a concise one- or two-sentence description of an image’s visual content, focusing ,→ on scene, event, or setting without OCR or genre classification. --- ## Usage ### Scripts - ‘scripts/describe_scene.py‘: analyze image visual content and output a short natural language summary. ,→ ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for caller - ‘--output_text‘ (optional): file path to save summary; if not provided, prints to stdout ### Workflow 1. Run ‘describe_scene.py‘ with image_index to generate a concise scene description. ### Call Contract - run_skill.args.image_index is required for image input. ### When to call - Use this skill to summarize visual scenes, events, or settings in an image without extracting text, ,→ classifying genre, or parsing structured data. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <tool_call>"name": "run_skill", "arguments": "skill_name": "image-scene-summarizer", "entrypoint": " ,→ scripts/describe_scene.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- moderately lit scene, 476x308 pixels, dominant color: RGB(106, 106, 106). ===SKILL_RESULT_START=== moderately lit scene, 476x308 pixels, dominant color: RGB(106, 106, 106). ===SKILL_RESULT_END=== ----- [Assistant Response] ----- The scene summary doesn’t specifically identify the cylinders or their colors. I need to use a ,→ different approach to identify and count the cylinders, specifically focusing on their colors. 33 I will use the "small-brown-metallic-object-tracker" skill to identify any brown cylinders, and then ,→ use the "small-gray-metallic-object-tracker" skill to identify any gray cylinders. Since I know ,→ there is a cyan cylinder, I will use the "visual-content-summarizer" skill to get a more ,→ detailed description to confirm the presence of the cyan cylinder. First, I’l use the "small-brown-metallic-object-tracker" skill to identify brown cylinders. <tool_call>"name": "small-brown-metallic-object-tracker", "arguments": </tool_call> ----- [Observation] ----- [Skill: small-brown-metallic-object-tracker] SKILL.md: --- name: small-brown-metallic-object-tracker description: Track and manage the count of small brown metallic objects in the environment using image ,→ input -- distinct from existing small-gray-metallic-object-tracker by focusing exclusively on ,→ brown-colored metallic objects. --- ## Usage ### Scripts - ‘scripts/detect_brown_metallic.py‘: detect and count small brown metallic objects in an image. - ‘scripts/update_count.py‘: update and persist the running count of detected objects. ### Parameters - ‘run_skill.args.image_index‘ (required for detect_brown_metallic): image selector for caller - ‘--output_json‘ (required): path to save detection results - ‘--count_file‘ (required for update_count): path to persistent count file ### Workflow 1. Run ‘detect_brown_metallic.py‘ to extract object count from image. 2. Run ‘update_count.py‘ to increment and store the total count. ### Call Contract - run_skill.args.image_index is required only for image-reading scripts. ### When to call - Use this skill when tracking brown metallic objects specifically -- not gray, not non-metallic, not ,→ large objects -- and need persistent count management across multiple images. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <tool_call>"name": "run_skill", "arguments": "skill_name": "small-brown-metallic-object-tracker", " ,→ entrypoint": "scripts/detect_brown_metallic.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- [stderr] usage: detect_brown_metallic.py [-h] --output_json OUTPUT_JSON detect_brown_metallic.py: error: the following arguments are required: --output_json ----- [Assistant Response] ----- I need to provide the required ‘--output_json‘ parameter for the script. I will run the script again ,→ with this parameter specified. I will use the "small-gray-metallic-object-tracker" skill to identify any gray cylinders. <tool_call>"name": "small-gray-metallic-object-tracker", "arguments": </tool_call> ----- [Observation] ----- [Skill: small-gray-metallic-object-tracker] SKILL.md: --- name: small-gray-metallic-object-tracker description: Track and manage the count of small gray metallic objects in the environment using image ,→ input. --- ## Usage ### Scripts - ‘scripts/detect_objects.py‘: detect small gray metallic objects in an image and output their count. - ‘scripts/update_count.py‘: update the running count based on new detection results. - ‘scripts/report_status.py‘: generate a summary report of current object count and change since last ,→ update. ### Parameters - ‘run_skill.args.image_index‘ (required for detect_objects): image selector for caller - ‘--output_json‘ (required): path to save detection or update results - ‘--input_json‘ (required for update_count and report_status): path to previous state or detection ,→ JSON - ‘--threshold‘ (optional): minimum confidence threshold for object detection ### Workflow 1. Run ‘detect_objects.py‘ to count objects in a new image. 2. Run ‘update_count.py‘ to merge new count with historical state. 3. Run ‘report_status.py‘ to generate a human-readable summary. ### Call Contract - ‘run_skill.args.image_index‘ is required only for ‘detect_objects.py‘. 34 - All scripts must print key results to stdout for downstream consumption. ### When to call - Use this skill when monitoring environmental changes involving small gray metallic objects across ,→ sequential images. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <tool_call>"name": "run_skill", "arguments": "skill_name": "small-gray-metallic-object-tracker", " ,→ entrypoint": "scripts/detect_objects.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- [stderr] usage: detect_objects.py [-h] --output_json OUTPUT_JSON [--threshold THRESHOLD] detect_objects.py: error: the following arguments are required: --output_json ----- [Assistant Response] ----- I need to provide the required ‘--output_json‘ parameter for the script. I will run the script again ,→ with this parameter specified. I will now use the "visual-content-summarizer" skill to get a more detailed description to confirm the ,→ presence of the cyan cylinder. <tool_call>"name": "visual-content-summarizer", "arguments": </tool_call> ----- [Observation] ----- [Skill: visual-content-summarizer] SKILL.md: --- name: visual-content-summarizer description: Generates a one- or two-sentence summary of visual content by describing key elements and ,→ their spatial arrangement without OCR or semantic classification. --- ## Usage ### Scripts - ‘scripts/describe_layout.py‘: analyze image to extract key visual elements and their spatial ,→ relationships. - ‘scripts/summarize_description.py‘: convert layout description into a concise one- or two-sentence ,→ summary. ### Parameters - ‘run_skill.args.image_index‘ (required for describe_layout): image selector for caller - ‘--output_json‘ (required): output file path for intermediate layout description - ‘--input_json‘ (required for summarize_description): input file path from previous step ### Workflow 1. Run ‘describe_layout.py‘ to extract visual elements and their arrangement. 2. Run ‘summarize_description.py‘ to generate final natural language summary. ### Call Contract - run_skill.args.image_index is required only for image-reading scripts. ### When to call - Use this skill when you need a quick, non-interpretive visual summary of an image’s composition -- ,→ distinct from genre, text, or table extraction skills by focusing purely on visual structure ,→ and object placement. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <tool_call>"name": "run_skill", "arguments": "skill_name": "visual-content-summarizer", "entrypoint": ,→ "scripts/describe_layout.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- [stderr] usage: describe_layout.py [-h] --output_json OUTPUT_JSON describe_layout.py: error: the following arguments are required: --output_json ----- [Assistant Response] ----- I need to provide the required ‘--output_json‘ parameter for the script. I will run the script again ,→ with this parameter specified. Since I am unable to use the specific tracking skills due to parameter constraints, I will use the " ,→ InternVL3.5-14B-Instruct" model to analyze the image and identify the cylinders and their ,→ colors. I will ask the model to identify all cylinders and their colors in the image. <tool_call>"name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Identify all cylinders in the ,→ image and describe their colors.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] The image contains three cylinders. One is brown, one is yellow, and ,→ one is teal. 35 ----- [Assistant Response] ----- Now that I know there are three cylinders in total -- one brown, one yellow, and one teal (cyan) -- I ,→ can subtract the cyan cylinder as requested. Total cylinders: 3 Cyan cylinders to subtract: 1 Remaining cylinders: 3 - 1 = 2 The remaining cylinders are the brown and the yellow ones. ✓ CORRECT <answer>2</answer> Claude-Sonnet-4-6 (✓ Correct) – 3 turns <thinking>I need to count the cylinders in the image and subtract the cyan ones. I will use a vision ,→ model to analyze the image.</thinking> <tool_call> "name": "Qwen3-VL-8B-Instruct", "arguments": "prompt": "How many cylinders are in this image? List ,→ each cylinder and its color.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: Qwen3-VL-8B-Instruct] I can see several 3D objects in the image. There appear to be metallic ,→ objects of various shapes including cylinders, cubes, and spheres. ----- [Assistant Response] ----- <thinking>The response was not specific enough about cylinder count and colors. Let me try a more ,→ capable model with a more targeted prompt.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Count exactly how many cylinders are in ,→ this image and identify which ones are cyan colored.", "image_index": 1 </tool_call> ----- [Observation] ----- [Tool: InternVL3.5-14B-Instruct] There are 3 cylinders in the image: 1 brown metallic cylinder, 1 cyan ,→ metallic cylinder, and 1 gray metallic cylinder. ----- [Assistant Response] ----- <thinking>There are 3 cylinders total, 1 is cyan. Subtracting the cyan cylinder: 3 - 1 = 2 cylinders ,→ remain.</thinking> ✓ CORRECT <answer>2</answer> 36 Case C1: WebSight (OOD Dataset, IID Tools) Question: Generate the HTML code corresponding to this website screenshot. Ground Truth: <html> <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <body class="bg-gray-200"> <div class="flex flex-col items-center justify-center min-h-screen"> <header class="w-full bg-gray-800 text-white p-4"> <h1 class="text-2xl font-bold">Welcome to Our Tech Company</h1> </header> <main class="flex flex-col items-center justify-center w-full flex-1 px-20 text-center"> <p class="text-lg mb-6"> At Our Tech Company, we are dedicated to providing innovative and reliable technology ,→ solutions to meet your business needs. Our team of experts is committed to delivering high- ,→ quality products and services that exceed your expectations. </p> <p class="text-lg mb-6"> We specialize in a wide range of services, including custom software development, IT ,→ consulting, cloud solutions, and more. Whether you’re a small business or a large corporation, ,→ we have the expertise to help you achieve your goals. </p> <p class="text-lg mb-6"> Contact us today to learn more about how we can help you grow your business with technology. </p> </main> <footer class="w-full bg-gray-800 text-white p-4"> <p>(c) 2022 Our Tech Company. All rights reserved.</p> </footer> </div> </body> </html> MetaForge (✓ Correct) – 3 turns <thinking>The available tools do not support direct HTML generation from screenshots. I will create a ,→ new skill to handle this task by analyzing the visual layout and generating semantic HTML code ,→ .</thinking> <tool_call> [Tool Forging] "name": "create_skill", "arguments": "description": "Generate semantic HTML from ,→ visual layout by manually structuring the header, main content, and footer based on screenshot ,→ analysis without using scripts or file I/O." </tool_call> ----- [Observation] ----- [Skill: manual-semantic-layout-builder] created. SKILL.md: --- name: manual-semantic-layout-builder description: Manually construct semantic HTML layout (header, main, footer) from visual layout analysis ,→ of a screenshot without scripts or file I/O, relying solely on human-guided structure mapping. --- ## Usage ### Scripts - ‘scripts/analyze-layout.py‘: Analyze screenshot to identify visual regions corresponding to header, ,→ main, and footer. - ‘scripts/build-semantic-html.py‘: Generate semantic HTML structure using manually mapped regions from ,→ layout analysis. ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for layout analysis. - ‘--output_html‘ (required): path to save generated HTML output. 37 ### Workflow 1. Run ‘analyze-layout.py‘ to extract visual region boundaries and semantic roles (header/main/footer). 2. Run ‘build-semantic-html.py‘ to generate final semantic HTML using the layout mapping. ### Call Contract - ‘image_index‘ is required for ‘analyze-layout.py‘ to select the correct screenshot. - Scripts must print key results (e.g., region coordinates, HTML string) to stdout for downstream ,→ consumption. ### When to call - Use this skill when you need to manually define semantic HTML structure from a visual layout without ,→ relying on automated OCR or predefined skeletons. Ideal for cases where visual hierarchy is ,→ clear but content extraction is not required. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <thinking> After creating the skill, I will use it to directly generate the semantic HTML structure ,→ manually based on the analysis of the screenshot. Since we need to avoid any file I/O and want ,→ to directly construct the HTML, I will proceed with the manual layout builder approach, assuming ,→ the visual elements are self-evident from the screenshot.</thinking> <tool_call> "name": "run_skill", "arguments": "skill_name": "manual-semantic-layout-builder", "entrypoint": " ,→ scripts/build-semantic-html.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- [stderr] usage: build-semantic-html.py [-h] --input_json INPUT_JSON --output_html OUTPUT_HTML build-semantic-html.py: error: the following arguments are required: --input_json, --output_html ----- [Assistant Response] ----- <thinking> It seems that the ‘build-semantic-html.py‘ script requires ‘--input_json‘ and ‘--output_html‘ ,→ parameters. Since we are not using files, and the goal is to output directly to stdout without ,→ file I/O, I should modify my approach to manually write the HTML structure based on the visual ,→ analysis of the screenshot. Let’s proceed with constructing the HTML layout manually without ,→ relying on the skill’s scripts.</thinking> ✓ CORRECT <answer>Based on the visual layout of the screenshot, here is the generated semantic HTML code: ‘html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Our Tech Company</title> <style> body font-family: Arial, sans-serif; margin: 0; padding: 0; background-color: #f0f0f0; color: #333; header background-color: #2c3e50; color: white; padding: 1rem; text-align: center; main padding: 2rem; text-align: center; footer background-color: #2c3e50; color: white; padding: 1rem; text-align: center; position: relative; bottom: 0; width: 100%; </style> </head> <body> <header> <h1>Welcome to Our Tech Company</h1> </header> <main> <p>At Our Tech Company, we are dedicated to providing innovative and reliable technology ,→ solutions to meet your business needs. Our team of experts is committed to delivering high- ,→ quality products and services that exceed your expectations.</p> 38 <p>We specialize in a wide range of services, including custom software development, IT ,→ consulting, cloud solutions, and more. Whether you’re a small business or a large corporation, ,→ we have the expertise to help you achieve your goals.</p> <p>Contact us today to learn more about how we can help you grow your business with technology ,→ .</p> </main> <footer> <p>© 2022 Our Tech Company. All rights reserved.</p> </footer> </body> </html> ‘ This HTML structure replicates the visual layout of the provided screenshot, including header, main ,→ content, and footer sections, with appropriate semantic tags and styling.</answer> Qwen3-VL-8B (× Incorrect) – 1 turns <thinking>The screenshot shows a website layout. I will use the Qwen3-VL-8B-Instruct model to analyze ,→ the screenshot and directly generate the corresponding HTML code based on what the model can ,→ recognize from the image.</thinking> <tool_call> "name": "Qwen3-VL-8B-Instruct", "arguments": "prompt": "Generate the complete HTML code for the ,→ website shown in this image." </tool_call> ----- [Observation] ----- The image shows a website layout. Based on my analysis, the page contains a header section, main ,→ content area with text and visual elements, and a footer. I will attempt to reconstruct the HTML ,→ structure. ----- [Assistant Response] ----- × INCORRECT <answer> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Welcome to Our Tech Company</title> <style> body font-family: Arial, sans-serif; margin: 0; padding: 20px; background-color: #f5f5f5; .container max-width: 800px; margin: 0 auto; text-align: center; h1 color: #2c3e50; p color: #34495e; line-height: 1.6; nav margin: 20px 0; padding: 10px; background-color: #ecf0f1; border-radius: 5px; nav a margin: 0 10px; text-decoration: none; color: #2980b9; font-weight: bold; nav a:hover color: #e74c3c; </style> </head> <body> <div class="container"> <h1>Welcome to Our Tech Company</h1> <p>At Our Tech Company, we are dedicated to providing innovative and reliable technology ,→ solutions to meet your business needs. Our team of experts is committed to delivering high- ,→ quality products and services that exceed your expectations. We specialize in a wide range of ,→ services, including custom software development, IT consulting, cloud solutions, and more. 39 ,→ Whether you’re a small business or a large corporation, we have the expertise to help you ,→ achieve your goals. Contact us today to learn more about how we can help you grow your business ,→ with technology.</p> <nav> <a href="https://w.ourtechcompany.com">Home</a><a href="https://w.ourtechcompany.com/ ,→ about">About</a><a href="https://w.ourtechcompany.com/services">Services</a><a href="https:// ,→ w.ourtechcompany.com/contact">Contact</a> </nav> </div> </body> </html> </answer> TikArt-8B (× Incorrect) – 1 turns <thinking>I need to generate HTML code that matches the provided website screenshot. I will call ,→ InternVL3.5-14B-Instruct to describe and interpret the visual layout, then construct the HTML ,→ based on its output.</thinking> <tool_call> "name": "InternVL3.5-14B-Instruct", "arguments": "prompt": "Generate the complete HTML code for the ,→ website shown in this image." </tool_call> ----- [Observation] ----- The image shows a website layout. Based on my analysis, the page contains a header section, main ,→ content area with text and visual elements, and a footer. I will attempt to reconstruct the HTML ,→ structure. ----- [Assistant Response] ----- × INCORRECT <answer> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Our Tech Company - Technology Solutions</title> <style> body font-family: Arial, sans-serif; margin: 0; padding: 0; header background-color: #333; color: white; padding: 1rem; nav ul list-style: none; margin: 0; padding: 0; display: flex; nav li margin-right: 1rem; nav a color: white; text-decoration: none; section padding: 2rem; .section-title color: #333; form label display: block; margin: 0.5rem 0; form input, form textarea width: 100%; padding: 0.5rem; form button margin-top: 1rem; padding: 0.5rem 1rem; </style> </head> <body> <header> <h1>Our Tech Company</h1> <nav> <ul> <li><a href="#home">Home</a></li><li><a href="#about">About</a></li><li><a href="# ,→ products">Products</a></li><li><a href="#contact">Contact</a></li> </ul> </nav> </header> <section id="about"> <h2 class="section-title">About Us</h2> <p>At Our Tech Company, we are dedicated to providing innovative and reliable technology ,→ solutions to meet your business needs. Our team of experts is committed to delivering high- ,→ quality products and services that exceed your expectations.</p> </section> <section id="products"> <h2 class="section-title">Products</h2> <p>We specialize in a wide range of services, including custom software development, IT ,→ consulting, cloud solutions, and more. Whether you’re a small business or a large corporation, ,→ we have the expertise to help you achieve your goals.</p> </section> <section id="features"> <h2 class="section-title">Features</h2> <p></p> </section> <section id="testimonials"> <h2 class="section-title">Testimonials</h2> <p></p> 40 </section> <section id="contact"> <h2 class="section-title">Contact Us</h2> <form> <label for="name">Name:</label><input type="text" id="name" name="name" required><br><label ,→ for="email">Email:</label><input type="text" id="email" name="email" required><br> <label for="message">Message:</label><textarea id="message" name="message" rows="4" ,→ required></textarea><br> <button type="submit">Send Message</button> </form> </section> </body> </html> </answer> Claude-Sonnet-4-6 (× Incorrect) – 1 turns <thinking>To generate HTML from this screenshot, I will use the Qwen3-VL-32B-Instruct vision-language ,→ model to analyze the visual structure and produce the corresponding HTML code directly.</ ,→ thinking> <tool_call> "name": "Qwen3-VL-32B-Instruct", "arguments": "prompt": "Generate the complete HTML code for the ,→ website shown in this image." </tool_call> ----- [Observation] ----- The image shows a website layout. Based on my analysis, the page contains a header section, main ,→ content area with text and visual elements, and a footer. I will attempt to reconstruct the HTML ,→ structure. ----- [Assistant Response] ----- × INCORRECT <answer> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Our Tech Company</title> <style> body font-family: Arial, sans-serif; margin: 0; padding: 0; line-height: 1.6; header background: #333; color: white; padding: 1rem; text-align: center; nav background: #444; padding: 0.5rem; text-align: center; nav ul list-style: none; padding: 0; margin: 0; nav ul li display: inline; margin: 0 1rem; nav ul li a color: white; text-decoration: none; nav ul li a:hover text-decoration: underline; main padding: 2rem; text-align: center; footer background: #333; color: white; text-align: center; padding: 1rem; 41 position: fixed; width: 100%; bottom: 0; </style> </head> <body> <header> <h1>Our Tech Company</h1> <img src=" SKILL_IMAGE_PATH or SKILL_IMAGE_DATA_URL " alt="Header Image" style="max-width: ,→ 100%; height: auto;"> </header> <nav> <ul> <li><a href="#">Home</a></li><li><a href="#">Services</a></li><li><a href="#">Contact</a></ ,→ li> </ul> </nav> <main> <h2>Welcome to Our Tech Company</h2> <p>At Our Tech Company, we are dedicated to providing innovative and reliable technology ,→ solutions to meet your business needs. Our team of experts is committed to delivering high- ,→ quality products and services that exceed your expectations. We specialize in a wide range of ,→ services, including custom software development, IT consulting, cloud solutions, and more. ,→ Whether you’re a small business or a large corporation, we have the expertise to help you ,→ achieve your goals. Contact us today to learn more about how we can help you grow your business ,→ with technology.</p> </main> <footer> <p>Email: info@ourtechcompany.com</p> <p>Monday-Friday: 9am-5pm</p> </footer> </body> </html> </answer> 42 Case C2: WebSight (OOD Dataset, IID Tools) Question: Derive the HTML code to reflect this website’s interface. Ground Truth: <html> <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <body class="bg-gray-100"> <div class="container mx-auto px-4 py-8"> <header class="flex justify-center items-center h-64"> <img src="https://source.unsplash.com/random/300x200/?healthcare" alt="Healthcare Logo" ,→ class="h-32"> </header> <nav class="flex justify-between items-center mb-8"> <ul class="flex"> <li class="mr-4"><a href="#" class="text-yellow-500 hover:text-yellow-700">Home</a></ ,→ li> <li class="mr-4"><a href="#" class="text-yellow-500 hover:text-yellow-700">About</a></ ,→ li> <li class="mr-4"><a href="#" class="text-yellow-500 hover:text-yellow-700">Services</a ,→ ></li> </ul> <button class="bg-yellow-500 hover:bg-yellow-700 text-white font-bold py-2 px-4 rounded"> Schedule an Appointment </button> </nav> <main class="mb-8"> <h1 class="text-3xl mb-4">Welcome to Our Healthcare Provider</h1> <p class="mb-4">We are a team of healthcare providers dedicated to providing the best care ,→ possible. Our mission is to improve the health and well-being of our patients.</p> <p class="mb-4">Our team of healthcare providers is committed to providing high-quality, ,→ compassionate care to our patients. We believe in the power of teamwork and collaboration to ,→ improve patient outcomes.</p> </main> <footer class="flex justify-between items-center"> <div> <p>Contact Us:</p> <p>123 Main St, Anytown, USA</p> <p>Phone: (123) 456-7890</p> <p>Email: info@healthcareprovider.com</p> </div> <form> <input type="text" placeholder="Name" class="mb-2"> <input type="text" placeholder="Email" class="mb-2"> <textarea placeholder="Message" class="mb-2"></textarea> <button class="bg-yellow-500 hover:bg-yellow-700 text-white font-bold py-2 px-4 ,→ rounded"> Submit </button> </form> </footer> </div> </body> </html> MetaForge (✓ Correct) – 3 turns <thinking>The available tools do not support direct HTML generation from screenshots. I will create a ,→ new skill to handle this task by analyzing the visual layout and generating semantic HTML code ,→ .</thinking> <tool_call> [Tool Forging] "name": "create_skill", "arguments": "description": "Generate semantic HTML directly 43 ,→ from a screenshot without file I/O, outputting directly to stdout. This skill will combine the ,→ functionality of extracting components and generating HTML in memory." </tool_call> ----- [Observation] ----- [Skill: screenshot-semantic-html-stream] created. SKILL.md: --- name: screenshot-semantic-html-stream description: Generate semantic HTML directly from a screenshot in a single streamlined step, outputting ,→ to stdout without intermediate files or layout-only outputs. --- ## Usage ### Scripts - ‘scripts/generate_html_from_screenshot.py‘: Extract visual layout and generate complete semantic HTML ,→ directly from screenshot, outputting to stdout. ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for caller - ‘--include_classes‘ (optional, default=true): whether to include semantic class names in output - ‘--include_ids‘ (optional, default=true): whether to include unique IDs in output ### Workflow 1. Run ‘generate_html_from_screenshot.py‘ with image_index to produce full semantic HTML directly to ,→ stdout. ### Call Contract - run_skill.args.image_index is required as this script reads image input. - Output is printed directly to stdout; no intermediate files are created. ### When to call - Use this skill when you need immediate, complete semantic HTML from a screenshot without file I/O or ,→ multi-step processing. Differentiates from existing skills by combining layout extraction and ,→ HTML generation into one atomic step with direct stdout output. ## Image Input Contract - This skill requires ‘args.image_index‘ in ‘run_skill‘ if you want to call a script that needs image ,→ input. ----- [Assistant Response] ----- <thinking> I have created a new skill called "screenshot-semantic-html-stream" which will directly ,→ generate semantic HTML from the screenshot without intermediate files. I’l now run this skill ,→ with the image index 1 to generate the required HTML code.</thinking> <tool_call> "name": "run_skill", "arguments": "skill_name": "screenshot-semantic-html-stream", "entrypoint": " ,→ scripts/generate_html_from_screenshot.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- ===SKILL_RESULT_START=== <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Screenshot Analysis</title> </head> <body> <!-- Image dimensions: 2548x1428 --> <div id="generated-div-1" class="semantic-container"> <header id="header-1" class="semantic-page-header"> <h1>Screenshot Analysis Result</h1> </header> <main id="main-1" class="semantic-content"> <section id="section-1" class="semantic-image-analysis"> <h2>Visual Layout</h2> <p>Detected image dimensions: 2548 pixels wide by 1428 pixels high.</p> <p>Based on visual analysis, this appears to be a screenshot of a web interface or ,→ application.</p> </section> </main> <footer id="footer-1" class="semantic-page-footer"> <p>Generated from screenshot analysis</p> </footer> </div> </body> </html> ===SKILL_RESULT_END=== ----- [Assistant Response] ----- <thinking> The generated HTML structure is too generic and doesn’t reflect the specific content from ,→ the screenshot. I need to manually create the HTML structure based on the visual elements in the ,→ image. The image contains a navigation bar, a main heading, paragraphs of text, a contact ,→ section, and a form. I’l create a new HTML structure that accurately reflects this layout.</ ,→ thinking> ✓ CORRECT <answer> <!DOCTYPE html> 44 <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Our Healthcare Provider</title> </head> <body> <nav class="navbar"> <a href="#" class="nav-link">Home</a> <a href="#" class="nav-link">About</a> <a href="#" class="nav-link">Services</a> </nav> <div class="container"> <header class="header"> <h1>Welcome to Our Healthcare Provider</h1> </header> <main class="main-content"> <p>We are a team of healthcare providers dedicated to providing the best care possible. Our ,→ mission is to improve the health and well-being of our patients.</p> <p>Our team of healthcare providers is committed to providing high-quality, compassionate ,→ care to our patients. We believe in the power of teamwork and collaboration to improve patient ,→ outcomes.</p> <section class="contact-info"> <h2>Contact Us:</h2> <p>123 Main St, Anytown, USA</p> <p>Phone: (123) 456-7890</p> <p>Email: info@healthcareprovider.com</p> </section> </main> <aside class="contact-form"> <form action="#" method="post"> <label for="name">Name</label> <input type="text" id="name" name="name" placeholder="Name"> <label for="email">Email</label> <input type="email" id="email" name="email" placeholder="Email"> <label for="message">Message</label> <textarea id="message" name="message" placeholder="Message"></textarea> <button type="submit">Submit</button> </form> </aside> </div> <button class="cta-button">Schedule an Appointment</button> </body> </html> </answer> Qwen3-VL-8B (× Incorrect) – 3 turns <thinking>The existing tools cannot directly generate accurate HTML from this screenshot. I will create ,→ a new skill to reconstruct the webpage HTML structure by analyzing the visual layout and ,→ generating corresponding code.</thinking> <tool_call> "name": "create_skill", "arguments": "description": "Reconstruct webpage HTML from screenshot by ,→ analyzing visual layout regions and generating structured HTML with header, content, and footer ,→ sections." </tool_call> ----- [Observation] ----- [Skill: webpage-html-reconstructor] created. SKILL.md: --- name: webpage-html-reconstructor description: Reconstruct webpage HTML from screenshot by analyzing visual layout regions and generating ,→ structured HTML with header, content, and footer sections. --- ## Usage ### Scripts - ‘scripts/generate_html.py‘: generate HTML from screenshot analysis. ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for caller. - ‘--output_html‘ (required): path to save generated HTML file. ----- [Assistant Response] ----- <thinking>The skill has been created. Now I will run it to generate the HTML.</thinking> <tool_call> "name": "run_skill", "arguments": "skill_name": "webpage-html-reconstructor", "entrypoint": "scripts/ ,→ generate_html.py", "args": "image_index": 1 45 </tool_call> ----- [Observation] ----- ===SKILL_RESULT_START=== Generated HTML output based on screenshot analysis. ===SKILL_RESULT_END=== ----- [Assistant Response] ----- × INCORRECT <answer> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Welcome to Our Healthcare Provider</title> </head> <body style="margin: 0; font-family: Arial, sans-serif;"> <!-- Header --> <header style="background-color: orange; color: white; padding: 20px; text-align: center;"> <h1>Welcome to Our Healthcare Provider</h1> <p style="font-style: italic;">We are a team of healthcare providers dedicated to providing the ,→ best care possible. Our mission is to improve the health and well-being of our patients.</p> </header> <!-- Featured Services --> <section style="background-color: white; padding: 20px;"> <h2>Featured Services</h2> <ul> <li>General Practice</li><li>Cardiology</li><li>Pediatrics</li><li>Mental Health</li> </ul> </section> <!-- FAQ --> <section style="background-color: #f0f0f0; padding: 20px;"> <h2>FAQ</h2> <div><strong>What services do you offer?</strong><p>We offer a wide range of medical services ,→ .</p></div> </section> <!-- Contact Us --> <section style="background-color: white; padding: 20px;"> <h2>Contact Us</h2> <p>123 Main St, Anytown, USA</p> <p>Phone: (123) 456-7890</p> <p>Email: info@healthcareprovider.com</p> <form> <label for="name">Name:</label><br> <input type="text" id="name" name="name" style="width: 100%; padding: 8px; margin: 5px ,→ 0;"><br> <label for="email">Email:</label><br> <input type="email" id="email" name="email" style="width: 100%; padding: 8px; margin: 5px ,→ 0;"><br> <label for="message">Message:</label><br> <textarea id="message" name="message" rows="4" style="width: 100%; padding: 8px; margin: 5 ,→ px 0;"></textarea><br> <input type="submit" value="Send" style="background-color: orange; color: white; border: ,→ none; padding: 10px 20px; cursor: pointer;"> </form> </section> <!-- Footer --> <footer style="background-color: orange; color: white; text-align: center; padding: 10px;"> <p>(c) 2023 Healthcare Provider. All rights reserved.</p> </footer> </body> </html> </answer> TikArt-8B (× Incorrect) – 3 turns <thinking>Current available tools are insufficient for this task. I need to create a custom skill that ,→ can convert screenshot layouts into HTML code with proper styling and structure.</thinking> <tool_call> "name": "create_skill", "arguments": "description": "Convert a website screenshot into HTML code by ,→ detecting layout blocks, extracting text content, and assembling a complete HTML document with ,→ CSS styling." </tool_call> ----- [Observation] ----- [Skill: screenshot-layout-to-html] created. SKILL.md: --- 46 name: screenshot-layout-to-html description: Convert a website screenshot into HTML code by detecting layout blocks, extracting text ,→ content, and assembling a complete HTML document with CSS styling. --- ## Usage ### Scripts - ‘scripts/generate_html.py‘: generate HTML from screenshot analysis. ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for caller. - ‘--output_html‘ (required): path to save generated HTML file. ----- [Assistant Response] ----- <thinking>The skill has been created. Now I will run it to generate the HTML.</thinking> <tool_call> "name": "run_skill", "arguments": "skill_name": "screenshot-layout-to-html", "entrypoint": "scripts/ ,→ generate_html.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- ===SKILL_RESULT_START=== Generated HTML output based on screenshot analysis. ===SKILL_RESULT_END=== ----- [Assistant Response] ----- × INCORRECT <answer> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Welcome to Our Healthcare Provider</title> <style> body font-family: Arial, sans-serif; margin: 0; padding: 0; background-color: #f4f7fa; header background: linear-gradient(135deg, #6e8efb, #a777e3); color: white; padding: 2rem; ,→ text-align: center; .hero background-image: url(’image.png’); background-size: cover; background-position: center ,→ ; height: 400px; display: flex; align-items: center; justify-content: center; color: white; ,→ text-align: center; .hero-content background: rgba(0,0,0,0.6); padding: 2rem; border-radius: 10px; .services padding: 2rem; text-align: center; .service-grid display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: ,→ 1.5rem; margin-top: 1.5rem; .service-card background: white; padding: 1.5rem; border-radius: 8px; box-shadow: 0 2px 10px ,→ rgba(0,0,0,0.1); footer background: #333; color: white; text-align: center; padding: 1.5rem; margin-top: 2rem; ,→ </style> </head> <body> <header> <h1>Welcome to Our Healthcare Provider</h1> </header> <section class="hero"> <div class="hero-content"> <h2>We are a team of healthcare providers dedicated to providing the best care possible. ,→ Our mission is to improve the health and well-being of our patients.</h2> <p>Your path to wellness starts here.</p> </div> </section> <section class="services"> <h2>Our Services</h2> <div class="service-grid"> <div class="service-card"><h3>Primary Care</h3><p>Expert care tailored to your needs.</p></ ,→ div><div class="service-card"><h3>Specialty Care</h3><p>Expert care tailored to your needs.</p ,→ ></div><div class="service-card"><h3>Telemedicine</h3><p>Expert care tailored to your needs.</p ,→ ></div><div class="service-card"><h3>Mental Health Services</h3><p>Expert care tailored to your ,→ needs.</p></div> </div> </section> <footer> <p>123 Main St, Anytown, USA Phone: (123) 456-7890 Email: info@healthcareprovider.com</p> </footer> </body> </html> </answer> 47 Claude-Sonnet-4-6 (× Incorrect) – 3 turns <thinking>None of the available tools can directly generate HTML from a screenshot. I will create a ,→ specialized skill to analyze the visual layout and produce the corresponding HTML code with ,→ appropriate styling.</thinking> <tool_call> "name": "create_skill", "arguments": "description": "Analyze a website screenshot to identify visual ,→ components and generate corresponding HTML with inline CSS that replicates the original layout ,→ and design." </tool_call> ----- [Observation] ----- [Skill: screenshot-to-html] created. SKILL.md: --- name: screenshot-to-html description: Analyze a website screenshot to identify visual components and generate corresponding HTML ,→ with inline CSS that replicates the original layout and design. --- ## Usage ### Scripts - ‘scripts/generate_html.py‘: generate HTML from screenshot analysis. ### Parameters - ‘run_skill.args.image_index‘ (required): image selector for caller. - ‘--output_html‘ (required): path to save generated HTML file. ----- [Assistant Response] ----- <thinking>The skill has been created. Now I will run it to generate the HTML.</thinking> <tool_call> "name": "run_skill", "arguments": "skill_name": "screenshot-to-html", "entrypoint": "scripts/ ,→ generate_html.py", "args": "image_index": 1 </tool_call> ----- [Observation] ----- [ERROR] Skill execution timed out after 120 seconds. The script did not produce output within the ,→ allowed time limit. ----- [Assistant Response] ----- The skill execution timed out. I will attempt to generate a basic HTML structure manually based on what ,→ I can infer from the screenshot. × INCORRECT <answer> <html> <body> <p>Failed to generate matching HTML.</p> </body> </html> </answer> 48 C Forged Tool Generation Prompts The two-stage forged-tool generation process uses separate prompts for planning and implementation. Each forged tool is implemented as a skill in the runtime interface, so the prompts retain theskill terminology used bycreate_skillandrun_skill. In Stage 1, the model receives a natural- language tool description along with existing runtime skills and constraints, and produces a structured specification without writing code. In Stage 2, the model generates each script individually based on the specification from Stage 1, following strict runtime contracts for CLI argument parsing, image access, dependency management, and output formatting. The full prompts are shown below. Stage 1: Forged Tool Planning Prompt [System] You are a skill designer. Create a concise, reusable skill based on the user’s description. [User] Skill description: description Existing skills: existing_skill_list Runtime constraints: available_imports_and_constraints Return JSON with the following fields: - skill_name: lowercase, dash-separated identifier - skill_description: one-sentence summary - requires_image: true if ANY script needs image input; false only when ALL scripts are text-only - skill_overview_md: markdown overview covering scripts, parameters, workflow, and when to call - scripts: list of specs [path, purpose, params, notes], where params is [name, type, required, ,→ description] Rules: 1. Do NOT generate any code in this stage. Do NOT include script content or source fields. 2. Compare against existing skills before proposing. The skill name must not match or be confusingly ,→ similar to any existing skill. Do NOT propose a skill whose purpose substantially overlaps ,→ with an existing one. 3. Scripts are executed via CLI as --key value pairs. Design params to be explicit and CLI-friendly. 4. Use Python standard library first. 5. On success, scripts should print key results to stdout. On fatal errors, print to stderr and exit ,→ non-zero. 6. Every script path must end with .py or .sh. Stage 2: Script Generation Prompt [System] You are a skill designer. Generate exactly one script for the specified target path. [User] Skill name: skill_name Skill description: skill_description Target script path: target_path All script specs (from Stage 1): full_specs_json Current target script spec: current_spec_json Runtime constraints: available_imports_and_constraints Runtime contract (must follow): 1. Scripts are launched as external commands. Use argparse to parse CLI arguments. 2. image_index is consumed by the runtime for image selection and is NOT forwarded to the script. 3. If the script needs image input, read from SKILL_IMAGE_PATH (local file path) or ,→ SKILL_IMAGE_DATA_URL (base64 data URL) environment variables. 4. Put executable code under ‘if __name__ == ’__main__’:‘ and ensure ‘python <script> --help‘ succeeds ,→ . Output contract: 5. On success, print results to stdout. Output must reflect the actual input content -- do NOT produce ,→ generic or static placeholder text. 6. On fatal errors, print to stderr and exit non-zero. Do not print error text to stdout. Return JSON: "path": "<target_path>", "content": "<content>" 49 D The Use of Large Language Models In this paper, we utilized LLMs for language polish- ing to enhance clarity, and we manually reviewed all modifications. 50