Paper deep dive

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Mesut Toruk

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 32

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/28/2026, 4:21:50 AM

Summary

The paper introduces BekchiAI, a comprehensive framework for evaluating Large Language Model (LLM) agents that combines a benchmark (BekchiAI-Benchmark) with an observability platform (BekchiAI-Platform). The benchmark consists of 13 tool-using ReAct agents across 7 categories (arithmetic, structured/SQL, security, URL grounding, planning, orchestration, tool-policy) with 2,057 deterministic tasks. It emphasizes verifier-checked gold answers, adversarial samples, and imperfect security scanners to measure true agentic skills rather than oracle copying. The platform provides real-time telemetry, remote control, and behavioral metrics like tool-call adherence and URL hallucination. The study evaluates four models (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b), revealing that aggregate accuracy masks significant performance variations across specific skill categories.

Entities (18)

BekchiAI-Benchmark · benchmark · 98%BekchiAI-Platform · software-platform · 98%gemma4:26b · llm-model · 95%gemma-4-31B-it · llm-model · 95%GPT-OSS-120B · llm-model · 95%Qwen3.7-Max · llm-model · 95%Arithmetic · task-category · 90%Orchestration · task-category · 90%Planning · task-category · 90%ReAct · agent-framework · 90%Security Detection · task-category · 90%SQL · technology · 90%Structured/SQL · task-category · 90%Tool-Call Adherence · metric · 90%Tool-Policy · task-category · 90%URL Grounding · task-category · 90%URL Hallucination · metric · 90%DAG · data-structure · 85%

Relation Signals (17)

BekchiAI-Benchmark → evaluates → gemma4:26b

confidence 95% · We report a four-model comparison (...gemma4:26b...)

BekchiAI-Benchmark → evaluates → GPT-OSS-120B

confidence 95% · We report a four-model comparison (...gpt-oss-120b)

BekchiAI-Benchmark → evaluates → Qwen3.7-Max

confidence 95% · We report a four-model comparison (Qwen3.7-Max...)

BekchiAI-Benchmark → evaluates → gemma-4-31B-it

confidence 95% · We report a four-model comparison (...gemma-4-31B-it...)

BekchiAI-Benchmark → includescategory → URL Grounding

confidence 95% · 7 task categories (...URL grounding...)

BekchiAI-Benchmark → includescategory → Security Detection

confidence 95% · 7 task categories (...security detection...)

BekchiAI-Benchmark → includescategory → Planning

confidence 95% · 7 task categories (...planning...)

BekchiAI-Benchmark → includescategory → Orchestration

confidence 95% · 7 task categories (...orchestration...)

BekchiAI-Benchmark → includescategory →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

28,878 characters extracted from source content.

Expand or collapse full text

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click Mesut Toruk Affiliation: Istanbul, Turkiye Email: m.mesut.toruk@gmail.com Abstract Large-language-model agents reason, call tools, and act autonomously over many steps, but their agentic skills—correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments—are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 1313 tool-using ReAct agents across 77 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,0572,057 deterministic, committed test tasks. Every task is verifier-checkable—gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model’s own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy—tool-call adherence, URL hallucination and source-match, and per-model token cost—and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released. Keywords LLM agents ⋅· benchmark ⋅· tool use ⋅· planning ⋅· agent security ⋅· evaluation ⋅· telemetry 1 Introduction Large language models (LLMs) are increasingly deployed not as single-shot text generators but as agents: systems that interleave model calls with tool use, memory, and multi-step planning to pursue goals with limited human supervision (Yao et al., 2023; Schick et al., 2023; Wang et al., 2024; Xi et al., 2023). This paradigm has moved quickly from research demonstrations into consequential applications. In software engineering, agents equipped with shell, editor, and search tools resolve real GitHub issues end-to-end (Jimenez et al., 2024; Yang et al., 2024); on the web, they navigate live sites to complete multi-step tasks such as booking, form-filling, and shopping (Deng et al., 2023; Zhou et al., 2024); and in the sciences, LLM-driven systems autonomously design, plan, and execute laboratory experiments (Boiko et al., 2023). Industry adoption is following: Gartner projects the share of enterprise applications embedding task-specific AI agents to rise from under 5%5\% in 2025 to roughly 40%40\% by 2026, with a growing fraction of routine decisions delegated to agents (Gartner, 2025). As agents take real actions—writing code, moving money, calling external services—the practical question shifts from “how fluent is the model?” to “how good are its agentic skills?” These skills—does the agent select and sequence the right tools, respect dependencies when planning, refuse a poisoned input, and ground the arguments (URLs, queries) it fabricates?—are precisely what a single accuracy number over a mixed test set fails to isolate. Worse, many popular agent tasks are easily gamed by pattern-matching or leak their gold through an oracle tool, so a high score need not reflect real competence or safe judgment. We address this with the BekchiAI-Benchmark: 1313 tool-using ReAct agents, each a self-contained task family with a verifier-checkable gold answer, spanning 77 categories and 2,0572,057 committed test tasks (Section 3). Three design choices make the suite hard to game: (i) gold is computed by a verifier—canonical SQL over a real database, the exact schedule of a directed acyclic graph (DAG), or a closed-form lambda—so it cannot drift; (i) the hard cases adversarial security inputs, homoglyph URL traps, deep dependency graphs) are hand-authored one by one, not templated; and (i) the security detectors the agent may call are deliberately imperfect signature scanners, so accuracy measures the model’s judgment rather than its ability to echo an oracle. The agents run on a small, purpose-built tool library (Section 4). We report each model’s accuracy and per-model token cost; the harness additionally computes behavioral metrics—tool-call adherence, and URL hallucination and source-match (Section 5)—that we release with the benchmark. The benchmark runs on a self-contained evaluation harness that logs per-call token and latency next to each outcome, so capability and cost come from a single run. Our four-model comparison (Section 6) shows that the ranking: a model that leads overall can collapse on structured or multi-step tool tasks while another inverts the order on grounding. Complementing the benchmark, the BekchiAI-Platform (Section 7) is a web-based observability and control layer for deployed agents—streaming the same per-call telemetry live and letting an operator stop, pause, resume, or revoke a running agent with one click. We run every rollout under the BekchiAI-Platform, an observability and control layer that instruments the benchmark drop-in—streaming per-call token/latency telemetry and enabling a remote kill-switch—so capability and cost are measured from one stream (Section 7). Our four-model comparison (Section 6) shows that the ranking lives in the per-family spread: a model that leads overall can collapse on structured or multi-step tool tasks while another inverts the order on grounding. Contributions. • The BekchiAI-Benchmark: 1313 verifier-checked, tool-using ReAct task families (2,0572,057 committed tasks) with hand-authored adversarial samples and imperfect-by-design security tools (Section 3). • A released tool library of arithmetic, SQL, security-scanner, web/URL, and business tools the agents call (Section 4), and a set of behavioral metrics beyond accuracy (Section 5). • A four-model agentic-skill comparison read from a single instrumented stream, capturing both capability (per family) and cost (Section 6). • The BekchiAI-Platform: a drop-in observability + control layer (agent-native event model, grounding and plan-vs-actual analytics, policy enforcement, remote control) used to run the benchmark (Section 7). 2 Related Work Agent and tool-use benchmarks. A growing body of benchmarks evaluates LLM agents. General suites such as AgentBench (Liu et al., 2024) and GAIA (Mialon et al., 2023) probe multi-step reasoning and tool use across heterogeneous environments, while domain benchmarks target software engineering (SWE-bench (Jimenez et al., 2024)), web navigation (Mind2Web (Deng et al., 2023), WebArena (Zhou et al., 2024)), and function/API calling (ToolLLM (Qin et al., 2024), τ-bench (Yao et al., 2024)). Most report a single end-to-end success rate per task, which conflates distinct skills and is sensitive to prompt formatting and to gold that can leak through an oracle tool. The BekchiAI-Benchmark is complementary: it isolates seven skill categories into verifier-checked families, computes gold from a real database, scheduling DAG, or closed-form lambda so it cannot drift, and reports behavioral metrics—tool-call adherence, plan-vs-actual consistency, and grounding—alongside accuracy. Agent security. InjecAgent (Zhan et al., 2024) and AgentDojo (Debenedetti et al., 2024) measure whether an agent is hijacked by malicious instructions injected into tool outputs or retrieved content. Our three security families ask a related but distinct question: given an untrusted artifact—a form value, a retrieved passage, a generated output, or a dependency—and a deliberately imperfect detector, does the model correctly decide to block or proceed? Because the scanner agrees with the truth only 4040–55%55\% of the time, the score reflects the model’s own judgment rather than oracle-echoing, and it penalises over-blocking benign inputs as well as missing real threats. Agent observability. Orthogonally, generic tracing and metrics stacks capture service calls but not agentic structure—sessions, tool trees, per-call token usage, and argument provenance—and offer no remote control of an autonomous process. The BekchiAI-Platform targets that gap and lets us read capability and cost from one instrumented stream. 3 The BekchiAI-Benchmark Table 1: The 1313 agents of the BekchiAI-Benchmark, their category, committed test-set size n, the tools they call, and the skill each isolates. Agent Category Tools n Skill probed ar_math arithmetic calc 154 multi-step computation ar_multistep arithmetic lookup, calc 152 constant lookup + compute sql_analytics structured list_tables, schema, run_sql 157 SQL analytics (joins, group-by) sec_sqli security scan_sql 154 SQL-injection judgment sec_context_poison security scan_context 154 prompt-injection judgment sec_guard security security_scan 151 LLM security-risk gating ug_fetch url-grounding websearch, fetch 155 argument generation (no URL given) ug_imposter url-grounding websearch, fetch 150 homoglyph/typosquat resistance dependency_planning planning customer-record lookups 180 dependency resolution multi_tool_planning planning customer, fx_rate, calc, discount 152 multi-tool orchestration orchestrator orchestration — 196 as-soon-as-possible scheduling tool_error_recovery tool-policy cache, mirror, authdb 150 failover / retry policy tool_arguments tool-policy 5 record-creation tools 152 correct argument construction Total 2,057 Structure. The suite is 1313 agents grouped into 77 categories (Table 1). Each agent is a strict-JSON ReAct loop: the model emits one JSON action per turn—a tool call with arguments, or a final answer—against a commodity LLM API, and the harness executes tools and feeds results back until the model answers or a step budget is reached. Every task family ships a deterministic test set committed as JSONL; the 1313 files total 2,0572,057 held-out tasks, so a run is exactly reproducible and inspectable. Verifier-checked gold. For sql_analytics, each question’s canonical SQL is run against the same in-memory database the agent queries, so the gold reflects the schema conventions (amounts in integer cents; cancelled orders excluded) exactly. For orchestrator, the gold is the unique as-soon-as-possible level-partition of a dependency DAG, computed from the scenario graph. Arithmetic families evaluate closed-form lambdas over the sampled constants. Because the verifier and the agent see the same world, gold cannot silently drift, and a numeric tolerance (a cent for money, tight relative tolerance for arithmetic) absorbs only formatting noise. Adversarial samples. Planning and orchestration scenarios use asymmetric-join DAGs, long idle-span dependencies, and mid-graph dead-end distractors that must be pruned by transitive reachability. The URL-grounding families pair each question with a different real host and, for ug_imposter, a pasted homoglyph link (e.g. examp1e.com) the agent must ignore in favour of the authoritative host. Imperfect-by-design security tools. The three security families (sec_sqli, sec_context_poison, and sec_guard, whose risk classes follow the OWASP Top-10 for LLM Applications) present a realistic operation (process this form value, use this retrieved context, ship this generated output) carrying an untrusted artifact, and expose a scanner tool the agent may call. Crucially the scanner is a signature heuristic that disagrees with the truth: it misses obfuscated attacks (false negatives) and fires on benign look-alikes (false positives), agreeing with the curated gold labels only ∼40 \!40–55%55\% of the time. A model that blindly trusts or inverts the tool therefore scores near chance; the correct behavior is to read the artifact and decide. This converts a would-be oracle into a genuine judgment test. 4 Benchmark Tools The agents run on a small, shared tool library which exist in github repo of benchmark; every agent imports its tools from this library rather than defining its own, so tool behavior is consistent and independently testable (Table 2). Two properties matter for the benchmark. First, the data-backed tools operate over real deterministic state: run_sql executes against an in-memory SQLite database of customers/products/orders, and fetch serves an offline index of real canonical web pages (404-ing everything else), so the verifier and the agent share one ground truth. Second, the security scanners are deliberately heuristic (Section 3), returning a hint (suspicious / no signal), never a verdict. Table 2: The shared tool library the agents call. Family Tools Backing arithmetic calc, lookup evaluator + constant table SQL list_tables, schema, run_sql in-memory SQLite (3 tables) security scan_sql, scan_context, security_scan signature scanners (imperfect) web / URL websearch, fetch offline index of real pages business find_customer, list_invoices, get_payment, fx_rate, account_discount customer / finance store failover cache_get, mirror_get, authdb_get tiered read with faults records create_invoice, schedule_shipment, book_appointment, configure_alert, register_device typed record creation 5 Metrics Beyond a single accuracy number, each rollout emits a small set of canonical behavioral metrics that aggregate across families: • Accuracy (success rate): fraction of tasks whose extracted answer matches the verifier gold, reported per family and per category with Wilson 95%95\% score intervals (Wilson, 1927; Brown et al., 2001), which are well-behaved at the small per-family sample sizes here. • URL grounding: for the grounding families, the hallucination rate (Ji et al., 2023) (a fabricated argument that is neither a real indexed URL nor was returned by any search) and the source-match rate (the cited answer host equals the gold host). • Safety: for the security families, a violation rate (proceeding on a truly unsafe artifact) and, where applicable, an over-refusal rate (blocking a benign one). • Cost: LLM calls and input/output token counts per model, captured by the platform on the same runs (Section 6). All metrics come from the github repo, so every reported number is reproducible and the released harness. 6 Results: Four Models on the Benchmark We evaluate all 1313 families across four models —Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, and gpt-oss-120b—each on the same 2,0572,057 held-out tasks. Task conventions (e.g. the SQL cents/cancelled rules) are stated in the prompt, so a score measures the underlying skill. Table 3: Test-set accuracy (success rate on [0,1][0,1]) per family across four models, computed from the committed test sets in data/eval (2,0572,057 tasks per model). Best per family in bold. The wide per-family spread—not an aggregate—is the benchmark’s ranking signal. Family Category Qwen3.7-Max gemma-4-31B gemma4:26b gpt-oss-120b ar_math arithmetic 0.90 0.86 0.91 0.69 ar_multistep arithmetic 0.97 0.97 0.85 0.65 sql_analytics structured 0.87 0.81 0.77 0.24 sec_sqli security 0.99 0.99 0.95 0.94 sec_guard security 0.99 0.96 0.93 0.85 sec_context_poison security 0.92 0.91 0.84 0.82 ug_fetch url-grounding 0.45 0.53 0.43 0.78 ug_imposter url-grounding 0.82 0.78 0.56 0.89 dependency_planning planning 0.99 0.99 0.96 0.68 multi_tool_planning planning 0.94 0.80 0.96 0.44 orchestrator orchestration 0.86 0.95 0.91 0.94 tool_error_recovery tool-policy 0.76 0.74 0.65 0.25 tool_arguments tool-policy 0.99 0.88 0.97 0.99 overall (task-weighted) 0.88 0.86 0.82 0.71 Table 4: The same accuracies collapsed to the seven task categories (each cell is the task-weighted mean over that category’s families; n is its total test tasks). This is the level the radar in Figure 1 plots; best per category in bold. Category n Qwen3.7-Max gemma-4-31B gemma4:26b gpt-oss-120b arithmetic 306 0.93 0.91 0.88 0.67 structured 157 0.87 0.81 0.77 0.24 security 459 0.97 0.95 0.91 0.87 url-grounding 305 0.63 0.65 0.49 0.83 planning 332 0.97 0.90 0.96 0.57 orchestration 196 0.86 0.95 0.91 0.94 tool-policy 302 0.88 0.81 0.81 0.62 overall 2,057 0.88 0.86 0.82 0.71 Figure 1: Zero-shot accuracy by task category for the four models—one axis per category (the task-weighted mean over its families of Table 4; radius == success rate on [0,1][0,1]). The polygons expose each model’s profile: gpt-oss-120b (red) caves in on the structured (0.240.24), planning (0.570.57), and tool-policy (0.620.62) categories yet reaches farthest on url-grounding (0.830.83), while the other three trace a broadly similar high envelope—a shape an overall score cannot convey. Table 5: Operational cost of the same runs, measured by the platform: LLM calls and token usage per model over the full 2,0572,057-task suite (input + output tokens in millions). Read alongside Table 3, this shows capability and cost from one instrumented stream: gemma-4-31B and gpt-oss-120b spend the most tokens without leading on accuracy, while gemma4:26b is the leanest. Model LLM calls input (M) output (M) total (M) Qwen3.7-Max 6,965 3.86 1.96 5.83 gemma-4-31B 7,715 8.32 1.80 10.12 gemma4:26b 7,258 4.61 0.82 5.43 gpt-oss-120b 7,158 6.56 2.25 8.81 Findings. Table 3 gives each model a distinct capability profile (Figure 1), and the story is in the per-family spread rather than the aggregate. Overall the four models rank Qwen3.7-Max (0.880.88) >> gemma-4-31B (0.860.86) >> gemma4:26b (0.820.82) >> gpt-oss-120b (0.710.71), but the top two are within two points and no model dominates everywhere. gpt-oss-120b is a telling outlier that inverts the usual ordering: it is the best of the four on both grounding families (ug_fetch 0.780.78, where every other model sits at 0.430.43–0.530.53; ug_imposter 0.890.89) yet by far the worst on the structured and multi-step families (sql_analytics 0.240.24, tool_error_recovery 0.250.25, multi_tool_planning 0.440.44)—a profile an overall score would hide entirely. The families that discriminate most widely are exactly these tool/structured tasks: sql_analytics spans 0.240.24–0.870.87, tool_error_recovery 0.250.25–0.760.76, and multi_tool_planning 0.440.44–0.960.96, so a single family separates the field by more than sixty points. gemma-4-31B leads on orchestrator (0.950.95) and ties for the top on the security and dependency-planning families, while the small local gemma4:26b is competitive throughout (0.820.82 overall) and even leads ar_math (0.910.91) and multi_tool_planning (0.960.96). A few families are saturated across all four (sec_sqli 0.940.94–0.990.99) and serve as sanity checks rather than ranking signal. The platform captures the operational side of the same runs (Table 5): every model issues 77k–88k LLM calls over the suite, but token usage varies almost 2×2×—gemma-4-31B spends the most (10.110.1M) and gemma4:26b the least (5.45.4M), and gpt-oss-120b burns 8.88.8M for the lowest accuracy. Capability and cost are thus read from one instrumented stream rather than separate experiments. 7 The BekchiAI-Platform BekchiAI-Platform is a drop-in observability and control layer. Enabling telemetry points a lightweight SDK at the platform; the agent’s task logic is unchanged. This is what lets us read capability and cost (Section 6) from the same run, and it adds security observability and a remote kill-switch on top. Architecture and event model. The platform has four parts: a client SDK that emits telemetry and polls for control commands; an ingestion + API and REST endpoints for agents, sessions, traces, and findings; a store with a wide events table (JSONB payloads plus relational metadata); and a React dashboard. Ingestion is idempotent on a event id, authenticated with per-project API keys, and multi-tenant. Agent activity is modelled as sessions of linked events forming a trace tree. Core types are session_start; llm_call (model, input/output tokens, latency); tool_call / tool_result (name, arguments, success); url_access (host, status, allowlisted); plan (intended tools and URLs); denied (policy events); and outcome. Table 6 shows how each benchmark behavior maps onto these events, so no separate evaluation harness is needed. Table 6: How each benchmark behavior maps onto BekchiAI-Platform signals. One instrumented run yields all four. Model behaviour Platform signal (event) Reveals task correctness outcome (success/mismatch) accuracy, per family model call llm_call (input/output tokens, latency) token usage and speed tool use tool_call, plan-vs-actual tool competence, adherence URL arguments url_access (grounded/hallucinated) URL-hallucination, source-match Behavioral-security analytics. Over the event store the platform computes two signals. Argument grounding: from each session’s plan and allowlisted accesses it builds a trusted-host set and labels each accessed host grounded, ungrounded, or imposter_url (an ungrounded host within Levenshtein distance ≤2≤ 2 of a trusted one—a likely typosquat), reporting the grounding rate. Plan-vs-actual consistency: it compares the intended tool set to the executed tool_call sequence, yielding intersection / unplanned / missing sets and a per-session verdict. These are the platform-side counterparts of the benchmark’s grounding and tool-adherence metrics. Policy enforcement and remote control. Operators define allow/block/alert policies over host globs and tool names; at ingest, block matches emit a synthetic denied event, giving a durable audit of prevented actions. The platform is also bidirectional: an operator issues stop/pause/resume/revoke from the dashboard, the agent polls the control endpoint, applies pending commands, and acknowledges; commands move pending→ → with timestamps. In the benchmark run this let us abort a running agent remotely within a single poll cycle. Enforcement is cooperative: an agent that cannot reach the channel is not controllable. 8 Conclusion We introduced the BekchiAI-Benchmark, a suite of 1313 tool-using ReAct agents across seven categories (2,0572,057 committed tasks) that measures the agentic skills of LLMs—tool selection and sequencing, planning under dependencies, security judgment, and argument grounding—rather than a single blended accuracy. Three design choices keep the suite honest: gold is computed by a verifier (canonical SQL, the exact schedule of a dependency DAG, or a closed-form lambda) and so cannot drift; the discriminating cases are hand-authored one by one rather than templated; and the security families replace the usual oracle detector with a deliberately imperfect signature scanner, so a score reflects the model’s own judgment rather than tool-copying. We pair accuracy with behavioral metrics—tool-call adherence, plan-vs-actual consistency, URL grounding, and per-model token cost—all read from one instrumented stream via the drop-in BekchiAI-Platform. Across four models, the results argue for reporting the per-family spread rather than an aggregate. The top two models finish within two points overall, yet no model dominates every skill: gpt-oss-120b leads both grounding families while collapsing on the structured and multi-step tool tasks (e.g. sql_analytics 0.240.24 vs. 0.870.87)—an inversion a single leaderboard number erases. The tool and structured families discriminate most sharply, separating the field by more than sixty points. Because the platform records token cost on the same runs, capability and efficiency can be weighed together—and the most accurate model is not the cheapest. As agents take on real, consequential actions, we argue that skill- and cost-resolved, hard-to-game evaluation of this kind is a prerequisite for trustworthy deployment. The benchmark is deliberately compact and evaluates one task per rollout; natural extensions include broader model coverage, additional adversarial rounds and skill categories, longer-horizon and multi-turn interactions, and applying the platform’s grounding and plan-vs-actual analytics to live agent deployments. We release the benchmark, tool library, metric scripts, and we released to platform on web and mobile markets to support this work. Reproducibility. The benchmark—provided test sets, the tool library, harness, and metrics—is released at https://github.com/bekchiai/bekchiai-benchmark. References Boiko et al. (2023) D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes Autonomous chemical research with large language models. Nature 624 (7992), p. 570–578. Cited by: §1. Brown et al. (2001) L. D. Brown, T. T. Cai, and A. DasGupta Interval estimation for a binomial proportion. Statistical Science 16 (2), p. 101–133. Cited by: 1st item. Debenedetti et al. (2024) E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Deng et al. (2023) X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su Mind2Web: towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. Gartner (2025) Gartner Gartner predicts 40% of enterprise apps will feature task-specific AI agents by 2026, up from less than 5% in 2025. Note: Gartner Newsroom, press release Cited by: §1. Ji et al. (2023) Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), p. 1–38. Cited by: 2nd item. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Liu et al. (2024) X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. AgentBench: evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), Cited by: §2. Mialon et al. (2023) G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom GAIA: a benchmark for general AI assistants. arXiv preprint arXiv:2311.12983. Cited by: §2. Qin et al. (2024) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. ToolLLM: facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations (ICLR), Cited by: §2. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §1. Wang et al. (2024) L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. Cited by: §1. Wilson (1927) E. B. Wilson Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), p. 209–212. Cited by: 1st item. Xi et al. (2023) Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: a survey. arXiv preprint arXiv:2309.07864. Cited by: §1. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-Bench: a benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. Cited by: §2. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), Cited by: §1. Zhan et al. (2024) Q. Zhan, Z. Liang, Z. Ying, and D. Kang InjecAgent: benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics (ACL), Cited by: §2. Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.

Instant research discovery