Paper deep dive
Unconstrained CTI-REALM with Open-Weights
Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The benchmark's published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude. Because the benchmark's checkpoint scores largely record what an agent did or mentioned, we also introduce stage-level metrics that score what the agent submits, by executing its detection queries in the telemetry store and judging its outputs against the answer key. Rather than combining them into one more overall reward, we show that different models lead different stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score. We also report how many tokens each model generates, and find that the share of output spent on thinking does not track the reward.
Tags
Links
- Source: https://arxiv.org/abs/2610.14180v1
- Canonical: https://arxiv.org/abs/2610.14180v1
Trouble viewing inline? Open PDF directly →
Full Text
54,479 characters extracted from source content.
Expand or collapse full text
Unconstrained CTI-REALM with Open-Weights Sahil Garg ∗ , Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma Averlon Redmond, WA Correspondence: sahil.garg@averlon.io, sahil.garg.cs@gmail.com Abstract Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM 2 , a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nine- teen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The benchmark’s published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude. Because the benchmark’s checkpoint scores largely record what an agent did or mentioned, we also introduce stage-level metrics that score what the agent submits, by executing its detection queries in the teleme- try store and judging its outputs against the answer key. We deliberately do not combine them into a new overall reward, which would be one more weighting among many defensible ones. Instead they show that different models lead differ- ent stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score. We also report how many tokens each model generates, and find that the share of output spent on thinking does not track the reward. 1 Introduction Security operations teams find intrusions by running detection rules over the logs their systems emit. Those logs are gathered in a security information and event management system (SIEM): a central store that collects telemetry from endpoints, servers, cloud services and identity providers, evaluates detection rules against it continuously, and raises an alert for an analyst whenever a rule matches. A SIEM is only as good as its rules, and writing them is the job of a detection engineer. Handed a threat-intelligence report on a new intrusion campaign, the engineer reads it and decides which adversary behaviours it describes, works out which of several hundred telemetry tables would have recorded them, queries that telemetry to check that the evidence is there, and only then writes the detection. Each step depends on the one before it, and only the last step produces a rule that is deployed. ∗ Author names are in descending order of contribution, not an indication of seniority. 2 A. Chakraborty, S. Ho, A. Cook and M. Mel ́ endez, CTI-REALM: Benchmark to evaluate agent perfor- mance on security detection rule generation capabilities, 2026, arXiv:2603.13517. Preprint. Under review. arXiv:2610.14180v1 [cs.AI] 9 Oct 2026 DeveloperModels evaluated Open-weight (19 models) Zhipu AIGLM-5.3-flash, GLM-5.3, GLM-5.2 DeepSeekDeepSeek-v4.1-flash, DeepSeek-v4-flash, DeepSeek-v4-pro AlibabaQwen3.8-2.4T, Qwen3.8-27B, Qwen3.6-35B-A3B Moonshot AIKimi-K3 MiniMaxMiniMax-M3 Google DeepMindGemma-4-31B, Gemma-4-26B-A4B NVIDIANemotron-3-Ultra Thinking Machines LabInkling, Inkling-Small Mistral AIMistral-Large-4 OpenAIGPT-OSS-120B, GPT-OSS-20B Closed (7 models) OpenAIGPT-5.6-sol, GPT-5.6-terra, GPT-5.6-luna AnthropicClaude-Opus-5, Claude-Sonnet-5, Claude-Haiku-4.5 xAIGrok-4.6 Table 1: The twenty-six models evaluated, grouped by whether their weights are published. The closed models are available only through their developers’ APIs. Mistral-Large-4 is counted as open- weight: its developer released it on 6 October 2026 with its weights announced for publication by the end of that month. This is the kind of multi-step work that organisations now want agentic language models to take on, and the deployment question is usually posed as a choice between open-weight and frontier closed models: can a model whose weights can be run and audited in-house do the work of a far more expensive proprietary one? CTI-REALM (Chakraborty et al., 2026) is a natural benchmark on which to ask this question. It follows the engineer’s workflow in five checkpoints, from reading the report to submitting a detection, over intrusions emulated on Linux hosts, cloud control planes and Kubernetes, and the agent must itself query a live log store in the Kusto Query Language (KQL), the query language of Microsoft’s Sentinel SIEM. We find that the answer depends on a setting that is easy to mistake for part of the benchmark. CTI-REALM caps every task at a fixed number of agent messages, and the cap is hard: a task that exhausts it never submits, and so scores zero on every checkpoint, including the ones it had already passed. At the published limit the strongest open-weight and closed models cannot be distinguished, and part of what is measured there is which models finish within the limit. With the limit relaxed, the open-weight models it was truncating recover, and an open-weight model matches the strongest closed models at a small fraction of their cost (Figure 1). With the limit relaxed, the two highest rewards belong to open-weight models. The differences among the leading models are small, and we do not claim an ordering among them (Section 3). Our claim is parity: the best open-weight models match the best closed models on this benchmark, and one of them does so at a cost low enough that the extra turns add little to it. A second obser- vation concerns price: among the leading models the cost of a run varies by more than an order of magnitude, so on this task price says little about how well a model does the work. A third observation concerns the benchmark’s stage scores, which mostly record what an agent did or mentioned rather than what it submitted. For four of the five stages we therefore introduce a metric that scores the submission itself against the benchmark’s answer key, by executing the submitted detection query in the telemetry store or by asking LLM judges to match the submission to the key. Where the benchmark’s score measures something else, such as the agent’s exploration, these metrics complement it; where it measures the same quantity poorly, they correct it. We do not combine these metrics into a new overall reward: the weighting would be one more choice among many defensible ones, and a single score would hide what the metrics show, namely that no model leads every stage. The best system for this workflow is therefore not necessarily one model chosen by its overall ranking, but a route that hands each stage to the model that performs it best. 2 1.1 Contributions 1. Open-weight models match the strongest closed models, one of them at a small fraction of the cost. Given enough turns to finish, the two best open-weight models score level with or above the two strongest closed models we measured, and one of them does so at less than a tenth of their cost. Section 3 gives the rewards, the costs and a paired bootstrap interval for the gap between the leading models. 2. Separation of the message budget from model capability. A task that exhausts the limit scores zero on every checkpoint, including ones it had already passed, so the published setting measures termination as well as capability. We therefore report every model at an unconstrained budget, at which no reported sample is truncated (Section 3). 3. Weak relation between price and capability among the leading models. Among the models that score close to the leader, the cost of a run varies by more than an order of magnitude (Section 3). 4. An open-against-closed evaluation under a common harness, scorer and judge panel. We evaluate twenty-six models, nineteen open-weight and seven closed (Table 1), and report each on each of the five checkpoints and on the overall reward (Sections 2 and 3). 5. Stage-level metrics grounded in the answer key. For four of the five stages we introduce metrics that score what the agent submits rather than what it did or mentioned: judge- matched ATT&CK mapping, the tables the submitted query reads against those that record the attack in the store, whether that query was tested before submission, and the detection itself, executed in the telemetry store and judged against the answer key. They complement the benchmark’s stage scores, or correct them where they mismeasure. We do not define a new overall reward. The metrics show that different models lead different stages, which argues for routing each stage to a different model rather than choosing one model by its overall reward, and building them exposes tasks whose answer key does not match the benchmark’s own data (Section 4). 6. Output token usage. We count the output tokens behind every reported reward, thinking included and checked against the visible text, and relate the volume and the thinking share to the reward (Section 3.1). Organisation. Section 2 describes the benchmark and how we ran it: the five checkpoints and the reward, the judge panel, the message budget and how cost is counted. A reader who wants only the outcome can skip it and go directly to Section 3, which gives the results: reward against cost at the unconstrained budget (Figure 1) and at the published limit (Figure 2), and the comparison between the leading models. Section 3.1 reports how many tokens a run generates and how much of that is thinking (Figure 3). Section 4 introduces our own metrics for C1 to C4 beside the benchmark’s own score and diagnostics for every checkpoint (Tables 3–7) and the answer-key defects they expose. Section 5 concludes, and Section 3 also lists, in Table 2, the benchmark’s scores of every model on every checkpoint. 2 Evaluation Setup All results are on CTI-REALM (Chakraborty et al., 2026), over the same fixed set of 50 agentic tasks spanning Linux, cloud and Kubernetes (Azure Kubernetes Service, AKS) emulations. Each task runs a five-checkpoint detection-engineering workflow: C0 retrieve and read the threat-intelligence report, C1 map it to techniques of MITRE ATT&CK, a public catalogue of adversary techniques, C2 discover which telemetry tables hold the evidence, C3 execute queries against the store, and C4 produce a KQL detection and a Sigma rule, a detection written in an open rule format that is portable across SIEMs. We run the benchmark’s public implementation, the cti realm50 task of inspectevals, with its tasks, answer keys and judge prompts. The one change is the judge: the judged checkpoints are graded by the panel described below in place of the implementation’s default judge, GPT-5-mini, so our absolute scores are not directly comparable with those reported by Chakraborty et al. (2026). 3 Reward. For one task, the benchmark’s overall reward is R = 1 10 4 X j=0 w j s j ,(w 0 , . . . , w 4 ) = (1.25, 0.75, 1.00, 0.50, 6.50).(1) Here, the checkpoint scores and their weights satisfy s j ∈ [0, 1], 4 X j=0 w j = 10. Table 2 reports each checkpoint score on its own 0–1 scale rather than as its weighted contribution to the reward, so that it can be read against the checkpoint’s own ceiling; overall is the reward. Note that the weight of C4 alone is 6.5 of the ten points, so the final detection carries more weight than the four preceding checkpoints combined. C4 itself has two parts: the F1, against the answer key, of the result rows the agent reports in its answer for its submitted KQL query (5.0 of its 6.5 points), and the judged quality of the Sigma rule (1.5 points): s 4 = 5.0 F1 query + 1.5 q Sigma 6.5 . Here, the numerator combines the query F1 and the judged quality of the Sigma rule. The C4 column of Table 2 reports the query F1; the Sigma part enters overall but has no column. A sample is one model’s attempt at one task, and it is truncated when it exhausts its message bud- get. A truncated sample enters Eq. (1) with every checkpoint score at zero, not merely the last: it never reaches submit(), and the scorer sees no submission to grade, so checkpoints the agent had already passed score zero as well. Truncation therefore acts on the reward through all five terms at once, which is why the number of truncated samples is the first thing to check when a reward at the published limit looks low. We report the sample mean of the reward over the 50 tasks, ˆ R = 1 50 X task i R i . This mean estimates the reward this workflow would obtain on the population of intrusions the 50 tasks are drawn from, and the estimate and that population value should not be conflated: Section 3 gives paired bootstrap intervals for the gaps between the leading models. Judging. C0 and the Sigma part of C4 (1.5 of its 6.5 points) are graded by LLM judges; the other checkpoints are scored against the answer key, so 72.5% of the reward weight involves no judge. The benchmark’s implementation grades each judged answer with a single call to one judge, GPT-5-mini, at temperature zero. One call is a noisy estimate of a judgement, and temperature zero is not recommended for current reasoning models; several, GPT-5-mini among them, no longer accept it. We therefore grade each judged answer with nine trials, three from each of three judges, Gemma-4-31B with its reasoning enabled, DeepSeek-v4-flash and GLM-5.3-flash, each otherwise at its default sampling and reasoning settings and allowed replies of up to 32,000 tokens rather than the implementation’s 5,000, which a reasoning judge can exceed, and average within each judge before averaging across the three, so that every judge carries equal weight: ˆs j = 1 |J | X k∈J 1 T T X t=1 s (k,t) j ,|J | = 3, T = 3,(2) where the outer sum runs over the three judges and the inner one over each judge’s three trials. A judge reply that carries no verdict is asked again, up to three times. Judges that are also evaluated. Three of the 26 models, DeepSeek-v4-flash, Gemma-4-31B and GLM-5.3-flash, are also judges on the panel and are graded by it like every other model. GLM-5.3- flash, the model with the highest reward, receives the same reward to two decimals whether or not it is among its own judges. 4 Thinking. Every model runs at its default reasoning setting, with two exceptions. Gemma-4-31B and Gemma-4-26B-A4B have thinking off by default, which would compare models that cannot think with models that can, so we run both with thinking enabled. With thinking off, they score 0.39 and 0.35 at the published limit, against 0.52 and 0.47 with thinking on. None of Gemma-4-31B’s 50 samples reached the cap, so its unconstrained result is carried under the rule stated next. One of Gemma-4-26B-A4B’s samples reached the cap at the published limit and was re-run at 300, the splice described below. Claude-Haiku-4.5 does not think at its default setting and is the one model evaluated without thinking. Message budget. Each task is capped at a number of agent messages, its message budget, and a sample that exhausts the budget has reached the cap. We report two settings: the published limit of 70 and an unconstrained budget of 300. Where no sample of a model reached the cap at a lower budget, a higher budget cannot change its result. Such a result is carried: it is drawn with a hollow marker in Figure 1 and is not a run at the budget shown. No budget is called “unlimited”: trajectory length is stochastic, and GLM-5.3-flash, which ran two samples to a 250 cap, finished those same two in 91 and 60 messages when given 500. Composition of the unconstrained results. Of the 26 models, 14 have samples run at budget 300 itself, 4 of them all 50 samples and 10 only the samples that had reached a lower cap, and 12 have a result carried from a lower budget under the rule above. Every carried result comes from a run with zero samples at the cap, so a higher budget could not have changed it, but these results are not runs at 300, and Figure 1 draws them with hollow markers. Of the 7 closed models, Claude-Opus-5 and Grok-4.6 have samples run at 300. Of those 14, 4 are complete 50-sample runs at budget 300 and 10 are spliced: GLM-5.3-flash, Kimi-K3, Grok-4.6, Claude-Opus-5, Qwen3.8-2.4T, DeepSeek-v4-pro, Nemotron-3-Ultra, Gemma- 4-26B-A4B, GPT-OSS-120B, GPT-OSS-20B. A splice applies the carry rule per sample rather than per model: the model was run at a lower budget, the samples that finished within that budget were kept unchanged because a higher budget cannot alter them, and only the samples that reached the cap were re-run at 300. The splice rests on the same argument as the carry rule, applied one sample at a time. We name the spliced models because the distinction is invisible in Table 2 and in Figure 1: a spliced model is not marked as carried, since part of its row is a run at 300. Cost. The cost of a run is the cost of its model calls over the 50 tasks behind the reported reward, at list prices and never at a discounted or blended rate. Table 2 shows it in whole dollars, or in cents below a dollar, and the text follows the same convention; a ratio between two costs is computed from the unrounded costs. No cache-read rate is published for the GPT-5.6 family, so we price its cache reads at zero; these runs are 81–82% cache reads, and their costs are therefore lower bounds rather than estimates. 3 Overall Results Figures 1 and 2 plot the overall reward of all 26 models against the cost of the run, first at the unconstrained budget of 300 messages and then at the published limit of 70. Table 2 lists the numbers behind the first figure, together with the benchmark’s five checkpoint scores. We observe that the reward varies far less across the models than the cost of a run does. At the unconstrained budget the 26 rewards span 0.27 to 0.74, a factor of about 2.8 from the lowest reward to the highest, while the cost of a fully priced run ranges from 51¢ (GPT-OSS-20B) to $50 (Claude-Opus-5), a factor of about 98; GPT-5.6-luna’s 28¢ is a lower bound. Among the leading models, additional spending yields no measurable gain in reward on this benchmark. The two figures have different leading models. We observe that at the unconstrained budget the highest reward is that of GLM-5.3-flash, 0.74 at $2, and that at the published limit it is that of Kimi- K3, 0.69 at $27. Both leaders are open-weight. At the published limit Kimi-K3 leads the best closed model, GPT-5.6-sol, by 0.010, with a 95% interval of [−0.042, 0.055] from the paired bootstrap described next. The lead at the unconstrained budget is also narrow: GLM-5.3-flash is 0.03 ahead of Kimi-K3, which ranks second. A paired bootstrap (10,000 replicates, resampling the 43 emulated intrusions the 50 tasks come from, so that tasks from one intrusion stay together) puts the gap at 0.025, with a 95% interval of [0.002, 0.057]. This interval reflects which tasks were sampled, not 5 25¢50¢$1$2$5$10$20$50 cost of the run (USD, log scale) 0.3 0.4 0.5 0.6 0.7 overall reward budget 300 GLM-5.3-flash Kimi-K3 Grok-4.6 GLM-5.3 Claude-Opus-5 Qwen3.8-27B GPT-5.6-sol Mistral-Large-4 GLM-5.2 Qwen3.8-2.4T DeepSeek-v4-pro GPT-5.6-terra DeepSeek-v4.1-flash Claude-Sonnet-5 DeepSeek-v4-flash MiniMax-M3 Inkling Nemotron-3-Ultra GPT-5.6-luna Gemma-4-31B Inkling-Small Qwen3.6-35B-A3B Claude-Haiku-4.5 Gemma-4-26B-A4B GPT-OSS-120B GPT-OSS-20B Figure 1: Overall reward against the cost of the run at the unconstrained budget of 300 agent mes- sages. This figure and Figure 2 share axes so that they can be compared. Open-weight models are drawn as blue circles and closed models as grey triangles. A hollow marker is a carried result: no sample of that model reached the cap at a lower budget, so the point is its result at that budget and not a run at 300. The highest reward belongs to GLM-5.3-flash, an open-weight model, at $2. Note that the least expensive open-weight run shown is GPT-OSS-20B at 51¢. the variation between runs of the same model. The gap also depends on which of GLM-5.3-flash’s runs is reported. Figure 1 and Table 2 report its budget-150 run with the two truncated samples re-run at 300. Its run made entirely at budget 300 scores 0.703, which would place it third, 0.010 behind Grok-4.6 (interval [−0.062, 0.027]) and 0.014 behind Kimi-K3 (interval [−0.051, 0.015]), intervals that both include zero. Kimi-K3, Claude-Opus-5 and Grok-4.6 have one run each above the published limit. We therefore regard the leading models as sharing the top of the ranking. The cost gap is far larger: at the unconstrained budget Kimi-K3’s run costs 19 times as much as GLM- 5.3-flash’s. 6 25¢50¢$1$2$5$10$20$50 cost of the run (USD, log scale) 0.3 0.4 0.5 0.6 0.7 overall reward budget 70 Kimi-K3 GPT-5.6-sol Claude-Opus-5 Grok-4.6 GPT-5.6-terra Qwen3.8-27B Qwen3.8-2.4T GLM-5.3 GLM-5.3-flash Claude-Sonnet-5 DeepSeek-v4-pro Mistral-Large-4 GLM-5.2 DeepSeek-v4.1-flash MiniMax-M3 Inkling GPT-5.6-luna Nemotron-3-Ultra Gemma-4-31B DeepSeek-v4-flash Inkling-Small Claude-Haiku-4.5 Qwen3.6-35B-A3B Gemma-4-26B-A4B GPT-OSS-120B GPT-OSS-20B Figure 2: Overall reward against the cost of the run at the benchmark’s published limit of 70 agent messages. This figure and Figure 1 share axes so that they can be compared. Open-weight models are drawn as blue circles and closed models as grey triangles. The highest reward belongs to Kimi-K3, an open-weight model, at $27. Note that the least expensive open-weight run shown is GPT-OSS- 20B at 41¢. The message cap is why we report the unconstrained budget. A truncated sample never reaches submit() and scores zero on every checkpoint, including ones it had already passed, so at the published limit a model with a dozen truncated samples is measured on termination as much as on capability. DeepSeek-v4-flash is the clearest case: 12 of its 50 samples were truncated at 70 messages and none at 250, the budget from which its unconstrained result is carried, and its reward rises from 0.51 to 0.62, the largest gain of any model after GLM-5.3-flash’s. A comparison at the published limit therefore measures termination and capability together, which is why we read the results at a budget where no reported sample is truncated. 7 Model C0C1C2C3C4overall cost judgedJaccardJaccardran≥2query F1 reward GLM-5.3-flash0.880.730.870.980.620.74$2 Kimi-K30.860.720.850.980.580.72$32 Grok-4.60.570.490.870.980.680.71$28 GLM-5.30.900.700.860.960.540.70$20 Claude-Opus-50.960.710.851.000.500.69$50 Qwen3.8-27B0.930.730.880.980.500.68$10 GPT-5.6-sol0.950.430.870.800.540.68$9 Mistral-Large-40.950.720.891.000.470.68$22 GLM-5.20.870.670.900.920.480.66$13 Qwen3.8-2.4T0.920.700.871.000.440.66$17 DeepSeek-v4-pro0.860.660.890.980.450.65$24 GPT-5.6-terra0.880.340.890.520.540.65$3 DeepSeek-v4.1-flash0.880.670.911.000.430.64$5 Claude-Sonnet-50.880.640.870.960.420.63$31 DeepSeek-v4-flash0.880.580.930.960.440.63$1 MiniMax-M30.870.630.870.900.360.58$5 Inkling0.750.460.870.820.390.57$9 Nemotron-3-Ultra0.730.480.880.780.360.56$17 GPT-5.6-luna0.860.370.900.560.350.5528¢ Gemma-4-31B0.730.230.870.540.380.5270¢ Inkling-Small0.700.300.790.500.360.50$2 Qwen3.6-35B-A3B0.840.420.890.820.240.49$2 Claude-Haiku-4.50.490.420.890.540.330.48$7 Gemma-4-26B-A4B0.570.260.840.480.350.47$1 GPT-OSS-120B0.490.200.740.160.180.34$3 GPT-OSS-20B0.530.140.610.160.110.2751¢ Table 2: Results for all 26 models at the unconstrained budget of 300 agent messages, ordered by overall reward. Columns C0 to C4 are the benchmark’s own five checkpoint scores, each on its own 0–1 scale: C0 is the judged score for reading the threat-intelligence report, C1 and C2 are the Jaccard scores for the ATT&CK techniques and the telemetry tables, C3 is the share of tasks in which at least two queries returned rows, and C4 is the F1 of the detection query. Overall is the weighted reward of Eq. (1); it also counts the judged quality of the Sigma rule, which has no column here. No sample reaches the message cap at this budget. Cost is the total cost of the run. In each score column the three best values are marked first, second and third, ranked on the precision shown so that equal printed values share a rank; the cost column, for which a lower value is better, is left unranked. Price and capability among the leading models. The ten highest-scoring models span 0.66 to 0.74 in reward. Their runs cost $2 to $50. We observe that within this group the cost of a run varies by more than an order of magnitude: GLM-5.3-flash at $2 and Claude-Opus-5 at $50 differ by 0.05 in reward (0.743 against 0.690). Note that we do not claim price never tracks capability. Across all 26 models it does (Spearman correlation 0.64 between cost and reward); among the ten leading models it does not (0.14). The claim is narrower and more useful to a buyer: among the models good enough to consider for this task, the price of a model is not evidence about which one to choose. 8 3.1 Output Token Usage 0250k500k750k1M1.25M1.5M1.75M output tokens per run of 50 tasks Mistral-Large-4 Qwen3.8-27B GLM-5.3-flash GLM-5.3 DeepSeek-v4.1-flash DeepSeek-v4-pro Claude-Sonnet-5† Kimi-K3 MiniMax-M3 Qwen3.8-2.4T Claude-Opus-5† GLM-5.2 Qwen3.6-35B-A3B Claude-Haiku-4.5 Grok-4.6 Nemotron-3-Ultra GPT-OSS-120B Gemma-4-26B-A4B DeepSeek-v4-flash GPT-5.6-sol Inkling GPT-OSS-20B Gemma-4-31B GPT-5.6-luna GPT-5.6-terra Inkling-Small 63% 61% 34% 36% 64% 63% ≥34% 15% 30% 54% ≥10% 12% 51% 0% 42% 18% 43% 46% 23% 18% 23% 11% 50% 18% 17% 18% (a) per run 010k20k30k40k50k60k70k80k output tokens per task (b) per task thinkinganswer and tool callsopen-weightclosed (grey, same split) Figure 3: Tokens generated per run and per task, at the unconstrained budget. (a) output tokens summed over the 50 tasks, split into thinking (dark) and the answer and tool calls (light); the number at the end of each bar is the thinking share, the fraction of output tokens spent on thinking. Blue marks open-weight models and grey closed ones, with the marker shapes of Figure 1. (b) the same output spread over the 50 tasks: the box spans the middle half of the tasks, the line is the median and the whiskers reach the 10th and 90th percentiles. The carried models are those drawn with hollow markers in Figure 1. † marks a thinking share that is a lower bound. Scorer and judge-panel calls are excluded. Figure 3 counts the output tokens each model generates at the unconstrained budget, thinking in- cluded: per run in panel (a) and per task in panel (b). A count is the reported output-token total, summed over every model call in the 50 trajectories behind the reward in Table 2; calls made by the benchmark’s in-run scorer and by the judge panel are excluded. We verify rather than assume that thinking is included in that count. On the 19 models that return their thinking as text, the visible thinking, answer and tool-call text comes to 3.6 to 4.2 characters per counted token, the usual rate for English text and code, and the reported thinking exceeds the reported output on at most 0.36% of any run’s output. Grok-4.6, GPT-5.6-sol, GPT-5.6-terra and GPT-5.6-luna return their thinking encrypted; for these models the reported output minus the reported thinking matches the visible text at 3.9 to 4.2 characters per token, so the reported thinking is exactly the hidden part. Claude-Opus-5 and Claude-Sonnet-5 return no thinking text, so their split cannot be checked and we report their thinking share as a lower bound. We observe that a run of 50 tasks generates 187k (Inkling-Small) to 1.59M (Mistral-Large-4) output tokens, an 8.5-fold range, and that volume tracks reward only moderately (Spearman correlation 0.59 over the 26 models). The ten leading models reach their rewards at very different volumes: 9 among them, output varies 5.1-fold, from 313k (GPT-5.6-sol) to 1.59M (Mistral-Large-4). Note that the relation does hold at the low end: the 6 runs that generate least have a median reward of 0.54, against 0.65 for the rest. Low volume is associated with a low reward, but high volume is not associated with a high one. Thinking takes 11% (GPT-OSS-20B) to 64% (DeepSeek-v4.1-flash) of the output among the 23 models whose share is measured, with lower bounds of 10% and 34% for Claude-Opus-5 and Claude-Sonnet-5, and Claude-Haiku-4.5 does not think at all. The thinking share says little about the reward (Spearman correlation 0.07 over the 26 models). Among the leading models, GLM-5.2 spends 12% of its output thinking and Mistral-Large-4 63%, and the two differ by 0.02 in reward. The thinking share describes how a model works, not how well it performs on this benchmark. Panel (b) shows that a run’s output is not spread evenly over its tasks. For the median model the 90th-percentile task generates 3.2 times the output of the 10th-percentile task, and the five heaviest tasks, a tenth of the run, account for 22% of its output. Per-task medians range from 3k (Inkling- Small) to 20k (Mistral-Large-4). GLM-5.3 has the widest spread, a ratio of 9.6 between those two percentiles, and MiniMax-M3 the most concentrated run, with 44% of its output in its five heaviest tasks. A total per run therefore hides a long tail of expensive tasks. Output is a small part of what a run processes. Each turn re-reads the growing transcript, so the input of a run comes to 10M to 62M tokens and output is 1.1% to 3.2% of all tokens processed; 4% to 90% of the input is read from cache. The cost of a run is therefore set by the price per token far more than by how much the model writes: GLM-5.3-flash generates 1.5 times as much output as Claude-Opus-5, and its run costs $2 against $50. 4 Results by Checkpoint At the unconstrained budget, each checkpoint’s table gives the benchmark’s scores, diagnostics we compute with its definitions and, for C1 to C4, our metrics, which score the agent’s submitted KQL query and Sigma rule. They complement the benchmark’s measures of exploration, querying and the form of the rule for C2, C3 and the Sigma rule, and correct how it measures the same quantity for C1 and the C4 query. They are computed from the stored trajectories without re-running any agent, do not enter the reward, and average per-task values, so a model’s F1 can lie below both its precision and its recall. For C1 to C4, a figure draws the key metrics of the stage, the benchmark’s and ours, for the five models that rank highest on it; the table keeps every metric. Of these, C1, the Sigma rule, the C4 queries whose rows cannot be matched to events and the C2 tables that the key’s reference queries leave open are graded as in Eq. (2) by Gemma-4-31B with reasoning enabled, DeepSeek-v4.1-flash and GLM-5.3-flash, the reward’s panel with DeepSeek-v4.1-flash in place of DeepSeek-v4-flash; each judge is also an evaluated model. 4.1 C0: Threat-Intelligence Research C0 asks the agent to find a relevant threat-intelligence report and explain why; the benchmark’s judge prompt, applied by the reward’s panel (Section 2), grades the research’s relevance to the detection objective. The answer key holds no reference for C0, and in 18 of the 50 tasks no report in the corpus is judged by both GLM-5.3-flash and DeepSeek-v4.1-flash to describe the target behaviour, so we add no metric of our own. Every model opens a report on at least 0.94 of its tasks, so access to the corpus does not explain the spread in the score; reports opened per task correlate moderately with it (Spearman correlation 0.57; Table 3). 10 Model Benchmark score and diagnostics score (judged) opened any reportreports per task GLM-5.3-flash0.881.0010.8 Kimi-K30.861.008.9 Grok-4.60.571.0016.0 GLM-5.30.901.0012.0 Claude-Opus-50.961.006.3 Qwen3.8-27B0.931.0010.9 GPT-5.6-sol0.951.006.9 Mistral-Large-40.951.0013.0 GLM-5.20.871.007.5 Qwen3.8-2.4T0.921.0010.6 DeepSeek-v4-pro0.861.0011.4 GPT-5.6-terra0.881.005.2 DeepSeek-v4.1-flash0.881.0018.3 Claude-Sonnet-50.881.0011.1 DeepSeek-v4-flash0.881.007.6 MiniMax-M30.870.985.5 Inkling0.750.987.9 Nemotron-3-Ultra0.731.006.9 GPT-5.6-luna0.861.005.0 Gemma-4-31B0.731.003.4 Inkling-Small0.700.964.7 Qwen3.6-35B-A3B0.841.007.4 Claude-Haiku-4.50.491.004.5 Gemma-4-26B-A4B0.570.942.1 GPT-OSS-120B0.490.943.0 GPT-OSS-20B0.530.983.3 Table 3: C0, threat-intelligence research, at the unconstrained budget. Benchmark: the score of the agent’s research, graded with the benchmark’s judge prompt by the reward’s panel (Section 2), and two diagnostics we compute, the share of tasks in which the agent opened at least one report and the reports it opened per task. The answer key holds no reference for C0, so there is no metric of ours. Rows follow the benchmark’s overall reward (Table 2); colours mark the first, second and third best values of the score, equal values sharing a colour. 11 judged score best 0.96 opened any report best 1.00 reports per task (over max, 18.3) best 1.00 0.2 0.4 0.6 0.8 1.0 best value of any of the 26 models on each axis 1. GLM-5.3-flash (overall reward 0.74) 2. Kimi-K3 (overall reward 0.72) 3. Grok-4.6 (overall reward 0.71) 4. GLM-5.3 (overall reward 0.70) 5. Claude-Opus-5 (overall reward 0.69) Figure 4: The three columns of Table 3 at the unconstrained budget, for the five models with the highest overall reward, in the colours of the figures for C1 to C4; reports per task are divided by the largest value any model reaches. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. 12 exact match best 0.68 macro F1 best 0.63 Sigma-tag technique F1 (ours) best 0.79 0.2 0.4 0.6 0.8 1.0 best value of any of the 26 models on each axis 1. GLM-5.3-flash (overall reward 0.74) 2. Kimi-K3 (overall reward 0.72) 3. Grok-4.6 (overall reward 0.71) 4. GLM-5.3 (overall reward 0.70) 5. Claude-Opus-5 (overall reward 0.69) Figure 5: Key metrics of C1 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 4 gives every metric for every model. 4.2 C1: ATT&CK Technique Mapping C1 concerns the mapping of the campaign to ATT&CK techniques, whose identifiers the task never asks for. The benchmark nonetheless collects the key’s identifiers anywhere in the agent’s output, reasoning included, so its Jaccard index with the key equals the recall of the key: a wrong identifier costs nothing, and with exact matching a sub-technique of a key technique earns no credit. We score instead the mapping the agent commits to, the techniques tagged in its submitted Sigma rule, as Sigma rules conventionally carry ATT&CK tags; an untagged rule, about one in eight, scores zero. A tag counts in proportion to the judge trials that match it to a key technique, whatever its identifier; one the judges accept but the key omits is neither credited nor penalised, as the key gives a single technique for 38 of the 50 tasks. A judge repeats its verdict in all three trials for 89 to 93% of tags, and the judges’ majority verdicts agree on 89%. The three GPT-5.6 models score 0.23 to 0.36 higher on our F1 than on the benchmark’s score (Table 4 and Figure 5). 13 Model Benchmark score and diagnosticsSigma-rule tags, judged (ours) Jaccard (key recall) exact matchmacro F1precisionrecall F1 (Sigma rule’s techniques) GLM-5.3-flash0.730.660.580.720.830.74 Kimi-K30.720.620.630.790.790.76 Grok-4.60.490.440.390.710.780.71 GLM-5.30.700.600.590.750.850.75 Claude-Opus-50.710.660.560.620.850.68 Qwen3.8-27B0.730.680.470.630.690.63 GPT-5.6-sol0.430.380.410.830.730.74 Mistral-Large-40.720.660.580.790.820.78 GLM-5.20.670.600.530.620.720.64 Qwen3.8-2.4T0.700.640.540.810.810.79 DeepSeek-v4-pro0.660.560.580.780.700.71 GPT-5.6-terra0.340.280.330.800.670.70 DeepSeek-v4.1-flash0.670.600.600.720.820.73 Claude-Sonnet-50.640.580.540.770.810.76 DeepSeek-v4-flash0.580.520.470.700.760.69 MiniMax-M30.630.580.450.600.690.60 Inkling0.460.400.330.620.620.59 Nemotron-3-Ultra0.480.440.390.610.610.59 GPT-5.6-luna0.370.340.310.630.610.60 Gemma-4-31B0.230.220.170.340.300.30 Inkling-Small0.300.260.220.520.440.44 Qwen3.6-35B-A3B0.420.360.410.430.430.40 Claude-Haiku-4.50.420.400.290.500.550.52 Gemma-4-26B-A4B0.260.240.190.280.280.26 GPT-OSS-120B0.200.160.170.150.140.14 GPT-OSS-20B0.140.140.120.060.060.06 Table 4: C1, ATT&CK technique mapping, at the unconstrained budget. Benchmark: the bench- mark’s Jaccard index between the key and the key’s technique identifiers found anywhere in the agent’s output, which therefore equals the recall of the key, and two diagnostics we compute, the share of tasks matched exactly and the macro F1 (F1 per technique class, averaged with equal weight). Sigma-rule technique tags (ours): F1 scores the technique mapping the agent commits to, the ATT&CK techniques tagged in its submitted Sigma rule, against the key. In each task, precision is the share of the rule’s tags that the judges match to a key technique, whatever the identifier; recall is the share of the key’s techniques that some tag matches; F1 is their harmonic mean. A tag counts in proportion to the trials, three LLM judges in three trials each, that match it; a tag the judges accept but the key omits counts in neither; an untagged rule scores zero. Per-task values are averaged over the 50 tasks, so F1 can lie below both precision and recall. Rows follow the benchmark’s overall reward (Table 2); colours mark the first, second and third best values of the Jaccard, the exact match, the macro F1 and our F1, equal values sharing a colour. 14 4.3 C2: Telemetry Table Discovery C2 asks the agent to find the telemetry tables that hold the evidence. The benchmark’s Jaccard index is again the recall of the key, here over the tables the agent passed to its two exploration tools, a set it never commits to, so inspecting all 12 tables scores 1 and a table queried without being inspected earns nothing. We score the tables the submitted query reads by precision, recall and F1 against the tables that record the attack in the store. A key table is one of them when its reference query returns events or, since a reference query can miss events that are there, when the judges, shown the rows of that table that meet the key’s conditions or carry its identifiers, find the attack in them. Of the 16 key tables whose reference queries return nothing, the judges find the attack in 9; in the two tasks on restricted administrative units, for instance, the attack’s AuditLogs events leave empty the actor field on which the reference query filters. A table that holds copies of a key table’s events, with the same event identifiers, stands in for it, as AKSAuditAdmin does for AKSAudit in 10 of the 16 AKS tasks in which a submission reads it; a table outside the key counts in proportion to the judge trials that find the attack in it and against precision otherwise, and is never required. The judges’ majority verdicts agree on 84% of the 19 tables they grade. Our F1 is at least 0.82 for every model except GPT-OSS-20B, at 0.48 (Table 5 and Figure 6). 15 Model Benchmark score and diagnosticsTables queried, judged (ours) Jaccard (key recall) exact matchmacro F1precisionrecall F1 (submitted query’s tables) GLM-5.3-flash0.870.720.831.000.940.96 Kimi-K30.850.700.831.000.910.94 Grok-4.60.870.720.830.980.900.92 GLM-5.30.860.680.790.990.930.95 Claude-Opus-50.850.680.771.000.920.95 Qwen3.8-27B0.880.740.850.980.890.92 GPT-5.6-sol0.870.720.810.960.920.92 Mistral-Large-40.890.780.880.960.940.94 GLM-5.20.900.780.890.980.920.94 Qwen3.8-2.4T0.870.720.810.980.890.92 DeepSeek-v4-pro0.890.760.870.990.920.94 GPT-5.6-terra0.890.760.860.990.920.94 DeepSeek-v4.1-flash0.910.800.910.990.920.94 Claude-Sonnet-50.870.740.830.990.900.93 DeepSeek-v4-flash0.930.840.931.000.910.94 MiniMax-M30.870.740.860.980.890.92 Inkling0.870.740.780.960.870.90 Nemotron-3-Ultra0.880.720.840.970.890.92 GPT-5.6-luna0.900.780.890.930.890.88 Gemma-4-31B0.870.700.731.000.910.94 Inkling-Small0.790.640.520.940.850.88 Qwen3.6-35B-A3B0.890.740.860.930.860.88 Claude-Haiku-4.50.890.740.851.000.880.92 Gemma-4-26B-A4B0.840.700.680.970.880.91 GPT-OSS-120B0.740.620.650.870.790.82 GPT-OSS-20B0.610.500.390.540.450.48 Table 5: C2, telemetry table discovery, at the unconstrained budget. Benchmark: the benchmark’s Jaccard index between the key and the key tables the agent passed to its two exploration tools, which therefore equals the recall of the key, and two diagnostics we compute, the share of tasks matched exactly and the macro F1 (F1 per table, averaged with equal weight). Tables read by the submitted query (ours): precision, recall and F1 of the tables the submitted query reads, against the tables that record the attack in the store: key tables whose reference queries return events or in which three LLM judges, in three trials each, find the attack, with copies of their events standing in for them, and tables outside the key in proportion to the judge trials that find the attack in them; per-task values averaged over the 50 tasks, so F1 can lie below both precision and recall. Rows follow the benchmark’s overall reward (Table 2); colours mark the first, second and third best values of the Jaccard, the exact match, the macro F1 and our F1, equal values sharing a colour. 16 exact match best 0.84 macro F1 best 0.93 submitted-query table F1 (ours) best 0.96 0.6 0.8 1.0 best value of any of the 26 models on each axis 1. GLM-5.3-flash (overall reward 0.74) 2. Kimi-K3 (overall reward 0.72) 3. Grok-4.6 (overall reward 0.71) 4. GLM-5.3 (overall reward 0.70) 5. Claude-Opus-5 (overall reward 0.69) Figure 6: Key metrics of C2 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 5 gives every metric for every model. 4.4 C3: Query Execution C3 asks the agent to execute queries against the telemetry store. The benchmark passes a task when at least two distinct queries returned rows, which shows that the agent queried the store, not that it tested its submission. We report the share of tasks in which it did: an executed query with the submission’s selection logic, ignoring formatting and operators that only shape the output, returned rows. A near miss, with a text similarity of at least 0.85 to the closest executed query that returned rows, also counts if both, run without row limits, return the same rows on at least two shared columns or an event identifier; 57 of the 81 near misses do. GPT-5.6-terra and GPT-5.6-luna pass the bench- mark’s check on 0.52 and 0.56 of their tasks but tested their submission on 1.00 and 0.98, while Nemotron-3-Ultra moves the other way, from 0.78 on the benchmark’s check to 0.48 (Table 6 and Figure 7). 17 Model Benchmark score and diagnostics Submitted query (ours) two distinct queries returned rows queries per task no query returned rows submitted query tested GLM-5.3-flash0.9811.70.000.88 Kimi-K30.986.60.000.98 Grok-4.60.988.50.000.96 GLM-5.30.9611.00.000.92 Claude-Opus-51.006.30.000.90 Qwen3.8-27B0.9812.30.000.98 GPT-5.6-sol0.804.70.001.00 Mistral-Large-41.0014.50.001.00 GLM-5.20.929.00.000.90 Qwen3.8-2.4T1.007.00.000.98 DeepSeek-v4-pro0.988.90.001.00 GPT-5.6-terra0.524.00.001.00 DeepSeek-v4.1-flash1.0012.00.000.96 Claude-Sonnet-50.968.90.000.96 DeepSeek-v4-flash0.969.70.040.70 MiniMax-M30.9011.20.020.78 Inkling0.825.30.020.74 Nemotron-3-Ultra0.786.90.020.48 GPT-5.6-luna0.564.10.000.98 Gemma-4-31B0.543.60.040.84 Inkling-Small0.502.80.060.70 Qwen3.6-35B-A3B0.827.30.040.74 Claude-Haiku-4.50.543.20.140.54 Gemma-4-26B-A4B0.484.10.060.46 GPT-OSS-120B0.162.50.100.62 GPT-OSS-20B0.162.70.360.40 Table 6: C3, query execution, at the unconstrained budget. Benchmark: the share of tasks in which at least two distinct queries returned rows, and two diagnostics we compute, the queries executed per task and the share of tasks in which no query returned rows. Submitted query tested (ours): the share of the 50 tasks in which a query the agent executed, with the selection logic of its submitted KQL query, returned rows, a near miss counting when it returns the same rows. Rows follow the benchmark’s overall reward (Table 2); colours mark the first, second and third best values of the benchmark’s score and of ours, equal values sharing a colour. 18 two distinct queries returned rows best 1.00 queries per task (over max, 14.5) best 1.00 submitted query tested (ours) best 1.00 0.4 0.6 0.8 1.0 best value of any of the 26 models on each axis 1. GLM-5.3-flash (overall reward 0.74) 2. Kimi-K3 (overall reward 0.72) 3. Grok-4.6 (overall reward 0.71) 4. GLM-5.3 (overall reward 0.70) 5. Claude-Opus-5 (overall reward 0.69) Figure 7: Key metrics of C3 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 6 gives every metric for every model. Queries per task are divided by the largest value any model reaches. 4.5 C4: Detection C4 asks for the deliverable, a KQL query and a Sigma rule. The benchmark never runs the query: it computes an F1 over the key’s field patterns on the rows the agent pasted into its answer, so the score depends on the output format, one fully matching row scores 1, and recall of the malicious events is never measured. We execute the submitted and reference queries in full in the telemetry store and report the precision, recall and F1 of the submission’s events against theirs, on the 36 tasks in which every reference query returns events. Submissions returning counts or other aggregates, about one in ten, cannot be matched to events; the judges grade them against the key as equivalent (1), broader or narrower (0.5), or a miss or invalid (0), and the grade serves as their precision, recall and F1. Precision falls with events the reference queries do not return, recall with those a query misses; their medians are 0.64 and 0.74. For the rule, the benchmark’s judge prompt, applied by the reward’s panel, rates syntax and speci- ficity by a generic rubric that never consults the answer key. Our judges grade it against the key’s reference queries and field patterns, on the scale used above for aggregate queries, without requiring the test environment’s marker strings, account names or IP addresses. A judge repeats its grade in 19 Model Benchmark scores and diagnosticEvents and Sigma grade (ours) query F1 zero query F1 Sigma (judged) event precision event recall event F1 Sigma grade (judged) GLM-5.3-flash0.620.180.880.680.850.700.88 Kimi-K30.580.240.870.720.800.700.90 Grok-4.60.680.120.840.780.860.780.88 GLM-5.30.540.260.870.570.700.570.90 Claude-Opus-50.500.280.880.720.800.710.85 Qwen3.8-27B0.500.320.860.650.770.660.85 GPT-5.6-sol0.540.300.900.780.780.740.91 Mistral-Large-40.470.400.870.660.830.680.88 GLM-5.20.480.340.820.650.740.640.85 Qwen3.8-2.4T0.440.320.890.640.730.630.88 DeepSeek-v4-pro0.450.240.860.750.860.750.88 GPT-5.6-terra0.540.320.870.740.790.700.89 DeepSeek-v4.1-flash0.430.280.860.640.780.650.90 Claude-Sonnet-50.420.400.850.720.770.710.86 DeepSeek-v4-flash0.440.380.760.510.570.490.76 MiniMax-M30.360.520.770.590.650.560.74 Inkling0.390.240.760.490.640.490.71 Nemotron-3-Ultra0.360.500.810.520.580.510.75 GPT-5.6-luna0.350.460.830.770.830.750.88 Gemma-4-31B0.380.340.730.590.720.570.80 Inkling-Small0.360.400.710.590.750.570.66 Qwen3.6-35B-A3B0.240.540.670.430.600.440.68 Claude-Haiku-4.50.330.380.690.490.510.480.62 Gemma-4-26B-A4B0.350.360.670.550.710.550.67 GPT-OSS-120B0.180.600.640.410.450.400.55 GPT-OSS-20B0.110.740.430.220.250.200.32 Table 7: C4, detection, at the unconstrained budget. Benchmark: the F1 over the key’s field patterns of the rows the agent pasted into its answer, without running the query; a diagnostic we compute, the share of tasks whose query F1 is zero; and the quality of the Sigma rule, graded with the benchmark’s judge prompt by the reward’s panel (Section 2). Executed submission and Sigma grade (ours): pre- cision, recall and F1 of the events the submitted query returns when executed in full, against those of the key’s reference queries, with submissions returning aggregates, about one in ten, graded by three LLM judges; per-task values averaged over the 36 tasks in which every reference query returns events, so F1 can lie below both precision and recall; and the grade of the Sigma rule against the key by the same judges, averaged over the 50 tasks. Rows follow the benchmark’s overall reward (Table 2); colours mark the first, second and third best values of the query F1, both Sigma columns and the event F1, equal values sharing a colour. all three trials for 73 to 87% of rules, and the judges’ majority grades agree on 72% (Table 7 and Figure 8). 20 query F1 best 0.68 event F1 (ours) best 0.78 Sigma grade, judged (ours) best 0.91 0.6 0.8 1.0 best value of any of the 26 models on each axis 1. GLM-5.3-flash (overall reward 0.74) 2. Kimi-K3 (overall reward 0.72) 3. Grok-4.6 (overall reward 0.71) 4. GLM-5.3 (overall reward 0.70) 5. Claude-Opus-5 (overall reward 0.69) Figure 8: Key metrics of C4 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 7 gives every metric for every model. 4.6 Effect on the Comparison Comparing medians, open-weight against closed, our metrics reverse the open-weight lead on the benchmark’s C1 (0.63 against 0.43) and C3 (0.92 against 0.80), to 0.63 against 0.70 and 0.84 against 0.96, and widen the closed models’ existing lead on both parts of C4, from 0.43 against 0.50 to 0.57 against 0.71 on the query and from 0.81 against 0.85 to 0.80 against 0.88 on the rule; on C2 the families are level (0.92 against 0.92). On each metric the best open-weight and the best closed model differ by at most 0.04, which should not be read as an ordering: open-weight models lead C1 (Qwen3.8-2.4T at 0.79 against 0.76) and C2 (GLM-5.3-flash at 0.96 against 0.95), Mistral-Large-4 and DeepSeek-v4-pro share 1.00 on C3 with GPT-5.6-sol and GPT-5.6-terra, and closed models lead the C4 query (0.75 against Grok-4.6 at 0.78) and the Sigma rule (0.90 against GPT-5.6-sol at 0.91). Because different models lead different stages (Tables 4 to 7), we do not combine our metrics into a new overall reward; this suggests routing each stage to the model that performs it best as the natural next step. 21 5 Conclusion Open-weight models match the strongest closed models on this benchmark, and GLM-5.3-flash does so for less than a tenth of what those closed models cost. GLM-5.3-flash reaches reward 0.74 at $2 a run against 0.71 for Grok-4.6 at $28 and 0.69 for Claude-Opus-5 at $50. The condition is that the model is given enough turns: at the benchmark’s published limit of 70 messages the leading open-weight and closed models cannot be separated. This dependence on the budget is itself a finding, not only a caveat. The published limit is hard, and a truncated sample scores zero on checkpoints it had already passed, so a comparison there partly measures which models terminate rather than which are capable. The limit constrains the models that take more turns to reach an answer, and these are not necessarily the models that answer worse: DeepSeek-v4-flash had 12 of its 50 samples truncated at 70 messages and none at 250, the budget from which its unconstrained result is carried. This is one reason the published setting orders the leading models poorly. The stage-level metrics of Section 4 add a second conclusion. No model leads every stage: the best technique mapping, table choice, query testing, detection query and Sigma rule come from six different models, three of them tied on query testing, and only GPT-5.6-sol leads two. A single overall reward ranks models for a choice that the workflow does not require, and routing each stage to the model that performs it best is the natural next step. Five of the seven closed models have results carried from a lower budget at which no sample reached the cap, so a higher budget could not have changed them. Each reported result rests on one measure- ment of each of the 50 tasks, so a difference of a few hundredths between two models should not be read as an ordering. In particular, the top of the ranking is shared rather than settled: GLM-5.3-flash’s run made entirely at the unconstrained budget would place it third, behind Kimi-K3 and Grok-4.6, by gaps whose intervals include zero. The cost claim holds under either run: a run at the uncon- strained budget costs little on GLM-5.3-flash and a great deal on the most expensive closed models. The claim therefore rests as much on cost as on reward. Author Contributions Sahil Garg set up and ran the experiments, developed the evaluation metrics introduced in this work, and wrote the paper. Ruhi Gottumukkala, as a summer research intern, carried out the error analy- sis and related the performance of the open-weight models to their architectures and to their scores on general benchmarks, to validate and interpret the results. Vishal Agarwal advised on the bench- mark as a cybersecurity expert and provided the resources that made the work possible. Shubham Vishwakarma led the engineering that set up the benchmark for experiments at scale. Acknowledgements We thank Sridhar Dutta and Manish Varma Datla for their feedback on this benchmark. References Chakraborty, A., Ho, S., Cook, A., and Mel ́ endez, M. (2026).CTI-REALM: Benchmark to evaluate agent performance on security detection rule generation capabilities. arXiv preprint arXiv:2603.13517. 22