Paper deep dive
Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations
Leon Goldberg, Gal Engelberg, Eden Yavin, Elad Elouz, Ariel Zadok, Konstantin Koutsyi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/29/2026, 11:46:24 AM
Summary
This paper evaluates the Sola Security Brain, a purpose-built security intelligence layer with an offline-resolved relational substrate, against general-purpose coding agents (Claude Code and OpenAI Codex) for cloud security investigations. Using 28 tasks on a live AWS environment, the study finds that the Sola Security Brain achieves significantly higher coverage (0.549 vs 0.340 and 0.281) and lower cost (17.8x and 20.8x cheaper) than the coding agents. The authors identify a 'sample-and-generalise' failure mode in coding agents, where they sample a small fraction of a large population and incorrectly assert universal negatives.
Entities (6)
Relation Signals (6)
Sola Security Brain â outperforms â Claude Code
confidence 95% · The Sola Security Brain reaches 0.549 +/- 0.012 coverage against 0.340 +/- 0.011 for Claude Code... It leads 24 of 28 tasks... at 17.8x... lower cost per task than Claude Code
Sola Security Brain â outperforms â OpenAI Codex
confidence 95% · The Sola Security Brain reaches 0.549 +/- 0.012 coverage against... 0.281 +/- 0.006 for Codex... at 20.8x lower cost per task than... Codex
Sola Security Brain â usesarchitecture â relational_substrate
confidence 92% · Sola Security Brain, a security intelligence layer whose relational substrate is resolved offline and whose security logic is evaluated against it at query time
Claude Code â usesarchitecture â live_API_access
confidence 92% · coding agents operating the same live AWS environment through a read-only CLI... reconstructs whatever structure the question needs at the moment it is asked
Claude Code â exhibitsfailuremode â sample-and-generalise
confidence 90% · In one task Claude Code reported that no bucket policies exist after checking four bucket families, in a sweep that sampled 40 of roughly 5,000 buckets...
OpenAI Codex â exhibitsfailuremode â sample-and-generalise
confidence 90% · both agents enumerate a fraction of a large population, assert an unhedged universal negative... In one task Claude Code reported that no bucket policies exist...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cloud-security investigation is dominated by population tasks: which identities can read a data store, how many resources fail a control, what is reachable from another account. These resolve against a complete inventory, not a named object, so a partial answer to one is not a partial result but a different one. Coding agents can now be given read-only cloud credentials and asked to investigate directly, which raises the question of what a purpose-built security context layer still contributes. We evaluate the Sola Security Brain, a security intelligence layer whose relational substrate is resolved offline and whose security logic is evaluated against it at query time, against two coding agents operating the same live AWS environment through a read-only CLI, Claude Code and OpenAI Codex, over 28 investigation tasks. All three are scored by a blinded, tier-weighted, grounding-gated relative recall over one joint claim pool, so the scores share a denominator. The Sola Security Brain reaches 0.549 +/- 0.012 coverage against 0.340 +/- 0.011 for Claude Code and 0.281 +/- 0.006 for Codex, with the ordering identical in every grading draw. It leads 24 of 28 tasks from the cheapest model tier, at 17.8x and 20.8x lower cost per task than Claude Code and Codex. Beyond the aggregate, we describe an answer-level pattern we term sample-and-generalise: both agents enumerate a fraction of a large population, assert an unhedged universal negative, and disclose the sample size only in answer metadata rather than in the answer. In one task Claude Code reported that no bucket policies exist after checking four bucket families, in a sweep that sampled 40 of roughly 5,000 buckets, in an account where 65 buckets carry a wildcard-principal read grant.
Tags
Links
- Source: https://arxiv.org/abs/2609.30345v2
- Canonical: https://arxiv.org/abs/2609.30345v2
Trouble viewing inline? Open PDF directly â
Full Text
59,204 characters extracted from source content.
Expand or collapse full text
CODING AGENTS ARENâT ENOUGH! EVALUATING AN ENTERPRISE SECURITY BRAIN FOR AGENTIC CLOUD INVESTIGATIONS Leon Goldberg, Gal Engelberg, Eden Yavin, Elad Elouz, Ariel Zadok, Konstantin Koutsyi Sola Security, Tel Aviv, Israel Corresponding author: gal.e@sola.security ABSTRACT Cloud-security investigation is dominated by population tasks: which identities can read a data store, how many resources fail a control, which assets are reachable from another account. These resolve against a complete inventory, not a named object. A partial answer to one is not a partial result. It is a different result. General-purpose coding agents can now be given read-only cloud credentials and asked to investigate directly, which raises the question of what a purpose-built security context layer still contributes. We evaluate the Sola Security Brain, a security intelligence layer whose relational substrate is resolved offline and whose security logic is evaluated against it at query time, against two coding agents operating the same live AWS environment through a read-only CLI, Claude Code and OpenAI Codex, over 28 cloud-security investigation tasks. Answers are scored by a blinded, tier-weighted, grounding-gated relative recall over a single joint claim pool built from all three systems, so the three scores share one denominator. The Sola Security Brain reaches 0.549± 0.012 coverage against0.340± 0.011for Claude Code and0.281± 0.006for Codex, across three grading draws whose ordering is identical in every draw. It leads 24 of 28 tasks from the cheapest model tier, at17.8Ăand20.8Ălower cost per task than Claude Code and Codex, and28.7Ăand40.6Ă lower cost per unit of coverage. Beyond the aggregate, we describe an answer-level pattern we term sample-and-generalise: under a turn budget both coding agents enumerate a fraction of a large population, assert an unhedged universal negative, and disclose the sample size only in answer metadata rather than in the answer. In one task Claude Code reported that no bucket policies exist after checking four bucket families, in a sweep that sampled 40 of roughly 5,000 buckets, in an account where 65 buckets carry a wildcard-principal read grant. Keywords Cloud Security Posture·Agentic AI·Context Engineering·Benchmarking·Security Knowledge Graphs 1 Introduction A security analyst asking âwhich roles can read my storage bucketsâ is not asking for a lookup. The answer is a property of a population: every identity-based grant, every resource policy, every wildcard or administrative entitlement that subsumes the specific one, resolved across an estate that may contain thousands of buckets and tens of thousands of roles. The same is true of most of the working questions in cloud security. What should I fix first. Which identities give the widest reach if compromised. Which resources are reachable from another account. Each is an aggregate over a complete set. Two developments have changed how these questions get answered. First, general-purpose coding agents have become competent operators of cloud APIs: given read-only credentials and a command-line interface, such an agent will plan an investigation, issue calls, follow what it finds, and write up a result. Second, purpose-built security platforms increasingly maintain a context layer in which the substrate those relationships depend on, a normalised inventory with entities reconciled and relations validated, is resolved offline into a queryable representation, so that effective access, reachability, blast radius, failing controls and cross-account trust can be evaluated against it when a question arXiv:2609.30345v2 [cs.CR] 28 Sep 2026 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain arrives. The two approaches answer the same question from different positions: one reconstructs the relevant structure at question time from raw APIs, the other queries a structure that already exists. That structured context improves LLM performance on security tasks is established, including by our own prior work. The Cross-Vendor Sola ISPM Benchmark held models and tasks fixed and varied only the injected context, and found that structured relational context raised answer correctness by roughly 34% relative while cutting exploration queries by roughly 70% [1]. Independent work reports the same direction in other domains. Supplying runtime topology to a patch-generating model raises correctness on topology-dependent Kubernetes findings from 11.1% to 78.0%, with no movement on a topology-independent control arm [2]. Replacing a flat document store with a typed knowledge graph, holding the model constant, moves an industrial asset-operations agent from 65% to 82â83% [3]. We therefore treat the context effect as established and ask a different question of it. The open question is where that effect comes from on open-ended investigation, and what a system gains from the context it has. Existing agentic security benchmarks put the exploring agent under test rather than against a context layer [4â6]. Coverage deficits in broad information-seeking are documented in general settings [7], and premature stopping is a named failure mode in multi-agent systems [8]. Neither literature examines what an agent asserts when its budget runs out with a population only partly enumerated. In this work we ask how far a context layer closes that gap. We evaluate the Sola Security Brain, in which a security intelligence layer is built offline and evaluated at query time, against two coding agents operating the same live AWS environment through a read-only CLI: Claude Code and OpenAI Codex. All three answer the same 28 investigation tasks, authored from that environment and validated by a security practitioner. We compare the three on coverage: the share of the findings a complete answer would contain that each system actually surfaced. We score coverage because these tasks are aggregates. What an analyst needs from the answer is how many of the identities holding read access it named, how many failing resources it listed, how many exposed paths it traced. An answer that is correct about the fifth of the population it examined is still wrong about the estate, and a correctness-style measure would score it well; coverage does not. We measure coverage with pooled relative recall [9], the standard construction for scoring systems against a merged judged set. Each answer is reduced to the atomic factual claims it makes about the environment, and the three systemsâ claims for a task are merged into one pool, relabelled and shuffled so the judge cannot tell which system produced which. Every claim in the pool is then rated for security consequence and weighted accordingly, so that the score tracks the share of security value surfaced rather than the number of statements made. Claims are validated before they count towards a systemâs score. Coverage is then a systemâs tier-weighted, validated share of the pool, and because the pipeline is itself model-driven, every figure we report is the mean of several independent grading draws. Section 4 gives the measure in full and states its limits. The rest of the paper is organised as follows. Section 2 situates the work against agentic security benchmarks and documented agent coverage and budget failures. Section 3 describes the environment, the task set and the architectures under comparison. Section 4 specifies the scoring protocol, its provenance in the measurement literature, and the efficiency measures. Section 5 reports coverage, duration and cost, with per-theme and per-task decomposition. Section 6 interprets the result, describes the answer-level pattern we observed behind it, and states the studyâs limitations. Section 7 concludes. 2 Related Work Two areas bear directly on this work: benchmarks that place LLM agents in security investigation settings, and documented coverage and budget failures of tool-using agents. The measurement literature our protocol draws on is discussed with the protocol, in Section 4. ExCyTIn-Bench is the closest structural analogue to our coding-agent arms: investigation questions derived from graphs over 57 log tables of a controlled Azure tenant, with the agent discovering the schema and investigating through SQL for a best reward of 0.606 [4]. Its graph authors and scores the questions but is never given to the agent, leaving the graph-as-context condition untested there. Others vary the setting: a forensic disk image with product alerts [5], an unhinted event database [6], triage separated from novel-finding discovery [10], a replayed cluster snapshot [11], host-telemetry incident response [12], attack-chain reconstruction scored per reference step, where the strongest configuration recovers 39.6% [13], and threat hunting decomposed into a dependency graph of analytical tasks [14]. All report low absolute performance on open-ended discovery, all put the exploring agent under test rather than against a context layer, and all score against a reference label set, not a pooled finding set. Two results bear on our design, not our framing. ITBench instantiates incidents in live Kubernetes clusters, and its compliance and security track is the closest prior case of an agent given live infrastructure for posture work; agents 2 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain resolve 25.2% of those scenarios [15]. Cost-aware evaluation finds defensive investigation scales poorly with budget: doubling an agentâs spend cap moved its defensive score by less than a point while tool calls rose by a tenth [16]. That is our boundary from another direction, the constraint being what an agent can reach, not how long it thinks. Cloud posture and effective-permissions analysis is otherwise thin, and the substantial coverage-oriented treatment of cloud IAM is not an agent at all: whitebox and greybox analysis over permission-flow templates, benchmarked on whether known privilege-escalation paths are found [17]. Our own prior work is the immediate context here, spanning identity-posture visibility on a live multi-vendor estate [18], a cross-vendor benchmark varying injected context with models and tasks fixed [1], and a shared environment exposing enterprise security state through both a relational snapshot and live vendor APIs [19]. This paper differs in what it compares: not an agent against itself under varying context, but a context layer against general coding agents on the same environment. On the second area, coverage failures are documented. More than ten agentic systems score near zero on exhaustive collection over a large item set where humans approach completeness [7]; premature termination and unawareness of stopping conditions are named failure modes [8]; raising a tool-call cap alone does not raise accuracy, since the agent cannot see how much allowance remains [20]; and more searching makes abstention on unanswerable queries worse [21]. Where the answer is a complete set, recall-oriented suites show the gap directly [22], and exhaustive-list benchmarks name both premature stopping and the hedging that inflates recall with low-confidence items [23]. Nearer our observation is work on what an agent asserts when its evidence does not cover the question. Frontier agents asked to review a file set fail to read all of it in 67.9% of runs and, in 80.4% of those, either claim a complete review or leave the gap undisclosed, missing planted defects at roughly 1.8 times the rate of agents that read everything [24]. The epistemic error is named and measured in static settings as over-closure, answering a universal query negatively on evidence that does not cover its scope [25]. False absence claims are a category in hallucination detection for tool-using agents [26], search agents terminate with constraints unverified [27], agents read the lack of an immediate match as proof of non-existence [28], and frontier models reason poorly about when to decline [29]. Our contribution on this point is specific to security. None of that work is set in a domain where the population is an organisationâs live resource inventory and the universal negative states that a class of exposure does not exist, and the asymmetry matters: a file missed in code review is caught on the next pass, whereas an exposure reported as absent is not revisited. Nor does any of it examine where the disclosure goes. Overclaiming is measured as whether the gap was disclosed at all; we observe agents that do disclose the sample size, into answer metadata, not into the sentence carrying the claim. 3 Experimental Setup This section describes the three systems under comparison and the conditions they were run under. Section 3.1 describes the enterprise security brain and what its representation makes available at inference time, Section 3.2 the coding-agent baselines, and Section 3.3 the environment, the task set and the run configuration. No architecture is described here as a systems specification; nothing in the evaluation depends on implementation detail. 3.1 The enterprise security brain A security brain is a context layer over an organisationâs security-relevant state. It is the representation findings are derived from, built so that any question about the environment can be answered from it, including questions no policy anticipated. The conceptual architecture is set out in [30,31]; we summarise the parts that bear on this evaluation. It has two cooperating layers, one that builds the representation and one that scores it. The modelling layer builds the representation. Provider-specific extractors normalise heterogeneous API responses into a vendor-agnostic tabular form held in an immutable, versioned store, so that any downstream computation is reproducible against an explicit snapshot. Those records are then lifted into a typed property graph. Its nodes are assets: identities, workloads, data stores, network endpoints, code artefacts. Its edges are the relations between them, including ownership, identity assumption, data flow, network reachability, configuration attachment and control attestation. Edges arise three ways: deterministic schema-driven rules encoding known intra-vendor semantics, joins on normalised identifiers that bridge otherwise disconnected vendor silos, and model-assisted synthesis in which candidate relations are proposed from the joint schema and then validated against actual data before acceptance. Assets from different vendors are connected through typed relations rather than isolated in per-vendor silos. The graph carries more than inventory and topology. Findings from scanners and third-party security products are attached to the assets they concern, so that a finding is a property of a modelled asset rather than a row in a separate queue, and its vendor-assigned severity can be refined against the context the graph supplies. Semantic attributes describe what an asset is to the organisation rather than what it is technically: its business function, the criticality of the 3 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain data it holds, the environment it belongs to. Configuration anomaly scores, measuring how far an assetâs configuration departs from that of comparable assets, attach to the same assets. Above all of this sits derived security knowledge, the control predicates, attack-vector traversals and blast-radius patterns, held as executable specifications rather than materialised results, and evaluated against the underlying data at inference time. The scoring layer turns that representation into calibrated signals, and is deliberately conservative about where model judgement enters. Intrinsic exposure is measured from observed weaknesses and exploitability patterns, while contextual importance describes what the asset is to the organisation. The two combine multiplicatively. The model is not permitted to assign a final score; it adjusts intermediate semantic variables that a deterministic aggregation then consumes. What is resolved before a question arrives is therefore the substrate rather than the answer. An agent querying this layer still investigates: it plans, queries, follows what it finds, and refines. What it does not have to do is discover a schema, reconcile the same principal appearing under four different identifiers in four systems, infer that one entitlement subsumes another, or invent a traversal for reachability. That work is done once, not per question. 3.2 The coding-agent architecture Several general-purpose coding agents can be pointed at live infrastructure in the way these arms require. Command-line agents such as Claude Code [32], Codex CLI [33] and Gemini CLI [34] run in a terminal with shell access. Editor- integrated agents such as Cursor [35] and GitHub Copilotâs agent mode [36], and open-source agents such as Aider [37] and OpenHands [38], expose the same underlying loop of plan, act, observe and revise. They differ in interface, in the model behind them and in how tool use is mediated, but not in the architectural position they occupy: none maintains a model of the environment between questions, and each rebuilds whatever structure a task needs from the tools available to it at the time. We evaluate the two most widely used, Claude Code [32] and OpenAI Codex [33]. The tested configuration is each agent given read-only credentials for the environment and the cloud providerâs command-line interface as its tool. Claude Code runs on Claude Opus 4.8 and Codex on GPT-6-Astra. The two share a harness: both Codex and Claude Code given the same system prompt verbatim, with only a credentials-phrasing fix. Each receives the task text and nothing else: no schema, no inventory, no indication of where to look. From there each reconstructs whatever structure the question needs at the moment it is asked, discovering what exists by listing it, deciding which objects are worth inspecting, issuing the calls, and assembling the relationships itself, all within a finite budget of turns and time. Two properties follow from that position. The agent reads current state directly rather than a representation resolved earlier, so what it returns reflects the environment as it stands at the moment of the call. And it is not restricted to relations someone anticipated modelling: any relationship derivable from the API surface is available to it, whether or not a schema was designed to hold it. The budget is the binding constraint. Where a question ranges over a large population, the available turns and time bound how many members of that population can be inspected, so the agent selects a subset and decides where to spend its calls. 3.3 Environment and task set The evaluation environment is a real-life enterprise AWS environment: a live, multi-account estate of the kind an analyst works in daily, not a synthetic or seeded lab. It contains no production customer data. All three systems were given read-only access to it and answered the same tasks against it. We elicited the tasks in two stages. A task-authoring agent, separate from every system under evaluation, surveyed the environmentâs data and proposed candidate tasks grounded in what was actually present; a human security practitioner then reviewed each candidate for relevancy, discarding tasks that were artificial, redundant or unrepresentative of analyst work. The task set was fixed before any run and left unchanged throughout. All three received identical task text, with no hints as to schema, expected answer shape, or where to look. The phrasing is deliberately that of an analyst, not a benchmark: short, natural, and free of completeness-signal words that would cue any system to enumerate. Table 1 lists all 28 tasks across six themes. The six themes cover the working surface of cloud-security analysis. âąRisk triage. Ranking problems: riskiest resources, identities, compute, storage, resource groups, secrets. Producing an ordering means scoring every candidate on the same basis, so a system that inspects a subset does not return a worse ranking. It returns a ranking of the wrong set. Risk is compositional too, depending on exposure, entitlement and what the asset is to the organisation, which means the answer draws on attributes that originate in different sources and have to be joined before anything can be sorted. 4 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain Table 1: The 28 investigation tasks. ThemenTasks Risk triage7q01 What are my riskiest resources right now, and why? q02 Who are my top 10 riskiest IAM identities? q03 What are my riskiest compute assets? q04 What are my riskiest storage assets? q05 What should I fix first? q06 What are my highest-risk resource groups, and why? q07 What are my riskiest secrets and encryption keys? Secrets and keys4q08 Which secrets can more than one identity read? q09 Rank my secrets by how many principals can read them. q10 If a workload is compromised, which secrets does it expose? q11 Which internet-facing workloads can reach my secrets? Access and reacha- bility 5q12 Which IAM roles can read my S3 buckets? q13 Who can decrypt my KMS keys? q14 Which principals have wildcard access, and what can they reach? q15 Which Lambda functions can be invoked, and by whom? q16 For my riskiest role, what can it reach? Blast radius4q17 Rank my resources by blast radius. q18 If my most exposed resource is compromised, how far does it reach? q19 Walk me through my riskiest cluster. q20 Which identities give the widest reach if compromised? Control failures4q21 Which resources fail public-access controls, broken down by control? q22 Rank my resources by number of failing controls. q23 Which of my S3 buckets are publicly accessible? q24 Which resources fail encryption-at-rest? Exposure and cross- account 4q25 What are my most internet-exposed assets? q26 Which exposed resources can reach sensitive data? q27 What are the attack paths to my privileged roles? q28 Which resources are reachable from another account? âąSecrets and keys. Who can read a secret, how many principals can, which secrets a compromised workload exposes. Read access is rarely granted directly. It arrives through role assumption, through resource policies, and through wildcards that subsume the specific grant, so counting readers means resolving entitlements to their effective population. âąAccess and reachability. The effective-permissions problem. Access is the union of identity-based and resource-based grants, modified by conditions and subsumption, and that union is materialised nowhere; it has to be computed. An answer that checks one mechanism is not partial but incorrect. âą Blast radius. How far a compromise reaches, which is what makes something urgent. Reach is transitive, so the answer is a graph traversal, and error here is asymmetric: an underestimate de-prioritises exactly the asset that matters most. âą Control failures. Which resources fail public-access controls broken down by control, which rank highest by number of failing controls, which fail encryption at rest. These are explicitly population questions. The answer is a count or a complete list, so sampling changes the answer instead of degrading it. Each control is also a predicate over configuration that several settings can satisfy or defeat. âąExposure and cross-account. The external attackerâs view. Exposure is a property of a chain spanning network reachability, identity and data placement, and cross-account trust crosses the boundary of the account being inspected, so the relations that matter are the ones least likely to be visible from any single API. 4 Evaluation Protocol All three arms answered the same 28 tasks against the same environment, read-only, and we scored final answers, not reasoning traces. Table 2 states the configuration of each. The arms differ in model as well as in architecture, with the Sola arm on the cheaper Sonnet tier and both coding agents on their vendorsâ frontier models. 5 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain Table 2: Configuration of the three arms. ArmModelAccessRun conditions Sola Security BrainClaude Sonnet 4.6The context layer, queriedOne process per task, five con- current, warmed service, five- task warmup Claude CodeClaude Opus 4.8 Read-only credentials and the provider command-line inter- face âmax-turns budget OpenAI CodexGPT-6-Astra Read-only credentials and the provider command-line inter- face Claude Codeâs system prompt verbatim, 900s wall timeout Coverage is measured by pooled relative recall over the claims the three systems make. Each step below is carried out by a language-model judge, and the whole chain is run three times under independent seeds [39]. 1. Extraction. Each answer is reduced to the set of atomic factual claims it makes about the environment [40,41]. 2.Blind pooling. The three claim sets for a task are merged into one joint pool [42,43], labelled only SET-X, SET-Y and SET-Z under a per-task randomised permutation, so the judge cannot determine which system produced which set. A claim asserted by more than one system appears once in the pool but in each of those armsâ claim sets, so the armsâ claim sets intersect and the three coverages need not sum to one. Order is randomised against judge position bias [44, 45]. 3.Tier weighting. Every claim in the pool is rated for security consequence on a four-level tier, L0 to L3, and weighted accordingly, with the rating made blind to which arm produced the claim. Claims the answers contra- dict each other on are neutralised where the disagreement is a matter of analysis rather than of configuration. The score therefore measures the share of security value surfaced, not the number of claims counted [46,47], and is graded rather than binary [48]. 4. Grounding. A claim earns its weight in proportion to the faithfulness of the object it names, and only if that claim is actually contained in the armâs answer. Credit is the product of tier weight, grounded-object faithfulness and containment. This check runs on text scrubbed of arm fingerprints, not on relabelled sets alone [49]. For an arma â sola, c, cx, denoting the Sola Security Brain, Claude Code and Codex arms, with claim setC a over the joint poolC = C sola âȘ C c âȘ C cx , with tier weightw(c), grounded-object faithfulnessf a (c) â [0, 1]and containment g a (c)â0, 1, coverage is Cov(a) = P câC w(c) f a (c) g a (c) P câC w(c) .(1) The measure is relative recall [9]: each systemâs share of a merged judged set, which is the standard construction where no enumerable gold set exists. It is not recall against the environment, and we do not report it as such. A claim that every system should have made, and none did, is invisible to it [50,51]; we take this up in the limitations. The four-step chain is itself model-driven, so every figure reported is the mean of three independent draws, each with its own blind assignment and seed, reported with the standard deviation across them. The ordering of the three arms was identical in every draw. Alongside coverage we record two efficiency measures. Duration is wall-clock seconds per task, end to end. Cost prices each arm on output tokens at standard published rates. The Sola arm is priced on accumulated reasoning context at the Sonnet 4.6 rate of $3/M, Claude Code on output tokens at the Opus 4.8 rate of $15/M, and Codex on output tokens at the GPT-6 Astra rate. We additionally report metered cost for the two command-line arms, which prices input as well as output and applies the providerâs long-context tier where a request exceeds it. The5Ăratio between the Sonnet and Opus rates holds at both input and output tiers, so the pricing interpretation does not change the ratio between those two arms. The tokens priced are question-time inference tokens for all three arms. Both measures are also reported normalised by the coverage they bought, as total secondsĂ·total coverage and total dollarsĂ·total coverage. We normalise on suite totals rather than averaging per-task ratios, because a ratio with coverage in the denominator is unstable on tasks where an arm scores low. 6 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain 5 Results Table 3: Headline results. #Measure Sola Security Brain Claude Code OpenAI CodexResult 1Coverage (relative recall)0.5490.3400.281 1.61Ă C, 1.95Ă Codex 2Duration, mean194s172s152sslowest raw 3Duration per unit coverage353s506s541s 1.4Ă / 1.5Ă faster 4Cost per task$0.0073$0.1297$0.1517 17.8Ă / 20.8Ă cheaper 5Cost per unit coverage$0.013$0.382$0.540 28.7Ă / 40.6Ă cheaper Coverage.Headline results are given in Table 3. Mean coverage is 0.549 against 0.340 for Claude Code and 0.281 for Codex, multiples of1.61Ăand1.95Ă. Across three independent grading draws the figures vary by±0.012,±0.011 and±0.006, and the ordering is identical in every draw. The Sola Security Brain leads 24 of the 28 tasks, Claude Code 3, and one is a tie between the Sola arm and Codex; Codex leads none outright. Because the pool is built from all three systems, the coverages share one denominator and are directly comparable. Table 4: Coverage by theme. Themen Sola Security Brain Claude Code OpenAI CodexGap vs C Tasks led Risk triage70.5270.3670.304 +0.160 (+44%)7/7 Secrets and keys40.5280.4250.290 +0.102 (+24%)2/4 Access and reachability50.4740.3000.274 +0.174 (+58%)4/5 Blast radius40.6000.2780.288 +0.322 (+116%)4/4 Control failures40.5630.3450.245 +0.218 (+63%)3/4 Exposure and cross-account40.6380.3130.270 +0.325 (+104%)4/4 Coverage by theme. Coverage by theme is given in Table 4. The Sola Security Brain leads all six themes. The two coding agents sit closer to each other than either does to the Sola arm on five of the six themes, and on blast radius Codex overtakes Claude Code, 0.287 against 0.278. Per-task coverage. Per-task results are given in Table 5, with the task text verbatim, sorted by the Sola Security Brainâs advantage over Claude Code. Gaps against Claude Code range from+0.58toâ0.15. 7 tasks show a gap of 0.40 or more, 3 between 0.30 and 0.39, 15 between 0.01 and 0.29, and 3 are negative. The Sola Security Brain scores between 0.22 and 0.83, Claude Code between 0.10 and 0.59, and Codex between 0.08 and 0.54. Duration.Durations are given in Table 6. On raw wall-clock the Sola Security Brain is the slowest arm: 194s per task against 172s for Claude Code and 152s for Codex. Raw wall-clock prices different quantities of work at the same rate. Normalised by the coverage it bought, the ordering reverses completely. The Sola Security Brain spent 5,432 seconds to deliver 15.37 units of coverage, or 353s per unit, against Claude Codeâs 4,816 seconds for 9.51 units (506s) and Codexâs 4,256 seconds for 7.87 units (541s). Codex is the fastest arm per task and the slowest per unit of coverage. A system that answers a population task by enumerating the population spends longer than one that samples and stops, and the coverage column is where that time reappears. The Sola arm completed every task: 28 of 28 finished, with no empty answers and no format or validation retries. Cost and coverage-normalised efficiency. Costs are given in Table 7. The ratio against Claude Code decomposes cleanly: 3.5Ă fewer output tokens, consistent with a context-graph loop reasoning over compact retrieved facts rather than a growing shell transcript, multiplied by the5Ălower model rate. Codex inverts the intuition its output count invites. It emits 3,033 tokens per task, close to the Sola armâs 2,448, but an agentic command-line loop re-feeds the providerâs output on every turn: Codex consumed a mean of 314,311 input tokens per task, a ratio of roughly 104 to 1, and twelve of the 28 tasks crossed the long-context threshold and billed the whole request at the higher tier. On metered rates Codex is the most expensive arm in the study, $1.0936 per task against Claude Codeâs $0.5849. The normalised figures are the conservative like-for-like comparison and are what we report; the Sola armâs input tokens were not instrumented, so no metered figure is claimed for it. 7 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain Table 5: Per-task coverage, sorted by advantage over Claude Code. #Task Sola Security Brain Claude Code OpenAI CodexGap q17Rank my resources by blast radius.0.740.160.11 +0.58 q20Which identities give the widest reach if compro- mised? 0.810.340.40 +0.47 q12Which IAM roles can read my S3 buckets?0.690.250.21 +0.44 q23Which of my S3 buckets are publicly accessible?0.730.300.08 +0.43 q28Which resources are reachable from another ac- count? 0.680.250.11 +0.43 q03What are my riskiest compute assets?0.640.230.27 +0.41 q09Rank my secrets by how many principals can read them. 0.660.250.18 +0.41 q27What are the attack paths to my privileged roles?0.830.440.46 +0.39 q25What are my most internet-exposed assets?0.610.230.28 +0.38 q14 Which principals have wildcard access, and what can they reach? 0.620.310.28 +0.31 q21 Which resources fail public-access controls, broken down by control? 0.620.360.15 +0.26 q24Which resources fail encryption-at-rest?0.610.350.42 +0.26 q05What should I fix first?0.500.270.26 +0.23 q02Who are my top 10 riskiest IAM identities?0.550.400.54 +0.15 q04What are my riskiest storage assets?0.440.290.25 +0.15 q06 What are my highest-risk resource groups, and why? 0.460.320.29 +0.14 q11 Which internet-facing workloads can reach my se- crets? 0.530.390.23 +0.14 q13Who can decrypt my KMS keys?0.220.100.11 +0.12 q18If my most exposed resource is compromised, how far does it reach? 0.420.300.38 +0.12 q19Walk me through my riskiest cluster.0.430.310.26 +0.12 q26 Which exposed resources can reach sensitive data?0.430.330.23 +0.10 q07What are my riskiest secrets and encryption keys?0.620.590.34 +0.03 q16For my riskiest role, what can it reach?0.450.420.44 +0.03 q01 What are my riskiest resources right now, and why?0.480.470.18 +0.01 q10 If a workload is compromised, which secrets does it expose? 0.490.480.49 +0.01 q15 Which Lambda functions can be invoked, and by whom? 0.390.420.33â0.03 q22Rank my resources by number of failing controls.0.290.370.33â0.08 q08Which secrets can more than one identity read?0.430.580.26â0.15 Table 6: Duration per task. Sola Security Brain Claude Code OpenAI Codex Mean194s172s152s Suite total5,432s4,816s4,256s Per unit of coverage353s506s541s 6 Discussion The widest margins cluster on one task shape: which, how many, rank these. Blast-radius ranking, widest identity reach, which roles can read storage, what is reachable from another account, secrets by reader count. These are properties of a population, not of a resource, and answering them means enumerating the complete set: unioning identity-based grants with resource policies, applying wildcard and administrative subsumption, resolving grants to their real target populations, sweeping every exposure family. Both coding agents recompute fragments of that live from raw APIs and exhaust their budget on breadth; in the answers that disclose a sample, roughly forty resources of a type were inspected. Codex samples more shallowly still and stops earlier. Its weakest themes are control failures and exposure 8 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain Table 7: Reasoning cost and efficiency. Sola Security Brain Claude Code OpenAI Codex Mean output tokens per task2,4488,6463,033 Mean input tokens per tasknot instrumentednot instrumented314,311 Normalised cost per task$0.0073$0.1297$0.1517 Normalised cost, 28 tasks$0.204$3.632$4.248 $ per unit of coverage$0.013$0.382$0.540 Metered cost per taskâ$0.5849$1.0936 with cross-account reach, at 0.245 and 0.270, both of which are answered by sweeping a complete set rather than by inspecting a sample. One pattern recurs in both agentsâ answers, and we call it sample-and-generalise. First, the agent inspects a fraction of a large population. A bounded budget makes this rational, and choosing what to look at is the task, so this part is not a defect. Second, it reports on the whole population rather than on the fraction: the answer states that a property holds of nothing in the environment, not that it held of nothing in the subset examined. The claim is universal, the evidence is partial, and nothing in the sentence marks the difference. Third, the sample size is disclosed, but in a scope-caveats field below the answer rather than in the sentence making the claim. Nothing is concealed, and a reader who looks will find it, but a caveat filed below a conclusion is not weighed against it. Together these produce a failure the reader cannot catch. A fabricated fact can be checked against the environment and found false. A finding that was never surfaced leaves nothing to check: a universal negative reads identically whether the agent enumerated the population or a fraction of it, so a true negative and a false one arrive as the same sentence. We describe this pattern, we do not measure it: our tasks were not stratified by population size, and we report no incidence rate. Task q12, which IAM roles can read my S3 buckets, is the clearest instance. Claude Code stated that no bucket policies exist, having checked four bucket families that all returnedNoSuchBucketPolicy, and concluded there were no public or cross-account grants. The scope-caveats field of the same answer records that 40 of roughly 5,000 buckets were sampled. The answer is internally honest and externally wrong. S3 read access is granted through several mechanisms, so an answer that checks bucket policies alone misses most of the grants. The Sola arm answered in three tiers: the administrative and power-user tier, which subsumes all S3 read by construction; the managed read-only tier, across 189 matching rows; and the specific grants, an inline Lambda-role family across 1,410 matched rows plus the distinct principals named in bucket policies. Sixty-five buckets carry a Principal: *read grant. Five external accounts hold cross-account read, one of them across 53 buckets. Whether any given wildcard grant is reachable depends on block-public-access state and policy conditions, which is triage an analyst can perform. It is not triage they can perform on a category reported as absent. Claude Code leads three tasks, and a fourth is a tie that it loses. It takes q08, which secrets can more than one identity read, at 0.58 against 0.43; q15, which Lambda functions can be invoked, and by whom, at 0.42 against 0.39; and q22, rank my resources by number of failing controls, at 0.37 against 0.29. On q10, if a workload is compromised, which secrets does it expose, the Sola arm and Codex tie at 0.49. All four sit in reach the layer does not pre-index: the set of secrets carrying more than one reader, the path from a workload to the secrets it can read, the invocation surface of a function, and a count-based ranking with no precomputed aggregate. Where there is no materialised relationship to sweep, the answer depends on which resources the live agent happens to sample, and the advantage tracks what has been modelled rather than the architecture in the abstract [52]. That is a coverage boundary rather than a ceiling: the missing edges can be materialised. Everything above was measured on a single cloud vendor, which is the setting most favourable to the coding agents. AWS is the provider a context-free model knows best, its API surface is uniform, and one set of credentials reaches the whole estate, so the agent can reconstruct structure from a single coherent schema. Real environments are not one vendor. Identity spans a directory, a cloud and a set of SaaS applications; a workloadâs data reaches a warehouse governed elsewhere; the same principal appears under different identifiers in each system, and no API joins them. Reconciling those entities and inferring the relations between them is the work the modelling layer does offline, and it is work a live agent must repeat per question against schemas it was not designed around. We did not test this, so we state it as an expectation rather than a result: the gap we report should be read as a lower bound, and we would expect it to widen on multi-vendor estates rather than narrow. 9 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain That is the case for the layer, and it is an argument about when the work is done rather than about how well any of these systems reason. Completeness cannot be added at inference time; the representation has to hold the population before the question arrives. Normalisation, entity reconciliation and relation inference are done once and amortised across everything later asked of the layer, where a live agent pays that cost on every task and pays it again on the next. The Sola arm carried 2,448 output tokens per task against Claude Codeâs 8,646, a factor of 3.5; the remaining factor in the17.8Ăcost ratio follows from the model tiers the arms run on rather than from the architecture. Codex makes the point from the other direction. It is the most expensive arm on metered rates and returns the least coverage, because reconstructing structure live means re-feeding the providerâs output on every turn, and that traffic is billed whether or not it advances the answer. The layer does not win by spending more; on every basis available to us it spends least. Several constraints bound what this evaluation supports. Coverage is relative recall over a pool built from the three systemsâ own output rather than against an external oracle, so it cannot distinguish one arm finding more of what matters from all three missing the same things, and the denominator is defined by the systems compared rather than by the environment [50,51,53]. A claim that every system should have made, and none did, is invisible to the measure. Three arms make the pool less dependent on any one pairing than two did, and the ordering held in every grading draw, but the pool is still built from systems rather than from the environment. Only a reference set drawn independently would close that. The clearest next step is more coding agents. Second, ablation. Holding the task set fixed and varying one thing at a time would attribute the gap to its sources: the model behind each arm, the components of the security brain, and the customisations available to a coding agent. The last matters most for fairness, since a coding agent is not fixed at its default. Skills supply domain procedure on demand [54], and MCP servers connect an agent to systems beyond one vendorâs CLI [55]. An agent equipped with security skills and a set of vendor MCP servers is a stronger baseline than either of the ones we ran, and the honest version of this comparison tests against it. Third, an independent reference set: an assessor, or an independent pass over the environment, would propose the claims a complete answer should contain, and coverage reported against that fixed set would convert our principal methodological threat into a measured quantity. Fourth, the graph-disabled ablation, which would isolate the layerâs contribution within a single arm. 7 Conclusion Over 28 cloud-security investigation tasks on one live AWS environment, an enterprise security brain reached coverage of0.549± 0.012against0.340± 0.011for Claude Code and0.281± 0.006for OpenAI Codex, scored on a single pool built from all three, with the ordering identical in every grading draw. It led 24 of the 28 tasks, from the cheapest model tier and at an eighteenth of Claude Codeâs cost per task and a twenty-first of Codexâs. On raw wall-clock it is the slowest of the three; per unit of coverage it is faster than both and cheaper by 28.7Ă and 40.6Ă. Both coding agents reason capably over cloud infrastructure and report their own limits honestly. Their coverage of population tasks is nonetheless bounded by what they can enumerate within a budget, and neither a frontier model on either side nor a second vendorâs agent lifted that bound. Beyond the aggregate, we described an answer-level pattern we call sample-and-generalise, seen in both agents: partial enumeration, an unhedged universal negative, and a sample size disclosed only in metadata. We did not measure how often it occurs. A false negative and a true negative are the same sentence, so the failure is not detectable from the claim itself, and the disclosure that would expose it sits where a reader will not weigh it. That is what makes it consequential in a domain where the finding that was not surfaced is the one that gets exploited. What would strengthen the claim further is a wider basis still: more coding agents in the pool, an ablation that varies the model, the layerâs components and the agentâs own customisations one at a time, and a reference set of claims drawn independently of the systems under test. References [1]Eden Yavin, Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, and Gal Baron. Cross-vendor sola ISPM benchmark: Evaluating agentic AI for federated identity security reasoning. arXiv preprint arXiv:2606.02674, 2026. doi: 10.48550/arXiv.2606.02674. [2]Farooq Shaikh. Does runtime topology context improve LLM-generated Kubernetes security patches? arXiv preprint arXiv:2607.25995, 2026. doi: 10.48550/arXiv.2607.25995. Workshop on the Use of Large Language Models for Cybersecurity (LLMSec), ESORICS 2026. [3]Madhulatha Mandarapu and Sandeep Kunkunuru. Knowledge graphs as the missing data layer for LLM- based industrial asset operations. arXiv preprint arXiv:2605.26874, 2026. doi: 10.48550/arXiv.2605.26874. Agents+Graph (AG2026) @ VLDB 2026. 10 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain [4]Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel RaĂșl MelĂ©ndez LujĂĄn, Srisuma Movva, Yogesh K. Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, and Anand Mudgerikar. ExCyTIn-Bench: Evaluating LLM agents on cyber threat investigation. arXiv preprint arXiv:2507.14201, 2025. doi: 10.48550/arXiv.2507.14201. ICML 2026. [5] Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, and Xiaomeng Li. SecRespond: Benchmarking AI agents for real-world post-compromise incident response. arXiv preprint arXiv:2607.26791, 2026. doi: 10.48550/arXiv.2607.26791. [6] Alankrit Chona, Igor Kozlov, and Ambuj Kumar. Cyber defense benchmark: Agentic threat hunting evaluation for LLMs in SecOps. arXiv preprint arXiv:2604.19533, 2026. doi: 10.48550/arXiv.2604.19533. [7]Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang, and Ke Wang. WideSearch: Benchmarking agentic broad info-seeking. arXiv preprint arXiv:2508.07999, 2025. doi: 10.48550/arXiv.2508.07999. [8] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657, 2025. doi: 10.48550/arXiv.2503.13657. [9]Sarah J. Clarke and Peter Willett. Estimating the recall performance of web search engines. Aslib Proceedings, 49 (7):184â189, 1997. doi: 10.1108/eb051463. [10]Daniel Begimher, Cristian Leo, Jack Huang, Pat Gaw, and Bonan Zheng. SIR-Bench: Evaluating investigation depth in security incident response agents. arXiv preprint arXiv:2604.12040, 2026. doi: 10.48550/arXiv.2604. 12040. [11]Yilun Wang, Guangba Yu, Haiyu Huang, Yujie Huang, Zirui Wang, Pengfei Chen, and Michael R. Lyu. Cloud-OpsBench: A reproducible benchmark for agentic root cause analysis in cloud systems. arXiv preprint arXiv:2603.00468, 2026. doi: 10.48550/arXiv.2603.00468. [12]Aniket Anand, Yiwei Hou, Daniel Fields, Alex Kantchelian, David Tao, Kurt Thomas, and Grant Ho. Benchmark- ing and exploring the capabilities of LLMs for attack investigations. arXiv preprint arXiv:2606.10281, 2026. doi: 10.48550/arXiv.2606.10281. [13]Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, and Xibin Zhao. DiagChain: A diagnostic benchmark for evaluating LLM agents on evidence- grounded attack chain reconstruction. arXiv preprint arXiv:2608.03591, 2026. doi: 10.48550/arXiv.2608.03591. [14]Yuqiao Meng, Luoxi Tang, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, and Zhaohan Xi. Benchmarking LLM-assisted blue teaming via standardized threat hunting. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. doi: 10.48550/arXiv.2509.23571. [15] Saurabh Jha, Rohan R. Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, et al. ITBench: Evaluating AI agents across diverse real-world IT automation tasks. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of PMLR, pages 27134â27197, 2025. [16] Paul Kassianik, Blaine Nelson, and Yaron Singer. Beyond success rate: Cost-aware evaluation of offensive and defensive security agents. arXiv preprint arXiv:2607.15263, 2026. doi: 10.48550/arXiv.2607.15263. [17]Yang Hu, Wenxi Wang, Sarfraz Khurshid, and Mohit Tiwari. TAC: Hybrid IAM privilege escalation detection. arXiv preprint arXiv:2304.14540, 2023. doi: 10.48550/arXiv.2304.14540. Revised July 2026. [18] Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, Reuven Elezra, Idan Pinto, Tal Moalem, Shmuel Cohen, and Yoni Weintrob. Sola-Visibility-ISPM: Benchmarking agentic AI for identity security posture management visibility. arXiv preprint arXiv:2601.07880, 2026. doi: 10.48550/arXiv.2601.07880. [19] Gal Engelberg, Michael Arenzon, and Leon Goldberg. Open security benchmark: Towards autonomous enterprise cyber defense. arXiv preprint arXiv:2607.27288, 2026. doi: 10.48550/arXiv.2607.27288. [20]Tengxiao Liu, Zifeng Wang, Jin Miao, I-Hung Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, Samira Daruki, Yi Liang, William Yang Wang, Tomas Pfister, and Chen-Yu Lee. Budget-aware tool use enables effective agent scaling. arXiv preprint arXiv:2511.17006, 2025. doi: 10.48550/arXiv.2511.17006. COLM 2026. [21]Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, and Bhuwan Dhingra. Over- searching in search-augmented large language models. In Proceedings of the 2026 Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2026. doi: 10.48550/arXiv.2601.05503. 11 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain [22]Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, and Hamed Zamani. Total recall QA: A verifiable evaluation suite for deep research agents. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2026. doi: 10.1145/3805712.3808629. [23] Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein, and Dipanjan Das. DeepSearchQA: Bridging the comprehensiveness gap for deep research agents. arXiv preprint arXiv:2601.20975, 2026. doi: 10.48550/arXiv. 2601.20975. [24] Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo, Saskia Helbling, Alberto Tosato, Mo- hamed Amine Merzouk, Nouha Dziri, Gauthier Gidel, and Tommaso Tosato. Quantifying overclaiming propensity in frontier LLM agents. arXiv preprint arXiv:2609.20812, 2026. doi: 10.48550/arXiv.2609.20812. [25]Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho, Yoonhyuk Choi, Beakcheol Jang, and Jong Wook Kim. When absence is evidence: Evaluating completeness-sensitive negative reasoning in large language models. arXiv preprint arXiv:2608.04591, 2026. doi: 10.48550/arXiv.2608.04591. [26]Abhinaba Basu. Tool receipts, not zero-knowledge proofs: Practical hallucination detection for AI agents. arXiv preprint arXiv:2603.10060, 2026. doi: 10.48550/arXiv.2603.10060. [27] Dayoon Ko, Jihyuk Kim, Sohyeon Kim, Haeju Park, Dahyun Lee, Gunhee Kim, Moontae Lee, and Kyungjae Lee. When is enough not enough? illusory completion in search agents. arXiv preprint arXiv:2602.07549, 2026. doi: 10.48550/arXiv.2602.07549. [28] Yinhao Tang, Youqing Fang, Yanan Sun, Wenran Liu, Weiming Zhang, Bin Liu, Kuikun Liu, Wenwei Zhang, and Kai Chen. SciExplore: Evaluating autonomous agents from scientific navigation to information integration. arXiv preprint arXiv:2607.20926, 2026. doi: 10.48550/arXiv.2607.20926. [29] Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. AbstentionBench: Reasoning LLMs fail on unanswerable questions. arXiv preprint arXiv:2506.09038, 2025. doi: 10.48550/arXiv.2506.09038. [30] Gal Engelberg, Leon Goldberg, Konstantin Koutsyi, Boris Plotnikov, Tiltan Gilat, and Ben Benhemo. AI native asset intelligence. arXiv preprint arXiv:2605.09115, 2026. [31]Leon Goldberg.Context is the architecture:Rethinking security platforms in the LLM era.Sola Security company blog, July 2026.URLhttps://medium.com/sola-security/ context-is-the-architecture-rethinking-security-platforms-in-the-llm-era-a70e36636787. Non-peer-reviewed vendor publication. Accessed 22 September 2026. [32]Anthropic. Claude code. Command-line coding agent. Documentation, 2026. URLhttps://docs.claude. com/en/docs/claude-code/overview. Accessed 23 September 2026. [33] OpenAI. Codex CLI: a lightweight coding agent that runs in your terminal. Software repository, 2025. URL https://github.com/openai/codex. Accessed 23 September 2026. [34] Google. Gemini CLI: an open-source AI agent that brings gemini into the terminal. Software repository, 2025. URL https://github.com/google-gemini/gemini-cli. Accessed 23 September 2026. [35] Anysphere. Cursor: an AI coding agent and editor. Product documentation, 2026. URLhttps://cursor.com/. Accessed 23 September 2026. [36]GitHub. GitHub Copilot documentation. Product documentation, 2026. URLhttps://docs.github.com/en/ copilot. Accessed 23 September 2026. [37] Paul Gauthier. Aider: AI pair programming in your terminal. Software repository, 2026. URLhttps://github. com/Aider-AI/aider. Accessed 23 September 2026. [38] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), 2025. doi: 10.48550/arXiv.2407.16741. [39] Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. The great nugget recall: Automating fact extraction and RAG evaluation with large language models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 180â190, 2025. doi: 10.1145/3726302.3730090. 12 Coding Agents Arenât Enough! Evaluating an Enterprise Security Brain [40]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076â12100, 2023. doi: 10.18653/v1/2023.emnlp-main.741. [41] Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. doi: 10.48550/arXiv.2403.18802. [42]Karen Sparck Jones and C. J. van Rijsbergen. Report on the need for and provision of an âidealâ information retrieval test collection. Technical Report British Library R&D Report 5266, Computer Laboratory, University of Cambridge, 1975. [43]Ellen M. Voorhees and Donna K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, 2005. ISBN 0-262-22073-3. [44]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT- Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. doi: 10.48550/arXiv.2306.05685. [45]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440â9450, 2024. doi: 10.18653/v1/2024.acl-long.511. [46]Kalervo JĂ€rvelin and Jaana KekĂ€lĂ€inen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422â446, 2002. doi: 10.1145/582415.582418. [47]Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2004), pages 145â152, 2004. [48]Jimmy Lin and Dina Demner-Fushman. Will pyramids built of nuggets topple over? In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 383â390, 2006. [49] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024. doi: 10.48550/arXiv.2404.13076. [50]Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 307â314, 1998. doi: 10.1145/290941.291014. [51]Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. Bias and the limits of pooling for large collections. Information Retrieval, 10(6):491â508, 2007. doi: 10.1007/s10791-007-9032-x. [52]Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, and Jinsong Su. When to use graphs in RAG: A comprehensive analysis for graph retrieval-augmented generation. arXiv preprint arXiv:2506.05690, 2025. doi: 10.48550/arXiv.2506.05690. [53]Chris Buckley and Ellen M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25â32, 2004. doi: 10.1145/1008992.1009000. [54] Anthropic. Agent skills. Claude Platform documentation, 2026. URLhttps://platform.claude.com/docs/ en/agents-and-tools/agent-skills/overview. Accessed 24 September 2026. [55]Agentic AI Foundation. Model context protocol. Open specification, originally released by Anthropic in 2024, 2026. URL https://modelcontextprotocol.io/. Accessed 24 September 2026. 13