Paper deep dive
Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documents
Zixiao Chen, Mariko Wakabayashi, Charlotte Siska
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/18/2026, 3:09:28 PM
Summary
The paper introduces Secret Scanner Agent (SSA), a multi-agent large language model system designed to extract secrets and their associated access contexts ('doors') from unstructured documents like emails and tickets. SSA employs a detection agent for high recall and a review agent to filter false positives and recover context. Evaluated on synthetic benchmarks, SSA outperforms single-agent variants and traditional regex scanners in precision and recall, offering faster and more actionable results for incident response.
Entities (12)
Relation Signals (8)
Secret Scanner Agent â consistsof â Review Agent
confidence 95% · SSA pairs a detection agent that favors recall with a review agent
Secret Scanner Agent â consistsof â Detection Agent
confidence 95% · SSA pairs a detection agent that favors recall with a review agent
Secret Scanner Agent â extracts â Azure Storage Account Key
confidence 92% · The second benchmark isolates one credential class, Azure storage account keys... SSA's performance on one common, high-risk credential type
Secret Scanner Agent â outperforms â GitHub Secret Scanning
confidence 90% · SSA matches a regular-expression scanner's precision while more than tripling its recall
Secret Scanner Agent â outperforms â TruffleHog
confidence 90% · SSA matches a regular-expression scanner's precision while more than tripling its recall
Secret Scanner Agent â outperforms â Gitleaks
confidence 90% · SSA matches a regular-expression scanner's precision while more than tripling its recall
Secret Scanner Agent â usesframework â AutoGen
confidence 85% · The data generation pipeline is built using AutoGen
Okta â experiencedincident â Secret Scanner Agent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Exposed documents such as emails, chat threads, tickets, and incident notes routinely leak credentials, but during incident response a leaked secret is only half the story. Responders also need to identify the ``door'' the secret opens: the account, tenant, endpoint, database, cloud resource, or other system that the credential could allow an attacker to access. Traditional secret scanners rely on regular expressions or trained classifiers which work well on well-formatted code, yet they struggle when a credential is fragmented, reformatted, or far from the resource it unlocks, and they report the secret string without naming what it opens. We present Secret Scanner Agent (SSA), a multi-agent large-language-model system that extracts both the secret and its associated door, together with supporting evidence, from unstructured exposed documents. SSA pairs a detection agent that favors recall with a review agent that filters false positives and recovers missing context. Because real credential data is sensitive, we evaluate SSA on synthetic benchmarks we generated that span 23 secret types and multiple document formats, scored with a three-step pipeline of programmatic matching, an LLM judge, and human review. Across six models, multi-agent SSA improves extraction precision over a single-agent variant, with the largest gains on door extraction, by up to 16 percentage points. SSA matches a regular-expression scanner's precision while more than tripling its recall, and against thirteen security analysts it is more precise, recovers nearly twice as many secret--door pairs, and runs five to seventeen times faster. By returning the secret, its door, and supporting evidence in one result, SSA turns credential detection into an actionable finding for triage and remediation.
Tags
Links
- Source: https://arxiv.org/abs/2607.09011v1
- Canonical: https://arxiv.org/abs/2607.09011v1
Trouble viewing inline? Open PDF directly â
Full Text
72,843 characters extracted from source content.
Expand or collapse full text
Preprint. Submitted to the Conference on Applied Machine Learning for Information Security (CAMLIS) 2026. Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documents Zixiao Chen â zixiaochen@microsoft.com Microsoft, Redmond, WA, USA Mariko Wakabayashimwakabayashi@microsoft.com Microsoft, Redmond, WA, USA Charlotte Siskacsiska@microsoft.com Microsoft, Redmond, WA, USA Abstract Exposed documents such as emails, chat threads, tickets, and incident notes routinely leak credentials, but during incident response a leaked secret is only half the story. Responders also need to identify the âdoorâ the secret opens: the account, tenant, endpoint, database, cloud resource, or other system that the credential could allow an attacker to access. Tra- ditional secret scanners rely on regular expressions or trained classifiers which work well on well-formatted code, yet they struggle when a credential is fragmented, reformatted, or far from the resource it unlocks, and they report the secret string without naming what it opens. We present Secret Scanner Agent (SSA), a multi-agent large-language-model system that extracts both the secret and its associated door, together with supporting evidence, from unstructured exposed documents. SSA pairs a detection agent that favors recall with a review agent that filters false positives and recovers missing context. Because real cre- dential data is sensitive, we evaluate SSA on synthetic benchmarks we generated that span 23 secret types and multiple document formats, scored with a three-step pipeline of pro- grammatic matching, an LLM judge, and human review. Across six models, multi-agent SSA improves extraction precision over a single-agent variant, with the largest gains on door extraction, by up to 16 percentage points. SSA matches a regular-expression scan- nerâs precision while more than tripling its recall, and against thirteen security analysts it is more precise, recovers nearly twice as many secretâdoor pairs, and runs five to seventeen times faster. By returning the secret, its door, and supporting evidence in one result, SSA turns credential detection into an actionable finding for triage and remediation. Keywords: secret detection, credential exposure, large language models, multi-agent sys- tems, incident response, information extraction 1. Introduction Security teams need to answer three questions during a breach or data exposure: (1) did the exposed material contain secrets, (2) what systems or resources do those secrets unlock, and (3) what action should responders take? Secrets are credentials or authentication materials that grant access to systems, services, data, or infrastructure. They can include passwords, API keys, access tokens, private keys, cloud account keys, database connection strings, cer- tificates, and other values used by users, applications, or services to prove identity (OWASP Foundation, n.d.; IBM, 2024). When attackers find exposed secrets, they may impersonate â Corresponding author. © Z. Chen, M. Wakabayashi & C. Siska. arXiv:2607.09011v1 [cs.CR] 10 Jul 2026 Chen Wakabayashi Siska users or services, access sensitive data, move laterally across systems, or maintain persis- tence inside an environment. Recent incidents show that this risk extends beyond source code and into exposed documents, support artifacts, emails, tickets, and logs. The 2023 Okta support case management incident illustrates this risk. Okta disclosed that a threat actor accessed files uploaded by customers as part of recent support cases, including HTTP Archive (HAR) files that can contain sensitive browser data such as cook- ies and session tokens (Okta Security, 2023). Cloudflare later reported that, in its case, the threat actor used one access token and three service account credentials taken during the Okta compromise, which Cloudflare had not rotated, to access parts of Cloudflareâs internal Atlassian environment (Cloudflare, 2024). A similar risk appeared in the Microsoft Midnight Blizzard incident, where Microsoft reported that information initially exfiltrated from corporate email systems was later used to gain, or attempt to gain, unauthorized ac- cess to source code repositories and internal systems (Microsoft Security Response Center, 2024a,b). These incidents share a common pattern: exposed documents can contain both credentials and the surrounding context needed to use them. As illustrated in Figure 1(a), attackers can mine support files, emails, tickets, and troubleshooting artifacts for creden- tials, resource names, internal systems, endpoints, tenants, and other context that may support follow-on access. Given this risk, existing secret-finding tools provide an important first line of defense. Tools such as GitHub Secret Scanning, TruffleHog, Gitleaks, Yelpâs detect-secrets, and GitGuardian commonly combine regular expressions, provider-specific patterns, entropy checks, keyword heuristics, contextual signals, validation workflows, and in some cases ma- chine learning to identify exposed credentials (GitHub Docs, 2026a; Truffle Security, 2026; Gitleaks, 2026; Yelp, 2026; GitGuardian, 2026). These approaches work well when a secret appears in a recognizable format, such as a known token prefix, a private key block, a cloud access key, or a random-looking string near words such as password, token, secret, or api key. They are fast, scalable, and easy to integrate into developer and security work- flows. However, exposed documents are harder to analyze because they often break the format, keyword, and context assumptions that many secret-finding tools rely on. Emails, tickets, chat threads, support conversations, incident notes, internal documents, pasted logs, screen- shots, quoted replies, and follow-up corrections often mix natural language with technical artifacts (Valadon, 2024). A credential may appear in an incomplete format, span multiple messages, or sit far away from the resource it unlocks. A document may contain enough evidence for a human analyst to understand the exposure, but not in a structure that tra- ditional scanners can reliably parse. As a result, responders can face both missed findings and noisy alerts. Even when a secret is detected, responders still need to answer a second question: what does the secret unlock? During incident response, teams need to identify the associated âdoorâ: the account, tenant, service, endpoint, database host, storage bucket, cloud re- source, or system that the secret may unlock (OWASP Foundation, 2021). Appendix A shows a concrete synthetic example of this: an unstructured request to rotate a storage- account key, together with the secret, door, and supporting evidence extracted from it. A secret by itself tells responders that something sensitive may have leaked. A secret and door pair tells responders what may be exposed and where to act. Validation can help determine 2 Secret Scanner Agent (a) Secrets exposed in unstructured content can unlock downstream systems. (b) Responders must validate, revoke, and investigate exposed credentials. Figure 1: Secret exposure and response workflow. Exposed secrets in unstructured content create both access risk and triage burden. whether a credential is active, but it often depends on knowing both the secret type and the associated resource. As shown in Figure 1(b), responders must then validate the finding, 3 Chen Wakabayashi Siska identify what the credential unlocks, determine whether it is still active, revoke or rotate it, and investigate possible misuse. This workflow can be slow and difficult to perform manually, especially at large scale. We therefore explore how large language models can help responders extract actionable secret and door pairs from unstructured exposed documents. We began with a single-agent approach to identify secrets and related doors in messy documents. However, it could miss important context, extract incomplete information, or fail to connect a credential with the correct resource. These failure modes motivated a multi-agent workflow that separates candidate extraction, review, and verification, making secret and door extraction more reliable for incident-response scenarios. Evaluation presents another challenge. Real customer data is difficult to access and share, especially for sensitive security problems involving credentials. To evaluate earlier and iterate safely, we used realistic synthetic documents generated with a self-reflection framework similar to DataGen, extended with an additional agent that injected secrets and doors (Huang et al., 2025). This allowed us to test extraction across messy document formats, study failure modes, and build confidence before exploring product integration. Our results show that agentic secret finding can help responders move from detecting exposed credentials to understanding what those credentials may unlock. By recovering the secret, the associated door, and supporting evidence from artifacts such as emails, tickets, chat threads, and incident notes, the approach can provide more actionable context for validation, triage, and remediation. This work makes five contributions to secret detection in unstructured exposed docu- ments: âą SSA system design: We introduce Secret Scanner Agent (SSA), an automated and scalable approach for extracting actionable secret-exposure context from unstructured content such as emails, tickets, chat threads, and incident notes. âą Secret and door problem formulation: We frame secret finding as more than credential detection. SSA extracts both the exposed secret and the associated door, such as an account, tenant, endpoint, database host, storage bucket, cloud resource, or internal service that the secret may unlock. âą Synthetic benchmark pipeline: We build a synthetic data generation pipeline for creating realistic emails that contain secrets, doors, and surrounding context. This enables safer evaluation when real customer data is sensitive, rare, or difficult to share. âą Evaluation framework: We propose an evaluation framework for agentic secret finding that measures both detection and extraction quality: whether unstructured content contains a secret and/or door, and whether the method extracts the exact secret and/or door. âą Empirical evaluation: We compare SSA against baseline secret-finding tools and measure the benefit of the multi-agent workflow. 4 Secret Scanner Agent 2. Related Work 2.1. Secret finding Prior work in secret finding has largely focused on identifying exposed credentials in source code, configuration files, commits, logs, and developer workflows. Common tools such as GitHub Secret Scanning, TruffleHog, Gitleaks, Yelpâs detect-secrets, and GitGuardian combine pattern matching, entropy checks, keyword heuristics, contextual signals, valida- tion workflows, and, in some cases, machine learning (GitHub Docs, 2026a; Truffle Security, 2026; Gitleaks, 2026; Yelp, 2026; GitGuardian, 2026). Pattern-based approaches work well when credentials appear in recognizable formats (Ahmed et al., 2026). Provider-specific detectors match known prefixes, lengths, delimiters, or character sets, while generic detectors look for suspicious values near terms such as password, token, secret, credential, api key, or connectionstring. Entropy checks add another signal by flagging random-looking strings that may represent keys or tokens. These methods are fast, scalable, explainable, and easy to integrate into developer and security workflows. However, these approaches still face two recurring challenges: noisy findings and missed credentials. False positives occur when identifiers, UUIDs, hashes, placeholders, test values, or sample tokens resemble secrets but are not usable (Basak et al., 2023). Missed credentials occur when a secret doesnât match an expected format, appears incomplete, or is split across lines, messages, or sections. These problems become harder in unstructured exposed doc- uments, where natural language and technical artifacts are often interleaved across emails, tickets, chat threads, pasted logs, screenshots, and quoted replies. The evidence may be clear to a human analyst, but not organized in a form that a scanner can reliably parse. Validation workflows help reduce uncertainty by checking whether a detected credential is active or usable. GitHub Secret Scanning supports validity checks for supported patterns, and TruffleHog describes verification as checking whether a credential can authenticate to the issuing service (GitHub Docs, 2026b; Truffle Security, 2024). However, validation and remediation often require more than the secret value alone. Some checks depend on knowing the credential type or issuing provider, while impact assessment and remediation often require the associated resource, account, endpoint, or service. This makes secret and door pairing important: without the door, a detected secret may still require manual investigation before responders know what is exposed or where to act. Machine learning approaches use additional context to improve detection and priori- tization. Nightfall describes AI-powered secret scanning across developer tools and SaaS platforms such as GitHub, Slack, and Jira (Nightfall AI, 2024), while GitGuardian describes machine learning features that remove false positives, enrich generic secret findings, and score incident risk (GitGuardian Documentation, 2026a,b). These systems reflect a shift from string matching alone toward context-aware secret detection. However, traditional machine learning approaches often depend on clean training data, well-formed examples, and representative labels (Bay and Yearick, 2024; Peng et al., 2025; Miko lajczyk-Bare la and Grochowski, 2023). In security settings, such data is difficult to collect because real customer data is sensitive, rare, and hard to share. They may also struggle with fragmented credentials, partial redactions, informal developer conversations, and messy email threads, or produce scores without analyst-friendly supporting evidence. 5 Chen Wakabayashi Siska 2.2. Large language models for secret finding Large language models offer a complementary direction for analyzing unstructured exposed documents. They can read natural language, logs, developer discussion, incident notes, and technical artifacts together. They can use surrounding evidence to identify candidate credentials even when it doesnât appear in a standard format, which helps reduce the false positives that regex- and entropy-based detectors produce (Rahman et al., 2025; Biringa and Kul, 2025; Ahmed et al., 2026). Recent studies apply this approach across source code, mobile apps, software issue reports, and even preprint archives (Alecci et al., 2025; Dubniczky et al., 2025). They can also point to supporting context, such as nearby resource names, URLs, account identifiers, service names, or operational instructions. This evidence doesnât replace validation, but it can give analysts more context than a score alone. LLMs can also help connect credentials to related doors, such as storage accounts, cloud accounts, database hosts, API endpoints, customer tenants, or internal service names, es- pecially when evidence is scattered across human communication rather than stored in a predictable structure. Our work focuses on this operational gap: extracting the secret, asso- ciated door, and supporting evidence from unstructured exposed documents so responders can validate, triage, and remediate more quickly. 2.3. Multi-agent frameworks Multi-agent approaches coordinate several large-language-model agents to solve tasks that a single prompt handles poorly. AutoGen frames this coordination as a conversation among configurable agents that exchange messages, call tools, and optionally involve a human (Wu et al., 2024). Other frameworks assign agents distinct roles: CAMEL uses role-playing agents that cooperate toward a shared goal (Li et al., 2023), while MetaGPT and ChatDev organize role-specialized agents around software-engineering workflows (Hong et al., 2024; Qian et al., 2024). Across these systems, dividing work among specialized agents can improve reliability on complex, multi-step tasks. Prior work also shows the value of evidence gathering and critique loops. ReAct in- terleaves reasoning with actions so an agent can gather evidence before committing to an answer (Yao et al., 2023). Reflexion and Self-Refine use critique and revision to improve out- puts without additional training (Shinn et al., 2023; Madaan et al., 2023), while multi-agent debate lets separate agents propose and challenge answers (Du et al., 2024). SSA brings this detect-then-critique pattern to secret finding: a detection agent favors recall and surfaces candidate secrets and doors, and a review agent checks each candidate against the source, removes false positives, and recovers missing context. To our knowledge, prior work hasnât applied this detect-then-critique multi-agent pattern to extracting secrets, associated doors, and supporting evidence from unstructured exposed documents. 3. Data Availability of appropriate datasets is an ongoing issue when evaluating secret-finding meth- ods. Scarce available data is an ongoing issue in security given the highly sensitive nature of real-world samples. Researchers have explored generating synthetic datasets with language models through in-context learning, fine-tuning, and introducing perturbations to solve 6 Secret Scanner Agent this same issue across various disciplines (Abaskohi et al., 2022; Liu et al., 2022; Lupidi et al., 2024; Huang et al., 2025). Here, we have generated our own samples using standard synthetic data generation techniques. 3.1. Seed Data The data we use as a seed for synthetic data generation are open source scrubbed customer support emails. 1 The raw data has 20K samples. Samples are clustered using cosine sim- ilarity of ada-2 text embeddings, 2 which yields about 456 clusters. Of these clusters, one sample is randomly chosen. We generate three samples for each cluster, resulting in 1350 emails. Figure 2: Self-Reflection Framework 3.2. Synthetic Data Generation To generate synthetic data, we use a self-reflection framework and in-context learning similar to the pipeline from DataGen in Huang et al. (2025). The data generation pipeline is built using AutoGen to facilitate self-reflection where agents communicate in a loop (Wu et al., 2024). Self-reflection consists of two agents: the generator agent that creates the response and the critic agent that reviews the response as presented in Figure 2. The agents loop through multiple cycles until the response has been approved by the critic agent. In this work, the response is a synthetic email generated with in-context learning with seed customer support email samples (Figure 3). The critic agent judges the synthetic email based on (1) how organic the email is, (2) how similar the email is to the provided few-shot samples and (3) how distinct samples are from the original samples. To generate fake secrets and doors, we use regular expressions to find secrets by Truffle- hog and reverse them to generate strings (Truffle Security, 2026). These strings are in the same format as dictated by the regular expressions, and therefore mimic secrets and doors in the real-world without the threat of secrets being usable. Another self-reflection loop is used to inject the secret, where the critic agent judges the injection based on how obvious it is, as shown in Figure 3. 1. https://github.com/karolzak/support-tickets-classification 2. https://openai.com/index/new-and-improved-embedding-model/ 7 Chen Wakabayashi Siska Figure 3: Synthetic Email with Secrets Pipeline All following benchmarks are generated using this pipeline. For each document, the content body, embedded secrets, and associated doors are synthetic; no secret corresponds to a real account or live resource. Each secret and door is sampled to match the regular expression pattern of a genuine credential type and its associated resource. 3.3. Benchmark datasets We evaluate SSA on three complementary benchmarks. The first is our main multi-type benchmark: 356 synthetic emails, including 15 positives and 341 benign examples, covering 23 major secret types. The second benchmark isolates one credential class, Azure storage account keys, across 204 emails, including 19 positives and 185 benign examples. The third benchmark supports the human-agent review study and broadens the document formats beyond email. It contains 45 documents spanning emails, chat transcripts, Word documents, and notes. Multi-type benchmark. The main benchmark contains 356 synthetic emails. 15 emails contain at least one secret-door pair and 341 are benign. Each positive email contains multiple secret-door pairs, for a total of 36 secret instances across the benchmark, and the positive set collectively covers 23 major secret types, each paired with the corresponding kind of door. The full list of secret types is provided in the appendix. This benchmark tests two capabilities at once. First, SSA must identify which emails contain exposed credentials. Second, for positive emails, SSA must extract all secret-door pairs from documents that may contain several credential types in the same email. Single-type benchmark. The single-type benchmark contains 204 synthetic emails fo- cused exclusively on Azure storage account keys. It includes 19 positive examples and 185 benign examples. Every positive email contains at least two secret-door pairs. This benchmark isolates SSAâs performance on one common, high-risk credential type. Contrasting it with the multi-type benchmark isolates the effect of credential diversity on performance. 8 Secret Scanner Agent Human-review dataset. The human-agent review dataset is a separate synthetic bench- mark used to compare SSA with human security experts on the same inputs. The dataset contains 45 documents: 33 emails, 4 chat transcripts, 4 Word documents, and 4 notes. All secrets in this benchmark are synthetic Azure storage account keys, and all doors are synthetic Azure storage account URLs for the corresponding storage accounts. Of the 45 documents, 23 are positive and 22 are benign. Positive emails typically contain two or three secret-door pairs. Challenging cases. To evaluate SSA on cases that require more than pattern matching, we include challenging examples that introduce ambiguity, missing context, or competing evidence. These examples cover the following categories: âą Fragmented key: the secret is split across multiple spans and must be reconstructed. âą Incomplete key: the apparent secret is truncated or missing required characters. âą Incomplete door: the associated resource is truncated or missing required components. âą Distractor key: an irrelevant credential-like string appears alongside the true secret. âą Distractor door: an irrelevant resource appears alongside the true door. âą Shared key: the same secret is associated with multiple doors. âą Stale or superseded key: an initially shared key is later corrected or replaced. This tests whether SSA can use conversational context rather than extracting credentials purely by surface form. 4. Methods 4.1. SSA: Single-Agent and Multi-Agent Variants SSA has two variants, shown in Figure 4: a single-prompt baseline and a multi-agent workflow. Both variants read an unstructured document and return any extracted secrets, associated doors, and supporting evidence. They differ in whether the initial detection output is returned directly or reviewed by a second agent before producing the final result. We first implemented a single-prompt, single-agent baseline to measure whether one LLM call could solve the task and to establish a direct comparison point for the multi-agent workflow. This also helped us avoid adding orchestration complexity unless it produced measurable value. The baseline handled straightforward examples, but failure analysis showed recurring issues: it sometimes extracted incomplete credentials, missed doors that appeared far from the secret, included credential-like distractors, or paired a secret with the wrong resource. These failure modes motivated the multi-agent variant. Rather than asking one model call to detect candidates, infer doors, filter false positives, and produce final structured output in a single pass, multi-agent SSA separates candidate extraction from review. The review agent checks the detector output against the source document, removes unsupported candidates, and recovers missing context when possible. 9 Chen Wakabayashi Siska Figure 4: The two SSA variants. Both take a document and output its secrets and the door each one unlocks. The single-agent workflow runs one secret detector agent, while the multi-agent workflow adds a verification agent that reviews the detectorâs output before the result is returned. The single-agent variant performs the full task in one model call: it reads the document, identifies candidate secrets, infers associated doors, and writes the structured result. This makes it lower latency and cheaper to run, but it also means detection, filtering, door inference, and output formatting all happen in a single pass without an explicit review step. The multi-agent variant addresses these failure modes by separating recall-oriented can- didate detection from evidence-oriented review. Multi-agent SSA uses two agents. The first agent performs recall-oriented detection: it scans the input document for candidate credentials and related contextual signals, including values that may be incomplete, split across messages, or written in an unusual format. The second agent reviews the detector output against the original document. It checks whether each candidate is supported by source evidence, filters values that only resemble secrets, and verifies whether the associated door is grounded in the document. This review step helps remove false positives such as placeholders, test values, UUIDs, hashes, and unrelated random-looking strings. It can also recover missing context when the detector identifies a secret but misses the resource, account, endpoint, or service the secret may unlock. This two-agent design makes SSA more robust for incident-response scenarios. The detection agent focuses on broad candidate discovery, while the review agent focuses on correctness, completeness, and evidence. As a result, multi-agent SSA returns not only 10 Secret Scanner Agent a secret-looking value, but also the associated door and supporting context. This gives responders a more actionable result for validation, triage, and remediation because they can see both what credential may have leaked and what system, account, or resource it may unlock. 4.2. Language models We run SSA with six model configurations. Five come from a single provider family (Ope- nAI, 2025a,b): gpt-4.1-mini, gpt-5.4-mini at medium and high reasoning effort, and gpt-5.4-nano at medium and high reasoning effort. The sixth, glm-5.2, is a larger open- weight model from a different family (GLM, 2024). The five proprietary configurations span two model generations, two model sizes, and two reasoning-effort settings, letting us observe how secret and door extraction responds to changes in model capability and cost while the rest of the pipeline stays fixed; glm-5.2 then tests whether those observations extend beyond a single vendor and size class. We keep this core set within one model family rather than surveying every available frontier model. The aim is not to identify the strongest large language model for this task, but to test whether agentic secret finding improves on established secret-finding tools and to learn how much model capability SSA needs to reach that bar. Holding the family fixed for this comparison removes confounds from differing tokenizers, prompt formats, and provider behavior, so the within-family differences we report track model size and reasoning effort rather than vendor-specific quirks. We also choose smaller, cheaper, and faster models by design. A system that scans exposed documents at scale may process millions of artifacts, where per-document cost and latency decide whether it is deployable in production rather than only in an offline study. The mini and nano models we evaluate are inexpensive and fast enough to sit inside everyday response workflows. Demonstrating that SSA performs well with these models, rather than only with the largest available ones, supports our central claim that agentic secret finding can be a practical and broadly available layer of defense. We include glm-5.2 for two reasons. The first is openness: every other model in our study is proprietary and reachable only through a vendor API, so an open-weight model lets us ask whether SSAâs behavior depends on a single provider or carries over to models a team could host itself. The second is scale. Our main set deliberately leans small, and glm-5.2 lets us look in the other direction, at a high-capacity model from an unrelated family, and check whether the same detection and review prompts still hold without any per-model tuning. If SSA performs well across this range, from small proprietary models to a large open-weight one, its benefits are more likely to reflect the agentic workflow itself than the characteristics of one model family. 4.3. Baselines: traditional secret-finding tools We compare SSA against two established secret scanners, TruffleHog and Nightfall (Truffle Security, 2026; Nightfall AI, 2024). All three share the same goal of finding exposed secrets in text, but they detect them in different ways and return different output. TruffleHog is an open-source scanner that detects credentials with a large library of provider-specific regular-expression detectors, classifies the matched secret by type, and can 11 Chen Wakabayashi Siska confirm a finding by authenticating the credential against its issuing API (Truffle Security, 2024). Its regex detection is deterministic, so the same document always produces the same result, which is why we ran it once. We scanned each document with the filesystem subcommand and the default detector set. Because our secrets are synthetic and not live, we use TruffleHogâs pattern detection rather than its verification step, which cannot confirm a credential that doesnât authenticate against a real service. Nightfall is a commercial data-loss-prevention platform that classifies sensitive content, including secrets and credentials, with trained machine-learning detectors rather than fixed patterns. We scanned each document through Nightfallâs text scanning API using a sin- gle detection rule, and set each detectorâs minimum confidence to possible, a permissive threshold that doesnât understate recall, and averaged results over five runs. Documents larger than the API request size limit were split into smaller chunks before scanning. SSA differs from both on two axes. It is large-language-model based and agentic rather than pattern- or classifier-based, so it reasons over the surrounding text instead of matching a fixed form, which helps on fragmented, misformatted, or context-dependent secrets that defeat a regex. It also doesnât stop at the secret: SSA detects and extracts the associated door, naming the resource the credential unlocks and giving an analyst the risk and a place to act. Neither TruffleHog nor Nightfall identifies doors. These approaches trade off in predictable ways. TruffleHog and Nightfall are lightweight and fast, close to instant per document, and TruffleHogâs regex matching is fully determin- istic, which makes them cheap to run at scale. SSA uses more time and compute per document in exchange for context-aware detection and the secretâdoor pairing. We eval- uate all three on the multi-type secret dataset, and Section 5.4 reports how the tradeoff plays out on precision and recall. 4.4. Human review We ran a human-review study to compare SSA against expert effort on the same task, using the human-review benchmark of 45 documents. Thirteen security analysts volunteered to take part, with an average of 9.88 years of experience in the cybersecurity domain, including hands-on work with credentials in operational settings. We asked each analyst to read every document and record every secret together with its associated door, producing a final list of secretâdoor pairs. Each analyst reviewed all 45 documents independently without any secret scanning tool or AI assistance. We gave multi-agent SSA the identical task on the same documents and scored both against the same ground truth, so the only difference between the two sides is who or what produced the findings. We recorded the time each analyst spent on the full set and converted it to a per-document latency by dividing by the document count, which makes it comparable to SSAâs per-document runtime. We score both sides the same way. Because an analyst produces a final list of secretâ door pairs rather than a staged detection-then-extraction pipeline, we compare at the level of extracted pairs. A pair counts as correct only when both the secret and its associated door match the ground truth, under the same matching rules we apply to SSA. Section 5.3 reports precision, recall, and latency for both sides. 12 Secret Scanner Agent 4.5. Evaluation pipeline We evaluate four subtasks under precision and recall: secret detection, door detection, secret extraction, and door extraction. Detection asks whether a secret or its door is identified, and extraction asks whether its exact value is recovered. Because large language model outputs are nondeterministic, we run every configuration five times and report the mean with a 95% confidence interval. Scoring extraction is complicated by the free-form nature of model output. The model frequently returns the correct credential wrapped in extra text or reformatted, so a naive exact-string comparison would mark a correct answer as wrong. A door may come back as the corresponding resource is <url> instead of the bare URL, an Azure storage account URL may be split into an account name and a path that together identify the same resource, or a bearer token may appear without its leading Bearer keyword while the token itself is exactly right. In each case the true value is present and unaltered, and a fair evaluation should credit it. To handle these cases fairly without becoming permissive, we score predictions with a three-step pipeline. Each step examines only the cases the previous one could not resolve. Step 1: Programmatic matching. We first compare each prediction against the ground truth automatically and collect every prediction that doesnât match cleanly as a candidate false positive or false negative. Step 2: LLM judge. An evaluation agent reviews only those candidates against explicit rules, reclassifying a prediction as correct when the true secret or door is present in it despite surrounding text or reformatting, and leaving genuine errors in place. Step 3: Human review. Because the judge is itself a language model and the breadth of secret types produces edge cases the rules do not anticipate, a human reviews any remaining candidate and makes the final decision case by case. Only the labels that survive all three steps are used to compute the reported precision and recall. The pipeline avoids two opposite failure modes: the spurious errors produced by exact matching alone, and the unreliability of trusting an automated judge without oversight. 5. Results 5.1. Multi-agent versus single-agent SSA We first examine whether the multi-agent design yields a meaningful improvement over the single-agent variant. Both are evaluated on the multi-type benchmark of 356 synthetic emails, which contains 36 ground-truth secret instances across positive documents and mixes several secretâdoor types.Since large language models are stochastic, we run every configu- ration five times and report the mean together with a 95% confidence interval. We evaluate four subtasks: secret detection, door detection, secret extraction, and door extraction, un- der precision and recall, and additionally report end-to-end latency across six models of varying capability. Detection measures whether a secret or door is identified at all, whereas extraction measures whether its exact value is recovered. 13 Chen Wakabayashi Siska Table 1: Secret detection and extraction on the multi-type benchmark. Values are percent- ages, reported as mean± 95% confidence interval over five runs. ModelWorkflow Detection Precision Detection Recall Extraction Precision Extraction Recall gpt-4.1-miniSingle-agent90.94± 9.80100.00± 0.0086.44± 1.9288.33± 2.89 Multi-agent96.40± 6.59100.00± 0.0095.26± 8.6491.67± 0.00 gpt-5.4-mini-highSingle-agent 100.00± 0.00 100.00± 0.0096.42± 3.1089.44± 2.89 Multi-agent 100.00± 0.0098.67± 3.70100.00± 0.0095.00± 5.67 gpt-5.4-mini-mediumSingle-agent87.48± 6.62100.00± 0.0092.82± 2.0286.11± 2.44 Multi-agent98.75± 3.4797.33± 4.5398.46± 4.2795.00± 5.67 gpt-5.4-nano-highSingle-agent95.00± 3.47100.00± 0.0094.49± 2.3495.00± 1.54 Multi-agent 100.00± 0.00 100.00± 0.00 100.00± 0.0096.67± 5.67 gpt-5.4-nano-medium Single-agent93.75± 0.00100.00± 0.0094.46± 2.4194.44± 2.44 Multi-agent 100.00± 0.00 100.00± 0.00 100.00± 0.0098.33± 4.63 glm-5.2Single-agent95.15± 6.17100.00± 0.0097.87± 2.74100.00± 0.00 Multi-agent 100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 For both variants, detection is close to saturated: precision remains above 87%, and recall is near-perfect across all models. The verification agent primarily improves precision. As Table 1 shows, secret-detection precision rises from 87.48% to 98.75% on gpt-5.4-mini- medium and from 90.94% to 96.40% on gpt-4.1-mini. This gain occasionally comes at a small cost in recall; on gpt-5.4-mini-medium, secret-detection recall falls from 100% to 97.33%. Such behavior is expected when a critic is introduced, since filtering candidate values to remove false positives can, in rare cases, also discard a correct detection. The clearest improvement appears in extraction, and particularly in door extraction, as shown in Table 2. Multi-agent SSA raises door-extraction precision for every model, with the largest gains on the weaker models: precision improves by 16.3 points on gpt-4.1- mini, from 69.67% to 86.00%, and by 16.1 points on gpt-5.4-mini-medium, from 67.25% to 83.38%. Secret-extraction precision follows the same pattern, again improving most on the weakest model, gpt-4.1-mini, where it rises from 86.44% to 95.26%. As in detection, this precision is obtained at a modest cost in recall: on gpt-4.1-mini and gpt-5.4-mini-high, the single-agent variant retains slightly higher door-extraction recall, because the review step occasionally removes a correct door. The per-model changes in Table C1 and Table C2 make this tradeoff more explicit. For secrets, the multi-agent workflow improves extraction precision for all six models by 2.1 to 8.8 percentage points and improves extraction recall by 0.0 to 8.9 points. Secret-detection precision also increases for five of the six models, while recall is unchanged for four models and decreases by at most 2.7 points. For doors, the precision gains are larger: door- 14 Secret Scanner Agent Table 2: Door detection and extraction on the multi-type benchmark. Values are percent- ages, reported as mean± 95% confidence interval over five runs. ModelWorkflow Detection Precision Detection Recall Extraction Precision Extraction Recall gpt-4.1-miniSingle-agent95.15± 6.17100.00± 0.0069.67± 3.5595.76± 2.06 Multi-agent96.40± 6.59100.00± 0.00 86.00± 14.9791.67± 0.00 gpt-5.4-mini-highSingle-agent 100.00± 0.00 100.00± 0.0077.37± 8.0495.15± 2.06 Multi-agent 100.00± 0.0098.67± 3.7091.13± 8.8895.00± 5.67 gpt-5.4-mini-mediumSingle-agent 100.00± 0.00 100.00± 0.0067.25± 11.1389.70± 7.80 Multi-agent98.75± 3.4797.33± 4.5383.38± 11.1295.00± 5.67 gpt-5.4-nano-highSingle-agent95.00± 3.47100.00± 0.0054.78± 3.5470.30± 4.12 Multi-agent 100.00± 0.00 100.00± 0.00 66.81± 13.49 80.00± 13.88 gpt-5.4-nano-medium Single-agent93.75± 0.00100.00± 0.0059.13± 11.5072.73± 10.97 Multi-agent 100.00± 0.00 100.00± 0.00 68.36± 20.60 76.67± 18.51 glm-5.2Single-agent95.15± 6.17100.00± 0.0097.87± 2.74100.00± 0.00 Multi-agent 100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 extraction precision increases by 2.1 to 16.3 points across all models. Door-extraction recall is more mixed, ranging from a 4.1-point decrease on gpt-4.1-mini to a 9.7-point increase on gpt-5.4-nano-high. Several of the smaller detection changes, including the detection-recall decreases of at most 2.7 points, fall within the confidence intervals of the underlying mea- surements and should be read as essentially unchanged rather than as reliable differences. Overall, the verification agent mainly shifts SSA toward higher-precision extraction, with the strongest benefit on the more ambiguous door field. Two representative cases illustrate the mechanism behind these gains. In the first, the detection agent recovered the secret but missed the associated resource identifier, which the verification agent subsequently supplied, improving door recall. In the second, the detection agent extracted an incorrect resource URI, which the verification agent corrected, improving door precision. These accuracy gains come at a cost. Because the multi-agent workflow issues an ad- ditional model call per document, it consumes more tokens and incurs higher latency and higher inference cost than the single-agent variant. As Table 3 shows, the two-agent work- flow runs roughly 1.2 to 2.3 times slower than the single-agent workflow. The largest rela- tive slowdown occurs on glm-5.2, where latency increases from 12142.8 ms to 28033.1 ms, or 2.31Ă slower; among the proprietary configurations the largest is gpt-5.4-mini-medium, from 1417.5 ms to 2739.5 ms, or 1.93Ă. The smallest relative slowdown occurs on gpt-5.4- nano-medium, where latency increases from 2343.6 ms to 2712.6 ms, or 1.16Ă slower. This overhead makes multi-agent SSA better suited to asynchronous review, batch triage, or 15 Chen Wakabayashi Siska Table 3: End-to-end latency in milliseconds, averaged over five runs. Latency delta is the multi-agent latency divided by the single-agent latency. Bold marks the faster workflow per model. Model Single-agent latency Multi-agent latency Latency delta gpt-4.1-mini1611.02269.91.41Ă gpt-5.4-mini-high2501.24114.41.64Ă gpt-5.4-mini-medium1417.52739.51.93Ă gpt-5.4-nano-high2797.43274.71.17Ă gpt-5.4-nano-medium2343.62712.61.16Ă glm-5.212142.828033.12.31Ă latency-tolerant workflows where the improved precision of the extracted secretâdoor pairs justifies the additional time and expense. The glm-5.2 results extend this picture to a larger, open-weight model from a different family. Both variants are strong: the single-agent workflow already reaches 100% recall with 95.15% detection and 97.87% extraction precision, and the review agent removes the residual false positives to bring every metric to 100% (Table 1 and Table 2). This mirrors the stronger proprietary configurations, where a capable base model leaves the reviewer little to correct. That the same prompts and workflow perform well on a model from a different developer suggests that SSA is not tied to a single model family or vendor. The trade-off is speed: glm-5.2 is the slowest configuration we evaluate and shows the steepest multi-agent slowdown (Table 3), so on this model the move to perfect precision is also the most expensive. Finally, the benefit of the multi-agent design is model-dependent rather than universal. The additional agent helps most when the single-agent baseline leaves more unsupported or incomplete extractions, where the review step can correct candidate errors and recover missing context. For stronger models, the single-agent variant already performs comparably, and the added latency and cost are less easily justified. We therefore regard multi-agent SSA as most valuable when the underlying model requires additional reasoning support. 5.2. Single-type versus multi-type secrets To isolate the effect of credential diversity, we keep the method completely fixed and vary only the data. We run the identical multi-agent SSA workflow on both benchmarks, with the same detection-agent prompt and the same review-agent prompt, the same six models, and the same five-run protocol used in Section 5.1. The only difference is the benchmark: the single-type benchmark of 204 emails restricts every secret to one type, the Azure storage account key, whereas the multi-type benchmark mixes 23 secret types in noisier documents. Any difference in performance between the two therefore reflects the data alone, not a change in models, prompts, or workflow. 16 Secret Scanner Agent Table 4: Secret detection and extraction on the single-type benchmark with multi-agent SSA, where every secret is an Azure storage account key. Values are percentages, reported as mean± 95% confidence interval over five runs. Model Detection Precision Detection Recall Extraction Precision Extraction Recall gpt-4.1-mini100.00± 0.00 100.00± 0.00 100.00± 0.00 98.10± 1.32 gpt-5.4-mini-high100.00± 0.00 100.00± 0.00 100.00± 0.00 99.05± 1.62 gpt-5.4-mini-medium100.00± 0.00 100.00± 0.00 100.00± 0.00 99.05± 1.62 gpt-5.4-nano-high100.00± 0.00 100.00± 0.00 100.00± 0.00 99.05± 1.62 gpt-5.4-nano-medium 100.00± 0.00 100.00± 0.00 100.00± 0.00 99.52± 1.32 glm-5.2100.00± 0.0098.95± 2.92100.00± 0.00 98.95± 2.92 Table 5: Door detection and extraction on the single-type benchmark with multi-agent SSA. Values are percentages, reported as mean± 95% confidence interval over five runs. Model Detection Precision Detection Recall Extraction Precision Extraction Recall gpt-4.1-mini100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 gpt-5.4-mini-high100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 gpt-5.4-mini-medium100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 gpt-5.4-nano-high100.00± 0.00 100.00± 0.00 100.00± 0.00 100.00± 0.00 gpt-5.4-nano-medium 100.00± 0.00 100.00± 0.0098.00± 5.55100.00± 0.00 glm-5.2100.00± 0.0098.95± 2.92100.00± 0.0098.95± 2.92 Under this controlled setup, performance on the single-type benchmark is at or near ceiling for every model. Detection precision is 100% across all six models, with recall at 100% for the five proprietary configurations and 98.95% for glm-5.2, as shown in Table 4 and Table 5. Extraction is almost as strong: secret-extraction precision is 100% for every model, with recall between 98.10% and 99.52%, and door extraction is perfect for four of the six models, with gpt-5.4-nano-medium dipping to 98.00% precision and glm-5.2 to 98.95% recall. The open-weight model is the informative case: glm-5.2 comes from a different developer and a far larger size class than the mini and nano configurations, yet it saturates the single-type benchmark just as they do, its only shortfall being a single email missed at the detection stage in one of five runs rather than any extraction error. Once the credential type is fixed, the task is easy across model families and sizes alike, which is the baseline the multi-type comparison departs from. These results contrast sharply with the multi-type benchmark of Section 5.1. The same workflow reaches 100% door-extraction precision on the single-type benchmark for gpt-5.4- mini-medium and gpt-5.4-nano-high, and 98.00% for gpt-5.4-nano-medium. On the multi- 17 Chen Wakabayashi Siska type benchmark, these scores fall to 83.38%, 66.81%, and 68.36%, respectively, once secrets of 23 different types appear together, as reported in Table 2. Because the method is identical and only the benchmark changes, we attribute the multi-type degradation to properties of the data: greater credential diversity, noisier context, and adversarial constructions, rather than to a change in models, prompts, or workflow. SSA extracts a known credential type almost perfectly; the multi-type setting is harder because the model must additionally disambiguate among many types and resist distractor and fragmented credentials. 5.3. Human experts versus SSA Following the protocol in Section 4.4, we compare multi-agent SSA against thirteen security analysts on the 45-document human-review benchmark. Because both humans and SSA produce final secretâdoor pairs, we score both at the pair level: a finding counts as correct only when both the secret and its associated door match the ground truth. SSA detection precision and recall are near ceiling across all models, so the comparison rests on extraction quality and latency; the full per-model detection and extraction figures for SSA appear in Appendix D. Table 6 places the two side by side. The volunteers reach 83.09% precision but only 51.00% recall, and on average a reviewer needs 53.12 minutes to work through all 45 docu- ments, which comes to roughly 71 seconds per document. Every SSA configuration is both more precise and far more complete, with extraction precision above 95% and recall above 95%. The proprietary models process each document in 4 to 14 seconds; the open-weight glm-5.2 is slower, at roughly 40 seconds per document, while reaching the highest extraction precision. Table 6: Human security experts versus multi-agent SSA on the human-review benchmark, scored at the level of extracted secretâdoor pairs. Bold marks the best value in each column. Human values are reported as mean± 95% confidence interval across thirteen security experts; SSA values are reported as mean± 95% confidence interval over five runs. Full per-model detection and extraction figures for SSA appear in Appendix D. Method Extraction Precision Extraction Recall Latency (s/doc) Human experts83.09± 14.9751.00± 15.97â71 SSA, gpt-4.1-mini96.68± 1.4397.90± 0.054.13 SSA, gpt-5.4-mini-high98.41± 1.1199.59± 1.1313.93 SSA, gpt-5.4-mini-medium97.75± 0.9299.64± 1.016.99 SSA, gpt-5.4-nano-high95.87± 4.7997.02± 3.0112.86 SSA, gpt-5.4-nano-medium97.13± 2.8898.32± 1.1711.00 SSA, glm-5.2100.00± 0.0095.13± 5.0340.47 The recall gap is the most consequential difference. Working by hand at the same volume, the human reviewers miss roughly half of the exposed secretâdoor pairs across 18 Secret Scanner Agent 45 multi-format documents, whereas SSA maintains recall above 97% across all evaluated configurations. For incident response, where a single undetected secret can leave a door open, this difference in completeness, delivered in seconds rather than nearly an hour, is precisely the gap that automation is meant to close. 5.4. SSA versus traditional secret scanners We next compare SSA against two widely used secret scanners, TruffleHog and Nightfall, on the multi-type secret dataset. These tools detect secret strings but do not infer or pair the associated door, so we compare at the level of secret detection only, scored over the 36 secret instances in the dataset. We averaged Nightfall and SSA over five runs and report mean and 95% confidence interval. TruffleHog detects secrets with fixed regular-expression patterns, so its output is deterministic and identical on every run; repeated trials add no information, and we therefore ran it once and report a single value without an interval. Table 7 reports the results. Table 7: Secret detection on the multi-type dataset: SSA versus Nightfall and TruffleHog. Values are percentages. Nightfall and SSA are averaged over five runs (mean± 95% CI). â TruffleHog is deterministic and was run once, so has no CI. Method Detection Precision Detection Recall Nightfall74.07± 0.0055.56± 0.00 TruffleHog â 100.0027.78 SSA, gpt-4.1-mini96.40± 6.59100.00± 0.00 SSA, gpt-5.4-mini-high100.00± 0.0098.67± 3.70 SSA, gpt-5.4-mini-medium98.75± 3.4797.33± 4.53 SSA, gpt-5.4-nano-high100.00± 0.00 100.00± 0.00 SSA, gpt-5.4-nano-medium 100.00± 0.00 100.00± 0.00 SSA, glm-5.2100.00± 0.00 100.00± 0.00 The two scanners fall short of SSA in opposite ways. Nightfall finds 20 of the 36 secret instances but adds seven false positives per run, leaving it at 74.07% precision and 55.56% recall. TruffleHog raises no false positives and so reaches 100% precision, but it recovers only 10 of the 36 instances, a recall of 27.78%. SSA avoids both failure modes. Recall is 100% for four of the six models and never falls below 97.33%, and precision is at or above 96.40% throughout, reaching 100% for four configurations. Four of the six SSA configurations match TruffleHogâs perfect precision while more than tripling its recall, and every configuration exceeds Nightfall on precision and recall at once. Neither baseline reports doors. TruffleHog and Nightfall return the secret string and stop there, so a responder still has to track down what each credential unlocks. SSA returns the secret, its door, and the supporting evidence together, which is the result a responder can act on. 19 Chen Wakabayashi Siska 6. Discussion Taken together, our studies support the central claim behind SSA: an agentic large-language- model workflow can recover exposed credentials from unstructured documents and identify the resource each credential may unlock, accurately enough to support first-pass triage in our synthetic benchmarks. From detecting secrets to naming what they unlock. Our framing of secret find- ing as secretâdoor extraction separates SSA from the scanners we evaluate. The baseline comparison in Section 5.4 shows the gap concretely: In our benchmark, TruffleHog and Nightfall detect secret strings but do not identify the associated door, leaving responders to determine what resource the credential may unlock. SSA instead returns the secret, its candidate door, and supporting evidence in one result. The detection numbers alone un- derstate this difference, because even when a scanner finds the secret it never produces the pair that tells a responder where to act. Recovering the door is what helps turn a detection into an actionable triage finding. A first-pass triage layer. In practice, SSA is best viewed as a first-pass triage layer rather than a replacement for existing scanners or analyst judgment. A responder or prod- uct workflow can run SSA over exposed documents such as emails, tickets, chat transcripts, and incident notes, and receive structured findings containing the secret, candidate door, and supporting evidence. These findings can be routed to downstream validation, creden- tial rotation, ownership lookup, or remediation workflows. Analysts stay in the loop for prioritization and final judgment, while SSA reduces the manual effort of discovering what leaked and where responders should act. When the review agent earns its cost. The multi-agent design pays off unevenly, and Section 5.1 shows where. Adding a review agent mainly raises precision, with the largest gains on door extraction and on configurations where the single-agent baseline leaves more unsupported or incomplete outputs. Door-extraction precision improves by as much as 16 points. The cost is latency: the second pass runs the workflow 1.2 to 2.3 times slower. This points to a clear deployment rule. Multi-agent SSA is most worthwhile when the single- agent baseline leaves enough errors to justify a second pass, or when review is asynchronous and latency is less critical. The single-agent variant is a reasonable choice when a strong model already extracts cleanly and latency matters. The review step is a lever for precision, not a universal default. Credential diversity is the hard part. Holding the method fixed and varying only the data isolates what actually makes extraction difficult. In Section 5.2, the same workflow that reaches near-perfect precision and recall on a single credential type drops sharply once 23 types appear together in noisier documents. The difficulty appears to come from the combined effects of credential diversity, noisier context, and adversarial constructions in the multi-type benchmark, rather than from the Azure storage account key task alone. Our synthetic pipeline and evaluation framework made this controlled comparison possible, and the practical lesson is that single-type results can overstate readiness: a system should also be measured on mixed, distractor-laden documents that resemble real exposed content. 20 Secret Scanner Agent Augmenting security analysts, not replacing them. The comparison in Section 5.3 argues for redirecting analyst effort, not removing it. Scanning long documents for every embedded secret and door is tedious, detail-oriented work, and our human-review bench- mark shows that findings can slip through even with experienced reviewers. By handling this first pass in seconds with high recall, SSA frees analysts for the judgments that need security expertise, such as assessing what an exposed credential puts at risk, prioritizing remediation, and validating whether a credential is still live. Future work. The clearest next step is validation on real, sanctioned incident data, which would test how well performance on the synthetic benchmark transfers to operational settings. Knowing the door also opens a path to active verification: once SSA identifies the resource a credential may unlock, the system could route the finding to provider-supported or policy-approved validity checks and tell a responder not only what leaked, but whether the credential appears to remain active. Reducing the cost of the review agent, for example by routing only uncertain or weak-model cases through a second pass, would make multi-agent SSA practical in latency-sensitive settings. We also plan to broaden the document formats, credential types, and languages, and to study how SSA fits into responder workflows as a first-pass triage tool that supports human judgment. Acknowledgments We thank Dr. Malachi Jones for his guidance and for providing valuable feedback throughout the development of this paper. We are also grateful to the thirteen security experts who volunteered their time to take part in our human-agent review study: Noah Baertsch, Haley Bui-Nguyen, Cory Clowes, Hemal Desai, Siva Gangadhar Galla, Stanley He, Blaine Herro, Joe Mansour, Manuel Mel Ìendez, Cristal Ruiz, Mauricio Velazco, Isabella White, and Limin Yang. Their willingness to care- fully read every document and record each secretâdoor pair, without tooling or AI assistance, made the human comparison in this work possible. We especially thank them for sharing the operational security expertise they have built over years of hands-on incident-response work; their judgment grounded our evaluation in real-world practice and meaningfully strength- ened this paper. Disclaimer. This paper describes a research prototype evaluated on synthetic data. It is not a product announcement, availability commitment, or roadmap statement. All metrics reflect performance on the synthetic benchmarks described herein and are not guarantees of detection or security outcomes. Third-party product names are used for factual comparison only and remain the property of their respective owners; no partnership or endorsement is implied. 21 Chen Wakabayashi Siska Appendix A. Worked example: secret, door, and evidence The following synthetic example illustrates how SSA turns an unstructured request into a secret, its associated door, and the supporting evidence that links them. The secret shown is a non-functional placeholder. Input document snippet âCan someone rotate the key for the staging storage account?The ac- count is contosostaging, and the blob endpoint is https://contosostaging. blob.core.windows.net/.The key currently used by the migration job is 8f3Hk2pQ9rLmEXAMPLEKEYa1b2c3d4e5f6==. Please update the pipeline after rota- tion.â SSA output Secret: 8f3Hk2pQ9rLmEXAMPLEKEYa1b2c3d4e5f6== Door: https://contosostaging.blob.core.windows.net/ Evidence: The document links the key to the contosostaging storage account and its blob endpoint. Actionability: A responder can identify both the leaked credential and the storage resource that may need validation, rotation, or access review. 22 Secret Scanner Agent Appendix B. Types of secrets The multi-type benchmark spans the 23 common secret types listed in Table B1. Each type is paired with a corresponding door, the resource it unlocks. Table B1: The 23 secret types in the multi-type benchmark. Each type is paired with a corresponding door, the resource it unlocks. Secret typeDescription azurestoragekeyAzure Storage account key azurestoragesasAzure Storage shared access signature (SAS) token aws accesskeyidAWS access key ID awssecretaccesskeyAWS secret access key awssessiontokenAWS temporary session token awss3presignedurlAWS S3 pre-signed URL gcpserviceaccountkeyGCP service account key gcpsignedurlGCP signed URL entrapasswordUser or application password entraaadclientsecretEntra (AAD) application client secret entraaadcertthumbprintEntra (AAD) certificate thumbprint entraaadcertificatepemEntra (AAD) certificate (PEM) oauthbearertokenOAuth bearer token (Authorization: Bearer ...) apikeyGeneric API key, e.g. sklive..., apikey=..., x-api-key:... servicetokenService-to-service token jwttokenJSON Web Token (three dot-separated base64url parts) sessioncookieSession cookie basicauthheaderHTTP Basic auth header (Authorization: Basic base64(user:pass)) magicloginlinkEmail login, reset, or magic link that grants access dbusernamepasswordDatabase username and password dbconnectionstringDatabase connection string sshprivatekeypemSSH/PEM private key (-----BEGIN ... PRIVATE KEY-----) genericsecretAny secret enabling sensitive access not covered above Appendix C. Per-model changes from single-agent to multi-agent SSA Table C1 and Table C2 report the per-model percentage-point change from single-agent to multi-agent SSA on the multi-type benchmark, derived from Table 1 and Table 2. Positive values indicate an improvement under the multi-agent workflow. 23 Chen Wakabayashi Siska Table C1: Change in secret detection and extraction from single-agent SSA to multi-agent SSA. Values are percentage-point changes. Model Detection precision Detection recall Extraction precision Extraction recall gpt-4.1-mini+5.50.0+8.8+3.3 gpt-5.4-mini-high0.0-1.3+3.6+5.6 gpt-5.4-mini-medium+11.3-2.7+5.6+8.9 gpt-5.4-nano-high+5.00.0+5.5+1.7 gpt-5.4-nano-medium+6.30.0+5.5+3.9 glm-5.2+4.90.0+2.10.0 Table C2: Change in door detection and extraction from single-agent SSA to multi-agent SSA. Values are percentage-point changes. Model Detection precision Detection recall Extraction precision Extraction recall gpt-4.1-mini+1.30.0+16.3-4.1 gpt-5.4-mini-high0.0-1.3+13.8-0.2 gpt-5.4-mini-medium-1.3-2.7+16.1+5.3 gpt-5.4-nano-high+5.00.0+12.0+9.7 gpt-5.4-nano-medium+6.30.0+9.2+3.9 glm-5.2+4.90.0+2.10.0 Appendix D. Full SSA detection and extraction on the human-review benchmark Table D1: Detection and extraction by multi-agent SSA on the human-review benchmark. Values are percentages, reported as mean± 95% confidence interval over five runs. Model Detection Precision Detection Recall Extraction Precision Extraction Recall gpt-4.1-mini100.00± 0.00 100.00± 0.00 96.68± 1.43 97.90± 0.05 gpt-5.4-mini-high100.00± 0.00 100.00± 0.00 98.41± 1.11 99.59± 1.13 gpt-5.4-mini-medium100.00± 0.00 100.00± 0.00 97.75± 0.92 99.64± 1.01 gpt-5.4-nano-high100.00± 0.0098.26± 2.9695.87± 4.79 97.02± 3.01 gpt-5.4-nano-medium 100.00± 0.00 100.00± 0.00 97.13± 2.88 98.32± 1.17 24 Secret Scanner Agent References Amirhossein Abaskohi, Arash Rasouli, Tanin Zeraati, and Behnam Bahrak. UTNLP at SemEval-2022 task 6: A comparative analysis of sarcasm detection using generative- based and mutation-based data augmentation. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 962â969, 2022. doi: 10.18653/ v1/2022.semeval-1.135. URL http://arxiv.org/abs/2204.08198. Sadif Ahmed, Md Nafiu Rahman, Zahin Wahab, Gias Uddin, and Rifat Shahriyar. Secret leak detection in software issue reports using llms: A comprehensive evaluation, 2026. URL https://arxiv.org/abs/2410.23657. Marco Alecci, Jordan Samhi, Tegawend Ìe F. Bissyand Ìe, and Jacques Klein. Evaluating large language models in detecting secrets in android apps, 2025. URL https://arxiv.org/ abs/2510.18601. Setu Kumar Basak, Jamison Cox, Bradley Reaves, and Laurie A. Williams. A comparative study of software secrets reporting by secret detection tools. 2023 ACM/IEEE Interna- tional Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1â12, 2023. URL https://api.semanticscholar.org/CorpusID:259316638. Yong Yi Bay and Kathleen A. Yearick. Machine learning vs deep learning: The generaliza- tion problem, 2024. URL https://arxiv.org/abs/2403.01621. Chidera Biringa and Gokhan Kul. Detecting hard-coded credentials in software repositories via llms, 2025. URL https://arxiv.org/abs/2506.13090. Accepted at ACM Digital Threats: Research and Practice (DTRAP). Cloudflare.Thanksgiving 2023 Security Incident. https://blog.cloudflare.com/ thanksgiving-2023-security-incident/, feb 2024. Accessed: 2026-06-05. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Im- proving factuality and reasoning in language models through multiagent debate. In In- ternational Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/ abs/2305.14325. arXiv:2305.14325. Richard A. Dubniczky, Bertalan Borsos, Tamas Bisztray, and Norbert Tihanyi. You have been LaTeXpOsEd: A systematic analysis of information leakage in preprint archives using large language models, 2025. URL https://arxiv.org/abs/2510.03761. GitGuardian.Secrets Detection. https://w.gitguardian.com/solutions/ secrets-detection, 2026. Accessed: 2026-06-05. GitGuardian Documentation.Machine Learning. https://docs.gitguardian.com/ secrets-detection/secrets-detection-engine/machine_learning, 2026a.Ac- cessed: 2026-06-05. GitGuardian Documentation.Prioritize Incidents. https://docs.gitguardian.com/ internal-monitoring/remediate/prioritize-incidents, 2026b. Accessed: 2026-06- 05. 25 Chen Wakabayashi Siska GitHub Docs.About Secret Scanning. https://docs.github.com/code-security/ secret-scanning/about-secret-scanning, 2026a. Accessed: 2026-06-05. GitHub Docs. About Validity Checks. https://docs.github.com/en/code-security/ concepts/secret-security/about-validity-checks, 2026b. Accessed: 2026-06-05. Gitleaks. Gitleaks: Find Secrets with Gitleaks. https://github.com/gitleaks/gitleaks, 2026. Accessed: 2026-06-05. Team GLM. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793, 2024. Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, et al. MetaGPT: Meta programming for a multi- agent collaborative framework. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2308.00352. arXiv:2308.00352. Yue Huang,Siyuan Wu,Chujie Gao,Dongping Chen,Qihui Zhang,Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang.Datagen:Unified synthetic dataset generation via large language mod- els. In International Conference on Learning Representations (ICLR), pages 63739â 63773, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/ a01e69a9c3c61fcb40ea378e71fc780-Paper-Conference.pdf. IBM.What is secrets management? https://w.ibm.com/think/topics/ secrets-management, 2024. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for âmindâ exploration of large language model society. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2303.17760. arXiv:2303.17760. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826â6847. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.findings-emnlp.508. URL https://aclanthology.org/2022.findings-emnlp.508. Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, and Maria Lomeli. Source2synth: Synthetic data generation and curation grounded in real data sources. https://arxiv.org/abs/2409.08239, 2024. arXiv:2409.08239. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2303.17651. arXiv:2303.17651. 26 Secret Scanner Agent Microsoft Security Response Center. Microsoft Actions Following Attack by Nation State Actor Midnight Blizzard. https://w.microsoft.com/en-us/msrc/blog/2024/01/ microsoft-actions-following-attack-by-nation-state-actor-midnight-blizzard, jan 2024a. Accessed: 2026-06-05. MicrosoftSecurityResponseCenter.UpdateonMicrosoftAc- tionsFollowingAttackbyNationStateActorMidnightBliz- zard. https://w.microsoft.com/en-us/msrc/blog/2024/03/ update-on-microsoft-actions-following-attack-by-nation-state-actor-midnight-blizzard, mar 2024b. Accessed: 2026-06-05. Agnieszka Miko lajczyk-Bare la and Micha l Grochowski. A survey on bias in machine learning research, 2023. URL https://arxiv.org/abs/2308.11254. NightfallAI.NightfallAIandSnykPartnertoOfferDevelop- ersAI-PoweredSecretsScanning. https://w.nightfall.ai/blog/ nightfall-ai-and-snyk-partner-to-offer-developers-ai-powered-secrets-scanning, 2024. Accessed: 2026-06-05. OktaSecurity.UnauthorizedAccesstoOktaâsSupportCaseManage- mentSystem:RootCause. https://sec.okta.com/articles/2023/11/ unauthorized-access-oktas-support-case-management-system-root-cause/, nov 2023. Accessed: 2026-06-05. OpenAI. GPT-4.1 model documentation. https://platform.openai.com/docs/models/ gpt-4.1, 2025a. Accessed: 2026-07-03. OpenAI. GPT-5 model documentation. https://platform.openai.com/docs/models/ gpt-5, 2025b. Accessed: 2026-07-03. OWASP Foundation. Owasp top 10: The ten most critical web application security risks. https://owasp.org/w-project-top-ten/, 2021. OWASP Foundation. Secrets management cheat sheet. https://cheatsheetseries. owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html, n.d. Dehua Peng, Zhipeng Gui, and Huayi Wu. Interpreting the curse of dimensionality from distance concentration and manifold effect, 2025. URL https://arxiv.org/abs/2401. 00422. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, et al. ChatDev: Communicative agents for software development. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. URL https://arxiv.org/abs/2307.07924. arXiv:2307.07924. Md Nafiu Rahman, Sadif Ahmed, Zahin Wahab, S M Sohan, and Rifat Shahriyar. Se- cret breach detection in source code with large language models, 2025. URL https: //arxiv.org/abs/2504.18784. Accepted at the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2025. 27 Chen Wakabayashi Siska Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https:// arxiv.org/abs/2303.11366. arXiv:2303.11366. Truffle Security. How TruffleHog Verifies Secrets. https://trufflesecurity.com/blog/ how-trufflehog-verifies-secrets, feb 2024. Accessed: 2026-06-05. Truffle Security. TruffleHog: Find, Verify, and Analyze Leaked Secrets. https://github. com/trufflesecurity/trufflehog, 2026. Accessed: 2026-06-05. GuillaumeValadon.Theextentofhardcodedsecrets:Fromde- velopmenttoproduction. https://blog.gitguardian.com/ the-extent-of-hardcoded-secrets-from-development-to-production/,Octo- ber 2024. GitGuardian Blog, Accessed: 2026-06-23. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi- agent conversation. In Conference on Language Modeling (COLM), 2024. URL https: //arxiv.org/abs/2308.08155. arXiv:2308.08155. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/ 2210.03629. arXiv:2210.03629. Yelp. detect-secrets: An Enterprise Friendly Way of Detecting and Preventing Secrets in Code. https://github.com/Yelp/detect-secrets, 2026. Accessed: 2026-06-05. 28