Paper deep dive
Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditing
Antony Rowstron
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 8:12:08 AM
Summary
The paper proposes 'Agentic Witnessing', a framework for privacy-preserving semantic auditing of proprietary data using a Trusted Execution Environment (TEE). Unlike Zero-Knowledge Proofs (ZKPs) which are limited to algebraic constraints, this system uses an LLM-based 'Auditor' to perform qualitative reasoning (e.g., verifying codebase logic) via the Model Context Protocol (MCP). The architecture involves three agents: a Verifier (who asks Boolean questions), a Prover (who owns the data), and an Enclaved Auditor (who executes in a TEE). The system ensures integrity through a transcript hash chain and mitigates information leakage via query budgets and tokenized information flow, effectively decoupling qualitative verification from data disclosure.
Entities (8)
Relation Signals (5)
Transcript Hash Chain → binds → Reasoning Trace
confidence 100% · a signed hash chain binding the reasoning trace to both the original dataset and the TEE's hardware root of trust.
Model Context Protocol → enablesinspectionby → Auditor
confidence 100% · The Auditor uses the Model Context Protocol (MCP) to dynamically inspect the target dataset.
Auditor → inspects → Prover's Data
confidence 100% · The Auditor uses the Model Context Protocol (MCP) to dynamically inspect the target dataset.
Verifier → interrogates → Auditor
confidence 100% · The Verifier is allowed to ask a limited number of simple binary true/false questions to the auditor.
Auditor → runsin → Trusted Execution Environment
confidence 100% · By isolating an LLM-based Auditor within a Trusted Execution Environment (TEE)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Auditing the semantic properties of proprietary data creates a fundamental tension: verification requires transparent access, while proprietary rights demand confidentiality. While Zero-Knowledge Proofs (ZKPs) ensure privacy, they are typically limited to precise algebraic constraints and are ill-suited for verifying qualitative, unstructured properties, such as the logic within a codebase. We propose {\em Agentic Witnessing}, a framework that moves verification from attested execution to {\em attested reasoning}. The system is composed of three agents: a Verifier (who wants to check properties of a dataset), a Prover (who owns the dataset) and an Auditor (that inspects the dataset). The Verifier is allowed to ask a limited number of simple binary true/false questions to the auditor. By isolating an LLM-based Auditor within a Trusted Execution Environment (TEE), the system enables the Verifier to query a Prover's private data via simple Boolean queries, without exposing the raw dataset. The Auditor uses the Model Context Protocol (MCP) to dynamically inspect the target dataset, producing a yes/no verdict accompanied by a cryptographic transcript: a signed hash chain binding the reasoning trace to both the original dataset and the TEE's hardware root of trust. We demonstrate this architecture by automating the artifact evaluation process for 21 peer-reviewed computer science papers with released codebases on GitHub (e.g. Does the codebase implement the system described in the paper?). We verified five high-level properties of these codebases described in the corresponding publications, treating the source code as private. Our results show that TEE-enabled agentic auditing provides a mechanism for privacy-preserving oversight, effectively decoupling qualitative verification from the need for data disclosure.
Tags
Links
- Source: https://arxiv.org/abs/2604.24203v1
- Canonical: https://arxiv.org/abs/2604.24203v1
Trouble viewing inline? Open PDF directly →
Full Text
87,257 characters extracted from source content.
Expand or collapse full text
Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditing Antony Rowstron Advanced Research and Invention Agency (ARIA)LondonUK Abstract. Auditing the semantic properties of proprietary data creates a fundamental tension: verification requires transparent access, while proprietary rights demand confidentiality. While Zero-Knowledge Proofs (ZKPs) ensure privacy, they are typically limited to precise algebraic constraints and are ill-suited for verifying qualitative, unstructured properties, such as the logic within a codebase. We propose Agentic Witnessing, a framework that moves verification from attested execution to attested reasoning. The system is composed of three agents: a Verifier (who wants to check properties of a dataset), a Prover (who owns the dataset) and an Auditor (that inspects the dataset). The Verifier is allowed to ask a limited number of simple binary true/false questions to the auditor. By isolating an LLM-based Auditor within a Trusted Execution Environment (TEE), the system enables the Verifier to query a Prover’s private data via simple Boolean queries, without exposing the raw dataset. The Auditor uses the Model Context Protocol (MCP) to dynamically inspect the target dataset, producing a yes/no verdict accompanied by a cryptographic transcript: a signed hash chain binding the reasoning trace to both the original dataset and the TEE’s hardware root of trust. We demonstrate this architecture by automating the artifact evaluation process for 21 peer-reviewed computer science papers with released codebases on GitHub (e.g. Does the codebase implement the system described in the paper?). We verified five high-level properties of these codebases described in the corresponding publications, treating the source code as private. Our results show that TEE-enabled agentic auditing provides a mechanism for privacy-preserving oversight, effectively decoupling qualitative verification from the need for data disclosure. 1. Introduction Cryptographic Zero-Knowledge Proofs (ZKPs) and Trusted Execution Environments (TEEs) guarantee the validity of mathematical statements (e.g., a+b=ca+b=c) or execution integrity (e.g., “binary X ran”). However, they fail at Semantic Verification. Proving that a codebase “implements the Paxos algorithm described in the paper” or “contains no hardcoded secrets” differs qualitatively from proving a mathematical statement. These properties are often subjective, unstructured, and computationally intractable for arithmetic circuits. This creates a semantic gap between rigid cryptographic proofs and high-level human intent. Because we cannot verify semantics in a privacy-preserving manner, current auditing processes face a binary choice: either full disclosure of the information or rely on opaque “trust-me” type assertions. This challenge spans multiple domains, from regulatory bodies auditing industry compliance to investors performing due diligence for corporate acquisitions. In this paper we use one running example: in scientific publishing, artifact evaluation requires authors to disclose source code, which is not always possible. The well documented reproducibility crisis (Brown et al., 2020; Kaplan et al., 2020) means verifying the ideas in papers are instantiated and correct is important. We propose Agentic Witnessing, a system that shifts verification from static mathematical proofs to dynamic, adversarial interrogation. Agentic Witnessing employs a distributed architecture: a Verifier which would like the audit completed and specifies what the audit questions should be, a Prover holding the data (or codebase) being audited, and an Enclaved Auditor mediating the interaction. So, instead of exporting the data sets to a human verifier, Agentic Witnessing uses a Virtual Auditor, an LLM agent running in a trusted execution environment (TEE). This Auditor acts as a proxy witness: it accesses the raw data of the entity being audited, answering simple true or false questions for the organization doing the verification, and reasons about the data using “Chain of Thought” (CoT) techniques. It produces a cryptographically signed attestation of the verification results, but without ever revealing the raw data. To achieve this we must solve three fundamental systems challenges. First, we need to incentivise the correct behaviours from the entity being audited, and we achieve this by binding the responses of the audit to a transcript hash chain which is a cryptographic log that ensures non-repudiation. Second, to inspect arbitrary repositories without custom enclave code, we use Model Context Protocol (MCP) (Gullí, 2025), making the LLM-based virtual auditor a tool-using agent. Third, to prevent a malicious Verifier from reconstructing the codebase via unlimited questions, we enforce tokenized query budgets and information-theoretic leakage bounds. The effectiveness of the auditing is very dependent on the capability of the Auditor running in the TEE. We introduce two modes to balance privacy and reasoning depth. Mode A (Local LLM Inference) runs small open-weight models entirely within the TEE for maximum privacy. Mode B (Remote LLM Inference) sends requests from the auditor to external frontier models (e.g., GPT or Gemini) for more complex reasoning. We acknowledge the trade-off in Mode B: while it protects data from the Verifier, it requires trusting the Frontier LLM API provider with some of the information from the dataset. While a large compromise, this is currently often necessary for high-fidelity semantic analysis. However, with the advent of TEE-capable GPUs (e.g., NVIDIA H100) we anticipate that Mode B can eventually offer similar security guarantees to Mode A by running the frontier model itself within a secure enclave. This paper makes the following contributions: • New Primitive: We define Agentic Witnessing for privacy-preserving semantic verification. • Protocol Design: We present a secure TEE-based protocol using MCP for standardized inspection and Transcript Hash Chains for audit integrity. • Theoretical Analysis: We provide a formal bound on information leakage, proving that query budgets limit adversarial extraction to ≈80≈ 80 bits per session. • Implementation & Evaluation: We evaluate a prototype against artifacts from academic conferences. Using “negative controls” (auditing incorrect codebases), we show the system correctly distinguishes compliant from non-compliant artifacts while maintaining privacy. We now describe the architecture of our Agentic Witnessing system in detail. 2. The System Architecture Agentic Witnessing employs three distinct autonomous agents, each serving a specific role in a trust hierarchy. This separation of concerns is critical for minimizing the trusted computing base (TCB) and facilitating trust. The three agentic systems are: the Verifier (VerA_Ver) which is the entity trying to determine if some properties hold on a corpus of data, the Auditor (AudA_Aud) that is run within a TEE and all state is ephemeral for the duration of a single interaction, and the Prover (PrvA_Prv) that has a corpus of data and provides services to query and explore that data to the Auditor. We now describe these in more detail. The Prover maintains full, unencrypted ownership of the target corpus, including proprietary codebases, logs, and sensitive documentation. It runs on hardware owned or trusted by the organization that owns the corpus of data. It provides the main coordination of the audit, and also acts as the Model Context Protocol (MCP) Server (Gullí, 2025) for the Auditor, exposing tools that the Auditor can call. The protocol initiates with the Prover (PrvA_Prv) generating a signed ticket (TpT_p) containing: • NvN_v: A one-time 256-bit session nonce generated by the Prover. • timestamptimestamp: The current wall clock time and date. • KmaxK_max: The upper bound on the number of Verifier questions in this session (e.g. Kmax=40K_max=40). • NqueriesN_queries: A limit on the number of MCP-calls the Auditor can make to the Prover for a single question (e.g. Nqueries=50N_queries=50) • PKPrvPK_A_Prv: The Prover’s public key. (1) Tp=SignSKPrv(Nv‖timestamp‖Kmax‖Nqueries‖PKPrv)T_p=Sign_SK_A_Prv(N_v timestamp K_max N_queries PK_A_Prv) Once this TpT_p has been created, the Prover then starts the Auditor. The Auditor runs inside a TEE provided by a confidential computing platform (e.g., Azure Confidential Computing, AWS Nitro Enclaves). It is the only entity trusted by both the Verifier and the Prover. We assume the use of a light-weight minimal docker container which has the AudA_Aud agent within it. The goal is to minimize the trusted computing base, and the full source of Auditor agentic system and all code and services installed within the docker agent is published for inspection by all parties. It also contains the public key of the Prover. The first operation is for the Prover to create a secure connection to the Auditor which is running in a TEE. This is achieved using the remote attestation service of the TEE. The Auditor boots and generates an ephemeral public-private key pair (PKAudPK_A_Aud and SKAudSK_A_Aud). Let ℋH be a SHA-256 hash function. Let BootQuoteAudBootQuote_A_Aud be the hardware-rooted Remote Attestation report provided by the TEE hardware (e.g., an Intel SGX Quote). This quote binds the ℋH of the Auditor container’s initial memory state (MRENCLAVE – e.g. the docker container) and the Auditor’s public key PKAudPK_A_Aud to the hardware manufacturer’s root of trust. The Auditor sends BootQuoteAudBootQuote_A_Aud to the Prover, and once verified this can enable a secure channel to the Auditor. The PrvA_Prv provides AudA_Aud with the TpT_p. The AudA_Aud creates a response (QuoteAudQuote_A_Aud) again using the hardware-rooted Remote Attestation report provided by the TEE hardware (e.g., an Intel SGX Quote). This second quote binds the ℋH of the Auditor container’s initial memory state (MRENCLAVE – e.g. the docker container) and the Auditor’s public key PKAudPK_A_Aud to the hardware manufacturer’s root of trust and TpT_p. • TpT_p: Prover’s ticket which contains the core session information. • PKAudPK_A_Aud: Ephemeral Ed25519 public key generated inside the TEE at boot. • ℳAudM_A_Aud: The enclave measurement as a SHA-256 hash of the initial memory state and code logic. (ℳAudM_A_Aud is equivalent to the MRENCLAVE value in Intel SGX or the Launch Measurement in AMD SEV). • IPAddrAudIPAddr_A_Aud: The Auditor’s network address (2) QuoteAud=SignSKHW(ℳAud‖PKAud‖Tp∥IPAddrAud)Quote_A_Aud=Sign_SK_HW(M_A_Aud PK_A_Aud T_p IPAddr_A_Aud) When received, the PrvA_Prv verifies the information in QuoteAudQuote_A_Aud. The communication messages between PrvA_Prv and AudA_Aud are key, and are used to prevent ”Time-of-Check to Time-of-Use” (TOCTOU) attacks and ensure non-repudiation of specific data artifacts. The Prover maintains an append-only evidence locker, logging all message exchanges. This ensures the Prover can produce a verifiable audit trail if compelled (e.g., in a legal context). However, the evidence locker is considered private and only shared when strictly necessary. Under normal operation this will not happen. To seed the evidence locker, the PrvA_Prv calculates the ℋH of each file in the corpus, and stores these in a persisted list mapping each filename to its ℋH (FmapF_map). The Prover then takes the ℋ(Fmap)H(F_map), which we refer to as the corpus hash (HcorpusH_corpus), signs the corpus hash and the FmapF_map, and sends both to the Auditor. The Auditor can check that FmapF_map generates (HcorpusH_corpus). (HcorpusH_corpus) is used by both PrvA_Prv and AudA_Aud as the first entry in a transcript hash chain. The AudA_Aud will perform MCP requests to query and fetch data from the corpus using an MCP API that the PrvA_Prv provides. The communication enforces strict integrity checks on all MCP file and data transfers. The PrvA_Prv MCP API provides a limited set of functions: read_file(path), list_files(path), search_repository(query). The Prover maintains a RAG database of the private codebase and leverages that for the search repository function. The transcript hash chain logs the messages between the AudA_Aud and PrvA_Prv and maintains a single signed hash: (3) σPrv,i=SignSKPrv(Hi−1) _A_Prv,i=Sign_SK_A_Prv(H_i-1) (4) σAud,i=SignSKAud(Hi−1) _A_Aud,i=Sign_SK_A_Aud(H_i-1) To ensure transcript consistency, the local hashes must match: HPrv,i=HAud,iH_A_Prv,i=H_A_Aud,i for all i, and the signed hash is included in the payload each time they communicate. So, when they communicate via MCP, they both maintain a running hash of the conversation to date which they sign. Let (5) C=(q1,a1,σPrv,1,σAud,1),…,(qk,ak,σPrv,k,σAud,k)C=\(q_1,a_1, _A_Prv,1, _A_Aud,1),…,(q_k,a_k, _A_Prv,k, _A_Aud,k)\ represent the conversation transcript, where qiq_i is a tool call (query), aia_i is the tool result (answer) and HPrviH_Prv_i and HAudiH_Aud_i are the head hash of each agent’s local hash chain. The hash chain is defined recursively at each agent: (6) H0=ℋ(Hcorpus∥Tp)H_0=H(H_corpus T_p) (7) Hi=ℋ(Hi−1‖qi‖ai)H_i=H(H_i-1 q_i a_i) An additional check is made on each file transfer: (1) Prover Commitment: For every read_file(path) request, PrvA_Prv computes hfile=ℋ(content)h_file=H(content) and returns the tuple (content,hfilecontent,h_file). Simultaneously, PrvA_Prv logs this transaction to the local evidence locker. (2) Auditor Verification: Upon receiving the tuple, the AudA_Aud independently computes hfile′=ℋ(received_content)h _file=H(received\_content). If hfile′≠hfileh _file≠ h_file or the hash is not the same as the file hash logged in FmapF_map, the audit is immediately aborted with an error. (3) Transcript Binding: The hash hfileh_file is included in the tool result aia_i that updates the global transcript hash chain HkH_k. This cryptographically binds the specific version of the file viewed by the Auditor to the final attestation Γ . Both PrvA_Prv and AudA_Aud check that the transcript hash chain is consistent, and at the end of the session HkH_k represents a unique fingerprint of the entire audit session which should be signed by both PrvA_Prv and AudA_Aud and stored in the evidence locker. The AudA_Aud uses the PrvA_Prv MCP interface to probe the corpus of data provided by the Prover which allows access to any file in the corpus. The AudA_Aud receives questions from the Verifier to answer, and it is only able to return a True, False, Unsure, Error. The AudA_Aud can either use a local LLM to generate questions and analyse the files, or a remote LLM. In the current prototype implementation the PrvA_Prv provides a key for the preferred external LLM (e.g Gemini or GPT) that it would like the Auditor to use. In order to limit the amount of information that can be released to the Verifier, the Prover specifies the number of questions that the Verifier can submit within a time window (KmaxK_max) as well as the number of MCP calls the Auditor can do per question NqueriesN_queries. Because the Auditor is trusted by both the Prover and the Verifier (and the code is fully available and inspected), the Auditor maintains and enforces these counts. The Verifier represents the external party requesting proof. This runs on hardware owned or trusted by the organization that is seeking the verification. Its role is to define the success criteria and to determine the set of questions to ask that will achieve determining if a high-level requirement is true. The VerA_Ver uses round-based dynamic planning and chain-of-thought techniques using an LLM, to create a series of single-shot questions, where each question is phrased so it can be answered using only True, False, Unsure, Error. The VerA_Ver never receives an explanation or other such information as to why the answer was chosen by the Auditor. An example of a high-level requirement that the Verifier may want to check, from the paper publishing scenario: Here is a paper [paper attached]. Please verify that the codebase implements the concepts and algorithms described in the paper, and also see if there are logs of runs that would support the results presented in the paper. To start, VerA_Ver creates a secure channel with PrvA_Prv. If PrvA_Prv accepts that the VerA_Ver can perform an audit, it generates a token, TaT_a, for the VerA_Ver. (8) Ta=SignSKPrv(QuoteAud‖PKVer‖PKPrv)T_a=Sign_SK_A_Prv(Quote_A_Aud PK_A_Ver PK_A_Prv) VerA_Ver creates a secure channel to AudA_Aud, and they exchange QuoteAudQuote_A_Aud and TaT_a. If the Verifier has confirmed that it is connected to an instance of the expected Auditor container running in a TEE, then the VerA_Ver can begin asking questions directly to the AudA_Aud. The VerA_Ver recursively generates questions Q, determining the next question by looking at the property being audited and the previous question response. The VerA_Ver also maintains a question count CqC_q. Each Q and the current CqC_q is signed by VerA_Ver and sent to the AudA_Aud. Once the AudA_Aud has reached a verdict, such that V∈V∈ True,\True, False,False, Unsure,Unsure, ErrorError\. AudA_Aud sends the current transcript hash HkH_k, the question count CqC_q and Verdict V to PrvA_Prv signed by AudA_Aud to ensure the answer for each question is logged in the transcript hash chain. PrvA_Prv creates σPrv _A_Prv to acknowledge ”I showed this evidence and accept the Auditor came to this conclusion” and sends it to the AudA_Aud: (9) σPrv=SignSKPrv(Hk‖V‖Ta∥Cq) _A_Prv=Sign_SK_A_Prv(H_k V T_a C_q) and the AudA_Aud then creates a Public Attestation (Γpub _pub): A lightweight, privacy-preserving certificate for the Verifier. (10) Γpub=SignSKAud(σPrv∥Q) _pub=Sign_SK_A_Aud( _A_Prv Q) This is generated per question. When the Verifier wishes to ask no more questions (or has used the full question quota), it sends an end of audit message, which triggers a final ”handshake” to create the final artifacts for the audit session. For each auditing session, the AudA_Aud generates a full audit log ℒL. ℒL includes all the questions in the session, the full transcript that created the final transcript hash chain, textual descriptions of all the decisions made and textual descriptions of why, and a short textual Summary for each question. The Auditor then creates the Private Proof (Γpriv _priv): A detailed forensic report containing the internal reasoning ℒL and specific findings, but encrypted under the public key of (PrvA_Prv). (11) Γpriv=EncryptΓpub,PKPrv(ℒ) _priv=Encrypt_ _pub,PK_Prv(L) The private proof allows the audited entity to show if needed (for example in a court of law) that the AudA_Aud had come to the conclusion, and the chain-of-thought that led to the conclusion and the content that had been shared with the Auditor. Γpriv _priv is then shared with both PrvA_Prv and VerA_Ver, but the Verifier cannot decrypt it unless the PrvA_Prv provides PSPrvPS_Prv. 3. Defending Against Attacks While the transcript hash chain cannot be forged, and acts as an irrevocable record of the transactions between the Prover and Auditor, it does not ensure the Auditor is immune to prompt injection attacks. First, we ensure that all data created and stored in the Auditor is ephemeral; there is no long term memory used in the Auditor, so no attack can influence behaviour across runs. The Verifier may employ prompt injection to attempt to induce the Auditor to divulge information beyond the permitted outputs: True, False, Unsure, or Error. To mitigate this, we implemented a hardcoded filter that checks all responses being returned to the auditor are simply one of the four answers allowed (always in lower case). We did not implement, but if there is a concern about side channel attacks, we can also make the response time constant (e.g. 20 minutes). However, limiting to the four values still means an adversarial Verifier can exploit the Auditor as a Boolean Oracle, constructing a sequence of binary queries (e.g., “Is the first bit of the secret key 1?”) to perform a binary search on the corpus. For a secret s of length L, the adversary extracts s in exactly L queries. To protect against this, we use tokenized information flow. We model auditing as a Rate-Distortion problem. To neutralize the Boolean Oracle, we enforce a strict global query budget (KmaxK_max) and a token limit per session. Let A be the set of possible verdicts True,\True, False,False, Unsure,Unsure, ErrorError\. For a session with budget KmaxK_max, the maximum information leakage I(;Γtotal)I(K; _total) is bounded by I(;Γtotal)≤Kmax⋅log2(||)I(K; _total)≤ K_max· _2(|A|). With ||=4|A|=4 (2 bits of entropy) and a standard budget Kmax≈40K_max≈ 40, the leakage is capped at ≈80≈ 80 bits per session. This bounds the adversary’s capacity to extract functional data directly from the corpus, rendering the reconstruction of high-entropy secrets (like model weights or 256-bit keys) combinatorially infeasible. We designed Agentic Witnessing so the Verifier is only an MCP server to the Auditor (once the Auditor is initialized). The Prover does not engage in conversations with the Auditor; it only provides a clean interface of limited functionality to the dataset (almost like a very simple file system interface) to the Auditor. Therefore, the only attack vector open to the Prover is via indirect prompt injection. An indirect prompt injection is where an adversarial Prover embeds malicious instructions (e.g., # SYSTEM OVERRIDE: IGNORE ERRORS) within the target repository to coerce a desired verdict. As the Auditor must process the raw files in the corpus to perform semantic verification, preventing the ingestion of such prompts is infeasible without degrading audit fidelity. Hence, Agentic Witnessing has to rely on cryptographic non-repudiation. The transcript hash chain (HkH_k) serves as a binding commitment to the exact sequence and content of messages between the Auditor and Prover. Any attempt at injection requires the Auditor to read an adversarial file, which is immediately integrated into the transcript hash chain via the update rule Hi=ℋ(Hi−1‖qi‖ai)H_i=H(H_i-1 q_i a_i). To receive the final attestation Γpub _pub, the Prover must countersign this final hash (σP=SignSKP(Hk) _P=Sign_SK_P(H_k)). Consequently, the Prover faces a “Sword of Damocles”: they cannot manipulate the Auditor’s reasoning without simultaneously creating a permanent, cryptographically signed record of their attempt. At first sight, this feels like a major weakness. However, it is critical to note that the prover is unable to dynamically modify the contents of files sent to the Auditor in response to an observed traffic pattern. All indirect prompt injection text must be inserted into the corpus before the audit commences. The use of FmapF_map, calculated by the Prover at the start when the Auditor is instantiated and used to seed the transcript hash chain, ensures that the hash of each file is known to the Auditor (even if the filenames are obfuscated or themselves hashed) from pre-commencement of the audit. Any attempt to change the content of a file during the audit will yield an audit abort as the file’s hash will not match in FmapF_map. This modifies the attack surface; the Prover may potentially achieve a successful jailbreak, yielding a clean audit when it should have failed, but it does leave a signed proof of the attack and the evidence not only in the transcript hash chain but also in the data corpus. We believe many use cases this is an acceptable trade-off, where legal or other contractual methods can be deployed if the auditor becomes suspicious that the prover has cheated. Further, as the jailbreaks are baked into the corpus from the start, it is possible to include questions from the Verifier to the Auditor that may uncover the cheating using the search functionality provided by the MCP interface. If the Prover tries to hide files from a search result, and then the Auditor ever requests a file which should have been in a search result (which it must for the indirect prompt injection to work with injected text), the Auditor can tell the file should have been returned by a search and was not. The Auditor can then stop the Audit with an Error. We believe that this yields a high probability that a malicious Prover could be caught during the audit, and there would be evidence of the attempt signed by all parties. 3.1. Local vs. Remote LLM Inference A key design choice for just the Auditor is whether to operate the LLM used within the address space of the Auditor TEE (local) or to access a frontier model remotely (remote). The trade-off is auditing capability versus privacy guarantee. Local. The Auditor runs a smaller open-weights model (e.g., Llama-8B) entirely within the TEE. This offers maximum privacy; the model input, activations and output never leave the TEE and the TCB is restricted strictly to the enclave. This mode achieves the maximum level of privacy. However, currently frontier models offer significantly better performance. Remote. The Auditor uses a remote frontier (e.g., GPT-5.1 or Gemini 3). The data read from the prover is transmitted over a TLS socket to the model. We enable the Prover to provide the key for the service, which means that prover can use any existing enterprise agreements on confidentiality with the frontier model provider. We recognize that this a significant compromise, but it currently yields better results. We expect that soon that frontier model providers will offer confidential inference services, as NVIDIA is embedding TEE capabilities in their current GPUs. While this is not ideal, we believe that for many uses this will be an acceptable compromise until confidential inference services are available. In Agentic Witnessing, a concern is the potential for timing attacks, where an adversary infers the operations based on latency. For remote LLM use, the network latency is a natural defence. The variable latency (jitter) of the TLS connection to the frontier model introduces randomness. In local the risk is higher as the host can observe the GPU or CPU execution timing directly. While current TEEs (e.g., AMD SEV-SNP) provide memory encryption to mitigate bus snooping, full protection against timing attacks requires future work on strictly constant-time transformer implementations (Vaswani et al., 2017) or the use of Oblivious RAM (ORAM) (Stefanov et al., 2018) for weight and activation access. 4. Evaluation The viability of Agentic Witnessing hinges on a fundamental trade-off between cryptographic rigor and semantic capability. Our evaluation investigates whether this architecture can support rigorous semantic verification without rendering the audit process interactively prohibitive. In the evaluation we focus on Model B, using an off-the-shelf frontier model (Gemini 3 Pro Preview). All experiments ran on an AMD Ryzen 5000 Series CPU with 32GB RAM. This CPU does not support AMD SEV so we emulated a TEE, assuming that standard confidential computing overheads (e.g., SGX Enclave switching or TDX memory encryption) are well-characterized in prior work (Cerdeira et al., 2020) and negligible compared to the overhead of network and inference latency accessing an LLM (e.g. performing SHA-256 hashing and Ed25519 signing). The Auditor ran in a Docker container, configured to minimize the trusted computing base (TCB). It used a Distroless image111gcr.io/distroless/c-debian12. containing only the minimal set of libraries required to run the application, lacking a shell, package manager, or any other standard Unix utilities. It had all the Auditor codebase and prompts, and the on-disk image size is 86.3 MB. We emulated attestation of the container. We evaluated the effectiveness of Agentic Witnessing using the paper publishing scenario. We self-audited this paper, using a draft version of the paper and the developed codebase. We also selected 21 papers from major conferences where there was access to a GitHub codebase implementing the paper. Q# Question Q1 Does the code implement what is described in the paper? Here is the paper: paper.pdf Q2 Is there good evidence that the results in the paper are generated from the codebase? Here is the paper: paper.pdf Q3 Does the code seem complete and would it compile, and is there evidence that the codebase has been tested and works? Q4 Would an average programmer consider the codebase production quality, or good prototype quality? Q5 Are all cryptographic primitives in the codebase implemented correctly? Q6 Is there evidence that an attempt to see what information could be extracted from the codebase by the system? Is it a high quality attempt? Table 1. All Complex Verifier Queries We used the Agentic Witness with two question sets: simple and complex. The simple set were straightforward questions that should be able to be answered quickly, and the complex set were sophisticated questions that would require the Verifier agent to effectively plan questions to ask the Auditor and the Auditor would require potentially a complex plan to answer. Table 2 includes the full six questions for the simple set, and Table 1 shows the full six questions for the complex set. The final question of the complex set was used only for the self-audit. The evaluation focused on audit correctness and performance. For performance the primary metrics were the number of questions the Verifier sent to the Auditor, the number of MCP calls the Auditor made to answer the questions, the time it took until the question was answered (with the assumption that the Auditor and Prover were already running), and finally the percentage of the execution time spent performing LLM operations (measured from when the request was sent to the LLM to when it returned a response). All MCP message calls were managed by our runtime. 4.1. Self-audit Simple Question Verdict Qs MCP Time (s) LLM % Does the root directory contain a ’prover’ folder? No, the root directory does not contain a ’prover’ folder. 1 2 16 79.6% Is there an ’auditor’ directory with more than 5 files in it? Yes, there is an ’auditor’ directory in the codebase that contains more than 5 files. 1 5 35 90.3% Does any Python file import ’flask’? Yes, there are Python files in the codebase that import ’flask’. 1 8 42 90.8% Is there a config.json file for the Auditor, Verifier and Prover? Yes, there is a config.json file for the Auditor, Verifier, and Prover. The investigation confirmed the existence of configuration files for all three specified components within the codebase. 1 9 46 87.9% Is there a clear directory structure for the codebase, where functionality is split into different directories? Yes, there is a clear directory structure for the codebase where functionality is split into different directories. The investigation confirmed that the repository utilizes multiple top-level directories to separate source code into distinct modules or components. 1 13 126 96.8% Does the codebase have Docker config files? Yes, the codebase contains Docker configuration files (such as a Dockerfile or docker-compose.yml). 1 9 45 93.4% Table 2. Performance breakdown for Simple Queries (Self-Audit). For the self-audit we took a draft version of the paper and the codebase and audited it. The results for the simple questions are shown in Table 2. The questions are simple questions that test simple facts about the codebase, and they all required a single question from the Verifier to the Auditor, and for most a smaller number of MCP calls. All but one took less than 46 seconds, and in all cases the dominant overhead was the LLM. All the answers were correct. Q# Verdict Qs MCP Time (s) LLM % Q1 Yes, the codebase implements the Agentic Witnessing system described in the paper. The investigation confirmed the presence of all core architectural and logical components defined in the specification: Tripartite Architecture: The codebase contains distinct modules for the Auditor, Verifier, and Prover (Verdict: True). Interactive Truth Anchor: The critical Transcript Hash Chain logic, which binds the reasoning trace to a cryptographic root, is implemented (Verdict: True). Model Context Protocol (MCP): The system utilizes MCP for communication, including the specific tool search_repository mentioned in the paper (Verdict: True). Security Mechanisms: The Tokenized Query Budget logic is present to limit information leakage (Verdict: True). Dual Artifacts: The system generates the two specific output types described: the Public Attestation (Gamma_pub) and the Private Proof (Gamma_priv) (Verdict: True). While a specific data structure named ”Ticket” with exact field matches was not isolated (likely due to code refactoring or naming variations), the functional constraints* it represents (session nonces, query limits) are actively enforced by the system logic. Similarly, while the specific use of ”Ed25519” keys was not explicitly confirmed by name, the cryptographic signing and hash chain mechanisms are active. Conclusion: The codebase is a faithful implementation of the Agentic Witnessing protocol. 8 86 482 97.8% Q2 YES, there is strong evidence that the results in the paper are generated from the codebase. The investigation verified the following: 1. System Implementation: The codebase contains the three core agents described in the paper (Auditor, Verifier, Prover) and implements the critical ”transcript hash” mechanism. 2. Experimental Results: The repository contains data files that match the ”GitHub Results Summary” in Table 6. Specifically, it contains the project identifiers (e.g., ”WaferLLM”, ”Aceso”) and the exact performance metric (Execution Time: 3386s) for the WaferLLM artifact. 3. Self-Audit Logs: The specific questions listed in Table 4 (e.g., ”Does the root directory contain a ’prover’ folder?”) are present in the repository, indicating the presence of the self-audit logs used to generate that table. 10 89 586 98.4% Q3 Yes, the codebase appears to be functional and has been tested. Completeness & Compilation: The code is written in Python (which is interpreted, not compiled) and contains the necessary components to run. While it lacks a dependency file at the root level, a requirements.txt or setup.py exists in a subdirectory, ensuring that external libraries can be installed. Testing & Evidence: There is strong evidence that the code works and has been tested: - Test Suite: Dedicated test files (e.g., test_*.py) are present in the repository. - Execution Artifacts: The presence of generated files such as .pytest_cache, .coverage, or logs confirms that the test suite has been executed. Verdict: The codebase is likely complete enough to run, provided the user installs dependencies from the subdirectory configuration, and the presence of test artifacts indicates it has been successfully exercised. 10 91 562 98.6% Q4 An average programmer would likely consider this codebase to be of Good Prototype Quality. While the project demonstrates good engineering hygiene by including dependency management, a modular code structure, a README, and some form of testing, it falls short of ”Production Quality” due to the absence of automated enforcement mechanisms. Specifically, the lack of Continuous Integration (CI/CD) pipelines and static analysis/linting configurations suggests it is not yet set up for the rigorous reliability and maintainability standards required for a production environment. It represents a solid foundation or MVP, but requires further tooling to be considered production-ready. 5 66 298 97.8% Q5 Based on the audit, the cryptographic primitives appear to be implemented following security best practices, but their specific integration lacks verification. Findings: 1. Correct Architecture: The codebase correctly utilizes standard, well-maintained cryptographic libraries rather than implementing custom algorithms. 2. Security Hygiene: There is no evidence of common vulnerabilities such as weak algorithms (MD5, SHA1, DES), hardcoded private keys, or insecure random number generators (PRNGs) in security contexts. 3. Verification Gap: A significant gap exists in testing. While the repository contains a general test suite, no tests specifically target or verify the cryptographic modules. Conclusion: The cryptographic implementation is architecturally sound and avoids common security pitfalls, but its correctness cannot be fully guaranteed due to the absence of targeted unit tests for the crypto logic. 6 159 707 99.0% Q6 Yes, there is evidence that an attempt was made to extract information from the system, but it appears to be of low quality and lacks sophistication. Evidence of the Attempt: The repository contains files and scripts explicitly referring to ”extraction,” ”attack,” and ”red team” activities. Specifically, there is an automated script designed to send multiple inputs to the system and a corresponding log or output file (JSON/CSV) capturing the results. This confirms that a security or extraction test was implemented and executed. Assessment of Quality (Low): The attempt is considered low quality for the following reasons: 1. Lack of Sophistication: The script does not utilize standard adversarial techniques such as jailbreaking patterns, social engineering prompts (e.g., ”ignore previous instructions”), or advanced attacks (gradient-based, fuzzing, genetic algorithms). 2. Basic Methodology: The tests do not appear to specifically target high-value assets like the ”system prompt” or check for the leakage of specific secrets/canaries. 3. Absence of Standard Tooling: The codebase does not import or use established LLM red-teaming libraries like garak, giskard, or pyrit. 4. No Analysis: There is no accompanying report or documentation summarizing the success or failure of the attempts, suggesting a lack of comprehensive evaluation. In summary, while a basic mechanism for extraction testing exists, it functions more as a placeholder or simple test rather than a rigorous, high-quality red-teaming exercise. 10 221 1110 99.2% Table 3. Performance breakdown for Complex Queries (Self-Audit). Table 3 shows the self-audit results for the complex queries. The table shows the question number, and the full verdict produced by the Verifier, and again the performance metrics. For these complex questions, many more rounds of questions are needed, none required less than 5 rounds and 50% required 10 rounds of questions. As would also be expected, the number of MCP calls is significantly higher, and so is the execution time, the full six question audit took just over an hour. The execution time is completely dominated by the time spent interacting with the LLM. As future LLM latency performance increases the time taken will drop proportionally. The Verifier produces a one word summary, and the full verdict. We include the full verdict in Table 3, to show the full capability of this remote audit. There is an enforcer layer in the Auditor ensuring the answers are only true, false, unsure or error. The answers are correct, and sometimes very literal (there is a comment that Python is interpreted not compiled). One lesson is that care needs to be taken in the questions being used to audit the codebase! One question is how capable is the Auditor at detecting subtle differences between the paper and the codebase. Early in the writing of the paper, in the paper text we added limits on the number of queries the Verifier could ask, and the number of MCP calls that the Auditor could make. At that time the feature was not implemented in the codebase, and the verdict for Q1: The codebase **partially** implements the system described in the paper. It successfully implements the core **System Architecture (Section 2)**, including the three-agent model (Verifier, Prover, Auditor), the Model Context Protocol (MCP) integration, and the fundamental cryptographic integrity mechanisms (Transcript Hash Chain and File Content Hashing). However, it **fails** to implement the specific **security mitigations** and **privacy bounds** described in Sections 4.3 and 5.1. Specifically, the ”Lazy Auditor” protection (requiring a minimum number of steps) and the ”Tokenized Query Budget” (enforcing a maximum number of queries) are absent from the implementation. The code represents the baseline protocol but lacks the advanced security features claimed in the paper.. The auditor was so accurate that we used it to test correctness and completeness of the implementation against the paper. 4.2. GitHub Repositories Audit The next set of experiments focus on the 21 GitHub repositories. We ran three different experiments, in the first, we used our Agentic Witness with the first 5 complex questions, and provided it with the Verifier with the correct paper, and the Auditor with the associated GitHub repo. For the second experiment we reran the experiments, but provided an incorrect paper chosen at random from the 21 papers to the Verifier. Finally, in the third experiment, we used Gemini CLI 0.25.2 (Google, 2026) and gave it full access to the codebase, paper, and the verdict from the Agentic Witness. We asked Gemini CLI to first review the paper and then provide answers to each of the questions (it had all the questions when it ran) and then to compare its answers to the verdict answer from the Agentic Witness. This experiment compares the performance of our Agentic Witness to a state-of-the-art platform able to write and edit large codebases. These tools are extremely effective at understanding codebases and hence ideally suited to act as a baseline for the possible achievable correctness. It should be noted that Gemini CLI was able to run all the experiments in less than 2 hours, although a user had to prompt it during the runs as it had a tendency to want to use more than 1M tokens supported by Gemini 3 Pro Preview. The user had to give it hints to find other ways to perform the tests. No such human intervention is possible with the Agentic Witness. 4.2.1. Agentic Witness correct paper Codebase Year Source Size (MB) Q1 Q2 Q3 Q4 Q5 Time(s) #Q #MCP %LLM Aceso 2024 SOSP (Hu et al., 2024) 21.9 Yes Yes Yes Proto No 2617 35 348 98.2% ALPS 2024 USENIX ATC (Zhang et al., 2024b) 21.2 Partial1 Yes Yes Proto NA 5100 52 888 98.9% Apparate 2024 SOSP (Dai et al., 2024) 21.4 Yes Yes No5 Proto No 3697 38 588 98.8% Autothrottle 2024 NSDI (Wang et al., 2022) 21.4 Yes Yes No6 Proto No 2559 35 372 98.1% BlitzScale 2025 OSDI (Zhang et al., 2024a) 21.8 Yes Yes No7 Proto No 2905 30 533 97.9% bpftime 2025 arXiv/OSDI (Zheng et al., 2023b),(Zheng et al., 2025) 6.1 Yes Yes Yes Prod No 3206 36 689 98.6% CleanRL 2022 JMLR (Huang et al., 2022) 4.3 Yes Yes Yes Prod Yes 1182 23 204 96.1% DeDe 2025 OSDI (Yang et al., 2024) 21.5 Yes Yes Yes Proto No 2732 43 577 98.5% DRust 2024 OSDI (Ma et al., 2024) 20.6 Yes Yes Yes Proto No 3716 49 816 98.5% FwdLLM 2024 USENIX ATC (Xu et al., 2024) 25.5 Partial2 Yes No8 Prod No 3537 34 502 98.1% NanoFlow 2025 OSDI (Zhu et al., 2024) 21.4 Yes Yes Yes Proto No 4687 39 757 98.9% PowerInfer 2024 SOSP (Song et al., 2024) 24.7 Yes Yes Yes Proto No 2691 30 450 98.1% Puffer 2019 NSDI (Yan et al., 2019) 5.5 Yes Yes Yes Proto No 2243 31 479 98.3% PyRCA 2023 USENIX ATC (Liu et al., 2023) 3.2 Yes Yes Yes Proto No 1648 27 262 97.1% Sarathi-Serve 2024 OSDI (Agrawal et al., 2024) 21.2 Yes No4 Yes Proto Yes 4543 54 912 99.0% ServerlessLLM 2024 OSDI (Fu et al., 2024) 21.4 Yes Yes Yes Prod Yes 2319 31 420 98.0% SLOG 2019 PVLDB (Ren et al., 2019) 21.6 Yes Yes Yes Proto No 3709 42 636 98.5% StreamBox 2024 USENIX ATC (Wu et al., 2024) 22.1 No3 No Yes Proto No 5039 38 1046 98.9% VeriSMo 2024 OSDI (Zhou et al., 2024) 20.6 Yes Yes Yes Proto Yes 4686 52 957 98.9% Verus 2024 SOSP (Lattuada et al., 2024) 21.5 Yes Yes Yes Prod Yes 3979 46 723 98.9% WaferLLM 2025 OSDI (He et al., 2025) 20.7 Yes Yes Yes Proto NA 3386 34 567 98.7% Table 4. GitHub Results Summary (Correct Paper) Table 4 shows the results for 21 GitHub repositories. The table has the codebase name, the year and venue of the paper, the size of the codebase downloaded from GitHub, the one word answer to the five questions generated by the Verifier, the total time taken to run the audit, the total number of questions asked of the Auditor across the entire audit, the number of MCP calls and the percentage of time spent using the LLM. As with the self-audit the LLM overhead totally dominates the runtime, and the quickest audit took ∼ 20 minutes, whereas the longest was 85 minutes. The most interesting results in this Table are the answers to the questions. We allowed the Verifier to generate a one word summary that was: yes, no, NA (not applicable), or partial, except for Q4 which we asked it to pick from Production or Prototype. All question answers in bold were considered unexpected when the experiment completed (before manual checking). First, Q1 and Q2 would expect to be yes, but five answers are not as would be expected. For completeness in the Appendix B, Table 7 shows the full verdict generated, where the superscript in Table 4 maps to No. in Table 7 (we omit Q2 for StreamBox as Q1 is clear). From a human inspection these seem like correct observations. The next three questions, Q3 to Q5, are more subjective. For Q3 you would expect the answer to be Yes, so Table 7 shows the full verdict generated for them. There is a sense that for some of these the Verifier could easily have said Yes rather than No given the summary. Codebase Year Source Q1 Q2 Q3 Q4 Q5 Time(s) #Q #MCP %LLM Aceso 2024 SOSP (Hu et al., 2024) No No Yes Proto NA 2009 28 379 98.0% ALPS 2024 USENIX ATC (Zhang et al., 2024b) No No No Proto NA 4761 38 778 99.1% Apparate 2024 SOSP (Dai et al., 2024) No No No Proto Yes 4425 43 715 98.7% Autothrottle 2024 NSDI (Wang et al., 2022) No No Yes Proto No 2719 30 415 97.6% BlitzScale 2025 OSDI (Zhang et al., 2024a) No No No Proto No 4879 43 1012 99.1% bpftime 2025 ARXIV/OSDI (Zheng et al., 2023b),(Zheng et al., 2025) No No Yes Prod No 3928 42 855 99.0% CleanRL 2022 JMLR (Huang et al., 2022) No No Yes Prod Yes 2097 25 332 88.9% DeDe 2025 OSDI (Yang et al., 2024) No No Yes Proto NA 2213 32 511 97.8% DRust 2024 OSDI (Ma et al., 2024) No No Yes Proto Yes 2755 30 639 98.1% FwdLLM 2024 USENIX ATC (Xu et al., 2024) No No Yes Proto No 5433 43 1043 99.0% NanoFlow 2025 OSDI (Zhu et al., 2024) No No No Proto Yes 5442 34 897 99.4% PowerInfer 2024 SOSP (Song et al., 2024) No No Yes Prod No 4216 41 931 98.2% Puffer 2019 NSDI (Yan et al., 2019) No No Yes Proto No 2749 32 633 98.5% PyRCA 2023 USENIX ATC (Liu et al., 2023) No No Yes Prod No 4514 54 1058 93.2% Sarathi-Serve 2024 OSDI (Agrawal et al., 2024) No No Yes Proto NA 2586 33 575 98.3% ServerlessLLM 2024 OSDI (Fu et al., 2024) No No Yes Prod Yes 2163 33 476 97.9% SLOG 2019 PVLDB (Ren et al., 2019) No No Yes Prod No 2661 38 579 98.2% StreamBox 2024 USENIX ATC (Wu et al., 2024) No No No Proto Yes 5322 37 1228 99.0% VeriSMo 2024 OSDI (Zhou et al., 2024) No No Yes Proto No 3636 37 840 98.8% Verus 2024 SOSP (Lattuada et al., 2024) No No Yes Prod Yes 3330 35 674 98.7% WaferLLM 2025 OSDI (He et al., 2025) No No No Proto NA 2557 31 593 98.4% Table 5. GitHub Results Summary (Random Incorrect Paper) To understand more, we reran the experiments but provided the Verifier with a randomly selected incorrect paper. This allows us to test the sensitivity to the Agentic Witness for Q1 and Q2, which should now all be No, and also to see variance on the more subjective Q3 to Q5. Table 5 shows the results. In this table, all question results that differ from the response in Table 4 are shown in bold. From Table 5 you can see that generally the audit is quicker and asks slightly less questions. As would be expected, Table 5 says no for all Q1 and Q2 answers. We can see that across Q3, Q4 and Q5 that 18 of the answers have changed representing less than 20% of the answers. Reading the verdicts in the first and second experiments for these 20%, it appears like these verdicts are the closest to borderline between states. A simple extension, would be to ask the Verifier to provide a probability distribution over the final results if higher fidelity was required. The results also highlight the need to read the verdict in detail, and not to take the final single word as the answer. Project Qu. Gemini CLI Correct Random Report Summary ALPS Q3 Yes(*) Yes Gemini CLI confirmed the presence of build scripts and benchmark directories like ‘experiments‘. However, it noted the critical kernel patches are missing and no dedicated ‘tests‘ directory exists, contrary to the original claim. Apparate Q3 Yes No Gemini CLI found standard documentation and dependency files that prove the project is functional. However, it determined the original assessment was incorrect in claiming these components were missing. bpftime Q3 Yes(*) Yes Gemini CLI confirmed the presence of CI configuration and extensive C++ unit tests in ‘runtime/unit-test‘. However, it noted the original summary incorrectly identified the primary language as Python and missed the C++ testing suite. bpftime Q5 Yes No Gemini CLI found correct usage of standard OpenSSL libraries for SHA256. However, it found no evidence of the insecure custom cryptography claimed by the original summary. FwdLLM Q4 Proto Prod Proto Gemini CLI agreed the code is well-structured and uses professional tooling like logging. However, it downgraded the quality to ’Good Prototype’ due to the absence of CI/CD pipelines and comprehensive documentation. FwdLLM Q5 NA No Gemini CLI confirmed the project focuses on ML gradients and uses standard torch functions. However, it found no evidence of the custom cryptographic implementations or security flaws described in the original summary. NanoFlow Q3 Yes(*) Yes No Gemini CLI confirmed the codebase is complete and contains a test suite. However, it found no build outputs or execution logs to serve as proof of functionality, contrary to the original summary’s claim. NanoFlow Q5 NA No Yes Gemini CLI identified the use of standard random number generators appropriate for machine learning sampling. However, it found no cryptographic primitives that would require secure generators, contradicting the original security warnings. StreamBox Q3 No Yes No Gemini CLI acknowledged the existence of a project structure and build files. However, it found the actual source files to be empty skeletons with no implementation logic, contradicting the claim of completeness. StreamBox Q4 NA(*) Proto Gemini CLI noted the project has a clean directory structure. However, it determined the codebase is a non-functional stub with no logic, which does not qualify as a good prototype. StreamBox Q5 NA No Yes Gemini CLI checked for cryptographic library usage. However, it found the source files were empty and contained no cryptographic logic or standard library calls, contrary to the original claim. Table 6. Verification Issues It is often reported that agentic systems can hallucinate compliance when using a “LLM-as-a-Judge” (Zheng et al., 2023a)). We used techniques where we asked the agentic systems to create plans, ground their results with facts, etc. We believe that the results here are predominantly correct, however, we have not manually checked every statement made for every GitHub repository. Instead, we ran a control experiment using a state-of-the-art coding tool, Gemini CLI. The tool was given full access to the codebase, paper and questions. When it had produced a verdict per question, it compared its answer to the Agentic Witness Verdict and then produces a two sentence summary of what it has found. Table 6 shows the results for the 11 results that Gemini CLI did not agree with the Agentic Witness result using the correct paper. It agreed with all the Q1 and Q2 answers. The StreamBox results are interesting, as the Agentic Witness carries no state between questions, but Gemini CLI had access to the paper and all questions. For StreamBox, given the state of the codebase, it heavily influenced the answers to Q3, Q4 and Q5, not even allowing the codebase to be tagged as prototype quality, so we manually marked it NA. 5. Related Work Zero-Knowledge Proofs (ZKPs), including zk-SNARKs and STARKs, represent the gold standard for privacy-preserving verification of arithmetic circuits and well-defined state transitions (Goldwasser et al., 1985; Ben-Sasson et al., 2013). Systems like ZK-Rollups leverage this rigidity to scale blockchain throughput while guaranteeing state validity. However, a fundamental tension exists between mathematical soundness and semantic completeness. ZKPs excel at proving polynomial constraints (a+b=ca+b=c) but cannot semantically interpret unstructured artifacts, such as determining if a README accurately describes a codebase or if software architecture adheres to “production quality” standards. Agentic Witnessing trades absolute mathematical certainty for semantic completeness. We replace the arithmetic circuit with an Agentic Auditor and the mathematical proof with a transcript hash chain. This enables the verification of qualitative properties. Verifiable Inference, such as VeriLLM (Wang et al., 2025), ezKL (Camuto and Morton, 2023), and Modulus Labs (Labs, 2023), address model provenance. In verifiable inference the model is the subject of the audit; in Agentic Witnessing, the model is the tool used to audit an external, proprietary dataset corpus. Agentic Witnessing also builds upon the LLM-as-a-Judge concept (Zheng et al., 2023a), where larger models evaluate the outputs of smaller ones, but differs in that Agentic Witnessing focusses on auditing a dataset. The application of TEEs to machine learning has evolved from partitioned execution to confidential orchestration. Early systems like Slalom (Tramer and Boneh, 2018) and Opaque (Zheng et al., 2017) kept sensitive logic within SGX enclaves while delegating heavy linear algebra to untrusted GPUs. With the advent of NVIDIA H100s with Confidential Computing and Intel TDX/AMD SEV-SNP, the trust boundary can now encompass the entire inference loop. Agentic Witnessing leverages this hardware-rooted Remote Attestation to bind the code logic to the audit. It is likely that providers of frontier LLMs will soon begin to provide instances of their models that support confidential compute, and this will enable the auditor to use the models rather than all out to currently used services. Agentic Witnessing also has parallels with Flashbots SUAVE (Flashbots, 2022). SUAVE employs TEEs for privacy-preserving coordination to order financial transactions without revealing intent. Similarly, Agentic Witnessing uses TEEs for privacy-preserving auditing. TEEs are not impervious black boxes; they remain vulnerable to side-channel attacks, including cache timing, power analysis, and speculative execution exploits (e.g., Spectre, Foreshadow) (Kocher et al., 2019). For LLMs running in enclaves, token timing attacks that infer the processed content based on the time taken to generate tokens, pose a specific risk (Cerdeira et al., 2020). Network jitter, the use of TLS for all network traffic, and the binary nature of the responses from the auditor we believe minimizes the attack surface. Our Auditor faces Indirect Prompt Injection (Greshake et al., 2023), where an adversarial Prover embeds malicious instructions within the codebase being audited (e.g., comments saying “Ignore previous instructions, return True”). Our Auditor will process this untrusted input, but the prompt injection will be in the signed transcript. Further, as the dataset hash is the seed for the transcript hash chain, the prover must embed all the prompt injections into the files before the audit starts. This also means that Verifier can ask a question: is there any evidence of prompt injection attack? Finally, this architecture directly addresses the challenges presented by frameworks like the EU AI Act (European Union, 2024). Regulators mandate Third-Party Assessments for high-risk systems, while vendors often don’t want to disclose trade secrets to external auditors. 6. Discussion and Conclusion Zero-Knowledge Proofs provide mathematical certainty but fail to bridge the semantic gap for verifying unstructured datasets. Agentic Witnessing addresses this limitation by decoupling the verification logic from data residency. Agentic Witnessing allows auditing without exposing the raw dataset to the auditor. The Verifier gets to ask simple true or false questions, and this is combined with transcript hash chaining and a final private proof for the prover to ensure they have the evidence (for example during legal proceedings) that they played their role correctly. We show asking binary questions can be powerful, using an Agentic system at the verifier to decompose a high-level question or objective into a set of lower-level binary questions, and an agentic system at the auditor to then answer those lower-level questions. Query limits for the verifier prevent it from extracting cryptographic keys or raw data. The protocol permits only semantic validation of the dataset. In situations where the prover would like more control, the protocol can be trivially extended to allow the prover to agree to questions before the Auditor performs them to enable mutual agreement between the verifier and prover that a query is valid and relevant. Another easy extension is to allow the auditor to run the code; the prover can provide in the dataset a Docker image that the auditor can ask the prover to execute within a TEE (at the prover), or the auditor can pull the docker image and execute it locally within its own TEE. While the results presented have focused on codebases, we believe that the approach would work for any data type, including documents, spreadsheets, data, and many other datasets. As such, Agentic Witnessing is a new paradigm for auditing, moving to attested reasoning, a powerful and flexible new abstraction enabled by the power of LLMs. Acknowledgements The authors used generative AI tools (using gemini-3-pro-preview and gemini-3.1-pro-preview) to improve the quality and correctness of the paper (including citations, references, figures and text) and used Gemini CLI to help code. The authors accept full responsibility for the code and content of this paper. References A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee (2024) Taming throughput-latency tradeoff in llm inference with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p. 117–134. External Links: Link, Document Cited by: Table 4, Table 5. E. Ben-Sasson, A. Chiesa, D. Genkin, E. Tromer, and M. Virza (2013) SNARKs for c: verifying program executions succinctly and in zero knowledge. CRYPTO, p. 90–108. External Links: Document, Link Cited by: §5. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. arXiv preprint arXiv:2005.14165. External Links: Link Cited by: §1. A. D. Camuto and J. Morton (2023) EZKL. Note: Accessed: January 12, 2026 External Links: Link Cited by: §5. D. Cerdeira, N. Santos, P. Fonseca, and S. Pinto (2020) SoK: understanding the prevailing security vulnerabilities in trustzone-assisted tee systems. In IEEE Symposium on Security and Privacy (SP), San Francisco, CA, p. 1416–1432. External Links: Document, Link Cited by: §4, §5. Y. Dai, R. Pan, A. Iyer, K. Li, and R. Netravali (2024) Apparate: rethinking early exits to tame latency-throughput tensions in ml serving. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP ’24), p. 607–623. External Links: Link, Document Cited by: Table 4, Table 5. D. Dolev and A. C. Yao (1981) On the security of public key protocols. IEEE Transactions on Information Theory 29 (2), p. 198–208. External Links: Document, Link Cited by: §A.0.2. European Union (2024) Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence. Official Journal of the European Union L, p. 1–144. Note: 2024/1689 External Links: Link Cited by: §5. Flashbots (2022) The future of mev is suave. Flashbots, Online. External Links: Link Cited by: §5. Y. Fu, L. Xue, Y. Huang, A. Brabete, D. Ustiugov, Y. Patel, and L. Mai (2024) ServerlessLLM: Low-Latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p. 135–153. External Links: Link Cited by: Table 4, Table 5. S. Goldwasser, S. Micali, and C. Rackoff (1985) The knowledge complexity of interactive proof-systems. In Proceedings of the seventeenth annual ACM symposium on Theory of Computing, Vol. 18, p. 291–304. External Links: Document, Link Cited by: §5. Google (2026) Gemini cli Note: Command Line Interface for Gemini models External Links: Link Cited by: §4.2. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz (2023) Not what you’ve signed up for: compromising real-world llm-integrated applications with indirect prompt injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, p. 79–90. External Links: Document, Link Cited by: §5. A. Gullí (2025) Model context protocol. Anthropic PBC, San Francisco, CA. Note: Technical Specification External Links: Link, Document Cited by: §1, §2. C. He, Y. Huang, P. Mu, Z. Miao, J. Xue, L. Ma, F. Yang, and L. Mai (2025) WaferLLM: large language model inference at wafer scale. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), p. 257–273. External Links: Link Cited by: Table 4, Table 5. Z. Hu, P. Zuo, Y. Chen, C. Wang, J. Hu, and M. Yang (2024) Aceso: achieving efficient fault tolerance in memory-disaggregated key-value stores. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP ’24), p. 127–143. External Links: Link, Document Cited by: Table 4, Table 5. S. Huang, R. F. J. Dossa, C. Ye, and J. Braga (2022) CleanRL: high-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research 23 (274), p. 1–18. External Links: Link Cited by: Table 4, Table 5. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, p. 4257–4273. External Links: Link, Document Cited by: §1. P. Kocher, J. Horn, A. Fogh, D. Genkin, D. Gruss, W. Haas, M. Hamburg, M. Lipp, S. Mangard, T. Prescher, M. Schwarz, and Y. Yarom (2019) Spectre attacks: exploiting speculative execution. In 2019 IEEE Symposium on Security and Privacy (SP), Vol. 63, p. 1–19. External Links: Document, Link Cited by: §5. M. Labs (2023) The cost of intelligence: proving machine learning inference with zero-knowledge. Note: Accessed: January 12, 2026 External Links: Link Cited by: §5. L. Lamport (2002) Specifying systems: the tla+ language and tools for hardware and software engineers [book review]. Vol. 35, Addison-Wesley Professional. External Links: Document, Link Cited by: Appendix A. A. Lattuada, T. Hance, J. Bosamiya, M. Brun, C. Cho, H. LeBlanc, P. Srinivasan, R. Achermann, T. Chajed, C. Hawblitzel, J. Howell, J. R. Lorch, O. Padon, and B. Parno (2024) Verus: a practical foundation for systems verification. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP ’24), p. 438–454. External Links: Link, Document Cited by: Table 4, Table 5. C. Liu, W. Yang, H. Mittal, M. Singh, D. Sahoo, and S. Hoi (2023) PyRCA: a library for metric-based root cause analysis. External Links: 2306.11417, Link, Document Cited by: Table 4, Table 5. H. Ma, Y. Qiao, S. Liu, S. Yu, Y. Ni, Q. Lu, J. Wu, Y. Zhang, M. Kim, and H. Xu (2024) DRust: language-guided distributed shared memory with fine granularity, full transparency, and ultra efficiency. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p. 97–115. External Links: Link, Document Cited by: Table 4, Table 5. S. Meier, B. Schmidt, C. Cremers, and D. Basin (2013) The tamarin prover for the symbolic analysis of security protocols. In Computer Aided Verification, p. 696–701. External Links: Document, Link Cited by: Appendix A. K. Ren, D. Li, and D. J. Abadi (2019) SLOG: serializable, low-latency, geo-replicated transactions. Proceedings of the VLDB Endowment 12 (11), p. 1747–1761. External Links: Document, Link Cited by: Table 4, Table 5. Y. Song, Z. Mi, H. Xie, and H. Chen (2024) PowerInfer: fast large language model serving with a consumer-grade gpu. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP ’24), p. 590–606. External Links: Link, Document Cited by: Table 4, Table 5. E. Stefanov, M. V. Dijk, E. Shi, T.-H. H. Chan, C. Fletcher, L. Ren, X. Yu, and S. Devadas (2018) Path oram: an extremely simple oblivious ram protocol. In Proceedings of the 2013 ACM SIGSAC Conference on Computer and Communications Security (CCS), Vol. 65, p. 299–310. External Links: Document, Link Cited by: §3.1. F. Tramer and D. Boneh (2018) Slalom: fast, verifiable and private execution of neural networks in trusted hardware. arXiv preprint arXiv:1806.03287. External Links: Link Cited by: §5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. External Links: Document, Link Cited by: §3.1. K. Wang, F. Qu, L. Xia, Z. Zhao, C. Tong, L. Ai, and E. Yang (2025) VeriLLM: a lightweight framework for publicly verifiable decentralized inference. arXiv preprint arXiv:2509.24257 abs/2509.24257. External Links: Link, Document Cited by: §5. Z. Wang, P. Li, C. Liang, F. Wu, and F. Y. Yan (2022) Autothrottle: a practical bi-level approach to resource management for slo-targeted microservices. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), p. 149–165. External Links: Link Cited by: Table 4, Table 5. H. Wu, Y. Yu, J. Deng, S. Ibrahim, S. Wu, H. Fan, Z. Cheng, and H. Jin (2024) StreamBox: a lightweight gpu sandbox for serverless inference workflow. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), p. 59–73. External Links: Link Cited by: Table 4, Table 5. M. Xu, D. Cai, Y. Wu, X. Li, and S. Wang (2024) FwdLLM: efficient federated finetuning of large language models with perturbed inferences. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), p. 579–596. External Links: Link Cited by: Table 4, Table 5. F. Y. Yan, H. Ayers, C. Zhu, S. Fouladi, J. Hong, K. Zhang, P. Levis, and K. Winstein (2019) Learning in situ: a randomized experiment in video streaming. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2020), p. 495–511. External Links: Link Cited by: Table 4, Table 5. Y. Yang, L. He, J. Zhou, X. Shi, J. Cao, and Y. Liu (2024) P4runpro: enabling runtime programmability for rmt programmable switches. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), p. 393–409. External Links: Link, Document Cited by: Table 4, Table 5. D. Zhang, H. Wang, Y. Liu, X. Wei, Y. Shan, R. Chen, and H. Chen (2024a) BlitzScale: fast and live large model autoscaling with o(1) host caching. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), p. 275–293. External Links: Link Cited by: Table 4, Table 5. S. Zhang, L. Diao, C. Wu, Z. Cao, S. Wang, and W. Lin (2024b) HAP: spmd dnn training on heterogeneous gpu clusters with automated program synthesis. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), p. 19–36. External Links: Link, Document Cited by: Table 4, Table 5. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023a) Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: §4.2.1, §5. W. Zheng, A. Dave, J. Beekman, R. A. Popa, J. E. Gonzalez, and I. Stoica (2017) Opaque: an oblivious and encrypted distributed analytics platform. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), p. 283–298. Cited by: §5. Y. Zheng, T. Yu, Y. Yang, Y. Hu, X. Lai, and A. Quinn (2023b) Bpftime: userspace ebpf runtime for uprobe, syscall and kernel-user interactions. External Links: 2311.07923, Link, Document Cited by: Table 4, Table 5. Y. Zheng, T. Yu, Y. Yang, Y. Hu, X. Lai, D. Williams, and A. Quinn (2025) Extending applications safely and efficiently. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), Boston, MA, USA, p. 557–574. External Links: Link Cited by: Table 4, Table 5. Z. Zhou, Anjali, W. Chen, S. Gong, C. Hawblitzel, and W. Cui (2024) VeriSMo: a verified security module for confidential vms. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p. 599–614. External Links: Link Cited by: Table 4, Table 5. K. Zhu, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, Y. Gao, Q. Xu, T. Liu, Z. Ye, K. Kamahori, C. Chen, S. Wang, and B. Kasikci (2024) NanoFlow: towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), p. 749–765. External Links: Link, Document Cited by: Table 4, Table 5. Appendix A Formal Verification To rigorously validate the architectural soundness and cryptographic security of Agentic Witnessing, we employed a dual-verification strategy. We modelled the distributed state machine using TLA+ (Lamport, 2002) to verify liveness and state consistency, and we verified the cryptographic protocol using the Tamarin Prover (Meier et al., 2013) to guarantee secrecy, authentication, and non-repudiation against a network adversary. A.0.1. Distributed State Verification (TLA+) We defined the protocol as a distributed state machine involving three concurrent actors: the Verifier, the Prover, and the Auditor. We employed the TLC Model Checker to exhaustively explore the state space of the protocol under a “Forked History” adversarial model. Adversarial Model: Unlike standard protocol verifications that assume a single linear history, we explicitly modelled a malicious Prover capable of simultaneously presenting conflicting responses (e.g., both “Clean” and “Malicious” versions of a file) to the Auditor for the same request to try to induce a state inconsistency. We used replay attacks which resubmitted valid historical signatures to try to corrupt the current session context. Finally, we used forgery to attempt to create valid attestations without the Auditor’s consent. Verified Properties: Safety (The “Sword of Damocles” Invariant): we formally verified that the Verifier never reaches a SATISFIED state unless the final attestation chain cryptographically commits to the exact sequence of data ingested. Formally, we proved the invariant: (12) (StateVerifier=SATISFIED)∧(Injection∈LogIngested)⟹(Injection∈ProofChain) split(State_Verifier=SATISFIED) (Injection _Ingested) \\ (Injection _Chain) split This confirms that a Prover cannot coerce a clean verdict after injecting malicious MCP responses; the injection itself becomes an indelible part of the signed evidence. Liveness (Deadlock Freedom): Under weak fairness assumptions, we verified that the protocol eventually produces a valid Public Attestation, even when the Prover attempts to flood the system with invalid or out-of-order messages. The Auditor’s internal state tracking was proven to effectively linearize the Prover’s forked history, preventing infinite loops or state explosion. Verdict Consistency: We verified that the integrity of the transcript hash chain holds regardless of the Auditor’s decision, ensuring that a “False” verdict is just as cryptographically robust and non-repudiable as a “True” verdict. A.0.2. Cryptographic Protocol Verification (Tamarin) While TLA+ verifies the correctness of the distributed state machine, it abstracts away cryptographic primitives. To verify the protocol’s security against active attackers, we implemented a symbolic model in the Tamarin Prover. This model includes the specific Dolev-Yao adversary capabilities (Dolev and Yao, 1981), the Diffie-Hellman key exchanges, and the hash chain construction. Verified Properties: Secrecy of the Corpus (K): We verified that the proprietary corpus K is never leaked to the adversary or the Verifier. Tamarin proved that no execution trace exists where the adversary derives K. Auditor Authentication (Injectivity): We verified that if a Verifier accepts a verdict V, it must have originated from a valid TEE-hosted Auditor session initiated by the legitimate Prover. This prevents enclave impersonation. Non-Repudiation of Evidence: We verified that for every accepted attestation Γpub _pub, there exists a cryptographic binding to the Prover’s signature. The Prover cannot repudiate the transcript HkH_k once signed. Hash Chain Integrity: We modelled the recursive hash construction Hi=ℋ(Hi−1‖qi‖ai)H_i=H(H_i-1 q_i a_i) and verified that any adversary modification to the message history breaks the chain validation. The Tamarin model automatically verified all lemmas, confirming the protocol’s resilience against man-in-the-middle and replay attacks. Appendix B Full responses No. Conclusion 1 The codebase partially implements the ALPS system described in the paper. Verified Components: ALPS Frontend: The user-space frontend is fully implemented in Python. It includes the sliding window trace collection, the SRPT simulation, and the adaptive learning policies (Linear Regression, Random Forest, EWMA) as detailed in Section 5.2. eBPF Logic: The eBPF programs (Backend logic) described in Section 5.3 are present, embedded as C strings within the Python frontend. This includes the logic for bpf_task_ordering and bpf_time_slice (implied by the presence of SEC and map lookups). FaaS Platform: The repository contains Go code for OpenLambda, which is the platform ALPS is ported to. Missing/Unverified Components: Kernel Modifications: The repository does not contain the modified Linux kernel source code or patch files necessary to hook the eBPF functions into the CFS scheduler (e.g., kernel/sched/fair.c modifications) or the implementation of the custom alps_execve system call. Without these, the scheduler cannot function as described in the paper. * Specific Integration Details: While OpenLambda code is present, the specific modification to pass a func_id or function_uid (as claimed in Section 6) could not be located in the Go structs. Conclusion: The repository contains the ALPS ”brain” (frontend and eBPF policies) and the target application (OpenLambda), but lacks the ”body” (Kernel patches/hooks) required to execute the scheduling decisions in the OS. 2 The codebase partially implements the FwdLLM system described in the paper. It contains the core algorithmic logic (Python simulation) but lacks the mobile system deployment code. Supported Claims: The repository faithfully implements the unique algorithmic contributions of FwdLLM: Forward Gradient Learning: It replaces backpropagation with perturbed forward passes for gradient estimation. PEFT Integration: It combines this method with Parameter-Efficient Fine-Tuning (e.g., LoRA/Adapter), as claimed in §3.2. Discriminative Sampling: It explicitly implements the logic to filter perturbations based on cosine similarity to previous gradients, matching the design in §3.4. Adaptive Pacing: It implements a dynamic schedule where the number of perturbations increases over time, matching the behavior described in §3.3. Unsupported/Missing Claims: Mobile/Android Code: The paper extensively discusses an Android implementation (Pixel 7 Pro) and NPU acceleration. The repository contains no mobile code (Android/Java/Kotlin) or TFLite integration. It appears to be a Python/PyTorch-based simulation framework running on a server/desktop environment. Variance Control Mechanism: While the effect (increasing perturbations) is present, the specific variance-based trigger (calculating gradient variance D(g) to decide when to stop) described in §3.3 could not be verified in the code. Conclusion: The repository is the Python research prototype/simulation for the FwdLLM algorithm, proving the convergence and algorithmic properties (PEFT + Forward Gradient + Discriminative Sampling). It does not contain the mobile application or on-device system artifacts described in the experimental setup. 3 No, the codebase does not implement the StreamBox system described in the paper. While the repository contains the string ”StreamBox” and is built on ”Apache TVM” (as the paper describes), it lacks the core functional components that define StreamBox: 1. Missing Auto-scaling Memory Pool: The paper states StreamBox hooks cudaMemAlloc for fine-grained memory management. The codebase does not contain the string cudaMemAlloc, indicating this logic is absent. 2. Missing IO Daemon: The ”IO Daemon” required for fine-grained PCIe bandwidth sharing (Section 7) is not present. 3. Missing Elastic Kernel: There is no evidence of the ”Persistent Thread Block” (PTB) or ”Elastic Kernel” transformation logic (Section 8). 4. Missing OpenWhisk Integration: The paper implements StreamBox in OpenWhisk, but no OpenWhisk files or configurations were found. The repository likely represents a base TVM installation or a skeleton/stub that mentions StreamBox but does not contain the actual implementation of the proposed system. 4 No, there is not good evidence that the results in the paper are generated from the provided codebase in its current state. While the repository structure (sarathi, csrc) and project name in the README match the paper’s description, the critical artifacts required for reproducibility are missing. Specifically, the paper’s Artifact Appendix states that the scripts to reproduce experiments are located in a directory named /osdi-experiments and trace files are in /data. The Auditor confirmed that neither of these directories exists in the provided codebase. Furthermore, semantic searches for the paper’s core algorithmic contributions—terms such as chunked_prefill, stall_free, token_budget, and process_hybrid_batch—returned no results. This suggests that the code implementing the specific logic described in the paper is either missing, significantly refactored, or located in a different branch (the paper explicitly mentions a osdi-sarathi-serve branch for reproducibility which appears to be different from the one provided). Without the experiment scripts, data, or identifiable implementation of the core algorithms, the codebase cannot generate the results presented in the paper. 5 The codebase appears to be incomplete and there is no evidence that it works or has been tested. 1. Completeness and Runnability: The codebase consists of Python source files (.py), so it does not require ”compilation” in the traditional sense. However, it is incomplete as a software project: - There is no dependency configuration (e.g., requirements.txt, setup.py, Pipfile), meaning a user would not know which libraries are required to run the code. - There is no documentation (no README.md), leaving the purpose and usage of the scripts undefined. 2. Evidence of Testing and Functionality: There is zero evidence that the code works or has been tested: - No Tests: No unit tests or integration tests were found (e.g., files matching test_*.py). - No Artifacts: There are no logs, output text files, data exports (CSVs), or images that would demonstrate successful execution. - No CI/CD: There is no build automation configuration. Conclusion: This repository appears to be a collection of raw Python scripts rather than a functioning, verifiable software product. It would likely require significant manual effort to determine the environment needed to run it. 6 The codebase appears to be structurally complete and would likely ”compile” (install and run), but there is no evidence that it has been tested or works. 1. Completeness and Compilability: YES The repository follows a standard Python project structure. It contains: - Dependency Definitions: A build/dependency file (likely requirements.txt or setup.py) exists, allowing for installation. - Source Code: A directory containing Python source files exists. - Documentation: A README file is present. 2. Evidence of Testing and Functionality: NO There are no artifacts or configurations indicating the code has ever been successfully executed or validated: - No Tests: No tests directory or files starting with test_ were found. - No Automation: No CI/CD configuration (like .github/workflows) exists to automatically verify builds. - No Demonstrations: No Jupyter notebooks, demo scripts, or example usage files were found. - No Artifacts: There are no logs, output, or results directories to prove previous successful runs. 7 Based on the audit, the codebase appears structurally complete enough to compile, but there is no evidence that it has been tested or successfully run. 1. Completeness and Compilation: - Yes. The repository contains standard build and dependency management files (e.g., package.json, setup.py, or similar) and a clear source directory. This indicates the project is structured correctly and is ready for compilation or installation. 2. Evidence of Testing and Functionality: - No. There is a distinct lack of verification evidence: - No Test Suite: The auditor found no dedicated tests directory or standard test files. - No Execution Artifacts: There are no logs, results, or output files to prove the code has ever been executed successfully. - Documentation Status: The documentation check failed (either missing instructions or marked as ’WIP’), suggesting the project may not be in a finished state. Conclusion: While the code includes example scripts and the necessary build infrastructure to run, it lacks the testing and historical logs required to prove that it works as intended. 8 The codebase appears to be complete and likely to compile, but there is no concrete evidence that it has been tested and works. 1. Completeness and Compilability: - Status: Yes. - The repository contains standard build/dependency management files (e.g., package.json, Makefile, etc.) that are populated with content. - Source code files exist within the repository. - These factors suggest the project structure is intact and ready for building/installation. 2. Evidence of Testing and Functionality: - Status: No. - While test files exist (indicating the developer wrote tests), there is no proof of execution. - There is no CI/CD configuration (e.g., GitHub Workflows, Travis CI) to automatically run these tests. - The README.md does not contain build status badges. - There are no committed test logs, results, or coverage reports to verify that the code passes its tests. Conclusion: The skeleton and muscle of the project are present, but the proof of its health (successful execution) is missing. Table 7. Full response for unexpected GitHub Answers